Bootstrap command refactoring: refactor BootstrapCommand (step 1) by Kobzol · Pull Request #126731 · rust-lang/rust (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation10 Commits8 Checks6 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

Kobzol

This PR is a first step towards https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap.

It refactors BoostrapCommand to get it closer to a state where it is an actual command wrapper that can be routed through a central place of command execution, and also to make the distinction between printing output vs handling output programatically clearer (since now it's a mess).

The existing usages of BootstrapCommand are complicated primarily because of different ways of handling output. There are commands that:

  1. Want to eagerly print stdout/stderr of the executed command, plus print an error message if the command fails (output mode PrintAll). Note that this error message attempts to print stdout/stderr of the command when -v is enabled, but that will always be empty, since this mode uses .status() and not .output().
  2. Want to eagerly print stdout/stderr of the executed command, but do not print any additional error message if it fails (output mode PrintOutput)
  3. Want to capture stdout/stderr of the executed command, but print an error message if it fails (output mode PrintFailure). This means that the user wants to either ignore the output or handle it programatically, but that's not obvious from the name.

The difference between 1) and 2) (unless explicitly specified) is determined dynamically based on the bootstrap verbosity level.

It is very difficult for me to wrap my head around all these modes. I think that in a future PR, we should split these axes into e.g. this:

  1. Do I want to handle the output programmatically or print it to the terminal? This should be a separate axis, true/false. (Note that "hiding the output" essentially just means saying that I handle it programmatically, and then I ignore the output).
  2. Do I want to print a message if the command fails? Yes/No/Based on verbosity (which would be the default).

Then there is also the failure mode, but that is relatively simple to handle, the command execution will just shutdown bootstrap (either eagerly or late) when the command fails.

Note that this is just a first refactoring steps, there are a lot of other things to be done, so some things might not look "final" yet. The next steps are (not necessarily in this order):

Best reviewed commit by commit.

r? @onur-ozkan

@Kobzol

This function both handles error printing and early/late failures, but it also always returns the actual output of the command

@Kobzol

@Kobzol

@Kobzol

@Kobzol

@rustbot rustbot added S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

T-bootstrap

Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

labels

Jun 20, 2024

Kobzol

@rust-log-analyzer

This comment has been minimized.

@Kobzol

@rust-log-analyzer

This comment has been minimized.

@Kobzol

onur-ozkan

@Kobzol

@Kobzol

@rustbot rustbot added S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

and removed S-waiting-on-author

Status: This is awaiting some action (such as code changes or more information) from the author.

labels

Jun 22, 2024

@onur-ozkan

This is a good first step, thanks!

@bors r+

@bors

📌 Commit 250586c has been approved by onur-ozkan

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors

Status: Waiting on bors to run and complete tests. Bors will change the label on completion.

and removed S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

labels

Jun 22, 2024

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request

Jun 22, 2024

@matthiaskrgr

…nur-ozkan

Bootstrap command refactoring: refactor BootstrapCommand (step 1)

This PR is a first step towards https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap.

It refactors BoostrapCommand to get it closer to a state where it is an actual command wrapper that can be routed through a central place of command execution, and also to make the distinction between printing output vs handling output programatically clearer (since now it's a mess).

The existing usages of BootstrapCommand are complicated primarily because of different ways of handling output. There are commands that:

  1. Want to eagerly print stdout/stderr of the executed command, plus print an error message if the command fails (output mode PrintAll). Note that this error message attempts to print stdout/stderr of the command when -v is enabled, but that will always be empty, since this mode uses .status() and not .output().
  2. Want to eagerly print stdout/stderr of the executed command, but do not print any additional error message if it fails (output mode PrintOutput)
  3. Want to capture stdout/stderr of the executed command, but print an error message if it fails (output mode PrintFailure). This means that the user wants to either ignore the output or handle it programatically, but that's not obvious from the name.

The difference between 1) and 2) (unless explicitly specified) is determined dynamically based on the bootstrap verbosity level.

It is very difficult for me to wrap my head around all these modes. I think that in a future PR, we should split these axes into e.g. this:

  1. Do I want to handle the output programmatically or print it to the terminal? This should be a separate axis, true/false. (Note that "hiding the output" essentially just means saying that I handle it programmatically, and then I ignore the output).
  2. Do I want to print a message if the command fails? Yes/No/Based on verbosity (which would be the default).

Then there is also the failure mode, but that is relatively simple to handle, the command execution will just shutdown bootstrap (either eagerly or late) when the command fails.

Note that this is just a first refactoring steps, there are a lot of other things to be done, so some things might not look "final" yet. The next steps are (not necessarily in this order):

Best reviewed commit by commit.

r? @onur-ozkan

bors added a commit to rust-lang-ci/rust that referenced this pull request

Jun 22, 2024

@bors

…iaskrgr

Rollup of 3 pull requests

Successful merges:

r? @ghost @rustbot modify labels: rollup

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request

Jun 22, 2024

@matthiaskrgr

…nur-ozkan

Bootstrap command refactoring: refactor BootstrapCommand (step 1)

This PR is a first step towards https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap.

It refactors BoostrapCommand to get it closer to a state where it is an actual command wrapper that can be routed through a central place of command execution, and also to make the distinction between printing output vs handling output programatically clearer (since now it's a mess).

The existing usages of BootstrapCommand are complicated primarily because of different ways of handling output. There are commands that:

  1. Want to eagerly print stdout/stderr of the executed command, plus print an error message if the command fails (output mode PrintAll). Note that this error message attempts to print stdout/stderr of the command when -v is enabled, but that will always be empty, since this mode uses .status() and not .output().
  2. Want to eagerly print stdout/stderr of the executed command, but do not print any additional error message if it fails (output mode PrintOutput)
  3. Want to capture stdout/stderr of the executed command, but print an error message if it fails (output mode PrintFailure). This means that the user wants to either ignore the output or handle it programatically, but that's not obvious from the name.

The difference between 1) and 2) (unless explicitly specified) is determined dynamically based on the bootstrap verbosity level.

It is very difficult for me to wrap my head around all these modes. I think that in a future PR, we should split these axes into e.g. this:

  1. Do I want to handle the output programmatically or print it to the terminal? This should be a separate axis, true/false. (Note that "hiding the output" essentially just means saying that I handle it programmatically, and then I ignore the output).
  2. Do I want to print a message if the command fails? Yes/No/Based on verbosity (which would be the default).

Then there is also the failure mode, but that is relatively simple to handle, the command execution will just shutdown bootstrap (either eagerly or late) when the command fails.

Note that this is just a first refactoring steps, there are a lot of other things to be done, so some things might not look "final" yet. The next steps are (not necessarily in this order):

Best reviewed commit by commit.

r? @onur-ozkan

bors added a commit to rust-lang-ci/rust that referenced this pull request

Jun 22, 2024

@bors

…llaumeGomez

Rollup of 5 pull requests

Successful merges:

r? @ghost @rustbot modify labels: rollup

rust-timer added a commit to rust-lang-ci/rust that referenced this pull request

Jun 22, 2024

@rust-timer

Rollup merge of rust-lang#126731 - Kobzol:bootstrap-cmd-refactor, r=onur-ozkan

Bootstrap command refactoring: refactor BootstrapCommand (step 1)

This PR is a first step towards https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap.

It refactors BoostrapCommand to get it closer to a state where it is an actual command wrapper that can be routed through a central place of command execution, and also to make the distinction between printing output vs handling output programatically clearer (since now it's a mess).

The existing usages of BootstrapCommand are complicated primarily because of different ways of handling output. There are commands that:

  1. Want to eagerly print stdout/stderr of the executed command, plus print an error message if the command fails (output mode PrintAll). Note that this error message attempts to print stdout/stderr of the command when -v is enabled, but that will always be empty, since this mode uses .status() and not .output().
  2. Want to eagerly print stdout/stderr of the executed command, but do not print any additional error message if it fails (output mode PrintOutput)
  3. Want to capture stdout/stderr of the executed command, but print an error message if it fails (output mode PrintFailure). This means that the user wants to either ignore the output or handle it programatically, but that's not obvious from the name.

The difference between 1) and 2) (unless explicitly specified) is determined dynamically based on the bootstrap verbosity level.

It is very difficult for me to wrap my head around all these modes. I think that in a future PR, we should split these axes into e.g. this:

  1. Do I want to handle the output programmatically or print it to the terminal? This should be a separate axis, true/false. (Note that "hiding the output" essentially just means saying that I handle it programmatically, and then I ignore the output).
  2. Do I want to print a message if the command fails? Yes/No/Based on verbosity (which would be the default).

Then there is also the failure mode, but that is relatively simple to handle, the command execution will just shutdown bootstrap (either eagerly or late) when the command fails.

Note that this is just a first refactoring steps, there are a lot of other things to be done, so some things might not look "final" yet. The next steps are (not necessarily in this order):

Best reviewed commit by commit.

r? @onur-ozkan

@Kobzol Kobzol deleted the bootstrap-cmd-refactor branch

June 22, 2024 15:16

bors added a commit to rust-lang-ci/rust that referenced this pull request

Jul 4, 2024

@bors

…nur-ozkan

Bootstrap command refactoring: consolidate output modes (step 3)

This PR is a continuation to rust-lang#126731. It consolidates the output modes of bootstrap (Print vs CaptureAll vs CaptureStdout) and simplifies the logic around error printing (now a command error is always printed if the failure is not ignored). It also ports even more usages of Command to BootstrapCommand, most notably the git helpers and many usages of the output function.

The last commit was added because the third commit made two variants of the Tool enum unused (no idea why, but it seems to have been a false positive that they were used before).

It can be reviewed now, but I would wait with merging until at least a few days after rust-lang#126731, just to catch any potential issues from that PR before we move further.

As a next step, I want to clean up the API of the command a little bit to make usage easier (currently it's a bit verbose), and then continue with the rest of the tasks from the tracking issue.

As always, best reviewed commit by commit.

Tracking issue: rust-lang#126819

r? @onur-ozkan

bors added a commit to rust-lang-ci/rust that referenced this pull request

Jul 4, 2024

@bors

…try>

Bootstrap command refactoring: consolidate output modes (step 3)

This PR is a continuation to rust-lang#126731. It consolidates the output modes of bootstrap (Print vs CaptureAll vs CaptureStdout) and simplifies the logic around error printing (now a command error is always printed if the failure is not ignored). It also ports even more usages of Command to BootstrapCommand, most notably the git helpers and many usages of the output function.

The last commit was added because the third commit made two variants of the Tool enum unused (no idea why, but it seems to have been a false positive that they were used before).

It can be reviewed now, but I would wait with merging until at least a few days after rust-lang#126731, just to catch any potential issues from that PR before we move further.

As a next step, I want to clean up the API of the command a little bit to make usage easier (currently it's a bit verbose), and then continue with the rest of the tasks from the tracking issue.

As always, best reviewed commit by commit.

Tracking issue: rust-lang#126819

r? @onur-ozkan

try-job: aarch64-apple

bors added a commit to rust-lang-ci/rust that referenced this pull request

Jul 4, 2024

@bors

…nur-ozkan

Bootstrap command refactoring: consolidate output modes (step 3)

This PR is a continuation to rust-lang#126731. It consolidates the output modes of bootstrap (Print vs CaptureAll vs CaptureStdout) and simplifies the logic around error printing (now a command error is always printed if the failure is not ignored). It also ports even more usages of Command to BootstrapCommand, most notably the git helpers and many usages of the output function.

The last commit was added because the third commit made two variants of the Tool enum unused (no idea why, but it seems to have been a false positive that they were used before).

It can be reviewed now, but I would wait with merging until at least a few days after rust-lang#126731, just to catch any potential issues from that PR before we move further.

As a next step, I want to clean up the API of the command a little bit to make usage easier (currently it's a bit verbose), and then continue with the rest of the tasks from the tracking issue.

As always, best reviewed commit by commit.

Tracking issue: rust-lang#126819

r? @onur-ozkan

try-job: aarch64-apple

jaisnan pushed a commit to jaisnan/rust-dev that referenced this pull request

Jul 29, 2024

@github-actions @celinval

Update Rust toolchain from nightly-2024-06-22 to nightly-2024-06-23 without any other source changes. This is an automatically generated pull request. If any of the CI checks fail, manual intervention is required. In such a case, review the changes at https://github.com/rust-lang/rust from rust-lang@c1b336c up to rust-lang@3cb521a. The log for this commit range is: rust-lang@3cb521a434 Auto merge of rust-lang#126761 - GuillaumeGomez:unsafe_extern_blocks, r=spastorino rust-lang@a0f01c3c10 Auto merge of rust-lang#126838 - matthiaskrgr:rollup-qkop22o, r=matthiaskrgr rust-lang@dc9a08f535 Rollup merge of rust-lang#126552 - fee1-dead-contrib:rmfx, r=compiler-errors rust-lang@162120b4fa Rollup merge of rust-lang#126318 - Kobzol:bootstrap-perf, r=onur-ozkan rust-lang@f3ced9d540 Rollup merge of rust-lang#126140 - eduardosm:stabilize-fs_try_exists, r=Amanieu rust-lang@f944afe380 Auto merge of rust-lang#116113 - kpreid:arcmut, r=dtolnay rust-lang@88c3db57e4 Generalize {Rc,Arc}::make_mut() to unsized types. rust-lang@a9a4830d25 Replace WriteCloneIntoRaw with CloneToUninit. rust-lang@ec201b8650 Add core::clone::CloneToUninit. rust-lang@81da6a6d40 Make effects an incomplete feature rust-lang@ac47dbad50 Auto merge of rust-lang#126824 - GuillaumeGomez:rollup-sybv8o7, r=GuillaumeGomez rust-lang@d265538016 Rollup merge of rust-lang#126823 - GuillaumeGomez:migrate-run-make-inline-always-many-cgu, r=Kobzol rust-lang@25bcc7d130 Rollup merge of rust-lang#126731 - Kobzol:bootstrap-cmd-refactor, r=onur-ozkan rust-lang@399c5cabdd Rollup merge of rust-lang#126723 - estebank:dot-dot-dot, r=Nadrieril rust-lang@3ed2cd74b5 Rollup merge of rust-lang#126686 - fmease:dump-preds-n-item-bounds, r=compiler-errors rust-lang@07e8b3ac01 Rollup merge of rust-lang#126555 - beetrees:f16-inline-asm-arm, r=Amanieu rust-lang@d03d6c0fea Auto merge of rust-lang#126750 - scottmcm:less-unlikely, r=jhpratt rust-lang@e7dfd4a913 Migrate run-make/inline-always-many-cgu to rmake.rs rust-lang@d9962bb4d8 Make read_dir method take a mutable callback rust-lang@f1b0d54ca9 Auto merge of rust-lang#126816 - weihanglo:update-cargo, r=weihanglo rust-lang@0bd58d8122 Apply review comments. rust-lang@250586cb2e Wrap std Output in CommandOutput rust-lang@f0aceed540 Auto merge of rust-lang#126817 - workingjubilee:rollup-0rg0k55, r=workingjubilee rust-lang@38bd7a0fcb Add #[rustc_dump_{predicates,item_bounds}] rust-lang@1916b3d57f Rollup merge of rust-lang#126811 - compiler-errors:tidy-ftl, r=estebank rust-lang@539090e5cd Rollup merge of rust-lang#126809 - estebank:wording-tweak, r=oli-obk rust-lang@b9ab6c3501 Rollup merge of rust-lang#126798 - miguelfrde:master, r=tmandry rust-lang@9498d5cf2f Rollup merge of rust-lang#126787 - Strophox:get-bytes, r=RalfJung rust-lang@1f9793f1aa Rollup merge of rust-lang#126722 - adwinwhite:ptr_fn_abi, r=celinval rust-lang@84b0922565 Rollup merge of rust-lang#126712 - Oneirical:bootest-constestllation, r=jieyouxu rust-lang@e7956cd994 Rollup merge of rust-lang#126530 - beetrees:f16-inline-asm-riscv, r=Amanieu rust-lang@10e1f5d212 Auto merge of rust-lang#124101 - the8472:pidfd-methods, r=cuviper rust-lang@2c65a24b8c Update cargo rust-lang@fcae62649e Auto merge of rust-lang#126758 - spastorino:avoid-safe-outside-unsafe-blocks, r=compiler-errors rust-lang@ffd72b1700 Fix remaining cases rust-lang@ea681ef281 Add a tidy rule to make sure that diagnostics don't end in periods rust-lang@8abf149bde to extract a pidfd we must consume the child rust-lang@0787c7308c Add PidFd::{kill, wait, try_wait} rust-lang@5d5892e966 Remove stray . from error message rust-lang@d94a40516e [fuchsia-test-runner] Remove usage of kw_only rust-lang@771e44ebd3 Add f16 inline ASM support for RISC-V rust-lang@753fb070bb Add f16 inline ASM support for 32-bit ARM rust-lang@22831ed117 Do not allow safe usafe on static and fn items rust-lang@a6a83d3d4e bless tests rust-lang@b512bf6f77 add as_ptr to trait AllocBytes, fix 2 impls; add pub fn get_bytes_unchecked_raw in allocation.rs; add pub fn get_alloc_bytes_unchecked_raw[_mut] in memory.rs rust-lang@02aaea1803 update intrinsic const param counting rust-lang@3b14b756d8 Remove feature(effects) from the standard library rust-lang@a314f7363a Stop using unlikely in strict_* methods rust-lang@225796a2df Add method to get FnAbi of function pointer rust-lang@630c3adb14 Add regression test for unsafe_extern_blocks rust-lang@bb9a3ef90c Implement unsafe_extern_blocks feature in rustdoc rust-lang@3c0a4bc915 rewrite crate-name-priority to rmake rust-lang@bc12972bcd Slightly refactor the dumping of HIR analysis data rust-lang@3fe4d134dd Appease clippy rust-lang@c15293407f Remove unused import rust-lang@5c4318d02c Implement run_cmd in terms of run_tracked rust-lang@0de7b92cc6 Remove run_delaying_failure rust-lang@e933cfb13c Remove run_quiet_delaying_failure rust-lang@949e667d3f Remove run_quiet rust-lang@a12f541a18 Implement new command execution logic rust-lang@9fd7784b97 Fix ... in multline code-skips in suggestions rust-lang@f22b5afa6a rewrite error-writing-dependencies to rmake rust-lang@75ee1d74a9 rewrite relocation-model to rmake rust-lang@87d2e61428 Add x perf command for profiling the compiler using rustc-perf rust-lang@fd44aca2aa Copy rustc-fake binary when building the rustc-perf tool rust-lang@9e0b76201b Add RustcPerf bootstrap tool rust-lang@9ec178df0b Add cargo_args to ToolBuild rust-lang@6a04dfe78c Rename std::fs::try_exists to std::fs::exists and stabilize fs_try_exists

Co-authored-by: celinval 35149715+celinval@users.noreply.github.com

Labels

S-waiting-on-bors

Status: Waiting on bors to run and complete tests. Bors will change the label on completion.

T-bootstrap

Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)