This is for a pre-existing FIXME, but it was easy enough to do.

- Remove download-rustc handling from bootstrap.py
- Allow a custom `pattern` in `builder.unpack()`
- Only download rustc once another part of bootstrap depends on it.

  This is somewhat necessary since the download functions rely on having a full
  `Builder`, which isn't available until after config parsing finishes.

This has no logic changes, just a move.

This statically prevents issues like https://github.com/rust-lang/rust/issues/59264,
where tools can only be built with the in-tree compiler and not beta.

Bootstrap currently provides `-Zunstable-options` for OpenBSD when using
split debuginfo - this commit provides it for all BSD targets.
Signed-off-by: NDavid Wood <david.wood@huawei.com>

Before:
```
thread 'main' panicked at 'error: no rules matched invalid', src/bootstrap/builder.rs:287:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```

After:
```
error: no `check` rules matched 'invalid'
help: run `x.py check --help --verbose` to show a list of available paths
note: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
```

after #95612, only linux and windows target are build with -Zunstable-options, 
but others platforms might use -Csplit-debuginfo

add "openbsd" target in the list of platforms using it.

This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.

Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.

Temporarily, only enable split debuginfo on Windows if not building with
the boostrap compiler as there is a bug that isn't fixed in the
bootstrap compiler which would result in `thorin` being run on Windows.
Signed-off-by: NDavid Wood <david.wood@huawei.com>

This breaks on submodules (see #96188). Disable the assertion for now until I can think of a proper
fix.

This doesn't revert any of the changes in `Step`s themselves, only what
`ShouldRun::paths` does.

Replace `run-dysutil` option with more general `split-debuginfo` option
that works on all platforms.
Signed-off-by: NDavid Wood <david.wood@huawei.com>

There's several places where the x.py command names are matched as
strings, leading to some inconsistencies and opportunities for cleanup.

* Add Format, Clean, and Setup variants to builder::Kind.
* Use Kind to parse the x.py subcommand name (including aliases)
* Match on the subcommand Kind rather than strings when handling
  options and help text.
* Several subcommands don't display any paths when run with `-h -v` even
  though the help text indicates that they should. Fix this and refactor
  so that manually keeping matches in sync isn't necessary.

Fixes #95937

This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
  - `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
  - `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
  - `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
  - `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
  - `install` was still using `src/librustc` instead of `compiler/rustc`.
  - None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.

Running `./x.py build -h -v` shows a list of available build targets,
but the short alias `./x.py b -h -v` does not. Fix so that the aliases
behave the same as their spelled out counterparts.

This PR attempts to clean up some minor spelling mistakes in comments

There were two fixes needed:
1. Use `top_stage` instead of `top_stage - 1`. There was a long and torturous comment about trying to match rustdoc's version, but it works better without the hard-coding than with.
2. Make sure that `ci-llvm/lib` is added to LD_LIBRARY_PATH. Previously the error index would be unable to load LLVM for stage0 builds.

At some point we should probably have a discussion about how rustdoc stages should be numbered;
confusion between 0/1/2 has come up several times in bootstrap now.

Note that this is still broken when using `download-rustc = true` and `--stage 1`,
but that's *really* a corner case and should affect almost no one. `--stage {0,2}`
work fine with download-rustc.

This was so verbose before that it made it hard to see what effect the flag actually had.

Before:
```
Set({test::src/tools/tidy}) not skipped for "bootstrap::test::Tidy" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Skipping Suite(test::src/test/ui) because it is excluded
Suite(test::src/test/run-pass-valgrind) not skipped for "bootstrap::test::RunPassValgrind" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Skipping Suite(test::src/test/mir-opt) because it is excluded
Suite(test::src/test/codegen) not skipped for "bootstrap::test::Codegen" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Suite(test::src/test/codegen-units) not skipped for "bootstrap::test::CodegenUnits" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Suite(test::src/test/assembly) not skipped for "bootstrap::test::Assembly" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Suite(test::src/test/incremental) not skipped for "bootstrap::test::Incremental" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Skipping Suite(test::src/test/debuginfo) because it is excluded
Skipping Suite(test::src/test/ui-fulldeps) because it is excluded
... about 100 more lines ...
```

After:
```
Skipping Suite(test::src/test/ui) because it is excluded
Skipping Suite(test::src/test/mir-opt) because it is excluded
Skipping Suite(test::src/test/debuginfo) because it is excluded
Skipping Suite(test::src/test/ui-fulldeps) because it is excluded
```

Signed-off-by: Ncodehorseman <cricis@yeah.net>

Fixes warnings from `clippy::useless_conversion` in `src/bootstrap`.

this also fixes a bug where bootstrap would try to use the fake `rustc` binary built by bootstrap -
cargo puts it in a different directory when using `cargo run` instead of x.py

The majority of the code is only used by either rustbuild or
rustc_llvm's build script. Rust_build is compiled once for rustbuild and
once for every stage. This means that the majority of the code in this
crate is needlessly compiled multiple times. By moving only the code
actually used by the respective crates to rustbuild and rustc_llvm's
build script, this needless duplicate compilation is avoided.

First, this reverts the `CFLAGS`/`CXXFLAGS` of #93918. Those flags are
already read by `cc` and populated into `Build` earlier on in the
process. We shouldn't be overriding that based on `CFLAGS`, since `cc`
also respects overrides like `CFLAGS_{TARGET}` and `HOST_CFLAGS`, which
we want to take into account.

Second, this adds the same capability to specify target-specific
versions of `LDFLAGS` as we have through `cc` for the `C*` flags:
https://github.com/alexcrichton/cc-rs#external-configuration-via-environment-variables

Note that this also necessitated an update to compiletest to treat
CXXFLAGS separately from CFLAGS.

x.py has support for excluding some steps from the invocation, but
unfortunately that's not granular enough: some steps have the same name
in different modules, and that prevents excluding only *some* of them.

As a practical example, let's say you need to run everything in `./x.py
test` except for the standard library tests, as those tests require IPv6
and need to be executed on a separate machine. Before this commit, if
you were to just run this:

    ./x.py test --exclude library/std

...the execution would fail, as that would not only exclude running the
tests for the standard library, it would also exclude generating its
documentation (breaking linkchecker).

This commit adds support for an optional module annotation in --exclude
paths, allowing the user to choose which module to exclude from:

    ./x.py test --exclude test::library/std

This maintains backward compatibility, but also allows for more ganular
exclusion. More examples on how this works:

| `--exclude`         | Docs    | Tests   |
| ------------------- | ------- | ------- |
| `library/std`       | Skipped | Skipped |
| `doc::library/std`  | Skipped | Run     |
| `test::library/std` | Run     | Skipped |

Note that the new behavior only works in the `--exclude` flag, and not
in other x.py arguments or flags yet.

Apparently it changes some tool sources and invalidates their fingerprints, forcing us to build them several times (before and after vendoring sources).
I have not dug into why vendoring actually invalidates the figreprints, but the moving the vendoring lower in the pipeline seems to avoid the issue.
I could imagine that we somehow write a .cargo/config somewhere which somehow makes subsequent builds use the vendored deps but I was not able to find anything.

I checked the sizes of generated archives pre and post patch and their are the same, so I hope there is not functional change.

Fixes #93033

DF_ORIGIN flag signifies that the object being loaded may make reference to the $ORIGIN substitution string.

Some implementations are just ignoring DF_ORIGIN and do substitution for $ORIGIN if present (whatever DF_ORIGIN pr

Set the flag inconditionally if rpath is wanted.

Previously, passing `-v` would emit an overwhelming amount of logging:

```
> Std { stage: 1, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
  > Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    > Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    < Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    > Rustc { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      > Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > StartupObjects { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        < StartupObjects { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > Libdir { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          > Sysroot { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
          < Sysroot { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        < Libdir { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Libdir { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Sysroot { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > StdLink { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Libdir { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Libdir { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        < StdLink { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      < Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
... continues for another 150 lines ...
```

This info is occasionally useful when debugging bootstrap itself, but not very useful for figuring
out why a config option was ignored or command wasn't run.  Demote it to `-vv` logging so that `-v`
is more useful.