this also fixes a bug where bootstrap would try to use the fake `rustc` binary built by bootstrap -
cargo puts it in a different directory when using `cargo run` instead of x.py

Same rationale as https://github.com/rust-lang/rust/pull/76544;
it would be nice to make python entirely optional at some point.

This also removes $ROOT as an option for the build directory; I haven't been using it, and like Alex
said in https://github.com/rust-lang/rust/pull/76544#discussion_r488248930 it seems like a
misfeature.

This allows running `cargo run` from src/bootstrap, although that still gives
lots of compile errors if you don't use the beta toolchain.

The majority of the code is only used by either rustbuild or
rustc_llvm's build script. Rust_build is compiled once for rustbuild and
once for every stage. This means that the majority of the code in this
crate is needlessly compiled multiple times. By moving only the code
actually used by the respective crates to rustbuild and rustc_llvm's
build script, this needless duplicate compilation is avoided.

x.py has support for excluding some steps from the invocation, but
unfortunately that's not granular enough: some steps have the same name
in different modules, and that prevents excluding only *some* of them.

As a practical example, let's say you need to run everything in `./x.py
test` except for the standard library tests, as those tests require IPv6
and need to be executed on a separate machine. Before this commit, if
you were to just run this:

    ./x.py test --exclude library/std

...the execution would fail, as that would not only exclude running the
tests for the standard library, it would also exclude generating its
documentation (breaking linkchecker).

This commit adds support for an optional module annotation in --exclude
paths, allowing the user to choose which module to exclude from:

    ./x.py test --exclude test::library/std

This maintains backward compatibility, but also allows for more ganular
exclusion. More examples on how this works:

| `--exclude`         | Docs    | Tests   |
| ------------------- | ------- | ------- |
| `library/std`       | Skipped | Skipped |
| `doc::library/std`  | Skipped | Run     |
| `test::library/std` | Run     | Skipped |

Note that the new behavior only works in the `--exclude` flag, and not
in other x.py arguments or flags yet.

The task of the macro is simple enough that a decl macro is almost ten
times shorter than the original proc macro. The proc macro is 159 lines
while the decl macro is just 18 lines.

This reduces the amount of dependencies of rustbuild from 45 to 37. It
also slight reduces compilation time from 47s to 44s for debug builds.

I'm working on some LLVM patches in concert with a Rust patch, and it's
helping me quite a bit to have this as an option. It doesn't seem that
hard, so I figured I'd formalize it in x.py and send it upstream.

On NixOS systems, bootstrap will patch rustc used in bootstrapping after
checking `/etc/os-release` (to confirm the current distribution is NixOS).
However, when using Nix on a non-NixOS system, it can be desirable for
bootstrap to patch rustc. In this commit, a `patch-binaries-for-nix`
option is added to `config.toml`, which allows for user opt-in to
bootstrap's Nix patching.
Signed-off-by: NDavid Wood <david.wood@huawei.com>

This shows up to 5% less instruction counts on multiple benchmarks, and up to
19% wins on the -j1 wall times for rustc self-compilation.

We can afford to spend the extra cycles building LLVM essentially once more for
the x86_64-unknown-linux-gnu CI build today. The builder finishes in around 50
minutes on average, and this adds just 10 more minutes. Given the sizeable
improvements in compiler performance, this is definitely worth it.

The options are `overflow-checks` and `overflow-checks-std`
defaulting to false.

Previously it would error out:

```
$ x check --dry-run
thread 'main' panicked at 'std::fs::read_to_string(ci_llvm.join("link-type.txt")) failed with No such file or directory (os error 2) ("CI llvm missing: /home/joshua/rustc3/build/tmp-dry-run/x86_64-unknown-linux-gnu/ci-llvm")', src/bootstrap/config.rs:795:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:00:10
```

- Add an assertion that `link-shared = true` when `thin-lto = true`.
  Previously, link-shared would be silently overwritten.

- Get rid of `Option<bool>` in bootstrap/config.rs. Set defaults
  immediately instead of delaying until later in bootstrap. This makes
  it easier to find what the default value is.

- Remove redundant `config.x = false` when the default was already false
- Set defaults for `bindir` in `default_opts()` instead of `parse()`
- Update `download-ci-llvm = if-supported` option to match bootstrap.py
- Remove redundant check for link_shared. Previously, it was checked twice.

- Update various options in config.toml.example to their defaults.
  Previously, some options showed an example value instead of the
  default value.

- Fix incorrect defaults in config.toml.example
  + `use-libcxx` defaults to false
  + Add missing `check-stage = 0`
  + Update several defaults to be conditional (e.g. `if incremental { 10 } else { 100 }`)

- Remove redundant defaults in prose
- Use the same comment for the default and target-dependent `musl-root`
- Fix typos
- Link to `cc_detect` for `cc` and `cxx`, since the logic is ... complicated.
- Update more defaults to better reflect how they actually get set
- Remove ignored `gpg-password-file` option

  This stopped being used in
  https://github.com/rust-lang/rust/commit/7704d35accfe1b587ce41ea09ca3bf6a47aca117,
  but was never removed from config.toml.

- Remove unused flags from `config.toml`
    + Disallow `infodir` and `localstatedir` in `config.toml`
    + Allow the flags in `./configure`, but give a warning that they will be
      ignored.
    + Fix incorrect comment that `datadir` will be ignored.

    Example output:

    ```
    $ ./configure --set install.infodir=xxx
    configure: processing command line
    configure:
    configure: install.infodir      := xxx
    configure: build.configure-args := ['--set', 'install.infodir=xxx']
    warning: infodir will be ignored
    configure:
    configure: writing `config.toml` in current directory
    configure:
    configure: run `python /home/joshua/rustc3/x.py --help`
    configure:
    ```

- Update CHANGELOG
- Add "as an example" where appropriate
- Link to an issue instead of to ephemeral chats

This allows keeping the setting to a fixed value without having to
toggle it when you want to work on the compiler instead of on tools.

 ## Motivation

This avoids having to rebuild bootstrap and tidy each time you rebase
over master. In particular, it makes rebasing and running `x.py fmt` on
each commit in a branch significantly faster. It also avoids having to
rebuild bootstrap after setting `download-rustc = true`.

 ## Implementation

Instead of extracting the CI artifacts directly to `stage0/`, extract
them to `ci-rustc/` instead. Continue to copy them to the proper
sysroots as necessary for all stages except stage 0.

This also requires `bootstrap.py` to download both stage0 and CI
artifacts and distinguish between the two when checking stamp files.

Note that since tools have to be built by the same compiler that built
`rustc-dev` and the standard library, the downloaded artifacts can't be
reused when building with the beta compiler. To make sure this is still
a good user experience, warn when building with the beta compiler, and
default to building with stage 2.

This way, you can debug rustdoc's JavaScript and CSS file
with normal F12 Dev Tools and you'll have useful line numbers
to work with.

Intended to piggy-back on output from existing build.print_step_timings setting.

- Use the same compiler for stage0 and stage1. This should be fixed at
  some point (so bootstrap isn't constantly rebuilt).
- Make sure `x.py build` and `x.py check` work.
- Use `git merge-base` to determine the most recent commit to download.
- Copy stage0 to the various sysroots in `Sysroot`, and delegate to
  Sysroot in Assemble. Leave all other code unchanged.
- Rename date -> key

  This can also be a commit hash, so 'date' is no longer a good name.

- Add the commented-out option to config.toml.example
- Disable all steps by default when `download-rustc` is enabled

  Most steps don't make sense when downloading a compiler, because they'll
  be pre-built in the sysroot. Only enable the ones that might be useful,
  in particular Rustdoc and all `check` steps.

  At some point, this should probably enable other tools, but rustdoc is
  enough to test out `download-rustc`.

- Don't print 'Skipping' twice in a row

  Bootstrap forcibly enables a dry run if it isn't already set, so
  previously it would print the message twice:

  ```
  Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
  Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
  ```

  Now it correctly only prints once.

 ## Future work

- Add FIXME about supporting beta commits
- Debug logging will never work. This should be fixed.

- Don't ignore packaging `llvm/lib/` for `rust-dev` when LLVM is linked
statically
- Add `link-type.txt` so bootstrap knows whether llvm was linked
  statically or dynamically
- Don't assume CI LLVM is linked dynamically in `bootstrap::config`
- Fall back to dynamic linking if `link-type.txt` doesn't exist
- Fix existing bug that split the output of `llvm-config` on lines, not spaces
- Enable building LLVM tests

  This works around the following llvm bug:

  ```
  llvm-config: error: component libraries and shared library

  llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest.a
  llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest_main.a
  llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVMTestingSupport.a
  thread 'main' panicked at 'command did not execute successfully: "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/bin/llvm-config" "--libfiles"
  ```

  I'm not sure why llvm-config thinks these are required, but to avoid
  the error, this builds them anyway.

- Temporarily set windows as the try builder. This should be reverted
  before merging.

- Bump version of `download-ci-llvm-stamp`

  `src/llvm-project` hasn't changed, but the generated tarball has.

- Only special case MacOS when dynamic linking. Static linking works fine.
- Store `link-type.txt` to the top-level of the tarball

  This allows writing the link type unconditionally. Previously, bootstrap
  had to keep track of whether the file IO *would* succeed (it would fail
  if `lib/` didn't exist), which was prone to bugs.

- Make `link-type.txt` required

  Anyone downloading this from CI should be using a version of bootstrap
  that matches the version of the uploaded artifacts. So a missing
  link-type indicates a bug in x.py.

addresses:

clippy::or_fun_call
clippy::single_char_add_str
clippy::comparison_to_empty
clippy::or_fun_call

The option allows to add or remove compression formats used while
producing dist tarballs.

This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.

For the same reason of avoiding greater time commitment, this currently avoids
implementing for platforms outside of x86_64-unknown-linux-gnu, though in
practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.

The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.

The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour up from 45 minutes).

The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.

`dsymutil` adds time to builds on Apple platforms for no clear benefit, and also
makes it more difficult for debuggers to find debug info. The compiler currently
defaults to running `dsymutil` to preserve its historical default, but when
compiling the compiler itself, we skip it by default since we know it's safe to
do so in that case.