Cache fuchsia checkout in a docker layer by erickt · Pull Request #138683 · rust-lang/rust (original) (raw)
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})
This allows the Fuchsia checkout to be cached and shared across builds, which avoids any rate limits with checking out Fuchsia. Note though that the Fuchsia checkout is about 27GiB as of checkout, which might cause separate problems.
try-job: x86_64-fuchsia
r? @marcoieni
rustbot has assigned @marcoieni.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.
Use r? to explicitly pick a reviewer
Area: The testsuite used to check the correctness of rustc
Status: Awaiting review from the assignee but also interested parties.
Relevant to the infrastructure team, which will review and decide on the PR/issue.
labels
bors added a commit to rust-lang-ci/rust that referenced this pull request
Cache fuchsia checkout in a docker layer
This allows the Fuchsia checkout to be cached and shared across builds, which avoids any rate limits with checking out Fuchsia. Note though that the Fuchsia checkout is about 27GiB as of checkout, which might cause separate problems.
try-job: x86_64-fuchsia
This comment has been minimized.
Status: This is awaiting some action (such as code changes or more information) from the author.
and removed S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
This allows the Fuchsia checkout to be cached and shared across builds, which avoids any rate limits with checking out Fuchsia. Note though that the Fuchsia checkout is about 27GiB as of checkout, which might cause separate problems.
try-job: x86_64-fuchsia
Fuchsia checkout is about 27GiB as of checkout
git sparse checkout?
Our Docker builds are automatically cached, and unless their inputs change, they are not rebuilt on CI. So I'm not sure how much more caching we could do, tbh.
Our Docker builds are automatically cached, and unless their inputs change, they are not rebuilt on CI. So I'm not sure how much more caching we could do, tbh.
@Kobzol - I'm not too familiar with how the docker run.sh scripts work, but if I'm reading src/ci/docker/host-x86_64/x86_64-fuchsia/Dockerfile, the ENV SCRIPT ... at the send suggests to me we're running the build-fuchsia.sh script outside of the cache. So factoring out the checkout might avoid all that IO.
Fuchsia checkout is about 27GiB as of checkout
git sparse checkout?
@klensy: we're working on a way to use sparse checkouts, but that's not actually the majority of the issue. Turns out most of our artifacts are prebuilt binaries, like our Go and Dart toolchains, which we really don't need to run our limited tests for Rust. I don't think we have an easy way to filter out these artifacts from being download, but I might be able to just delete them from the docker directory to remove unnecessary things though. I'll experiment with this.
Our Docker builds are automatically cached, and unless their inputs change, they are not rebuilt on CI. So I'm not sure how much more caching we could do, tbh.
@Kobzol - I'm not too familiar with how the docker run.sh scripts work, but if I'm reading src/ci/docker/host-x86_64/x86_64-fuchsia/Dockerfile, the
ENV SCRIPT ...at the send suggests to me we're running thebuild-fuchsia.shscript outside of the cache. So factoring out the checkout might avoid all that IO.Fuchsia checkout is about 27GiB as of checkout
git sparse checkout?
@klensy: we're working on a way to use sparse checkouts, but that's not actually the majority of the issue. Turns out most of our artifacts are prebuilt binaries, like our Go and Dart toolchains, which we really don't need to run our limited tests for Rust. I don't think we have an easy way to filter out these artifacts from being download, but I might be able to just delete them from the docker directory to remove unnecessary things though. I'll experiment with this.
Oh, I see, I didn't notice that. Well, that actually seems quite wasteful! I suppose that we could move the download to a Docker layer (so run it during docker build, which is cached), and then only do bash scripts/rust/build_fuchsia_from_rust_ci.sh when the container runs (so run it during docker run? I don't know if the recent errors were in the download or build phase though.
Labels
Area: The testsuite used to check the correctness of rustc
Status: This is awaiting some action (such as code changes or more information) from the author.
Relevant to the infrastructure team, which will review and decide on the PR/issue.