debuginfo: add compiler option to allow compressed debuginfo sections by durin42 · Pull Request #115358 · rust-lang/rust (original) (raw)
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Conversation27 Commits6 Checks0 Files changed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})
LLVM already supports emitting compressed debuginfo. In debuginfo=full builds, the debug section is often a large amount of data, and it typically compresses very well (3x is not unreasonable.) We add a new knob to allow debuginfo to be compressed when the matching LLVM functionality is present. Like clang, if a known-but-disabled compression mechanism is requested, we disable compression and emit uncompressed debuginfo sections.
The API is different enough on older LLVMs we just pretend the support
is missing on LLVM older than 16.
This comment has been minimized.
r? @oli-obk
(rustbot has picked a reviewer for you, use r? to override)
Status: Awaiting review from the assignee but also interested parties.
Relevant to the compiler team, which will review and decide on the PR/issue.
labels
This comment has been minimized.
This failure is what I'd expect given no way to detect missing zlib support, so maybe someone has an idea how we should check that in the test?
On Tue, Aug 29, 2023, 20:34 Rust Log Analyzer ***@***.***> wrote: The job *x86_64-gnu-llvm-15* failed! Check out the build log: (web) <https://github.com/rust-lang/rust/actions/runs/6018812772/job/16327664411> (plain) <https://github.com/rust-lang/rust/commit/d1f923b0632d4d8b656a1a2f970fb2910288ae9f/checks/16327664411/logs> *Click to see the possible cause of the failure (guessed by this bot)* ---- [run-make] tests/run-make/compressed-debuginfo stdout ---- error: make failed status: exit status: 2 command: cd "/checkout/tests/run-make/compressed-debuginfo" && AR="ar" CC="cc -ffunction-sections -fdata-sections -fPIC -m64" CXX="c++ -ffunction-sections -fdata-sections -fPIC -m64" HOST_RPATH_DIR="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" LD_LIB_PATH_ENVVAR="LD_LIBRARY_PATH" LLVM_BIN_DIR="/usr/lib/llvm-15/bin" LLVM_COMPONENTS="aarch64 aarch64asmparser aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils aggressiveinstcombine all all-targets amdgpu amdgpuasmparser amdgpucodegen amdgpudesc amdgpudisassembler amdgpuinfo amdgputargetmca amdgpuutils analysis arm armasmparser armcodegen armdesc armdisassembler arminfo armutils asmparser asmprinter avr avrasmparser avrcodegen avrdesc avrdisassembler avrinfo binaryformat bitreader bitstreamreader bitwriter bpf bpfasmparser bpfcodegen bpfdesc bpfdisassembler bpfinfo cfguard codegen core coroutines coverage debuginfocodeview debuginfodwarf debuginfogsym debuginfomsf debuginfopdb demangle dlltooldriver dwarflinker dwp engine executionengine extensions filecheck frontendopenacc frontendopenmp fuzzercli fuzzmutate globalisel hexagon hexagonasmparser hexagoncodegen hexagondesc hexagondisassembler hexagoninfo instcombine instrumentation interfacestub interpreter ipo irreader jitlink lanai lanaiasmparser lanaicodegen lanaidesc lanaidisassembler lanaiinfo libdriver lineeditor linker lto m68k m68kasmparser m68kcodegen m68kdesc m68kdisassembler m68kinfo mc mca mcdisassembler mcjit mcparser mips mipsasmparser mipscodegen mipsdesc mipsdisassembler mipsinfo mirparser msp430 msp430asmparser msp430codegen msp430desc msp430disassembler msp430info native nativecodegen nvptx nvptxcodegen nvptxdesc nvptxinfo objcarcopts objcopy object objectyaml option orcjit orcshared orctargetprocess passes perfjitevents powerpc powerpcasmparser powerpccodegen powerpcdesc powerpcdisassembler powerpcinfo profiledata remarks riscv riscvasmparser riscvcodegen riscvdesc riscvdisassembler riscvinfo runtimedyld scalaropts selectiondag sparc sparcasmparser sparccodegen sparcdesc sparcdisassembler sparcinfo support symbolize systemz systemzasmparser systemzcodegen systemzdesc systemzdisassembler systemzinfo tablegen target textapi transformutils ve veasmparser vecodegen vectorize vedesc vedisassembler veinfo webassembly webassemblyasmparser webassemblycodegen webassemblydesc webassemblydisassembler webassemblyinfo webassemblyutils windowsdriver windowsmanifest x86 x86asmparser x86codegen x86desc x86disassembler x86info x86targetmca xcore xcorecodegen xcoredesc xcoredisassembler xcoreinfo xray" LLVM_FILECHECK="/usr/lib/llvm-15/bin/FileCheck" NODE="/usr/bin/node" PYTHON="/usr/bin/python3" RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" RUSTDOC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" RUST_BUILD_STAGE="stage2-x86_64-unknown-linux-gnu" RUST_DEMANGLER="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools-bin/rust-demangler" S="/checkout" TARGET="x86_64-unknown-linux-gnu" TARGET_RPATH_DIR="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" TMPDIR="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/compressed-debuginfo/compressed-debuginfo" "make" --- stdout ------------------------------- LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/compressed-debuginfo/compressed-debuginfo:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/compressed-debuginfo/compressed-debuginfo -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/compressed-debuginfo/compressed-debuginfo -Ainternal_features --crate-name=foo --crate-type=lib --emit=obj -C debuginfo=full -Z debuginfo-compression=zlib foo.rs readelf -t /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/compressed-debuginfo/compressed-debuginfo/foo.o | grep -q ZLIB --- stderr ------------------------------- --- stderr ------------------------------- make: *** [Makefile:12: zlib] Error 1 Build completed unsuccessfully in 0:25:21 — Reply to this email directly, view it on GitHub <#115358 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAE6LPCZENHNF2UQSMI4WLXX2C7ZANCNFSM6AAAAAA4DPEYM4> . You are receiving this because you authored the thread.Message ID: ***@***.***>
is this failure because our CI does not have zlib installed?
That's my assumption - I'm not familiar enough with the Rust project's LLVM setup to know how to check.
I'm also not sure how to make the test gracefully degrade if zlib is missing. :(
If compression is requested but not available, a warning might be appropriate. Then you could check for presence of the warning.
Right - I'm not sure how I should be printing a warning from where I've got access to the LLVM API in question. Any pointers?
Alternatively, if I can run ld.lld in the test, I can check what compression methods it supports.
Oh, I might have had the dumb. I may see what I missed on my last attempt to print the warning...
Ah, I remember the stumbling block I hit: rustc_codegen_llvm isn't currently used by rustc_session, and that seems like a layering violation to me naively, so I can't call into the FFI functions to look if the requested compression is available from there. But by the time we're in a location where we can check that in rustc_codegen_llvm, I don't seem to have a way to emit warnings at all (eg from target_machine_factory) - so I'm stumped where I should do this - if there's an example on how to emit the warning based on the codegen backend, I'm sadly missing it. :(
Any hints?
Manishearth got me sorted as far as I can tell on printing the diagnostic, and I came up with some Shell Crimes for the test in the makefile, so I think this is ready for review.
This comment has been minimized.
This comment has been minimized.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just a style nit, then this lgtm
This comment has been minimized.
📌 Commit 8577e3e has been approved by oli-obk
It is now in the queue for this repository.
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
fmease added a commit to fmease/rust that referenced this pull request
debuginfo: add compiler option to allow compressed debuginfo sections
LLVM already supports emitting compressed debuginfo. In debuginfo=full builds, the debug section is often a large amount of data, and it typically compresses very well (3x is not unreasonable.) We add a new knob to allow debuginfo to be compressed when the matching LLVM functionality is present. Like clang, if a known-but-disabled compression mechanism is requested, we disable compression and emit uncompressed debuginfo sections.
The API is different enough on older LLVMs we just pretend the support is missing on LLVM older than 16.
Using ld.lld may have been clever, but that was getting the /system/
ld.lld, not one we may have built as part of building llvm. By using the
warning message coming directly from rustc we now correctly skip the
zlib and zstd tests when the support is missing.
Rebased (thus the force push) and reworked test to actually do what it says on the tin.
@rustbot label -S-waiting-on-author +S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
and removed S-waiting-on-author
Status: This is awaiting some action (such as code changes or more information) from the author.
labels
📌 Commit 78d805c has been approved by oli-obk
It is now in the queue for this repository.
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
bors added a commit to rust-lang-ci/rust that referenced this pull request
…iaskrgr
Rollup of 5 pull requests
Successful merges:
- rust-lang#113807 (Tests crash from inappropriate use of common linkage)
- rust-lang#115358 (debuginfo: add compiler option to allow compressed debuginfo sections)
- rust-lang#115630 (Dont suggest use between
useand cfg attr) - rust-lang#115662 (Improve "associated type not found" diagnostics)
- rust-lang#115673 (Fix sanitize/cfg.rs test)
r? @ghost
@rustbot modify labels: rollup
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request
Rollup merge of rust-lang#115358 - durin42:compress-debuginfo, r=oli-obk
debuginfo: add compiler option to allow compressed debuginfo sections
LLVM already supports emitting compressed debuginfo. In debuginfo=full builds, the debug section is often a large amount of data, and it typically compresses very well (3x is not unreasonable.) We add a new knob to allow debuginfo to be compressed when the matching LLVM functionality is present. Like clang, if a known-but-disabled compression mechanism is requested, we disable compression and emit uncompressed debuginfo sections.
The API is different enough on older LLVMs we just pretend the support is missing on LLVM older than 16.
Just tried this option got warning: unknown debuginfo compression algorithm zstd - will fall back to uncompressed debuginfo, how to use zstd as compression method?
That means the LLVM your rustc was compiled against didn't have zstandard support enabled. zlib is probably already on, maybe try that?
This was referenced
Feb 11, 2024
fbq pushed a commit to Rust-for-Linux/linux that referenced this pull request
fbq pushed a commit to Rust-for-Linux/linux that referenced this pull request
fbq pushed a commit to Rust-for-Linux/linux that referenced this pull request
fbq pushed a commit to Rust-for-Linux/linux that referenced this pull request
ojeda added a commit to ojeda/linux that referenced this pull request
ojeda added a commit to Rust-for-Linux/linux that referenced this pull request
Rust 1.74.0 introduced (unstable) support for the
-Zdebuginfo-compression flag, thus use it.
Note that the releases built by the Rust project (i.e. the ones provided by rustup) do not enable support for zstd in their bundled LLVM (yet, at least), thus the Rust compiler will warn, but the build will proceed:
warning: unknown debuginfo compression algorithm zstd - will fall
back to uncompressed debuginfoLink: rust-lang/rust#120953 Link: rust-lang/rust#115358 Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org [ Added note about zstd support in Rust-provided binaries. ] Signed-off-by: Miguel Ojeda ojeda@kernel.org
ojeda added a commit to Rust-for-Linux/linux that referenced this pull request
Rust 1.74.0 introduced (unstable) support for the
-Zdebuginfo-compression flag, thus use it.
Note that the releases built by the Rust project (i.e. the ones provided by rustup) do not enable support for zstd in their bundled LLVM (yet, at least), thus the Rust compiler will warn, but the build will proceed:
warning: unknown debuginfo compression algorithm zstd - will fall
back to uncompressed debuginfoLink: rust-lang/rust#120953 Link: rust-lang/rust#115358 Reviewed-by: Alice Ryhl aliceryhl@google.com Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org [ Added note about zstd support in Rust-provided binaries. ] Signed-off-by: Miguel Ojeda ojeda@kernel.org
sweettea pushed a commit to sweettea/btrfs-fscrypt that referenced this pull request
Rust 1.74.0 introduced (unstable) support for the
-Zdebuginfo-compression flag, thus use it.
Note that the releases built by the Rust project (i.e. the ones provided by rustup) do not enable support for zstd in their bundled LLVM (yet, at least), thus the Rust compiler will warn, but the build will proceed:
warning: unknown debuginfo compression algorithm zstd - will fall
back to uncompressed debuginfoLink: rust-lang/rust#120953 Link: rust-lang/rust#115358 Reviewed-by: Alice Ryhl aliceryhl@google.com Link: https://lore.kernel.org/r/20240217002622.57322-1-ojeda@kernel.org [ Added note about zstd support in Rust-provided binaries. ] Signed-off-by: Miguel Ojeda ojeda@kernel.org
Labels
Unstable option: debuginfo compression
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Relevant to the compiler team, which will review and decide on the PR/issue.