gh-99108: Build the hashlib HACL* code as a static library. (fix wasm builds) by gpshead · Pull Request #101917 · python/cpython (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation8 Commits4 Checks0 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

gpshead

A followup to #101707 which broke some builds. This builds HACL* as a library in one place.

This doesn't solve all the problems, the wasm-wasi build is fixed by this. The wasm-enscripten build is not. Because it has a linker that can't cope with the same .a file being listed twice on its command line and decides to ignorantly process that as two inputs with duplicate symbols.

@gpshead

@gpshead

@gpshead

@bedevere-bot

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 3c7e18e 🤖

The command will test the builders whose names match following regular expression: wasm

The builders matched are:

@gpshead

The wasm-wasi bot using llvm-ar to build libpython3.12.a appears happy with this.
The wasm-enscripten ones that use emscripten/emar to build libpython3.12.a still does not.

@gpshead

is this the problem? the .a file is showing up on the emcc linking command line twice, is that really the source of the "duplicate" symbols? is emcc that dumb?

@gpshead

@bedevere-bot

🤖 New build scheduled with the buildbot fleet by @gpshead for commit a5bb63e 🤖

The command will test the builders whose names match following regular expression: wasm

The builders matched are:

@gpshead

Emscripten emcc which calls wasm-ld winds up with a duplicate of -LModules/_hacl -lHacl_Streaming_SHA2 on its command line or a direct Modules/_hacl/libHacl_Streaming_SHA2.a on its command line (the emcc script turns the first into the latter regardless). And whatever version of enscripten wasm-ld this is... is brain dead and does not deduplicate input files. Thus reporting "duplicate symbol" because the same .a is listed twice. 💩

Real linkers (like clang and thus llvm's ld.lld) do not have a problem with this.

@gpshead gpshead changed the titleBuild the hashlib HACL* code as a static library. gh:99108: Build the hashlib HACL* code as a static library.

Feb 14, 2023

@gpshead gpshead changed the titlegh:99108: Build the hashlib HACL* code as a static library. gh-99108: Build the hashlib HACL* code as a static library.

Feb 14, 2023

@gpshead gpshead marked this pull request as ready for review

February 14, 2023 22:44

@gpshead

@gpshead

Realistically: We should probably just combine _sha256 and _sha512 into a single _sha2 module as another way to work around this. The .so files for each of those are carrying the code for both regardless given how we link the shared libraries.

@gpshead gpshead changed the titlegh-99108: Build the hashlib HACL* code as a static library. gh-99108: Build the hashlib HACL* code as a static library. (fix wasm builds)

Feb 14, 2023

@gpshead

Realistically: We should probably just combine _sha256 and _sha512 into a single _sha2 module as another way to work around this. The .so files for each of those are carrying the code for both regardless given how we link the shared libraries.

I'll do that as its own follow on PR. This one at least addresses 2 of the 4 bot failures and defines the static library as we want regardless.

2 participants

@gpshead @bedevere-bot