Remove test harnesses, defunct tooling by cpu · Pull Request #127 · C2SP/wycheproof (original) (raw)
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Conversation8 Commits3 Checks0 Files changed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})
This PR removes the Java and Javascript test harnesses as well as associated defunct tooling and leftover Google-specific CI bits.
What remains are the bits #105 convincingly articulates as the future of the repository: JSON test vectors (testvectors/, testvectors_v1/), the associated schemas (schemas/) and supporting documentation (doc/).
Resolves #105 (and a variety of related issues that can be closed manually after this is merged).
Upstream library authors are best positioned to maintain this sort of code, leaving the community managed Wycheproof contributors to focus on JSON test vectors.
For the same reasons articulated in the previous commit.
This relied on the now-removed Java harnesses, and was specific to an internal Google CI system that's not in use now that the repo has transitioned to community management.
I disagree with the deletion of the tests. A significant fraction of the code contains tests that are not covered by test vectors.
What should happen instead is to move the java and javascript tests into a separate component, so that the remainder of the repo does not depend on them.
A significant fraction of the code contains tests that are not covered by test vectors.
I think the best way to address that is to add new JSON vectors for what's missing.
What should happen instead is to move the java and javascript tests into a separate component, so that the remainder of the repo does not depend on them.
I believe that's best done in a separate repository. Is there a reason you think that approach would be unmanageable?
A significant fraction of the code contains tests that are not covered by test vectors.
I think the best way to address that is to add new JSON vectors for what's missing.
There are a lot of tests that cannot be translated into test vectors. Prominent test are for timing attacks,
bias in ECDSA/DSA signatures, potentially weak interfaces such as CipherInputStream and CipherOutputStream in java,
edge cases such as encrypting long messages with GCM, etc.
A main issue with the current code is that it uses a particular framework, bazel and junit, which makes it difficult to integrate
the tests into some other environment. There is already some work ongoing trying to achieve this goal.
What should happen instead is to move the java and javascript tests into a separate component, so that the remainder of the repo does not depend on them.
I believe that's best done in a separate repository. Is there a reason you think that approach would be unmanageable?
I think this is what I just proposed.
There are a lot of tests that cannot be translated into test vectors. Prominent test are for timing attacks,
bias in ECDSA/DSA signatures, potentially weak interfaces such as CipherInputStream and CipherOutputStream in java,
edge cases such as encrypting long messages with GCM, etc.
That makes sense, but I think those are great examples of things that end up being tightly coupled to specific implementations, requiring both significant maintenance and per-project/language expertise. My perspective is that this repository is better used as a place to house test data that isn't tied to any one implementation or API.
I believe that's best done in a separate repository. Is there a reason you think that approach would be unmanageable?
I think this is what I just proposed.
Ah ok! It sounds like we're on the same page then:
- Remove the test harnesses/supporting code from here
- Add it back in a separate repository
Step 2 can be done by whoever finds the removed pieces valuable and wants to own maintaining them. It doesn't need to block Step 1 since the history remains in-repo.
I think we are all on the same page that 1) dynamic tests are valuable and do increase coverage, and 2) they are different enough from test vectors that they belong in a different repository. An important part of (2) is that the expertise and resources necessary to maintain them are different, and we don't feel like we're the right set of people to do that.
The LICENSE file is very clear, so anyone can take the tests and maintain them in a different repository.
P.S. I would love to find portable ways to test at least some of these things with vectors. For example,
bias in ECDSA/DSA signatures
can be tested with vectors if the implementation uses draft-irtf-cfrg-det-sigs-with-noise-05 like we do in Go
edge cases such as encrypting long messages with GCM
can be tested with vectors if the input can be described like "generate X bytes from SHAKE" and the output can be hashed. We have such a test for cSHAKE in Go, and I'd love to port more tests like this to Wycheproof.
Filed #134 to discuss strategies for large vectors.
Member Author
cpu commented
• Loading
I think there are a handful of issues we could close now that the harnesses are removed. I'll close those today.
This was referenced
Mar 19, 2025