s2n-quic (original) (raw)

s2n-quic Continuous Integration

s2n-quic executes a comprehensive suite of tests, linters, benchmarks, simulations and other checks on each pull request and merge into main. Information about these checks is provided below.

Tests

s2n-quic defines many tests that can be executed with cargo test. These tests, described below, are executed across a variety of operating systems, architectures, and Rust versions.

Unit Tests

Unit tests validate the expected behavior of individual components of s2n-quic. Typically, unit tests will be located in a tests module at the end of the file being tested. When there are a significant number of unit test functions for given component, the tests module may be located in a separate file.

Integration Tests

s2n-quic integration tests use the public API to validate the end-to-end behavior of the library under specific scenarios and configurations. Integration tests are located in the top-level s2n-quic crate, in the tests module.

Snapshot Tests

Snapshot tests use insta to assert complex output remains consistent with expected references values. In s2n-quic, snapshot tests are typically constructed in one of two ways:

using the insta::assert_debug_snapshot macro to compare the Debug representation of an instance to the snapshot
using the event::testing::Publisher::snapshot() event publisher to assert against a snapshot of events emitted by the s2n-quic event framework

Property Tests

Property tests assert that specific properties of the output of functions and components are upheld under a variety of inputs. s2n-quic uses the Bolero property-testing framework to execute property-testing with multiple fuzzing engines as well as the Kani Rust Verifier. For more details on how Bolero and Kani are used together in s2n-quic, see the following blog posts from the Kani Rust Verifier Blog:

Fuzz Tests

Fuzz tests provide large amounts of varied input data to assert s2n-quic behaves as expected regardless of the input. Fuzz testing in s2n-quic comes in three flavors:

Component-level fuzz testing using the Bolero property-testing framework. These tests generate a corpus of inputs that is included in the s2n-quic repository to allow the fuzz tests to be replayed when executing cargo test.
End-to-end QUIC protocol-level fuzzing using quic-attack. quic-attack is a collection of features that collectively turn s2n-quic into an online QUIC protocol fuzzer. It allows incoming and outgoing datagrams and packets, as well as the port number on incoming datagrams, to be intercepted and manipulated.
UDP protocol-level fuzzing using udp-attack. udp-attack generates random UDP packets and transmits them to s2n-quic to catch issues with packet handling.

Concurrency Permutation Tests

loom is used to validate the behavior of concurrent code in s2n-quic. loom executes concurrent code using a simulation of the operating system scheduler and Rust memory model to evaluate concurrent code under all possible thread interleavings.

Interoperability

The quic-interop-runner defines a suite of test cases that ensure compatibility between QUIC implementations. s2n-quic publishes a report with the results.

Compliance

s2n-quic annotates source code with inline references to requirements in IETF RFC specifications. Duvet is used to generate a report, which makes it easy to track compliance with each requirement.

Simulations

A Monte Carlo simulation tool is used to execute thousands of randomized simulations of s2n-quic that vary one or more network variables, such as bandwidth, jitter, and round trip time. The report output provides a visual representation of the relationship between the input variables and overall performance.

A loss recovery simulation tool plots the growth of the congestion window over time under various simulated loss scenarios and publishes the results in a report to visualize changes to the congestion control algorithm.

Performance & Efficiency Profiling

Flame graphs are generated and published in a report to visualize stack traces produced under a variety of data transfer scenarios.

dhat performs heap profiling and publishes the results in a report.

Clippy

clippy is a rust linter which catches common mistakes.

Rustfmt

rustfmt ensures code is consistently formatted.

Miri

Miri detects undefined behavior and memory leaks.

Code Coverage

LLVM source-based code coverage measures how much of s2n-quic code is executed by tests. s2n-quic publishes a report with the results.