libminizinc/tests at master · MiniZinc/libminizinc (original) (raw)
MiniZinc Testing
Setup
Requires Python 3 (on Windows 3.8 is required). Make sure you're in the tests/
directory.
pip install -r requirements.txt
Running Test Suite
To run the full test suite:
An HTML report will be generated at output/report.html
.
Use the --solvers
option to specify a subset of solvers to use in the tests. Tests which use a solver not in this list will be skipped.
pytest --solvers gecode,chuffed
Use the --driver
option to specify the directory containing the minizinc executable. Otherwise, default directories and PATH will be used to locate it.
Multiple test suites
To facilitate running the test suite with different minizinc options, specs/suites.yml
contains configurations for running tests, or a subset of tests using different options.
my-test-suite: !Suite includes: ['*'] # Globs for included .mzn files solvers: [gecode, chuffed] # The allowed solvers (if the test case itself specifies a different solver it will be skipped) strict: false # Allow tests to pass if they check against another solver (default true) options: -O3: true # Default options to pass to minizinc (merged and overwritten by individual test cases)
For example, to run the optimize-2
and no-mip-domains
configurations only:
pytest --suite optimize-2 --suite no-mip-domains
Creating/editing test cases with the web interface
A web interface can be used to create and edit test cases graphically. Python 3.8.1 or newer and Flask is required.
Start the web interface with
cd tests python -m minizinc_testing.create
Then open http://localhost:5000
.
The web interface detects .mzn
files in the tests/spec
directory. To add a new test, simply create a .mzn
containing the model, and then open the file in the web interface. Test cases can then be generated accordingly.
Creating/editing test cases manually
The test cases are defined using YAML
inside the minizinc .mzn
files. This YAML definition must be inside a block comment like the following:
/*** !Test expected: !Result solution: x: 1 ***/
Multiple cases can be specified for one .mzn
file:
/*** --- !Test ... --- !Test ... --- !Test ... ***/
YAML format
The format of the test case spec is as follows:
!Test solvers: [gecode, cbc, chuffed] # List of solvers to use (omit if all solvers should be tested) check_against: [gecode, cbc, chuffed] # List of solvers used to check results (omit if no checking is needed) extra_files: [datafile.dzn] # Data files to use if any options: # Options passed to minizinc-python's solve(), usually all_solutions if present all_solutions: true timeout: !Duration 10s expected: # The obtained result must match one of these
- !Result status: SATISFIED # Result status solution: !Solution s: 1 t: !!set {1, 2, 3} # The set containing 1, 2 and 3 u: !Range 1..10 # The range 1 to 10 (inclusive) v: [1, 2, 3] # The array with 1, 2, 3 x: !Unordered [3, 2, 1] # Ignore the order of elements in this array _output_item: !Trim | trimmed output item gets leading/trailing whitespace ignored
- !Error type: MiniZincError # Name of the error type message: Exact error message # Exact error message string (avoid using this as it's generally not portable) regex: .type-inst must be par set. # Regex the start of the string must match (run with M and S flags)
For a test to pass, at least one expected result must be a subset of the obtained result. That is, the obtained result can have more attributes, but not less, and corresponding attributes must match.
If a solution is produced that does not match any given expected output, the result is checked using another solver. If this check passes then the test passes with a warning.
Multiple solutions
When setting all_solutions: true
and the order of the returned solutions often does not matter, use !SolutionSet
for the list of solutions:
!Result status: ALL_SOLUTIONS solution: !SolutionSet
- !Solution x: 1
- !Solution x: 2
Testing FlatZinc output
Use type: compile
on a test to enable only flattening. Then !FlatZinc filename.fzn
to give files with expected results.
!Test solvers: [gecode] type: compile expected: !FlatZinc expected.fzn
TODO
- Tool for generating test cases
- Tweak YAML parsing so not so many !Tags are required
- Better documentation of how the framework operates