Submit results
A submission is a set of test cases. The maintainers approve each one by hand, which can take a few working days, then replay it on each program’s pinned build. SUBMITTING.md has the steps and commands.
- Run the tool. Choose any programs; each needs three runs, six hours by default. Keep every test case and its time. No fee.
- Convert and validate. An adapter turns each run into a bundle and
commonreplay validatechecks it. - Check for crashes. Run
commonreplay signatureson each bundle. If it reports any, restrict that program’s archive or leave out the crashing cases until the disclosure period is over. - Deposit the bundles. Each goes to a permanent archive that issues a DOI, such as Zenodo, with its SHA-256.
- Open a submission. The submission form asks for the tool, track, budget, model, bundle DOIs and hashes and your consent to publish under CC BY 4.0.
Propose a program
The program proposal form asks for the program and version, the source archive and its SHA-256, an OSI-approved licence, the input surfaces and a build recipe. PROPOSING-A-PROGRAM.md has the conditions.
Report a problem
| What is wrong | Where to report it |
|---|---|
| The runner, validator, adapters or specification | The bug form. Changes to the specification go through the specification change form. |
| A wrong or mislabelled number | The data form. The correction goes into the next release. |
| A security problem in the runner | A private report, as SECURITY.md describes. |
| A bug in one of the 18 programs | The project’s maintainers through its own reporting channel, under the disclosure policy below. |
| The website | The website form. |
Disclosure policy
Error kinds and counts of sanitiser findings are published for every program. Locations, stack traces and inputs of bugs not yet fixed upstream stay unpublished. Each count is of untriaged signatures.
A finding that looks like a real bug is reported to the project’s maintainers through its own reporting channel. Details may become public with the fix or 90 days after that report and the maintainers may then credit the finders. Submitters do not publish details of an unfixed bug. Crashing inputs go in a restricted record or stay out. DISCLOSURE.md has the full policy.