diff
3.12Terms
- Run
- One campaign of one tool on one program.
- Cores
- The parallel processes of one campaign.
- Budget
- The longest a campaign may run.
- Fixed track
- The command line is given and the tool supplies only the input. Method
- Open track
- The tool also chooses arguments, files and environment. Method
- Median
- The middle value over a tool's runs; the median of two runs is their mean.
- standard
- Conventional gcov replay of every retained test case, same counting code and denominators. Method
- Unreached
- A branch that none of the main-group tools that ran on the program reaches (4 to 12 tools).
- Union of runs
- The branches that at least one of a tool's runs reaches, each counted once.
- Output tokens
- Model output of a whole campaign, reasoning included, as self-reported in the release.
- Sanitiser signature
- The sanitiser, the error kind and the innermost frame in the program's own source; two can share a cause.
- Language model
- A system in which a language model does the central reasoning of a classical technique, such as concolic execution.
- Symbolic
- Runs the program on symbolic inputs and asks a constraint solver for concrete inputs that reach new paths.
- Fuzzer
- Coverage-guided fuzzing: mutates inputs and keeps those that reach new code.
- Hybrid
- A fuzzer paired with a concolic executor; each supplies the other with inputs.
- Main
- Shown by default: the tools a reader is most likely to compare.
- Ablation
- Variants of the agent that switch off or change one part, to show what that part adds.
- Seeding
- Runs of a tool that start from seeds produced by another system.
- Control
- Systems that run without the agent, as a reference for the main group.
- Release
- A dated snapshot of the data, named like 2026.1. It never changes; corrections go into the next release.
- Superseded
- An entry that a newer one replaces. It stays in the release and shows under all versions.
- Withdrawn
- An entry taken out of the comparison after it was published.
GNU diff compares two files or the matching files in two directories line by line and prints the differences.
- Branches
- 4,36930.8% unreached (6 tools)
- Language
- C
- Inputs
- command-line arguments, files, standard input, environment
- Fixed track
- none agreed
- Tools with runs
- 106 main
- Runs
- 3636 with branch sets
Branch map
1 tool 2352 to 3 9604 tools 4505 tools 433all 6 945Unreached 1,346
Coverage over time
Use the left and right arrow keys, Home and End to move along the time axis. The value of every line at that time is read out.
The chart needs JavaScript. The final values are in the Final coverage table below.
Showing 6 of 6
Showing 3 of 4
Showing 0 of 0
Showing 6 of 6
Curves:
curves/ in the results repositoryFinal coverage
| Tool | Median | Min to max | Share (median) | Runs | Median curve | Output tokens |
|---|---|---|---|---|---|---|
| Main group, 6 tools | ||||||
Agolic KLEE, GPT-5.5agent, open track, six cores, version 1.0.1+ | 2,072 | 1,905–2,144 | 47.4% | 3 | 227,662 | |
Agolic KLEE, DeepSeek V4.1 Flashagent, open track, six cores, version 1.0.1+ | 2,409 | 2,230–2,522 | 55.1% | 3 | 1,361,361 | |
Agolic Direct-Input, DeepSeek V4.1 Flashagent, open track, six cores, version 1.0.1+ | 2,716 | 2,716–2,727 | 62.2% | 3 | 3,258,583 | |
KLEE fixed options, 6 coressymbolic, open track, six cores, version efa1e052 | 936 | 781–982 | 21.4% | 3 | not applicable | |
KLEE uniform options, 6 coressymbolic, open track, six cores, version efa1e052 | 1,395 | 1,238–1,869 | 31.9% | 3 | not applicable | |
SymTuner, 6 coressymbolic, open track, six cores, version 71806b45 | 1,340 | 1,157–1,515 | 30.7% | 3 | not applicable | |
| Ablation group, 1 tool | ||||||
Agolic KLEE, GPT-5.4 supersededagent, open track, six cores, version unreleased+ | 1,282 | 1,154–1,775 | 29.3% | 3 | not reported | |
| Control group, 3 tools | ||||||
KLEE fixed options, 1 coresymbolic, open track, one core, version efa1e052 | 680 | 637–755 | 15.6% | 5 | not applicable | |
KLEE uniform options, 1 coresymbolic, open track, one core, version efa1e052 | 1,070 | 967–1,375 | 24.5% | 5 | not applicable | |
SymTuner, 1 coresymbolic, open track, one core, version 71806b45 | 1,240 | 847–1,894 | 28.4% | 5 | not applicable | |
finals.csv in the results repositoryBranches reached by one tool only
| Tool | Runs in union | Branches reached | Share (union) | Reached by this tool only |
|---|---|---|---|---|
Agolic KLEE, GPT-5.5 | 3 | 2,311 | 52.9% | 3 |
Agolic KLEE, DeepSeek V4.1 Flash | 3 | 2,644 | 60.5% | 24 |
Agolic Direct-Input, DeepSeek V4.1 Flash | 3 | 2,809 | 64.3% | 200 |
KLEE fixed options, 6 cores | 3 | 1,048 | 24.0% | 0 |
KLEE uniform options, 6 cores | 3 | 1,937 | 44.3% | 6 |
SymTuner, 6 cores | 3 | 1,569 | 35.9% | 2 |
Sanitiser findings
Signatures in each tool’s final test cases.
| Tool | Runs replayed | AddressSanitizerkind, signatures | UndefinedBehaviorSanitizerkind, signatures |
|---|---|---|---|
Agolic KLEE, GPT-5.5 | 3 | none found | none found |
Agolic KLEE, DeepSeek V4.1 Flash | 3 | none found | none found |
Agolic Direct-Input, DeepSeek V4.1 Flash | 3 | none found |
|
KLEE fixed options, 6 cores | 3 | none found | none found |
KLEE uniform options, 6 cores | 3 | none found | none found |
SymTuner, 6 cores | 3 | none found | none found |
KLEE fixed options, 1 core | 5 | none found | none found |
KLEE uniform options, 1 core | 5 | none found | none found |
SymTuner, 1 core | 5 | none found | none found |
Agolic KLEE, GPT-5.4 | 3 | none found |
|
Counts are of untriaged signatures. † marks fewer than three runs.