Confetti
1.0.0-beta.4Terms
- Run
- One campaign of one tool on one program.
- Cores
- The parallel processes of one campaign.
- Budget
- The longest a campaign may run.
- Fixed track
- The command line is given and the tool supplies only the input. Method
- Open track
- The tool also chooses arguments, files and environment. Method
- Median
- The middle value over a tool's runs; the median of two runs is their mean.
- standard
- Conventional gcov replay of every retained test case, same counting code and denominators. Method
- Unreached
- A branch that none of the main-group tools that ran on the program reaches (4 to 12 tools).
- Union of runs
- The branches that at least one of a tool's runs reaches, each counted once.
- Output tokens
- Model output of a whole campaign, reasoning included, as self-reported in the release.
- Sanitiser signature
- The sanitiser, the error kind and the innermost frame in the program's own source; two can share a cause.
- Language model
- A system in which a language model does the central reasoning of a classical technique, such as concolic execution.
- Symbolic
- Runs the program on symbolic inputs and asks a constraint solver for concrete inputs that reach new paths.
- Fuzzer
- Coverage-guided fuzzing: mutates inputs and keeps those that reach new code.
- Hybrid
- A fuzzer paired with a concolic executor; each supplies the other with inputs.
- Main
- Shown by default: the tools a reader is most likely to compare.
- Ablation
- Variants of the agent that switch off or change one part, to show what that part adds.
- Seeding
- Runs of a tool that start from seeds produced by another system.
- Control
- Systems that run without the agent, as a reference for the main group.
- Release
- A dated snapshot of the data, named like 2026.1. It never changes; corrections go into the next release.
- Superseded
- An entry that a newer one replaces. It stays in the release and shows under all versions.
- Withdrawn
- An entry taken out of the comparison after it was published.
Confetti is a configuration-file language and a parser library written in C.
- Branches
- 63248.9% unreached (12 tools)
- Language
- C
- Inputs
- standard input
- Fixed track
- standard input
- Tools with runs
- 2812 main
- Runs
- 8989 with branch sets
Branch map
1 tool 02 to 5 06 to 8 69 to 11 114all 12 203Unreached 309
Coverage over time
Use the left and right arrow keys, Home and End to move along the time axis. The value of every line at that time is read out.
The chart needs JavaScript. The final values are in the Final coverage table below.
Showing 12 of 12
Showing 13 of 14
Showing 2 of 2
Showing 8 of 8
Curves:
curves/ in the results repositoryFinal coverage
† marks fewer than three runs.
| Tool | Median | Min to max | Share (median) | Runs | Median curve | Output tokens |
|---|---|---|---|---|---|---|
| Main group, 12 tools | ||||||
Agolic KLEE, GPT-5.5agent, open track, six cores, version 1.0.1+ | 316 | 315–319 | 50.0% | 3 | 46,743 | |
Agolic KLEE, DeepSeek V4.1 Flashagent, open track, six cores, version 1.0.1+ | 323 | 315–323 | 51.1% | 3 | 217,832 | |
Agolic Direct-Input, DeepSeek V4.1 Flashagent, open track, six cores, version 1.0.1+ | 323 | 323–323 | 51.1% | 3 | 108,478 | |
Agolic AFL++, DeepSeek V4.1 Flashagent, open track, six cores, version 1.0.1+ | 323 | 323–323 | 51.1% | 3 | 38,835 | |
ConcoLLMic, GPT-5.5language model, open track, version 9634e686+ | 311.5† | 311–312 | 49.3% | 2† | not reported | |
ConcoLLMic, DeepSeek V4.1 Flashlanguage model, open track, version 9634e686+ | 313 | 307–314 | 49.5% | 3 | 802,214 | |
KLEE fixed options, 6 coressymbolic, open track, six cores, version efa1e052 | 204 | 154–204 | 32.3% | 3 | not applicable | |
KLEE uniform options, 6 coressymbolic, open track, six cores, version efa1e052 | 257 | 231–258 | 40.7% | 3 | not applicable | |
SymTuner, 6 coressymbolic, open track, six cores, version 71806b45 | 261 | 254–272 | 41.3% | 3 | not applicable | |
AFL++, 1 core fixed trackfuzzer, fixed track, one core, version 5.02c | 323 | 307–323 | 51.1% | 3 | not applicable | |
SymCC + AFL++, 1 core fixed trackhybrid, fixed track, one core, version 07c8895f+ | 320 | 320–320 | 50.6% | 3 | not applicable | |
SymSan + AFL++, 1 core fixed trackhybrid, fixed track, one core, version 2937d1c0+ | 323 | 323–323 | 51.1% | 3 | not applicable | |
| Ablation group, 10 tools | ||||||
Agolic KLEE, GPT-5.5, effort lowagent, open track, six cores, version 1.0.1+ | 311 | 303–312 | 49.2% | 3 | 47,808 | |
Agolic KLEE, GPT-5.5, effort extra highagent, open track, six cores, version 1.0.1+ | 322 | 319–323 | 50.9% | 3 | 71,020 | |
Agolic KLEE, GPT-5.5, Harness-Entry Mode onlyagent, open track, six cores, version 1.0.1+ | 222 | 216–252 | 35.1% | 3 | 88,441 | |
Agolic KLEE, GPT-5.5, Seeded-Prefix Mode onlyagent, open track, six cores, version 1.0.1+ | 323 | 307–323 | 51.1% | 3 | 34,185 | |
Agolic KLEE, GPT-5.5, seed finder offagent, open track, six cores, version 1.0.1+ | 322 | 313–322 | 50.9% | 3 | 53,141 | |
Agolic KLEE, GPT-5.5, proposal review offagent, open track, six cores, version 1.0.1+ | 320 | 306–323 | 50.6% | 3 | 48,676 | |
Agolic KLEE, GPT-5.5, run history withheldagent, open track, six cores, version unknown | 322 | 319–322 | 50.9% | 3 | 179,266 | |
Agolic KLEE, DeepSeek V4.1 Flash, effort lowagent, open track, six cores, version 1.0.1+ | 321 | 321–323 | 50.8% | 3 | 214,927 | |
Agolic KLEE, DeepSeek V4.1 Flash, effort highagent, open track, six cores, version 1.0.1+ | 323 | 323–323 | 51.1% | 3 | 265,385 | |
Agolic KLEE, GPT-5.4 supersededagent, open track, six cores, version unreleased+ | 318 | 314–320 | 50.3% | 3 | not reported | |
| Seeding group, 2 tools | ||||||
SymTuner rerun, 6 coressymbolic, open track, six cores, version 71806b45 | 255 | 254–264 | 40.3% | 3 | not applicable | |
SymTuner from the agent's seeds, 6 cores, its own testssymbolic, open track, six cores, version 71806b45 | 315 | 312–316 | 49.8% | 3 | not applicable | |
| Control group, 4 tools | ||||||
AFL++ rebuilt targets fixed trackfuzzer, fixed track, version 5.02c | 323 | 323–323 | 51.1% | 3 | not applicable | |
KLEE fixed options, 1 coresymbolic, open track, one core, version efa1e052 | 145 | 138–199 | 22.9% | 5 | not applicable | |
KLEE uniform options, 1 coresymbolic, open track, one core, version efa1e052 | 215 | 213–258 | 34.0% | 5 | not applicable | |
SymTuner, 1 coresymbolic, open track, one core, version 71806b45 | 251 | 214–260 | 39.7% | 5 | not applicable | |
finals.csv in the results repositoryBranches reached by one tool only
| Tool | Runs in union | Branches reached | Share (union) | Reached by this tool only |
|---|---|---|---|---|
Agolic KLEE, GPT-5.5 | 3 | 322 | 50.9% | 0 |
Agolic KLEE, DeepSeek V4.1 Flash | 3 | 323 | 51.1% | 0 |
Agolic Direct-Input, DeepSeek V4.1 Flash | 3 | 323 | 51.1% | 0 |
Agolic AFL++, DeepSeek V4.1 Flash | 3 | 323 | 51.1% | 0 |
ConcoLLMic, GPT-5.5 | 2† | 315† | 49.8% | 0 |
ConcoLLMic, DeepSeek V4.1 Flash | 3 | 316 | 50.0% | 0 |
KLEE fixed options, 6 cores | 3 | 205 | 32.4% | 0 |
KLEE uniform options, 6 cores | 3 | 261 | 41.3% | 0 |
SymTuner, 6 cores | 3 | 278 | 44.0% | 0 |
AFL++, 1 core | 3 | 323 | 51.1% | 0 |
SymCC + AFL++, 1 core | 3 | 320 | 50.6% | 0 |
SymSan + AFL++, 1 core | 3 | 323 | 51.1% | 0 |
Sanitiser findings
Signatures in each tool’s final test cases.
| Tool | Runs replayed | AddressSanitizerkind, signatures | UndefinedBehaviorSanitizerkind, signatures |
|---|---|---|---|
Agolic KLEE, GPT-5.5 | 3 | none found | none found |
Agolic KLEE, DeepSeek V4.1 Flash | 3 | none found | none found |
Agolic Direct-Input, DeepSeek V4.1 Flash | 3 | none found | none found |
Agolic AFL++, DeepSeek V4.1 Flash | 3 | none found | none found |
ConcoLLMic, GPT-5.5 | 2† |
|
|
ConcoLLMic, DeepSeek V4.1 Flash | 3 |
|
|
KLEE fixed options, 6 cores | 3 | none found | none found |
KLEE uniform options, 6 cores | 3 | none found | none found |
SymTuner, 6 cores | 3 | none found | none found |
AFL++, 1 core | 3 | none found | none found |
SymCC + AFL++, 1 core | 3 | none found | none found |
SymSan + AFL++, 1 core | 3 | none found | none found |
AFL++ rebuilt targets | 3 | none found | none found |
KLEE fixed options, 1 core | 5 | none found | none found |
KLEE uniform options, 1 core | 5 | none found | none found |
SymTuner, 1 core | 5 | none found | none found |
Agolic KLEE, GPT-5.4 | 3 | none found | none found |
Agolic KLEE, GPT-5.5, effort low | 3 | none found | none found |
Agolic KLEE, GPT-5.5, effort extra high | 3 | none found | none found |
Agolic KLEE, GPT-5.5, Harness-Entry Mode only | 3 | none found | none found |
Agolic KLEE, GPT-5.5, Seeded-Prefix Mode only | 3 | none found | none found |
Agolic KLEE, GPT-5.5, seed finder off | 3 | none found | none found |
Agolic KLEE, GPT-5.5, proposal review off | 3 | none found | none found |
Agolic KLEE, GPT-5.5, run history withheld | 3 | none found | none found |
Agolic KLEE, DeepSeek V4.1 Flash, effort low | 3 | none found | none found |
Agolic KLEE, DeepSeek V4.1 Flash, effort high | 3 | none found | none found |
SymTuner rerun, 6 cores | 3 | none found | none found |
SymTuner from the agent's seeds, 6 cores, its own tests | 3 | none found | none found |
Counts are of untriaged signatures. † marks fewer than three runs.