Results
Median final branch coverage of test cases replayed on one pinned gcov build per program.
Versions
Group
Show
Terms
- Run
- One campaign of one tool on one program.
- Cores
- The parallel processes of one campaign.
- Fixed track
- The command line is given and the tool supplies only the input. Method
- Open track
- The tool also chooses arguments, files and environment. Method
- Median
- The middle value over a tool's runs; the median of two runs is their mean.
- standard
- Conventional gcov replay of every retained test case, same counting code and denominators. Method
- Unreached
- A branch that none of the main-group tools that ran on the program reaches (4 to 12 tools).
- Agent
- A language model that plans a campaign of other tools or inputs and reads the results.
- Language model
- A system in which a language model does the central reasoning of a classical technique, such as concolic execution.
- Symbolic
- Runs the program on symbolic inputs and asks a constraint solver for concrete inputs that reach new paths.
- Fuzzer
- Coverage-guided fuzzing: mutates inputs and keeps those that reach new code.
- Hybrid
- A fuzzer paired with a concolic executor; each supplies the other with inputs.
- Main
- Shown by default: the tools a reader is most likely to compare.
- Ablation
- Variants of the agent that switch off or change one part, to show what that part adds.
- Seeding
- Runs of a tool that start from seeds produced by another system.
- Control
- Systems that run without the agent, as a reference for the main group.
- Release
- A dated snapshot of the data, named like 2026.1. It never changes; corrections go into the next release.
- Entry
- A tool in one version and configuration, with a model, a track and a budget. One row of the data.
- Superseded
- An entry that a newer one replaces. It stays in the release and shows under all versions.
- Withdrawn
- An entry taken out of the comparison after it was published.
finals.csv in the results repository
Latest: the current entries of each tool. All adds the superseded and withdrawn ones.
Main group
| Programbranchesunreached | Total branchesunreached (tools that ran) | Agent | Language model | Symbolic | Fuzzer | Hybrid | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Agolic KLEEGPT-5.5, version 1.0.1+2407fd07open track | Agolic KLEEDeepSeek V4.1 Flash, version 1.0.1+8a86838eopen track | Agolic Direct-InputDeepSeek V4.1 Flash, version 1.0.1+8a86838eopen track | Agolic AFL++DeepSeek V4.1 Flash, version 1.0.1+8a86838eopen track | ConcoLLMicGPT-5.5, version 9634e686+f3fd7c36open track | ConcoLLMicDeepSeek V4.1 Flash, version 9634e686+f3fd7c36open track | KLEE fixed options6 cores, version efa1e052open track | KLEE uniform options6 cores, version efa1e052open track | SymTuner6 cores, version 71806b45open track | AFL++1 core, version 5.02cfixed track | SymCC + AFL++1 core, version 07c8895f+06a515fafixed track | SymSan + AFL++1 core, version 2937d1c0+499bf218fixed track | ||
| bc1,922 branches, 16.6% unreached by 5 tools | 1,92216.6% (5 tools) | 75.4%1,449, 3 runs | no runs | no runs | no runs | 69.4%1,333, 3 runs | no runs | 22.4%430, 3 runs | 27.4%527, 3 runs | 27.9%537, 3 runs | no fixed track | no fixed track | no fixed track |
| Confetti632 branches, 48.9% unreached by 12 tools | 63248.9% (12 tools) | 50.0%316, 3 runs | 51.1%323, 3 runs | 51.1%323, 3 runs | 51.1%323, 3 runs | 49.3%311.5, 2 runs | 49.5%313, 3 runs | 32.3%204, 3 runs | 40.7%257, 3 runs | 41.3%261, 3 runs | 51.1%323, 3 runs | 50.6%320, 3 runs | 51.1%323, 3 runs |
| diff4,369 branches, 30.8% unreached by 6 tools | 4,36930.8% (6 tools) | 47.4%2,072, 3 runs | 55.1%2,409, 3 runs | 62.2%2,716, 3 runs | no runs | no runs | no runs | 21.4%936, 3 runs | 31.9%1,395, 3 runs | 30.7%1,340, 3 runs | no fixed track | no fixed track | no fixed track |
| du7,331 branches, 72.5% unreached by 6 tools | 7,33172.5% (6 tools) | 22.7%1,667, 3 runs | 24.1%1,765, 3 runs | 24.8%1,815, 3 runs | no runs | no runs | no runs | 14.5%1,061, 3 runs | 18.0%1,319, 3 runs | 18.0%1,320, 3 runs | no fixed track | no fixed track | no fixed track |
| gawk20,356 branches, 29.1% unreached by 6 tools | 20,35629.1% (6 tools) | 37.5%7,634, 3 runs | 55.7%11,335, 3 runs | 60.8%12,375, 3 runs | no runs | no runs | no runs | 10.1%2,057, 3 runs | 23.8%4,849, 3 runs | 24.1%4,897, 3 runs | no fixed track | no fixed track | no fixed track |
| gcal15,632 branches, 34.6% unreached by 4 tools | 15,63234.6% (4 tools) | 41.9%6,552, 3 runs | no runs | no runs | no runs | no runs | no runs | 22.6%3,539, 3 runs | 43.4%6,790, 3 runs | 44.5%6,953, 3 runs | no fixed track | no fixed track | no fixed track |
| grep4,806 branches, 37.1% unreached by 4 tools | 4,80637.1% (4 tools) | 55.8%2,680, 3 runs | no runs | no runs | no runs | no runs | no runs | 36.2%1,739, 3 runs | 47.8%2,296, 3 runs | 47.0%2,257, 3 runs | no fixed track | no fixed track | no fixed track |
| gzip3,083 branches, 38.9% unreached by 6 tools | 3,08338.9% (6 tools) | 50.8%1,566, 3 runs | 53.1%1,636, 3 runs | 58.1%1,790, 3 runs | no runs | no runs | no runs | 35.6%1,099, 3 runs | 46.4%1,430, 3 runs | 47.1%1,451, 3 runs | no fixed track | no fixed track | no fixed track |
| jq18,280 branches, 61.0% unreached by 4 tools | 18,28061.0% (4 tools) | 30.5%5,574, 3 runs | no runs | no runs | no runs | no runs | no runs | 8.6%1,570, 3 runs | 13.4%2,442, 3 runs | 13.2%2,419, 3 runs | no fixed track | no fixed track | no fixed track |
| Krep1,231 branches, 19.8% unreached by 12 tools | 1,23119.8% (12 tools) | 67.2%827, 3 runs | 66.0%812, 3 runs | 77.0%948, 3 runs | 63.4%780, 3 runs | 72.7%895, 3 runs | 71.4%879, 3 runs | 47.1%580, 3 runs | 55.0%677, 3 runs | 55.9%688, 3 runs | 16.0%197, 3 runs | 16.0%197, 3 runs | 15.9%196, 3 runs |
| Libmatheval591 branches, 41.6% unreached by 8 tools | 59141.6% (8 tools) | 56.9%336, 3 runs | no runs | no runs | no runs | 56.5%334, 2 runs | no runs | 24.7%146, 3 runs | 44.5%263, 3 runs | 44.7%264, 3 runs | 58.0%343, 3 runs | 45.9%271, 3 runs | 50.1%296, 3 runs |
| LibYAML4,275 branches, 30.2% unreached by 8 tools | 4,27530.2% (8 tools) | 63.7%2,723, 3 runs | no runs | no runs | no runs | 42.4%1,813, 3 runs | no runs | 41.0%1,752, 3 runs | 52.5%2,244, 3 runs | 54.6%2,333, 3 runs | 68.7%2,936, 3 runs | 60.0%2,566, 3 runs | 67.4%2,880, 3 runs |
| ls4,441 branches, 34.4% unreached by 6 tools | 4,44134.4% (6 tools) | 47.4%2,106, 3 runs | 58.0%2,578, 3 runs | 53.7%2,383, 3 runs | no runs | no runs | no runs | 28.8%1,279, 3 runs | 39.6%1,757, 3 runs | 39.9%1,771, 3 runs | no fixed track | no fixed track | no fixed track |
| Oggenc1,138 branches, 26.0% unreached by 9 tools | 1,13826.0% (9 tools) | 71.0%808, 3 runs | no runs | no runs | 71.4%812, 3 runs | 57.8%658, 3 runs | no runs | 17.7%201, 3 runs | 29.6%337, 3 runs | 29.5%336, 3 runs | 15.7%179, 3 runs | 15.9%181, 3 runs | 16.0%182, 3 runs |
| sed6,958 branches, 37.6% unreached by 4 tools | 6,95837.6% (4 tools) | 55.3%3,848, 3 runs | no runs | no runs | no runs | no runs | no runs | 17.9%1,246, 3 runs | 42.0%2,923, 3 runs | 42.4%2,950, 3 runs | no fixed track | no fixed track | no fixed track |
| WOFF211,412 branches, 80.0% unreached by 12 tools | 11,41280.0% (12 tools) | 16.0%1,830, 3 runs | 16.1%1,833, 3 runs | 18.7%2,134, 3 runs | 19.3%2,197, 3 runs | 15.4%1,754, 1 run | 16.5%1,880, 3 runs | 1.4%163, 3 runs | 7.6%866, 3 runs | 7.9%899, 3 runs | 17.3%1,977, 3 runs | 15.8%1,800, 3 runs | 15.6%1,780, 3 runs |
| xmlwf6,293 branches, 25.5% unreached by 6 tools | 6,29325.5% (6 tools) | 61.5%3,869, 3 runs | 66.6%4,191, 3 runs | 62.7%3,945, 3 runs | no runs | no runs | no runs | 21.6%1,360, 3 runs | 31.9%2,006, 3 runs | 32.8%2,062, 3 runs | no fixed track | no fixed track | no fixed track |
| xorriso54,607 branches, 75.6% unreached by 4 tools | 54,60775.6% (4 tools) | 10.7%5,848, 3 runs | no runs | no runs | no runs | no runs | no runs | 6.4%3,502, 3 runs | 6.5%3,525, 3 runs | 8.3%4,552, 3 runs | no fixed track | no fixed track | no fixed track |
Ablation group
| Programbranchesunreached | Total branchesunreached (tools that ran) | Agent (main) | Agent | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Agolic KLEEGPT-5.5, version 1.0.1+2407fd07open track | Agolic KLEEDeepSeek V4.1 Flash, version 1.0.1+8a86838eopen track | Agolic Direct-InputDeepSeek V4.1 Flash, version 1.0.1+8a86838eopen track | Agolic AFL++DeepSeek V4.1 Flash, version 1.0.1+8a86838eopen track | Agolic KLEEGPT-5.5, effort low, version 1.0.1+2407fd07open track | Agolic KLEEGPT-5.5, effort extra high, version 1.0.1+2407fd07open track | Agolic KLEEGPT-5.5, Harness-Entry Mode only, version 1.0.1+2407fd07open track | Agolic KLEEGPT-5.5, Seeded-Prefix Mode only, version 1.0.1+2407fd07open track | Agolic KLEEGPT-5.5, seed finder off, version 1.0.1+2407fd07open track | Agolic KLEEGPT-5.5, proposal review off, version 1.0.1+2407fd07open track | Agolic KLEEGPT-5.5, run history withheld, version unknownopen track | Agolic KLEEDeepSeek V4.1 Flash, effort low, version 1.0.1+8a86838eopen track | Agolic KLEEDeepSeek V4.1 Flash, effort high, version 1.0.1+8a86838eopen track | ||
| Confetti632 branches, 48.9% unreached by 12 tools | 63248.9% (12 tools) | 50.0%316, 3 runs | 51.1%323, 3 runs | 51.1%323, 3 runs | 51.1%323, 3 runs | 49.2%311, 3 runs | 50.9%322, 3 runs | 35.1%222, 3 runs | 51.1%323, 3 runs | 50.9%322, 3 runs | 50.6%320, 3 runs | 50.9%322, 3 runs | 50.8%321, 3 runs | 51.1%323, 3 runs |
| gzip3,083 branches, 38.9% unreached by 6 tools | 3,08338.9% (6 tools) | 50.8%1,566, 3 runs | 53.1%1,636, 3 runs | 58.1%1,790, 3 runs | no runs | 49.8%1,534, 3 runs | 50.7%1,564, 3 runs | 44.0%1,356, 3 runs | 49.4%1,522, 3 runs | 50.0%1,543, 3 runs | 51.4%1,584, 3 runs | 48.2%1,487, 3 runs | 54.3%1,674, 3 runs | 55.3%1,706, 3 runs |
| Krep1,231 branches, 19.8% unreached by 12 tools | 1,23119.8% (12 tools) | 67.2%827, 3 runs | 66.0%812, 3 runs | 77.0%948, 3 runs | 63.4%780, 3 runs | 64.3%792, 3 runs | 68.8%847, 3 runs | 66.7%821, 3 runs | 69.2%852, 3 runs | 66.0%812, 3 runs | 68.7%846, 3 runs | 66.2%815, 3 runs | 64.0%788, 3 runs | 64.3%791, 3 runs |
| WOFF211,412 branches, 80.0% unreached by 12 tools | 11,41280.0% (12 tools) | 16.0%1,830, 3 runs | 16.1%1,833, 3 runs | 18.7%2,134, 3 runs | 19.3%2,197, 3 runs | 15.5%1,772, 3 runs | 16.0%1,828, 3 runs | 1.9%213, 3 runs | 14.9%1,696, 3 runs | 14.4%1,649, 3 runs | 15.0%1,710, 3 runs | 13.7%1,566, 3 runs | 16.5%1,880, 3 runs | 17.2%1,966, 3 runs |
Seeding group
| Programbranchesunreached | Total branchesunreached (tools that ran) | Symbolic | |
|---|---|---|---|
| SymTuner rerun6 cores, version 71806b45open track | SymTuner from the agent's seeds6 cores, its own tests, version 71806b45open track | ||
| bc1,922 branches, 16.6% unreached by 5 tools | 1,92216.6% (5 tools) | 28.4%546, 3 runs | 26.9%517, 3 runs |
| Confetti632 branches, 48.9% unreached by 12 tools | 63248.9% (12 tools) | 40.3%255, 3 runs | 49.8%315, 3 runs |
| gzip3,083 branches, 38.9% unreached by 6 tools | 3,08338.9% (6 tools) | 47.2%1,456, 3 runs | 48.6%1,497, 3 runs |
| jq18,280 branches, 61.0% unreached by 4 tools | 18,28061.0% (4 tools) | 13.3%2,434, 3 runs | 12.9%2,354, 3 runs |
| Krep1,231 branches, 19.8% unreached by 12 tools | 1,23119.8% (12 tools) | 54.0%665, 3 runs | 52.7%649, 3 runs |
| WOFF211,412 branches, 80.0% unreached by 12 tools | 11,41280.0% (12 tools) | 5.9%675, 3 runs | 8.7%998, 3 runs |
| xmlwf6,293 branches, 25.5% unreached by 6 tools | 6,29325.5% (6 tools) | 32.1%2,020, 3 runs | 31.2%1,964, 3 runs |
Control group
| Programbranchesunreached | Total branchesunreached (tools that ran) | Fuzzer | Fuzzer (main) | Symbolic | Symbolic (main) | ||||
|---|---|---|---|---|---|---|---|---|---|
| AFL++ rebuilt targets, version 5.02cfixed track | AFL++1 core, version 5.02cfixed track | KLEE fixed options1 core, version efa1e052open track | KLEE uniform options1 core, version efa1e052open track | SymTuner1 core, version 71806b45open track | KLEE fixed options6 cores, version efa1e052open track | KLEE uniform options6 cores, version efa1e052open track | SymTuner6 cores, version 71806b45open track | ||
| bc1,922 branches, 16.6% unreached by 5 tools | 1,92216.6% (5 tools) | no fixed track | no fixed track | 16.9%325, 5 runs | 23.9%460, 5 runs | 22.9%441, 5 runs | 22.4%430, 3 runs | 27.4%527, 3 runs | 27.9%537, 3 runs |
| Confetti632 branches, 48.9% unreached by 12 tools | 63248.9% (12 tools) | 51.1%323, 3 runs | 51.1%323, 3 runs | 22.9%145, 5 runs | 34.0%215, 5 runs | 39.7%251, 5 runs | 32.3%204, 3 runs | 40.7%257, 3 runs | 41.3%261, 3 runs |
| diff4,369 branches, 30.8% unreached by 6 tools | 4,36930.8% (6 tools) | no fixed track | no fixed track | 15.6%680, 5 runs | 24.5%1,070, 5 runs | 28.4%1,240, 5 runs | 21.4%936, 3 runs | 31.9%1,395, 3 runs | 30.7%1,340, 3 runs |
| du7,331 branches, 72.5% unreached by 6 tools | 7,33172.5% (6 tools) | no fixed track | no fixed track | 13.7%1,005, 5 runs | 17.0%1,245, 5 runs | 17.4%1,272, 5 runs | 14.5%1,061, 3 runs | 18.0%1,319, 3 runs | 18.0%1,320, 3 runs |
| gawk20,356 branches, 29.1% unreached by 6 tools | 20,35629.1% (6 tools) | no fixed track | no fixed track | 10.1%2,059, 5 runs | 17.6%3,590, 5 runs | 15.0%3,054, 5 runs | 10.1%2,057, 3 runs | 23.8%4,849, 3 runs | 24.1%4,897, 3 runs |
| gcal15,632 branches, 34.6% unreached by 4 tools | 15,63234.6% (4 tools) | no fixed track | no fixed track | 17.1%2,676, 5 runs | 26.9%4,198, 5 runs | 37.8%5,909, 5 runs | 22.6%3,539, 3 runs | 43.4%6,790, 3 runs | 44.5%6,953, 3 runs |
| grep4,806 branches, 37.1% unreached by 4 tools | 4,80637.1% (4 tools) | no fixed track | no fixed track | 32.3%1,552, 5 runs | 35.7%1,718, 5 runs | 32.9%1,581, 5 runs | 36.2%1,739, 3 runs | 47.8%2,296, 3 runs | 47.0%2,257, 3 runs |
| gzip3,083 branches, 38.9% unreached by 6 tools | 3,08338.9% (6 tools) | no fixed track | no fixed track | 22.0%678, 5 runs | 39.0%1,203, 5 runs | 42.5%1,311, 5 runs | 35.6%1,099, 3 runs | 46.4%1,430, 3 runs | 47.1%1,451, 3 runs |
| jq18,280 branches, 61.0% unreached by 4 tools | 18,28061.0% (4 tools) | no fixed track | no fixed track | 8.4%1,530, 5 runs | 11.5%2,111, 5 runs | 11.4%2,090, 5 runs | 8.6%1,570, 3 runs | 13.4%2,442, 3 runs | 13.2%2,419, 3 runs |
| Krep1,231 branches, 19.8% unreached by 12 tools | 1,23119.8% (12 tools) | 16.0%197, 3 runs | 16.0%197, 3 runs | 42.7%526, 5 runs | 48.6%598, 5 runs | 49.9%614, 5 runs | 47.1%580, 3 runs | 55.0%677, 3 runs | 55.9%688, 3 runs |
| Libmatheval591 branches, 41.6% unreached by 8 tools | 59141.6% (8 tools) | no runs | 58.0%343, 3 runs | 24.7%146, 5 runs | 43.1%255, 5 runs | 43.7%258, 5 runs | 24.7%146, 3 runs | 44.5%263, 3 runs | 44.7%264, 3 runs |
| LibYAML4,275 branches, 30.2% unreached by 8 tools | 4,27530.2% (8 tools) | no runs | 68.7%2,936, 3 runs | 21.0%896, 5 runs | 42.8%1,828, 5 runs | 42.8%1,828, 5 runs | 41.0%1,752, 3 runs | 52.5%2,244, 3 runs | 54.6%2,333, 3 runs |
| ls4,441 branches, 34.4% unreached by 6 tools | 4,44134.4% (6 tools) | no fixed track | no fixed track | 24.7%1,099, 5 runs | 37.0%1,645, 5 runs | 37.9%1,681, 5 runs | 28.8%1,279, 3 runs | 39.6%1,757, 3 runs | 39.9%1,771, 3 runs |
| Oggenc1,138 branches, 26.0% unreached by 9 tools | 1,13826.0% (9 tools) | 16.0%182, 3 runs | 15.7%179, 3 runs | 14.9%170, 5 runs | 27.2%309, 5 runs | 26.8%305, 5 runs | 17.7%201, 3 runs | 29.6%337, 3 runs | 29.5%336, 3 runs |
| sed6,958 branches, 37.6% unreached by 4 tools | 6,95837.6% (4 tools) | no fixed track | no fixed track | 14.5%1,012, 5 runs | 35.6%2,475, 5 runs | 34.8%2,424, 5 runs | 17.9%1,246, 3 runs | 42.0%2,923, 3 runs | 42.4%2,950, 3 runs |
| WOFF211,412 branches, 80.0% unreached by 12 tools | 11,41280.0% (12 tools) | 17.7%2,024, 3 runs | 17.3%1,977, 3 runs | 1.3%152, 5 runs | 3.0%338, 5 runs | 2.2%249, 5 runs | 1.4%163, 3 runs | 7.6%866, 3 runs | 7.9%899, 3 runs |
| xmlwf6,293 branches, 25.5% unreached by 6 tools | 6,29325.5% (6 tools) | no fixed track | no fixed track | 18.8%1,186, 5 runs | 28.6%1,800, 5 runs | 27.7%1,746, 5 runs | 21.6%1,360, 3 runs | 31.9%2,006, 3 runs | 32.8%2,062, 3 runs |
| xorriso54,607 branches, 75.6% unreached by 4 tools | 54,60775.6% (4 tools) | no fixed track | no fixed track | 4.4%2,380, 5 runs | 4.9%2,669, 5 runs | 5.1%2,803, 5 runs | 6.4%3,502, 3 runs | 6.5%3,525, 3 runs | 8.3%4,552, 3 runs |