Libmatheval
1.1.11Terms
- Run
- One campaign of one tool on one program.
- Cores
- The parallel processes of one campaign.
- Budget
- The longest a campaign may run.
- Fixed track
- The command line is given and the tool supplies only the input. Method
- Open track
- The tool also chooses arguments, files and environment. Method
- Median
- The middle value over a tool's runs; the median of two runs is their mean.
- standard
- Conventional gcov replay of every retained test case, same counting code and denominators. Method
- Unreached
- A branch that none of the main-group tools that ran on the program reaches (4 to 12 tools).
- Union of runs
- The branches that at least one of a tool's runs reaches, each counted once.
- Output tokens
- Model output of a whole campaign, reasoning included, as self-reported in the release.
- Sanitiser signature
- The sanitiser, the error kind and the innermost frame in the program's own source; two can share a cause.
- Language model
- A system in which a language model does the central reasoning of a classical technique, such as concolic execution.
- Symbolic
- Runs the program on symbolic inputs and asks a constraint solver for concrete inputs that reach new paths.
- Fuzzer
- Coverage-guided fuzzing: mutates inputs and keeps those that reach new code.
- Hybrid
- A fuzzer paired with a concolic executor; each supplies the other with inputs.
- Main
- Shown by default: the tools a reader is most likely to compare.
- Ablation
- Variants of the agent that switch off or change one part, to show what that part adds.
- Seeding
- Runs of a tool that start from seeds produced by another system.
- Control
- Systems that run without the agent, as a reference for the main group.
- Release
- A dated snapshot of the data, named like 2026.1. It never changes; corrections go into the next release.
- Superseded
- An entry that a newer one replaces. It stays in the release and shows under all versions.
- Withdrawn
- An entry taken out of the comparison after it was published.
GNU libmatheval is a library that parses and evaluates mathematical expressions written as text.
- Branches
- 59141.6% unreached (8 tools)
- Language
- C
- Inputs
- standard input
- Fixed track
- standard input
- Tools with runs
- 128 main
- Runs
- 4141 with branch sets
Branch map
1 tool 02 to 3 274 to 5 586 to 7 113all 8 147Unreached 246
Coverage over time
Use the left and right arrow keys, Home and End to move along the time axis. The value of every line at that time is read out.
The chart needs JavaScript. The final values are in the Final coverage table below.
Showing 8 of 8
Showing 1 of 2
Showing 0 of 0
Showing 7 of 7
Curves:
curves/ in the results repositoryFinal coverage
† marks fewer than three runs.
| Tool | Median | Min to max | Share (median) | Runs | Median curve | Output tokens |
|---|---|---|---|---|---|---|
| Main group, 8 tools | ||||||
Agolic KLEE, GPT-5.5agent, open track, six cores, version 1.0.1+ | 336 | 336–336 | 56.9% | 3 | 40,865 | |
ConcoLLMic, GPT-5.5language model, open track, version 9634e686+ | 334† | 333–335 | 56.5% | 2† | not reported | |
KLEE fixed options, 6 coressymbolic, open track, six cores, version efa1e052 | 146 | 146–149 | 24.7% | 3 | not applicable | |
KLEE uniform options, 6 coressymbolic, open track, six cores, version efa1e052 | 263 | 263–263 | 44.5% | 3 | not applicable | |
SymTuner, 6 coressymbolic, open track, six cores, version 71806b45 | 264 | 263–264 | 44.7% | 3 | not applicable | |
AFL++, 1 core fixed trackfuzzer, fixed track, one core, version 5.02c | 343 | 343–343 | 58.0% | 3 | not applicable | |
SymCC + AFL++, 1 core fixed trackhybrid, fixed track, one core, version 07c8895f+ | 271 | 269–303 | 45.9% | 3 | not applicable | |
SymSan + AFL++, 1 core fixed trackhybrid, fixed track, one core, version 2937d1c0+ | 296 | 286–301 | 50.1% | 3 | not applicable | |
| Ablation group, 1 tool | ||||||
Agolic KLEE, GPT-5.4 supersededagent, open track, six cores, version unreleased+ | 334 | 333–336 | 56.5% | 3 | not reported | |
| Control group, 3 tools | ||||||
KLEE fixed options, 1 coresymbolic, open track, one core, version efa1e052 | 146 | 145–149 | 24.7% | 5 | not applicable | |
KLEE uniform options, 1 coresymbolic, open track, one core, version efa1e052 | 255 | 253–258 | 43.1% | 5 | not applicable | |
SymTuner, 1 coresymbolic, open track, one core, version 71806b45 | 258 | 257–258 | 43.7% | 5 | not applicable | |
finals.csv in the results repositoryBranches reached by one tool only
| Tool | Runs in union | Branches reached | Share (union) | Reached by this tool only |
|---|---|---|---|---|
Agolic KLEE, GPT-5.5 | 3 | 336 | 56.9% | 0 |
ConcoLLMic, GPT-5.5 | 2† | 335† | 56.7% | 0 |
KLEE fixed options, 6 cores | 3 | 149 | 25.2% | 0 |
KLEE uniform options, 6 cores | 3 | 264 | 44.7% | 0 |
SymTuner, 6 cores | 3 | 264 | 44.7% | 0 |
AFL++, 1 core | 3 | 343 | 58.0% | 0 |
SymCC + AFL++, 1 core | 3 | 304 | 51.4% | 0 |
SymSan + AFL++, 1 core | 3 | 308 | 52.1% | 0 |
Sanitiser findings
Signatures in each tool’s final test cases.
| Tool | Runs replayed | AddressSanitizerkind, signatures | UndefinedBehaviorSanitizerkind, signatures |
|---|---|---|---|
Agolic KLEE, GPT-5.5 | 3 | none found | none found |
ConcoLLMic, GPT-5.5 | 2† | none found | none found |
KLEE fixed options, 6 cores | 3 | none found | none found |
KLEE uniform options, 6 cores | 3 | none found | none found |
SymTuner, 6 cores | 3 | none found | none found |
AFL++, 1 core | 3 | none found | none found |
SymCC + AFL++, 1 core | 3 | none found | none found |
SymSan + AFL++, 1 core | 3 | none found | none found |
KLEE fixed options, 1 core | 5 | none found | none found |
KLEE uniform options, 1 core | 5 | none found | none found |
SymTuner, 1 core | 5 | none found | none found |
Agolic KLEE, GPT-5.4 | 3 | none found | none found |
Counts are of untriaged signatures. † marks fewer than three runs.