Agent Software Testing

Compare

Release 2026.1

Two tools side by side on the programs both ran on.

Terms
Run
One campaign of one tool on one program.
Cores
The parallel processes of one campaign.
Budget
The longest a campaign may run.
Fixed track
The command line is given and the tool supplies only the input. Method
Open track
The tool also chooses arguments, files and environment. Method
Median
The middle value over a tool's runs; the median of two runs is their mean.
standard
Conventional gcov replay of every retained test case, same counting code and denominators. Method
Output tokens
Model output of a whole campaign, reasoning included, as self-reported in the release.
Agent
A language model that plans a campaign of other tools or inputs and reads the results.
Language model
A system in which a language model does the central reasoning of a classical technique, such as concolic execution.
Symbolic
Runs the program on symbolic inputs and asks a constraint solver for concrete inputs that reach new paths.
Fuzzer
Coverage-guided fuzzing: mutates inputs and keeps those that reach new code.
Hybrid
A fuzzer paired with a concolic executor; each supplies the other with inputs.
Main
Shown by default: the tools a reader is most likely to compare.
Ablation
Variants of the agent that switch off or change one part, to show what that part adds.
Seeding
Runs of a tool that start from seeds produced by another system.
Control
Systems that run without the agent, as a reference for the main group.
Release
A dated snapshot of the data, named like 2026.1. It never changes; corrections go into the next release.
A12
The probability that a run of A covers more branches than a run of B, ties counting half. Method
A: KLEE fixed options, 6 coresB: SymTuner, 6 cores

Both ran on 18 programs.

Median final coverage

Median final coverage of A and B by program, with A12
ProgramMedian shareA medianB medianB − AA12Runs A / BCurves
bc1,922 branches
A22.4%
B27.9%
430537+1070.003 / 3
Confetti632 branches
A32.3%
B41.3%
204261+570.003 / 3
diff4,369 branches
A21.4%
B30.7%
9361,340+4040.003 / 3
du7,331 branches
A14.5%
B18.0%
1,0611,320+2590.003 / 3
gawk20,356 branches
A10.1%
B24.1%
2,0574,897+2,8400.003 / 3
gcal15,632 branches
A22.6%
B44.5%
3,5396,953+3,4140.003 / 3
grep4,806 branches
A36.2%
B47.0%
1,7392,257+5180.003 / 3
gzip3,083 branches
A35.6%
B47.1%
1,0991,451+3520.003 / 3
jq18,280 branches
A8.6%
B13.2%
1,5702,419+8490.003 / 3
Krep1,231 branches
A47.1%
B55.9%
580688+1080.003 / 3
Libmatheval591 branches
A24.7%
B44.7%
146264+1180.003 / 3
LibYAML4,275 branches
A41.0%
B54.6%
1,7522,333+5810.003 / 3
ls4,441 branches
A28.8%
B39.9%
1,2791,771+4920.003 / 3
Oggenc1,138 branches
A17.7%
B29.5%
201336+1350.003 / 3
sed6,958 branches
A17.9%
B42.4%
1,2462,950+1,7040.003 / 3
WOFF211,412 branches
A1.4%
B7.9%
163899+7360.003 / 3
xmlwf6,293 branches
A21.6%
B32.8%
1,3602,062+7020.003 / 3
xorriso54,607 branches
A6.4%
B8.3%
3,5024,552+1,0500.223 / 3

Branches only each tool reaches

Branches reached by only A, by both and by only B
ProgramOnly ABothOnly BSplit of the union
bc4147998
Confetti020573
diff261,022547
du01,078374
gawk02,1533,413
gcal13,7754,273
grep152,140310
gzip01,172324
jq21,630934
Krep19594161
Libmatheval0149115
LibYAML332,050428
ls01,347467
Oggenc0210127
sed01,4131,843
WOFF20163916
xmlwf01,463753
xorriso403,6041,424

Resources

Resources of tool A and tool B
ResourceKLEE fixed options, 6 coresSymTuner, 6 cores
FamilySymbolicSymbolic
BackendKLEEKLEE
Modelnonenone
Trackopenopen
Cores per campaign66
Budget per campaign6 h6 h
Runs, on the shared programs5454
Output tokens per run, median, on the shared programsnot applicable (no model)not applicable (no model)