Tools

Release 2026.1

A row expands to its results on each program it ran.

Terms

Run: One campaign of one tool on one program.
Cores: The parallel processes of one campaign.
Budget: The longest a campaign may run.
Fixed track: The command line is given and the tool supplies only the input. Method
Open track: The tool also chooses arguments, files and environment. Method
Median: The middle value over a tool's runs; the median of two runs is their mean.
standard: Conventional gcov replay of every retained test case, same counting code and denominators. Method
Output tokens: Model output of a whole campaign, reasoning included, as self-reported in the release.
Agent: A language model that plans a campaign of other tools or inputs and reads the results.
Language model: A system in which a language model does the central reasoning of a classical technique, such as concolic execution.
Symbolic: Runs the program on symbolic inputs and asks a constraint solver for concrete inputs that reach new paths.
Fuzzer: Coverage-guided fuzzing: mutates inputs and keeps those that reach new code.
Hybrid: A fuzzer paired with a concolic executor; each supplies the other with inputs.
Main: Shown by default: the tools a reader is most likely to compare.
Ablation: Variants of the agent that switch off or change one part, to show what that part adds.
Seeding: Runs of a tool that start from seeds produced by another system.
Control: Systems that run without the agent, as a reference for the main group.
Release: A dated snapshot of the data, named like 2026.1. It never changes; corrections go into the next release.
Entry: A tool in one version and configuration, with a model, a track and a budget. One row of the data.
Superseded: An entry that a newer one replaces. It stays in the release and shows under all versions.
Withdrawn: An entry taken out of the comparison after it was published.

runs.csv in the results repository

Tool Family Model Version Cores Budget Track Group Programs

Agolic KLEE, GPT-5.5

AgentKLEE

GPT-5.5

1.0.1+2407fd07

≤ 6 h

open

Main

A language model agent plans a campaign as a series of short KLEE invocations. For each invocation it chooses the target function, the command-line arguments, which inputs are symbolic, the environment and the concrete seed inputs. It decides from the program's source code and the coverage of earlier runs.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs; model: GPT-5.5.

Version 1.0.1+2407fd07, submitted 2026-10-16, added in release 2026.1, current, supersedes Agolic KLEE, GPT-5.4.

System id in the CSV files: agolic-klee-gpt55

Compare with Agolic KLEE, DeepSeek V4.1 Flash All versions of Agolic

Results of Agolic KLEE, GPT-5.5 by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	3	1,449	1,443–1,541	75.4%	248,934
Confetti	3	316	315–319	50.0%	46,743
diff	3	2,072	1,905–2,144	47.4%	227,662
du	3	1,667	1,667–1,741	22.7%	205,769
gawk	3	7,634	6,735–8,001	37.5%	198,827
gcal	3	6,552	6,306–7,412	41.9%	99,058
grep	3	2,680	2,524–2,778	55.8%	179,867
gzip	3	1,566	1,563–1,571	50.8%	265,374
jq	3	5,574	5,178–6,625	30.5%	210,302
Krep	3	827	805–835	67.2%	332,737
Libmatheval	3	336	336–336	56.9%	40,865
LibYAML	3	2,723	2,685–2,806	63.7%	148,310
ls	3	2,106	2,041–2,107	47.4%	290,493
Oggenc	3	808	790–810	71.0%	366,443
sed	3	3,848	3,806–3,935	55.3%	175,368
WOFF2	3	1,830	1,604–1,840	16.0%	233,916
xmlwf	3	3,869	3,698–4,070	61.5%	190,000
xorriso	3	5,848	5,193–12,113	10.7%	178,298

Agolic KLEE, DeepSeek V4.1 Flash

AgentKLEE

DeepSeek V4.1 Flash

1.0.1+8a86838e

≤ 6 h

open

Main

The same agent and code as the system with GPT-5.5, with DeepSeek V4.1 Flash as the model.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs; model: DeepSeek V4.1 Flash.

Version 1.0.1+8a86838e, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-deepseek

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, DeepSeek V4.1 Flash by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	323	315–323	51.1%	217,832
diff	3	2,409	2,230–2,522	55.1%	1,361,361
du	3	1,765	1,757–1,836	24.1%	1,365,454
gawk	3	11,335	10,726–11,361	55.7%	1,378,412
gzip	3	1,636	1,625–1,662	53.1%	1,417,262
Krep	3	812	809–876	66.0%	1,662,241
ls	3	2,578	1,996–2,685	58.0%	1,141,799
WOFF2	3	1,833	1,729–1,870	16.1%	791,985
xmlwf	3	4,191	4,151–4,210	66.6%	1,515,624

Agolic Direct-Input, DeepSeek V4.1 Flash

AgentDirect-Input

DeepSeek V4.1 Flash

1.0.1+8a86838e

≤ 6 h

open

Main

The agent plans invocations that execute whole concrete inputs directly, with command-line arguments, standard input and files, up to 32 inputs per invocation. It keeps every input as a test. It uses no symbolic execution and no solver.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: DeepSeek V4.1 Flash.

Version 1.0.1+8a86838e, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-direct-deepseek

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic Direct-Input, DeepSeek V4.1 Flash by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	323	323–323	51.1%	108,478
diff	3	2,716	2,716–2,727	62.2%	3,258,583
du	3	1,815	1,792–1,872	24.8%	1,569,537
gawk	3	12,375	12,357–12,396	60.8%	2,931,054
gzip	3	1,790	1,718–1,836	58.1%	3,208,166
Krep	3	948	939–950	77.0%	726,241
ls	3	2,383	2,328–2,446	53.7%	1,513,740
WOFF2	3	2,134	2,063–2,181	18.7%	1,594,432
xmlwf	3	3,945	3,927–3,980	62.7%	2,967,197

Agolic AFL++, DeepSeek V4.1 Flash

AgentAFL++

DeepSeek V4.1 Flash

1.0.1+8a86838e

≤ 6 h

open

Main

The agent plans AFL++ invocations. It chooses the length of each, the input channel, the command-line arguments, the seeds and whether to continue from the queue of an earlier invocation.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; AFL++: Combining Incremental Steps of Fuzzing Research; model: DeepSeek V4.1 Flash.

Version 1.0.1+8a86838e, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-aflpp-deepseek

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic AFL++, DeepSeek V4.1 Flash by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	323	323–323	51.1%	38,835
Krep	3	780	759–822	63.4%	361,587
Oggenc	3	812	807–819	71.4%	485,177
WOFF2	3	2,197	2,193–2,210	19.3%	440,886

ConcoLLMic, GPT-5.5

Language modelConcolic loop

GPT-5.5

9634e686+f3fd7c36

–

≤ 6 h

open

Main

ConcoLLMic is a concolic executor in which a language model does the central reasoning. It follows concrete runs of the program and asks a language-model agent to model the conditions along each path, so it needs no hand-written symbolic model of the language or the environment. It then proposes inputs that reach other paths.

Citation: Agentic Concolic Execution. 2026 IEEE Symposium on Security and Privacy (SP), 2026; model: GPT-5.5.

Version 9634e686+f3fd7c36, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: concollmic-gpt55

Compare with ConcoLLMic, DeepSeek V4.1 Flash All versions of ConcoLLMic

Results of ConcoLLMic, GPT-5.5 by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	3	1,333	1,306–1,339	69.4%	796,474
Confetti	2^†	311.5	311–312	49.3%	not reported
Krep	3	895	893–898	72.7%	601,687
Libmatheval	2^†	334	333–335	56.5%	not reported
LibYAML	3	1,813	1,669–1,838	42.4%	687,570
Oggenc	3	658	645–708	57.8%	533,239
WOFF2	1^†	1,754	1,754–1,754	15.4%	not reported

ConcoLLMic, DeepSeek V4.1 Flash

Language modelConcolic loop

DeepSeek V4.1 Flash

9634e686+f3fd7c36

–

≤ 6 h

open

Main

ConcoLLMic with DeepSeek V4.1 Flash as the model.

Citation: Agentic Concolic Execution. 2026 IEEE Symposium on Security and Privacy (SP), 2026; model: DeepSeek V4.1 Flash.

Version 9634e686+f3fd7c36, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: concollmic-deepseek

Compare with ConcoLLMic, GPT-5.5 All versions of ConcoLLMic

Results of ConcoLLMic, DeepSeek V4.1 Flash by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	313	307–314	49.5%	802,214
Krep	3	879	865–904	71.4%	2,300,636
WOFF2	3	1,880	1,759–1,916	16.5%	6,954,649

KLEE fixed options, 6 cores

SymbolicKLEE

–

efa1e052

≤ 6 h

open

Main

KLEE is a symbolic execution engine. It runs a program on symbolic inputs, explores its paths and asks a constraint solver for concrete inputs that reach them. This system uses KLEE's default search and fixed options.

Citation: KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI 08), 2008.

Version efa1e052, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: klee-fixed-6w

Compare with KLEE uniform options, 6 cores All versions of KLEE

Results of KLEE fixed options, 6 cores by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	3	430	394–481	22.4%	not applicable
Confetti	3	204	154–204	32.3%	not applicable
diff	3	936	781–982	21.4%	not applicable
du	3	1,061	1,039–1,062	14.5%	not applicable
gawk	3	2,057	2,042–2,112	10.1%	not applicable
gcal	3	3,539	3,518–3,587	22.6%	not applicable
grep	3	1,739	1,710–2,090	36.2%	not applicable
gzip	3	1,099	1,088–1,102	35.6%	not applicable
jq	3	1,570	1,545–1,604	8.6%	not applicable
Krep	3	580	572–588	47.1%	not applicable
Libmatheval	3	146	146–149	24.7%	not applicable
LibYAML	3	1,752	1,639–2,006	41.0%	not applicable
ls	3	1,279	1,276–1,328	28.8%	not applicable
Oggenc	3	201	200–208	17.7%	not applicable
sed	3	1,246	1,231–1,336	17.9%	not applicable
WOFF2	3	163	163–163	1.4%	not applicable
xmlwf	3	1,360	1,348–1,383	21.6%	not applicable
xorriso	3	3,502	2,729–3,506	6.4%	not applicable

KLEE uniform options, 6 cores

SymbolicKLEE

–

efa1e052

≤ 6 h

open

Main

KLEE with the options of each invocation sampled uniformly from SymTuner's parameter space.

Citation: KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI 08), 2008; SymTuner: Maximizing the Power of Symbolic Execution by Adaptively Tuning External Parameters.

Version efa1e052, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: klee-uniform-6w

Compare with KLEE fixed options, 6 cores All versions of KLEE

Results of KLEE uniform options, 6 cores by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	3	527	505–556	27.4%	not applicable
Confetti	3	257	231–258	40.7%	not applicable
diff	3	1,395	1,238–1,869	31.9%	not applicable
du	3	1,319	1,305–1,344	18.0%	not applicable
gawk	3	4,849	4,562–5,079	23.8%	not applicable
gcal	3	6,790	6,490–6,878	43.4%	not applicable
grep	3	2,296	2,122–2,364	47.8%	not applicable
gzip	3	1,430	1,419–1,434	46.4%	not applicable
jq	3	2,442	2,436–2,469	13.4%	not applicable
Krep	3	677	663–681	55.0%	not applicable
Libmatheval	3	263	263–263	44.5%	not applicable
LibYAML	3	2,244	2,153–2,343	52.5%	not applicable
ls	3	1,757	1,751–1,773	39.6%	not applicable
Oggenc	3	337	337–337	29.6%	not applicable
sed	3	2,923	2,887–2,937	42.0%	not applicable
WOFF2	3	866	808–922	7.6%	not applicable
xmlwf	3	2,006	1,929–2,018	31.9%	not applicable
xorriso	3	3,525	3,349–3,619	6.5%	not applicable

SymTuner, 6 cores

SymbolicKLEE

–

71806b45

≤ 6 h

open

Main

SymTuner is an independent tool that tunes KLEE's options adaptively. It observes how earlier invocations behaved and shifts how it samples the options for later ones. It was ported to the KLEE build of this study.

Citation: SymTuner: Maximizing the Power of Symbolic Execution by Adaptively Tuning External Parameters. Proceedings of the 44th International Conference on Software Engineering (ICSE 2022), 2022; KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs.

Version 71806b45, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: symtuner-6w

Compare with KLEE fixed options, 6 cores All versions of SymTuner

Results of SymTuner, 6 cores by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	3	537	530–538	27.9%	not applicable
Confetti	3	261	254–272	41.3%	not applicable
diff	3	1,340	1,157–1,515	30.7%	not applicable
du	3	1,320	1,320–1,430	18.0%	not applicable
gawk	3	4,897	4,888–5,057	24.1%	not applicable
gcal	3	6,953	6,626–7,838	44.5%	not applicable
grep	3	2,257	2,137–2,301	47.0%	not applicable
gzip	3	1,451	1,433–1,461	47.1%	not applicable
jq	3	2,419	2,409–2,473	13.2%	not applicable
Krep	3	688	669–730	55.9%	not applicable
Libmatheval	3	264	263–264	44.7%	not applicable
LibYAML	3	2,333	2,270–2,372	54.6%	not applicable
ls	3	1,771	1,757–1,782	39.9%	not applicable
Oggenc	3	336	336–336	29.5%	not applicable
sed	3	2,950	2,888–3,142	42.4%	not applicable
WOFF2	3	899	877–970	7.9%	not applicable
xmlwf	3	2,062	2,038–2,075	32.8%	not applicable
xorriso	3	4,552	3,262–4,653	8.3%	not applicable

AFL++, 1 core

FuzzerAFL++

–

5.02c

≤ 6 h

fixed

Main

AFL++ is a coverage-guided fuzzer. It mutates inputs, keeps those that reach new code, then mutates those again.

Citation: AFL++: Combining Incremental Steps of Fuzzing Research. 14th USENIX Workshop on Offensive Technologies (WOOT 20), 2020.

Version 5.02c, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: aflpp

Compare with AFL++ rebuilt targets All versions of AFL++

Results of AFL++, 1 core by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	323	307–323	51.1%	not applicable
Krep	3	197	197–197	16.0%	not applicable
Libmatheval	3	343	343–343	58.0%	not applicable
LibYAML	3	2,936	2,914–2,938	68.7%	not applicable
Oggenc	3	179	179–180	15.7%	not applicable
WOFF2	3	1,977	1,897–1,991	17.3%	not applicable

SymCC + AFL++, 1 core

HybridSymCC with AFL++

–

07c8895f+06a515fa

≤ 6 h

fixed

Main

SymCC is a concolic executor that builds the symbolic reasoning into the program at compile time. Here it is paired with AFL++ as a hybrid.

Citation: Symbolic execution with SymCC: Don't interpret, compile. 29th USENIX Security Symposium (USENIX Security 20), 2020; AFL++: Combining Incremental Steps of Fuzzing Research.

Version 07c8895f+06a515fa, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: symcc-aflpp

Compare with SymSan + AFL++, 1 core All versions of SymCC

Results of SymCC + AFL++, 1 core by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	320	320–320	50.6%	not applicable
Krep	3	197	196–204	16.0%	not applicable
Libmatheval	3	271	269–303	45.9%	not applicable
LibYAML	3	2,566	2,542–2,575	60.0%	not applicable
Oggenc	3	181	181–181	15.9%	not applicable
WOFF2	3	1,800	1,640–1,887	15.8%	not applicable

SymSan + AFL++, 1 core

HybridSymSan with AFL++

–

2937d1c0+499bf218

≤ 6 h

fixed

Main

SymSan is a concolic executor built on LLVM's data-flow sanitiser; this build is a reconstruction from its source code. It is designed to reduce the time and memory spent on managing symbolic expressions. Here it is paired with AFL++ as a hybrid.

Citation: SYMSAN: Time and Space Efficient Concolic Execution via Dynamic Data-flow Analysis. 31st USENIX Security Symposium (USENIX Security 22), 2022; AFL++: Combining Incremental Steps of Fuzzing Research.

Version 2937d1c0+499bf218, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: symsan-aflpp

Compare with SymCC + AFL++, 1 core All versions of SymSan

Results of SymSan + AFL++, 1 core by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	323	323–323	51.1%	not applicable
Krep	3	196	196–203	15.9%	not applicable
Libmatheval	3	296	286–301	50.1%	not applicable
LibYAML	3	2,880	2,879–2,946	67.4%	not applicable
Oggenc	3	182	180–190	16.0%	not applicable
WOFF2	3	1,780	1,641–1,826	15.6%	not applicable

AFL++ rebuilt targets

FuzzerAFL++

–

5.02c

–

≤ 6 h

fixed

Control

AFL++ without an agent, on the same rebuilt builds, seeds and dictionaries that the agent's AFL++ runs used. It is the control for those runs.

Citation: AFL++: Combining Incremental Steps of Fuzzing Research. 14th USENIX Workshop on Offensive Technologies (WOOT 20), 2020.

Version 5.02c, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: aflpp-rebuilt-control

Compare with AFL++, 1 core All versions of AFL++

Results of AFL++ rebuilt targets by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	323	323–323	51.1%	not applicable
Krep	3	197	197–197	16.0%	not applicable
Oggenc	3	182	182–187	16.0%	not applicable
WOFF2	3	2,024	1,995–2,046	17.7%	not applicable

KLEE fixed options, 1 core

SymbolicKLEE

–

efa1e052

≤ 6 h

open

Control

KLEE with fixed options, as in the main group's entry.

Citation: KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI 08), 2008.

Version efa1e052, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: klee-fixed-1w

Compare with KLEE fixed options, 6 cores All versions of KLEE

Results of KLEE fixed options, 1 core by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	5	325	312–350	16.9%	not applicable
Confetti	5	145	138–199	22.9%	not applicable
diff	5	680	637–755	15.6%	not applicable
du	5	1,005	948–1,031	13.7%	not applicable
gawk	5	2,059	1,787–2,098	10.1%	not applicable
gcal	5	2,676	2,515–3,006	17.1%	not applicable
grep	5	1,552	864–1,666	32.3%	not applicable
gzip	5	678	583–1,003	22.0%	not applicable
jq	5	1,530	1,176–1,570	8.4%	not applicable
Krep	5	526	434–561	42.7%	not applicable
Libmatheval	5	146	145–149	24.7%	not applicable
LibYAML	5	896	835–1,310	21.0%	not applicable
ls	5	1,099	1,029–1,224	24.7%	not applicable
Oggenc	5	170	166–183	14.9%	not applicable
sed	5	1,012	994–1,134	14.5%	not applicable
WOFF2	5	152	67–163	1.3%	not applicable
xmlwf	5	1,186	1,112–1,423	18.8%	not applicable
xorriso	5	2,380	2,296–2,540	4.4%	not applicable

KLEE uniform options, 1 core

SymbolicKLEE

–

efa1e052

≤ 6 h

open

Control

KLEE with uniform options, sampled from SymTuner's parameter space, as in the main group's entry.

Version efa1e052, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: klee-uniform-1w

Compare with KLEE fixed options, 6 cores All versions of KLEE

Results of KLEE uniform options, 1 core by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	5	460	448–490	23.9%	not applicable
Confetti	5	215	213–258	34.0%	not applicable
diff	5	1,070	967–1,375	24.5%	not applicable
du	5	1,245	1,106–1,283	17.0%	not applicable
gawk	5	3,590	2,657–4,207	17.6%	not applicable
gcal	5	4,198	4,041–4,434	26.9%	not applicable
grep	5	1,718	1,513–1,836	35.7%	not applicable
gzip	5	1,203	1,186–1,341	39.0%	not applicable
jq	5	2,111	1,873–2,267	11.5%	not applicable
Krep	5	598	572–632	48.6%	not applicable
Libmatheval	5	255	253–258	43.1%	not applicable
LibYAML	5	1,828	1,682–2,093	42.8%	not applicable
ls	5	1,645	1,626–1,679	37.0%	not applicable
Oggenc	5	309	305–330	27.2%	not applicable
sed	5	2,475	2,201–2,531	35.6%	not applicable
WOFF2	5	338	197–779	3.0%	not applicable
xmlwf	5	1,800	1,618–1,833	28.6%	not applicable
xorriso	5	2,669	2,572–3,803	4.9%	not applicable

SymTuner, 1 core

SymbolicKLEE

–

71806b45

≤ 6 h

open

Control

SymTuner, as in the main group's entry.

Version 71806b45, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: symtuner-1w

Compare with KLEE fixed options, 6 cores All versions of SymTuner

Results of SymTuner, 1 core by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	5	441	424–479	22.9%	not applicable
Confetti	5	251	214–260	39.7%	not applicable
diff	5	1,240	847–1,894	28.4%	not applicable
du	5	1,272	1,233–1,395	17.4%	not applicable
gawk	5	3,054	2,698–4,007	15.0%	not applicable
gcal	5	5,909	4,279–6,264	37.8%	not applicable
grep	5	1,581	1,565–1,848	32.9%	not applicable
gzip	5	1,311	1,198–1,422	42.5%	not applicable
jq	5	2,090	1,842–2,199	11.4%	not applicable
Krep	5	614	538–686	49.9%	not applicable
Libmatheval	5	258	257–258	43.7%	not applicable
LibYAML	5	1,828	1,645–2,069	42.8%	not applicable
ls	5	1,681	1,629–1,803	37.9%	not applicable
Oggenc	5	305	302–315	26.8%	not applicable
sed	5	2,424	2,199–2,554	34.8%	not applicable
WOFF2	5	249	196–785	2.2%	not applicable
xmlwf	5	1,746	1,593–1,852	27.7%	not applicable
xorriso	5	2,803	2,645–3,746	5.1%	not applicable

Agolic KLEE, GPT-5.4 superseded

AgentKLEE

GPT-5.4

unreleased+cfd2c75f

≤ 6 h

open

Ablation

The agent plans with GPT-5.4. These campaigns were run before the GPT-5.5 campaigns, on an earlier version of the code, so the model and the code differ from the reference configuration.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: GPT-5.4.

Version unreleased+cfd2c75f, submitted 2026-10-22, added in release 2026.1, superseded, superseded by Agolic KLEE, GPT-5.5.

System id in the CSV files: agolic-klee-gpt54

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, GPT-5.4 by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	3	1,415	1,324–1,514	73.6%	not reported
Confetti	3	318	314–320	50.3%	not reported
diff	3	1,282	1,154–1,775	29.3%	not reported
du	3	1,407	1,368–1,423	19.2%	not reported
gawk	3	4,621	4,565–6,315	22.7%	not reported
gcal	3	5,524	5,080–6,573	35.3%	not reported
grep	3	2,466	2,137–2,471	51.3%	not reported
gzip	3	1,534	1,450–1,582	49.8%	not reported
jq	3	4,685	4,120–4,995	25.6%	not reported
Krep	3	829	779–834	67.3%	not reported
Libmatheval	3	334	333–336	56.5%	not reported
LibYAML	3	2,658	2,626–2,668	62.2%	not reported
ls	3	1,723	1,587–2,003	38.8%	not reported
Oggenc	3	799	779–803	70.2%	not reported
sed	3	3,894	3,892–4,080	56.0%	not reported
WOFF2	3	1,725	1,653–1,776	15.1%	not reported
xmlwf	3	3,047	2,766–3,104	48.4%	not reported
xorriso	3	4,572	4,126–6,409	8.4%	not reported

Agolic KLEE, GPT-5.5, effort low

AgentKLEE

GPT-5.5

1.0.1+2407fd07

≤ 6 h

open

Ablation

The reference configuration with the agent's reasoning effort set to low.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: GPT-5.5.

Version 1.0.1+2407fd07, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-gpt55-effort-low

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, GPT-5.5, effort low by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	311	303–312	49.2%	47,808
gzip	3	1,534	1,502–1,553	49.8%	160,285
Krep	3	792	789–799	64.3%	192,480
WOFF2	3	1,772	1,600–1,816	15.5%	125,853

Agolic KLEE, GPT-5.5, effort extra high

AgentKLEE

GPT-5.5

1.0.1+2407fd07

≤ 6 h

open

Ablation

The reference configuration with the agent's reasoning effort set to extra high.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: GPT-5.5.

Version 1.0.1+2407fd07, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-gpt55-effort-xhigh

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, GPT-5.5, effort extra high by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	322	319–323	50.9%	71,020
gzip	3	1,564	1,539–1,659	50.7%	296,347
Krep	3	847	839–871	68.8%	596,149
WOFF2	3	1,828	1,810–1,902	16.0%	536,634

Agolic KLEE, GPT-5.5, Harness-Entry Mode only

AgentKLEE

GPT-5.5

1.0.1+2407fd07

≤ 6 h

open

Ablation

The agent may start symbolic exploration only at the entry of the KLEE harness, as an ordinary KLEE invocation does. The mode that runs a concrete seed up to a chosen start function (Seeded-Prefix Mode) is turned off.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: GPT-5.5.

Version 1.0.1+2407fd07, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-gpt55-harness-entry-only

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, GPT-5.5, Harness-Entry Mode only by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	222	216–252	35.1%	88,441
gzip	3	1,356	1,343–1,390	44.0%	213,053
Krep	3	821	797–822	66.7%	290,512
WOFF2	3	213	199–222	1.9%	173,281

Agolic KLEE, GPT-5.5, Seeded-Prefix Mode only

AgentKLEE

GPT-5.5

1.0.1+2407fd07

≤ 6 h

open

Ablation

The agent is limited to Seeded-Prefix Mode. Every planned invocation executes a concrete seed up to a start function and symbolic exploration begins there.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: GPT-5.5.

Version 1.0.1+2407fd07, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-gpt55-seeded-prefix-only

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, GPT-5.5, Seeded-Prefix Mode only by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	323	307–323	51.1%	34,185
gzip	3	1,522	1,499–1,530	49.4%	167,709
Krep	3	852	835–881	69.2%	361,957
WOFF2	3	1,696	1,600–1,846	14.9%	226,624

Agolic KLEE, GPT-5.5, seed finder off

AgentKLEE

GPT-5.5

1.0.1+2407fd07

≤ 6 h

open

Ablation

The seed finder is turned off. The seed finder is a second agent that searches the web for example input files.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: GPT-5.5.

Version 1.0.1+2407fd07, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-gpt55-seed-finder-off

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, GPT-5.5, seed finder off by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	322	313–322	50.9%	53,141
gzip	3	1,543	1,494–1,545	50.0%	252,480
Krep	3	812	807–829	66.0%	337,815
WOFF2	3	1,649	1,626–1,721	14.4%	130,470

Agolic KLEE, GPT-5.5, proposal review off

AgentKLEE

GPT-5.5

1.0.1+2407fd07

≤ 6 h

open

Ablation

The review step that checks the agent's proposed invocation specifications before they are dispatched is turned off.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: GPT-5.5.

Version 1.0.1+2407fd07, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-gpt55-proposal-review-off

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, GPT-5.5, proposal review off by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	320	306–323	50.6%	48,676
gzip	3	1,584	1,543–1,607	51.4%	157,408
Krep	3	846	813–854	68.7%	395,447
WOFF2	3	1,710	1,672–1,829	15.0%	238,968

Agolic KLEE, GPT-5.5, run history withheld

AgentKLEE

GPT-5.5

unknown

≤ 6 h

open

Ablation

The record of earlier runs and their outcomes is withheld from the agent, so each plan is made without knowing what the previous runs reached.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: GPT-5.5.

Version unknown, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-gpt55-history-withheld

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, GPT-5.5, run history withheld by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	322	319–322	50.9%	179,266
gzip	3	1,487	1,485–1,526	48.2%	193,802
Krep	3	815	796–888	66.2%	270,350
WOFF2	3	1,566	1,539–1,573	13.7%	239,028

Agolic KLEE, DeepSeek V4.1 Flash, effort low

AgentKLEE

DeepSeek V4.1 Flash

1.0.1+8a86838e

≤ 6 h

open

Ablation

The DeepSeek configuration with the model's reasoning effort set to low.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: DeepSeek V4.1 Flash.

Version 1.0.1+8a86838e, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-deepseek-effort-low

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, DeepSeek V4.1 Flash, effort low by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	321	321–323	50.8%	214,927
gzip	3	1,674	1,650–1,718	54.3%	1,600,709
Krep	3	788	783–799	64.0%	1,691,874
WOFF2	3	1,880	1,844–2,027	16.5%	1,133,503

Agolic KLEE, DeepSeek V4.1 Flash, effort high

AgentKLEE

DeepSeek V4.1 Flash

1.0.1+8a86838e

≤ 6 h

open

Ablation

The DeepSeek configuration with the model's reasoning effort set to high.

Citation: Agentic Planning for Symbolic Execution. arXiv 2608.06397, 2026; model: DeepSeek V4.1 Flash.

Version 1.0.1+8a86838e, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: agolic-klee-deepseek-effort-high

Compare with Agolic KLEE, GPT-5.5 All versions of Agolic

Results of Agolic KLEE, DeepSeek V4.1 Flash, effort high by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
Confetti	3	323	323–323	51.1%	265,385
gzip	3	1,706	1,637–1,771	55.3%	2,120,385
Krep	3	791	777–827	64.3%	2,423,342
WOFF2	3	1,966	1,946–2,018	17.2%	1,464,639

SymTuner rerun, 6 cores

SymbolicKLEE

–

71806b45

≤ 6 h

open

Seeding

A second run of SymTuner with the baseline's master seeds. The count is of the tests SymTuner kept.

Citation: SymTuner: Maximizing the Power of Symbolic Execution by Adaptively Tuning External Parameters. Proceedings of the 44th International Conference on Software Engineering (ICSE 2022), 2022.

Version 71806b45, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: symtuner-6w-rerun

Compare with KLEE fixed options, 6 cores All versions of SymTuner

Results of SymTuner rerun, 6 cores by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	3	546	523–563	28.4%	not applicable
Confetti	3	255	254–264	40.3%	not applicable
gzip	3	1,456	1,452–1,459	47.2%	not applicable
jq	3	2,434	2,420–2,468	13.3%	not applicable
Krep	3	665	662–665	54.0%	not applicable
WOFF2	3	675	314–822	5.9%	not applicable
xmlwf	3	2,020	1,963–2,081	32.1%	not applicable

SymTuner from the agent's seeds, 6 cores, its own tests

SymbolicKLEE

–

71806b45

≤ 6 h

open

Seeding

SymTuner started from the seeds that an Agolic campaign with GPT-5.5 produced for the same program. Only the tests that SymTuner itself generated are counted.

Citation: SymTuner: Maximizing the Power of Symbolic Execution by Adaptively Tuning External Parameters. Proceedings of the 44th International Conference on Software Engineering (ICSE 2022), 2022.

Version 71806b45, submitted 2026-10-16, added in release 2026.1, current.

System id in the CSV files: symtuner-6w-rerun-seeded-tests-only

Compare with KLEE fixed options, 6 cores All versions of SymTuner

Results of SymTuner from the agent's seeds, 6 cores, its own tests by program
Program	Runs	Median	Min to max	Share (median)	Output tokens
bc	3	517	504–518	26.9%	not applicable
Confetti	3	315	312–316	49.8%	not applicable
gzip	3	1,497	1,482–1,522	48.6%	not applicable
jq	3	2,354	2,280–2,498	12.9%	not applicable
Krep	3	649	639–714	52.7%	not applicable
WOFF2	3	998	967–1,139	8.7%	not applicable
xmlwf	3	1,964	1,914–2,036	31.2%	not applicable