Agolic
An LLM agent that plans a campaign of short runs on a backend (KLEE, Direct-Input or AFL++) and chooses for each run its command-line arguments, inputs and seeds from the source code and the results so far.
Terms
- Release
- A dated snapshot of the data, named like 2026.1. It never changes; corrections go into the next release.
- Entry
- A tool in one version and configuration, with a model, a track and a budget. One row of the data.
- Superseded
- An entry that a newer one replaces. It stays in the release and shows under all versions.
- Withdrawn
- An entry taken out of the comparison after it was published.
Reference: arxiv.org/abs/2608.06397
Versions
| Entry | Version | Model | Configuration | Submitted | First release | Programs covered | Status |
|---|---|---|---|---|---|---|---|
| Agolic KLEE, GPT-5.4 | unreleased+ | GPT-5.4 | KLEE; reasoning effort medium; six slots; 40 turns per round | 2026-10-22 | 2026.1 | 18 of 18 | superseded by Agolic KLEE, GPT-5.5 |
| Agolic KLEE, GPT-5.5 | 1.0.1+ | GPT-5.5 | KLEE; reasoning effort medium; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 18 of 18 | current |
| Agolic KLEE, DeepSeek V4.1 Flash | 1.0.1+ | DeepSeek V4.1 Flash | KLEE; reasoning effort medium; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 9 of 18 | current |
| Agolic Direct-Input, DeepSeek V4.1 Flash | 1.0.1+ | DeepSeek V4.1 Flash | Direct-Input; reasoning effort medium; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 9 of 18 | current |
| Agolic AFL++, DeepSeek V4.1 Flash | 1.0.1+ | DeepSeek V4.1 Flash | AFL++; reasoning effort medium; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
| Agolic KLEE, GPT-5.5, effort low | 1.0.1+ | GPT-5.5 | KLEE; reasoning effort low; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
| Agolic KLEE, GPT-5.5, effort extra high | 1.0.1+ | GPT-5.5 | KLEE; reasoning effort extra high; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
| Agolic KLEE, GPT-5.5, Harness-Entry Mode only | 1.0.1+ | GPT-5.5 | KLEE; Harness-Entry Mode only; reasoning effort medium; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
| Agolic KLEE, GPT-5.5, Seeded-Prefix Mode only | 1.0.1+ | GPT-5.5 | KLEE; Seeded-Prefix Mode only; reasoning effort medium; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
| Agolic KLEE, GPT-5.5, seed finder off | 1.0.1+ | GPT-5.5 | KLEE; seed finder off; reasoning effort medium; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
| Agolic KLEE, GPT-5.5, proposal review off | 1.0.1+ | GPT-5.5 | KLEE; proposal review off; reasoning effort medium; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
| Agolic KLEE, GPT-5.5, run history withheld | unknown | GPT-5.5 | KLEE; run history withheld; reasoning effort medium; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
| Agolic KLEE, DeepSeek V4.1 Flash, effort low | 1.0.1+ | DeepSeek V4.1 Flash | KLEE; reasoning effort low; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
| Agolic KLEE, DeepSeek V4.1 Flash, effort high | 1.0.1+ | DeepSeek V4.1 Flash | KLEE; reasoning effort high; six slots; 60 turns per round | 2026-10-16 | 2026.1 | 4 of 18 | current |
Median coverage by version
| Program | Agolic KLEEGPT-5.51.0.1+ | Agolic KLEEDeepSeek V4.1 Flash1.0.1+ | Agolic Direct-InputDeepSeek V4.1 Flash1.0.1+ | Agolic AFL++DeepSeek V4.1 Flash1.0.1+ |
|---|---|---|---|---|
| bc | 75.4%, 3 runs | no runs | no runs | no runs |
| Confetti | 50.0%, 3 runs | 51.1%, 3 runs | 51.1%, 3 runs | 51.1%, 3 runs |
| diff | 47.4%, 3 runs | 55.1%, 3 runs | 62.2%, 3 runs | no runs |
| du | 22.7%, 3 runs | 24.1%, 3 runs | 24.8%, 3 runs | no runs |
| gawk | 37.5%, 3 runs | 55.7%, 3 runs | 60.8%, 3 runs | no runs |
| gcal | 41.9%, 3 runs | no runs | no runs | no runs |
| grep | 55.8%, 3 runs | no runs | no runs | no runs |
| gzip | 50.8%, 3 runs | 53.1%, 3 runs | 58.1%, 3 runs | no runs |
| jq | 30.5%, 3 runs | no runs | no runs | no runs |
| Krep | 67.2%, 3 runs | 66.0%, 3 runs | 77.0%, 3 runs | 63.4%, 3 runs |
| Libmatheval | 56.9%, 3 runs | no runs | no runs | no runs |
| LibYAML | 63.7%, 3 runs | no runs | no runs | no runs |
| ls | 47.4%, 3 runs | 58.0%, 3 runs | 53.7%, 3 runs | no runs |
| Oggenc | 71.0%, 3 runs | no runs | no runs | 71.4%, 3 runs |
| sed | 55.3%, 3 runs | no runs | no runs | no runs |
| WOFF2 | 16.0%, 3 runs | 16.1%, 3 runs | 18.7%, 3 runs | 19.3%, 3 runs |
| xmlwf | 61.5%, 3 runs | 66.6%, 3 runs | 62.7%, 3 runs | no runs |
| xorriso | 10.7%, 3 runs | no runs | no runs | no runs |