Terms
- Release
- A dated snapshot of the data, named like 2026.1. It never changes; corrections go into the next release.
- Entry
- A tool in one version and configuration, with a model, a track and a budget. One row of the data.
- Superseded
- An entry that a newer one replaces. It stays in the release and shows under all versions.
- Withdrawn
- An entry taken out of the comparison after it was published.
Project: github.com/ConcoLLMic/ConcoLLMic
Reference: doi.org/10.1109/SP63933.2026.00003
Versions
| Entry | Version | Model | Configuration | Submitted | First release | Programs covered | Status |
|---|---|---|---|---|---|---|---|
| ConcoLLMic, GPT-5.5 | 9634e686+ | GPT-5.5 | --timeout 10 --plateau_slot 30 --parallel_num 5; reasoning effort medium | 2026-10-16 | 2026.1 | 7 of 18 | current |
| ConcoLLMic, DeepSeek V4.1 Flash | 9634e686+ | DeepSeek V4.1 Flash | --timeout 10 --plateau_slot 30 --parallel_num 5; reasoning effort medium | 2026-10-16 | 2026.1 | 3 of 18 | current |
Median coverage by version
| Program | ConcoLLMicGPT-5.59634e686+ | ConcoLLMicDeepSeek V4.1 Flash9634e686+ |
|---|---|---|
| bc | 69.4%, 3 runs | no runs |
| Confetti | 49.3%†, 2 runs | 49.5%, 3 runs |
| Krep | 72.7%, 3 runs | 71.4%, 3 runs |
| Libmatheval | 56.5%†, 2 runs | no runs |
| LibYAML | 42.4%, 3 runs | no runs |
| Oggenc | 57.8%, 3 runs | no runs |
| WOFF2 | 15.4%†, 1 run | 16.5%, 3 runs |