| glm-5.1 (Ark) | kimi-2.6 | deepseek-v4-flash@max | MiniMax-M3 | deepseek-v4-pro@max (Ark) | |
|---|---|---|---|---|---|
| Score | 57/89 · 64.0% | 53/89 · 59.6% | 50/89 · 56.2% | 46/89 · 51.7% | 41/89 · 46.1% |
| Resolved% | 65.5% | 63.9% | 61.7% | 53.5% | 46.1% |
| Failed | 30 | 30 | 31 | 40 | 48 |
| Errored | 2 | 6 | 8 | 3 | 0 |
| Exception breakdown | 10 TimedOut, 2 VerifierTimeoutError | 6 RateLimited, 14 TimedOut | 7 ConnectionDropped, 5 TimedOut, 1 VerifierTimeoutError | 12 TimedOut, 2 ConnectionDropped, 1 VerifierTimeoutError | 34 TimedOut |
| Input tokens | 31.8M | 119.0M | 328.2M | 91.9M | 24.4M |
| Output tokens | 648.7K | 1.9M | 3.1M | 2.2M | 1.2M |
| Cache hit | 0.0% | 97.3% | 99.0% | 93.8% | 75.8% |
| Input tokens / passed task | 557.7K | 2.2M | 6.6M | 2.0M | 595.3K |
| Est. cost ‡ | ≈$33.16 | ≈$28.19 | ≈$2.28 | ≈$9.39 | ≈$14.69 |
| Cost / passed task ‡ | ≈$0.58 | ≈$0.53 | ≈$0.05 | ≈$0.20 | ≈$0.36 |
‡ Cost = (input − cache) × miss-price + cache × hit-price + output × out-price, summed across all trials and converted at ≈1 USD = 7.10 CNY. Per-run prices configured via --price-cny in CNY/1M tokens.
Sorted by divergence (most-disagreed first). Click a column header to re-sort.
| Task | glm-5.1 (Ark) | kimi-2.6 | deepseek-v4-flash@max | MiniMax-M3 | deepseek-v4-pro@max (Ark) | Agreement |
|---|---|---|---|---|---|---|
| circuit-fibsqrt | failed | failed · TimedOut | failed | failed | failed · TimedOut | none of 5 passed |
| dna-assembly | failed | failed | errored · ConnectionDropped | failed | failed · TimedOut | none of 5 passed |
| dna-insert | failed | failed | failed | failed | failed | none of 5 passed |
| extract-moves-from-video | failed · TimedOut | failed · TimedOut | failed · TimedOut | failed | failed · TimedOut | none of 5 passed |
| gcode-to-text | failed | failed · TimedOut | failed | failed · TimedOut | failed | none of 5 passed |
| gpt2-codegolf | failed · TimedOut | failed · TimedOut | failed · TimedOut | failed | failed · TimedOut | none of 5 passed |
| install-windows-3.11 | failed | failed | failed | failed | failed · TimedOut | none of 5 passed |
| make-doom-for-mips | failed · TimedOut | failed · TimedOut | failed | failed | failed · TimedOut | none of 5 passed |
| model-extraction-relu-logits | failed | failed | failed | errored · ConnectionDropped | failed · TimedOut | none of 5 passed |
| path-tracing | failed · TimedOut | failed · TimedOut | errored · ConnectionDropped | failed | failed · TimedOut | none of 5 passed |
| protein-assembly | failed · TimedOut | failed | errored · ConnectionDropped | failed | failed · TimedOut | none of 5 passed |
| raman-fitting | failed | failed | failed | failed · TimedOut | failed · TimedOut | none of 5 passed |
| regex-chess | failed | failed · TimedOut | failed | failed | failed · TimedOut | none of 5 passed |
| train-fasttext | failed · TimedOut | failed | errored · ConnectionDropped | failed | failed · TimedOut | none of 5 passed |
| video-processing | failed | failed · TimedOut | failed | failed | failed · TimedOut | none of 5 passed |
| build-pmars | passed | passed | passed | passed | passed | all 5 passed |
| cobol-modernization | passed | passed | passed | passed | passed | all 5 passed |
| code-from-image | passed | passed | passed | passed | passed | all 5 passed |
| custom-memory-heap-crash | passed | passed | passed | passed | passed | all 5 passed |
| distribution-search | passed | passed | passed | passed | passed | all 5 passed |
| fix-git | passed | passed | passed | passed | passed | all 5 passed |
| fix-ocaml-gc | passed | passed | passed | passed | passed | all 5 passed |
| git-leak-recovery | passed | passed | passed | passed | passed | all 5 passed |
| hf-model-inference | passed | passed | passed | passed | passed | all 5 passed |
| log-summary-date-ranges | passed | passed | passed | passed | passed | all 5 passed |
| merge-diff-arc-agi-task | passed | passed | passed | passed | passed | all 5 passed |
| modernize-scientific-stack | passed | passed | passed | passed | passed | all 5 passed |
| multi-source-data-merger | passed | passed | passed | passed | passed | all 5 passed |
| nginx-request-logging | passed | passed | passed | passed | passed | all 5 passed |
| prove-plus-comm | passed | passed | passed | passed | passed | all 5 passed |
| regex-log | passed | passed | passed | passed | passed | all 5 passed |
| sqlite-db-truncate | passed | passed | passed | passed | passed | all 5 passed |
| vulnerable-secret | passed | passed | passed | passed | passed | all 5 passed |
| adaptive-rejection-sampler | failed · TimedOut | passed | failed | failed · TimedOut | failed · TimedOut | 1 of 5 passed |
| cancel-async-tasks | failed | failed | failed | failed | passed | 1 of 5 passed |
| chess-best-move | failed | failed | passed | failed · TimedOut | failed · TimedOut | 1 of 5 passed |
| db-wal-recovery | passed | failed · TimedOut | failed | failed | failed · TimedOut | 1 of 5 passed |
| feal-linear-cryptanalysis | failed | passed | errored · ConnectionDropped | failed · TimedOut | failed · TimedOut | 1 of 5 passed |
| filter-js-from-html | passed | failed · TimedOut | failed | failed | failed | 1 of 5 passed |
| pytorch-model-cli | failed | failed | failed | failed | passed | 1 of 5 passed |
| query-optimize | failed | failed | passed | failed | failed · TimedOut | 1 of 5 passed |
| rstan-to-pystan | failed · TimedOut | passed | failed · TimedOut | failed · TimedOut | failed · TimedOut | 1 of 5 passed |
| torch-pipeline-parallelism | passed | failed | failed | failed · TimedOut | failed · TimedOut | 1 of 5 passed |
| tune-mjcf | failed · TimedOut | errored · RateLimited | passed | failed · TimedOut | failed · TimedOut | 1 of 5 passed |
| write-compressor | failed | passed | failed · TimedOut | failed | failed · TimedOut | 1 of 5 passed |
| break-filter-js-from-html | passed | failed | passed | passed | passed | 4 of 5 passed |
| build-cython-ext | passed | passed | passed | passed | failed | 4 of 5 passed |
| compile-compcert | failed · TimedOut | passed | passed | passed | passed | 4 of 5 passed |
| configure-git-webserver | passed | passed | passed | failed | passed | 4 of 5 passed |
| constraints-scheduling | passed | errored · RateLimited | passed | passed | passed | 4 of 5 passed |
| crack-7z-hash | passed | passed | passed | passed | failed · TimedOut | 4 of 5 passed |
| extract-elf | passed | passed | passed | failed · TimedOut | passed | 4 of 5 passed |
| feal-differential-cryptanalysis | failed | passed | passed | passed | passed | 4 of 5 passed |
| fix-code-vulnerability | passed | errored · RateLimited | passed | passed | passed | 4 of 5 passed |
| git-multibranch | passed | passed | passed | passed | failed | 4 of 5 passed |
| headless-terminal | failed | passed | passed | passed | passed | 4 of 5 passed |
| large-scale-text-editing | passed | passed | passed | passed | failed · TimedOut | 4 of 5 passed |
| mcmc-sampling-stan | errored · VerifierTimeoutError | passed | passed | passed | passed | 4 of 5 passed |
| password-recovery | passed | passed | passed | failed · TimedOut | passed | 4 of 5 passed |
| polyglot-c-py | passed | passed | failed | passed | passed | 4 of 5 passed |
| qemu-startup | passed | passed | errored · ConnectionDropped | passed | passed | 4 of 5 passed |
| reshard-c4-data | passed | passed | passed | failed | passed | 4 of 5 passed |
| sam-cell-seg | errored · VerifierTimeoutError | passed | passed | passed | passed | 4 of 5 passed |
| sparql-university | passed | passed | failed | passed | passed | 4 of 5 passed |
| torch-tensor-parallelism | passed | passed | passed | passed | failed | 4 of 5 passed |
| bn-fit-modify | passed | errored · RateLimited | failed | passed | failed | 2 of 5 passed |
| build-pov-ray | passed | failed | failed | passed | failed | 2 of 5 passed |
| count-dataset-tokens | failed | passed | passed | failed | failed | 2 of 5 passed |
| financial-document-processor | passed | failed · TimedOut | failed | failed | passed | 2 of 5 passed |
| make-mips-interpreter | passed | failed · TimedOut | errored · ConnectionDropped | passed | failed · TimedOut | 2 of 5 passed |
| pytorch-model-recovery | passed | passed | errored · VerifierTimeoutError | errored · VerifierTimeoutError | failed · TimedOut | 2 of 5 passed |
| winning-avg-corewars | passed | errored · RateLimited | passed | failed | failed · TimedOut | 2 of 5 passed |
| caffe-cifar-10 | passed | passed | failed | passed | failed · TimedOut | 3 of 5 passed |
| kv-store-grpc | passed | passed | failed | failed | passed | 3 of 5 passed |
| largest-eigenval | passed | failed | passed | passed | failed · TimedOut | 3 of 5 passed |
| llm-inference-batching-scheduler | passed | passed | passed | failed · TimedOut | failed · TimedOut | 3 of 5 passed |
| mailman | passed | failed | passed | failed | passed | 3 of 5 passed |
| mteb-leaderboard | passed | passed | failed | passed | failed | 3 of 5 passed |
| mteb-retrieve | failed | passed | passed | passed | failed | 3 of 5 passed |
| openssl-selfsigned-cert | passed | passed | failed | failed | passed | 3 of 5 passed |
| overfull-hbox | passed | passed | passed | failed · TimedOut | failed · TimedOut | 3 of 5 passed |
| path-tracing-reverse | passed | failed · TimedOut | passed | passed | failed · TimedOut | 3 of 5 passed |
| polyglot-rust-c | failed | passed | passed | passed | failed · TimedOut | 3 of 5 passed |
| portfolio-optimization | passed | errored · RateLimited | passed | failed | passed | 3 of 5 passed |
| pypi-server | passed | passed | failed | passed | failed | 3 of 5 passed |
| qemu-alpine-ssh | passed | failed · TimedOut | failed · TimedOut | passed | passed | 3 of 5 passed |
| sanitize-git-repo | passed | passed | failed | passed | failed | 3 of 5 passed |
| schemelike-metacircular-eval | passed | passed | passed | failed | failed · TimedOut | 3 of 5 passed |
| sqlite-with-gcov | passed | passed | passed | errored · ConnectionDropped | failed | 3 of 5 passed |