QuantBench
An independent 4-bit quantization benchmark that publishes its failures
R3 · measurementIndependent, July 2026. Runs on Modal; data on Hugging Face.
- Question
- GPTQ and AWQ for small instruct models: Δ perplexity against fp16 on wikitext2, measured against throughput, VRAM and artifact size on Modal A10 and T4.
- Method
- Stack-specific baselines, median of three runs, no extrapolation.
- Result
- Seven configurations failed and are documented as results. Full sweep dataset published.
- Implication
- A benchmark that only reports what worked is an advertisement. Failures are results too.
7failing configurations, documented as results