01/

Research Index

Index of findings — F-01…F-04

Four studies on where modern perception stacks break: calibration under uncertainty, benchmark contamination, extreme quantization, and the sim-to-real gap. Each entry is a paper-style abstract — open a finding for its shareable permalink.

F-01 VLM Calibration · Modal A100

Selective Refusal as Protective Calibration

Abstract
Which VLM should a safety-critical robot trust when it's uncertain?
Method
Six-pipeline benchmark — Qwen2.5-VL 3B / 7B / 72B + InternVL2-2B — on Modal A100s over robot perception frames.
Key Result
Failure modes are model-family-specific: one family refuses under uncertainty, another confabulates on identical inputs. The refusing family reaches ~2× precision and 5–10× recall on engaged frames.
Implication
Refusal is calibration, not weakness — pick model families by failure mode, not leaderboard rank.
Fig. 1 — Precision / recall vs confabulating family
precision ~2× recall 5–10× engaged frames · refusing family vs baseline
Permalink · /research/selective-refusal →
F-02 Benchmark Ablation

Prompt Priming Masquerading as Visual Attribution

Abstract
An apparent visual-misattribution effect in detector-grounded VLM benchmarks.
Method
Marker × prompt ablation isolating the visual channel from the text channel.
Key Result
The effect was prompt-text priming — wording shifted results 9.4pp; visual markers ≤0.2pp.
Implication
Benchmark conclusions can flip when input channels aren't isolated.
Fig. 2 — Effect size by input channel
prompt text 9.4pp visual marker ≤0.2pp shift in benchmark outcome (percentage points)
Permalink · /research/prompt-priming →
F-03 1-bit VLMs · Edge

Vision on 1-bit LLMs & a Silent Training Failure

Abstract
Can vision be grafted onto extreme-quantized (1-bit) language models?
Method
Vision-grafting pipeline onto BitNet / Falcon3 1-bit LLMs.
Key Result
Identified a previously-unnamed boundary-supervision failure — silent at train time, degenerate at inference — fixed with one line. Measured a CPU/GPU efficiency inversion: ~3× CPU speedup, ~6× lower memory vs 4-bit baselines.
Implication
1-bit VLMs are natively suited to CPU/ARM edge hardware.
Fig. 3 — 1-bit vs 4-bit baseline (CPU)
speed ~3× memory ÷6 grey = 4-bit baseline · cyan = 1-bit
Permalink · /research/1bit-vision →
F-04 Manuscript in preparation

The Sim-to-Real Gap in Indoor Segmentation

Abstract
Do synthetic validation scores predict real-world performance?
Method
SegFormer-B0 fine-tuned on a 1,458-image labeled real indoor dataset.
Key Result
0.676 mIoU on real-frame validation — while a synthetic-trained counterpart scored 0.988 val mIoU yet failed on real footage.
Implication
Synthetic metrics can wildly overstate deployability.
Fig. 4 — Validation mIoU: sim vs real
0.988 sim (synthetic val) 0.676 real (deployed) same architecture · different training reality
Permalink · /research/sim-to-real →