02 · Measured evidence
The first controlled run changes the decision.
These are exploratory ten-way next-page ranking and perception diagnostics, not a title-level result. They tell us what to build next—and which attractive idea has not yet earned promotion.
433NPP task rows283 train, 84 validation, and 66 test rows after prefix, target, split, and candidate-order repairs.
38unique test targetsOnly two chapters from one title; the 10-page condition has 28 eligible targets.
10%random top-1Each item has ten candidates; expected random mean reciprocal rank is .293.
28.9%frozen SigLIP · 5 pagesA task-local, image-only similarity reference; it falls to 21.4% with 10 pages.
Perception · viable front end
MagiV2 sees characters well; text labels expose a granularity mismatch.
Across all 308 pages, standard IoU@.5 character detection reaches .7727 precision, .9479 recall, and .8514 F1. Standard text detection is only .0807 precision, .0967 recall, and .0879 F1. The matched prediction-to-gold area ratio has a .263 median, consistent with tight lettering predictions being compared with larger Re:Verse bubble regions.
A separate intersection-over-smaller containment diagnostic gives text .8317 precision, .9968 recall, and .9068 F1. That .9068 is IoS, not standard IoU, AP, or a substitute detection score; it only shows that tight predicted text is usually contained by a released bubble box.
The four-page OCR pilot compared 25 outputs with 11 adapted teacher turns: mean best token-F1 per teacher turn was .8328, with all 11 turns reaching at least .5; mean best token-F1 per OCR string was .4727. This is a lexical proxy—not OCR ground truth, CER, WER, precision, or recall.
State utility · no-go
The current deterministic state loses useful signal.
Privileged local text and gold identity/layout improve candidate ranking, so narrative and structural information can matter. But the matched Story State serialization trails the frozen visual reference, and its linear head collapses to 7.1% with 10 pages.
In the held-constant fusion test, local text beats matched state by 18.4 top-1 points at five pages (descriptive 95% interval +5.3 to +31.6) and 17.9 points at ten (+3.6 to +28.6). Candidate representations and the SigLIP prefix sequence are identical; only the prefix text representation changes.
The current deterministic state has not earned promotion. Branch-aware graphs, uncertain relations, open-thread stores, and predictive latent slots remain candidate data structures—a hypothesis to test after exact retrieval and a strong task reader, not an established architecture win.
Protocol lesson
Seemingly small leakage channels can dominate NPP.
If page identifiers escape the boundary, chronology solves all 433 rows. Reused candidates also permit transductive strategies. Scientific runs therefore detach pixels, strip file and page metadata, expose only neutral A–J aliases, and require a stateless task-local scorer.
The repaired manifest excludes prefix/target gaps and keeps identical candidates for a shared target across 5- and 10-page comparisons. The public manifest is an audit artifact, not by itself a sealed benchmark.
Qwen2.5-VL · protocol repair
The first zero-shot run was invalidated by answer-position bias.
The superseded three-seed validation run scored 16.7% top-1 / .367 MRR, but accuracy changed drastically with the gold letter: D reached 46.9%, H 32.4%, A 4.2%, and J 0%. Those numbers are not model evidence.
Version 2 cycles every item through A–J exactly once and shuffles negatives independently. Validation is 13.4% top-1 / .329 MRR with five pages and 10.0% / .316 with ten. The frozen test is 17.1% / .371 and 17.1% / .360, with 99.7%/100% parse rates.
That is above chance on this fixed test cohort, but 11.8 and 4.3 points below frozen SigLIP; paired descriptive intervals include zero. Preferred canonical images still change across a mean 4.2 of ten permutations. Balancing neutralizes aggregate letter preference—it does not make Qwen order-invariant or establish a gain over simple image similarity.