Ranking under each sentiment pathway

Same cached inputs for every pathway (MR_OFFLINE=1): RT, BOM, CinemaScore and the same 1,381 comments; only the per-comment sentiment/stance step differs. Rank = position in the shortlist (bucket first, then score). Rank change = live rank minus pathway rank (positive = moves up). Labels for accuracy are mine (Grok Bot), single annotator. Live index.html uses GPT-6.1 Sol (from cached verdicts) since 2026-10-25; rank changes are vs that live pathway. Every pathway uses the same bucket rule v2: Watch needs score >= max(75, Q3 of that pathway's slate).

Cell = #rank · score · change vs live (↑ moves up) · bucket · R = Reddit community score. Colour = bucket (Watch Depends Skip). Click a cell for that movie's comments with each pathway's verdict and rationale. The live shortlist on the home page uses GPT-6.1 Sol (since 2026-10-25; previously siebert+opinion). Buckets use rule v2 for every pathway: Watch needs score ≥ max(75, Q3 of that pathway's own slate); per-pathway bars: VADER (old v1) 80.05, siebert+opinion (previous live) 77.25, Local NLI stance (deberta-v3-large) 75, GPT-6.1 Sol (LIVE) 90.45, Claude Opus 5.5 91.05, Claude Sonnet 5.5 90, Gemini 4 Argon 91.3, Grok 4.7 90.55, GPT-6 Luna (cheap) 89.05, Claude Haiku 5.5 (cheap) 89.45, Gemini 3.5 Flash-Lite (cheap) 88.55. Skip < 55 (see Formula).

Stance pathways ask: did the commenter like or recommend the film? (positive / negative / mixed / no verdict, JSON, temperature 0). Mixed and no-verdict count as neutral. OpenRouter spend for these evaluations and runs: $14.47. Not run: Local Qwen2.5-3B stance

movie (ordered by live rank)VADER (old v1)siebert+opinion (previous live)Local NLI stance (deberta-v3-large)GPT-6.1 Sol (LIVE)Claude Opus 5.5Claude Sonnet 5.5Gemini 4 ArgonGrok 4.7GPT-6 Luna (cheap)Claude Haiku 5.5 (cheap)Gemini 3.5 Flash-Lite (cheap)
You Can See Everything#3 81 ↓2
Watch · R 66
#8 71 ↓7
Depends · R 47
#9 56 ↓8
Depends · R 20
#1 95
Watch · R 93
#1 95 =
Watch · R 93
#1 93 =
Watch · R 89
#1 95 =
Watch · R 93
#2 92 ↓1
Watch · R 88
#5 89 ↓4
Depends · R 82
#5 89 ↓4
Depends · R 81
#1 93 =
Watch · R 89
Wicker#2 81 =
Watch · R 75
#4 79 ↓2
Watch · R 71
#10 52 ↓8
Skip · R 14
#2 92
Watch · R 98
#4 92 ↓2
Watch · R 98
#3 92 ↓1
Watch · R 98
#4 92 ↓2
Watch · R 98
#3 92 ↓1
Watch · R 98
#2 92 =
Watch · R 98
#2 92 =
Watch · R 98
#4 89 ↓2
Watch · R 93
Wildwood#5 80 ↓2
Depends · R 68
#1 81 ↑2
Watch · R 70
#1 84 ↑2
Watch · R 76
#3 92
Watch · R 94
#2 92 ↑1
Watch · R 96
#2 92 ↑1
Watch · R 95
#2 92 ↑1
Watch · R 95
#1 92 ↑2
Watch · R 95
#1 92 ↑2
Watch · R 96
#3 91 =
Watch · R 92
#3 92 =
Watch · R 94
Resident Evil#9 71 ↓5
Depends · R 55
#7 72 ↓3
Depends · R 58
#2 73 ↑2
Depends · R 59
#4 91
Watch · R 98
#3 92 ↑1
Watch · R 100
#5 89 ↓1
Depends · R 94
#3 92 ↑1
Watch · R 100
#5 90 ↓1
Depends · R 96
#3 91 ↑1
Watch · R 97
#1 92 ↑3
Watch · R 100
#2 92 ↑2
Watch · R 100
Primetime#4 80 ↑1
Watch · R 79
#2 81 ↑3
Watch · R 80
#3 68 ↑2
Depends · R 56
#5 90
Depends · R 97
#5 90 =
Depends · R 98
#4 91 ↑1
Watch · R 99
#5 91 =
Depends · R 99
#4 91 ↑1
Watch · R 99
#4 89 ↑1
Watch · R 96
#4 90 ↑1
Watch · R 98
#5 88 =
Depends · R 93
Clayface#7 76 ↓1
Depends · R 72
#6 73 =
Depends · R 63
#4 67 ↑2
Depends · R 50
#6 88
Depends · R 97
#6 89 =
Depends · R 99
#6 89 =
Depends · R 99
#6 89 =
Depends · R 99
#6 88 =
Depends · R 98
#6 89 =
Depends · R 99
#6 88 =
Depends · R 98
#6 87 =
Depends · R 96
Forgotten Island#1 83 ↑6
Watch · R 82
#3 79 ↑4
Watch · R 75
#8 58 ↓1
Depends · R 27
#7 88
Depends · R 93
#7 89 =
Depends · R 95
#7 87 =
Depends · R 92
#7 89 =
Depends · R 95
#7 87 =
Depends · R 91
#7 88 =
Depends · R 94
#7 87 =
Depends · R 92
#7 84 =
Depends · R 86
Heart of the Beast#8 74 =
Depends · R 66
#9 67 ↓1
Depends · R 50
#6 62 ↑2
Depends · R 40
#8 81
Depends · R 82
#8 83 =
Depends · R 85
#8 83 =
Depends · R 85
#8 85 =
Depends · R 90
#8 83 =
Depends · R 85
#8 83 =
Depends · R 84
#8 81 =
Depends · R 81
#8 80 =
Depends · R 80
Sense and Sensibility#6 79 ↑3
Depends · R 88
#5 75 ↑4
Depends · R 79
#7 59 ↑2
Depends · R 44
#9 74
Depends · R 76
#10 74 ↓1
Depends · R 76
#10 77 ↓1
Depends · R 82
#9 76 =
Depends · R 80
#9 75 =
Depends · R 78
#9 75 =
Depends · R 78
#9 75 =
Depends · R 79
#9 74 =
Depends · R 76
Whalefall#11 63 ↓1
Depends · R 58
#11 66 ↓1
Depends · R 65
#11 51 ↓1
Skip · R 33
#10 72
Depends · R 80
#9 75 ↑1
Depends · R 85
#9 77 ↑1
Depends · R 91
#10 75 =
Depends · R 85
#10 74 =
Depends · R 84
#10 70 =
Depends · R 75
#10 75 =
Depends · R 84
#10 74 =
Depends · R 83
A Talent for Murder#10 66 ↑1
Depends · R —
#10 66 ↑1
Depends · R —
#5 66 ↑6
Depends · R —
#11 66
Depends · R —
#11 66 =
Depends · R —
#11 66 =
Depends · R —
#11 66 =
Depends · R —
#11 66 =
Depends · R —
#11 66 =
Depends · R —
#11 66 =
Depends · R —
#11 66 =
Depends · R —
Other Mommy#15 42 ↓3
Skip · R 51
#13 35 ↓1
Skip · R 36
#13 31 ↓1
Skip · R 25
#12 44
Skip · R 56
#12 47 =
Skip · R 61
#12 48 =
Skip · R 64
#12 47 =
Skip · R 61
#12 42 =
Skip · R 51
#12 46 =
Skip · R 60
#12 47 =
Skip · R 62
#12 43 =
Skip · R 52
Verity#13 52 =
Skip · R 52
#14 35 ↓1
Skip · R 16
#12 31 ↑1
Skip · R 6
#13 36
Skip · R 18
#13 36 =
Skip · R 18
#13 36 =
Skip · R 18
#13 37 =
Skip · R 19
#13 36 =
Skip · R 18
#13 35 =
Skip · R 15
#13 38 =
Skip · R 21
#14 33 ↓1
Skip · R 11
Street Fighter#14 50 =
Skip · R 49
#12 40 ↑2
Skip · R 30
#14 28 =
Skip · R 4
#14 36
Skip · R 20
#14 34 =
Skip · R 16
#14 35 =
Skip · R 18
#14 35 =
Skip · R 18
#14 34 =
Skip · R 18
#14 34 =
Skip · R 16
#14 36 =
Skip · R 22
#13 34 ↑1
Skip · R 17
Klara and the Sun#12 52 ↑3
Skip · R 67
#15 28 =
Skip · R 27
#15 12 =
Skip · R 0
#15 25
Skip · R 22
#15 29 =
Skip · R 29
#15 27 =
Skip · R 26
#15 29 =
Skip · R 28
#15 28 =
Skip · R 27
#15 24 =
Skip · R 20
#15 28 =
Skip · R 28
#15 22 =
Skip · R 18
Per-pathway summary (labels are mine, single annotator)
Held-out acc / macro-F1 (n=112)38% / 0.3459% / 0.5853% / 0.4687% / 0.8684% / 0.8183% / 0.8180% / 0.7981% / 0.7980% / 0.8078% / 0.7780% / 0.79
Challenge acc (n=71)55%76%66%93%90%89%89%89%93%90%86%
Held-out vs siebert+opinion (McNemar p)+9/−32, p=0.0004baseline+18/−25, p=0.3604+37/−6, p=0.0+36/−8, p=0.0+37/−10, p=0.0001+33/−9, p=0.0003+32/−7, p=0.0001+30/−6, p=0.0001+31/−10, p=0.0015+29/−6, p=0.0001
API calls (all runs) / cached this runlocallocal1161 / 13811161 / 13811161 / 13811161 / 13811161 / 13811161 / 13811161 / 13811161 / 13811161 / 1381
Cost per full cold run$0 (local)$0 (local)$0 (local)$1.47$4.98$2.31$1.72$3.44$0.10$0.12$0.26
Latency per comment——0.9 s2.2 s3.0 s1.2 s3.4 s3.2 s1.8 s1.1 s0.9 s
Cold run, scoring time (serial)1 s6 min20 min51 min69 min28 min78 min73 min41 min25 min20 min
Stance prompt (shared by every LLM pathway, stance-v1)
You classify a Reddit comment from a movie's discussion thread by the commenter's VERDICT ON THE FILM ITSELF: did they like it / would they recommend it?
Judge the verdict, not the tone or vocabulary.
- Praise of dark, scary, violent, gross, sad or disturbing content is POSITIVE ("the villain is terrifying", "that scene was brutal, incredible", "I cried the whole time, beautiful").
- Hating a character, calling a character evil/a psycho, or describing a horrifying scene is NOT a negative verdict unless they criticise the film.
- Polite or soft disappointment is NEGATIVE ("I really wanted to like it", "it fell flat for me", "beautiful but I didn't care").
- Clear praise and clear criticism with no overall lean is MIXED. If they lean one way overall ("I liked it despite flaws"), pick that side.
- Plot discussion, theories, quotes, jokes, trivia, comments about the book/director/marketing without a judgement of this film, or not having seen it: NO_VERDICT.
Reply with JSON only: {"verdict": "positive" | "negative" | "mixed" | "no_verdict", "rationale": "<max 20 words>"}
NLI hypotheses
Challenge set by type (correct / n)
pathwaydark words, positive verdictpolite words, negative verdictdark description, no verdictmixed
VADER (old v1)28/3910/191/110/2
siebert+opinion (previous live)35/3917/192/110/2
Local NLI stance (deberta-v3-large)26/3915/195/111/2
GPT-6.1 Sol (LIVE)36/3918/1911/111/2
Claude Opus 5.536/3916/1910/112/2
Claude Sonnet 5.535/3915/1911/112/2
Gemini 4 Argon36/3915/1910/112/2
Grok 4.736/3916/1910/111/2
GPT-6 Luna (cheap)36/3918/1910/112/2
Claude Haiku 5.5 (cheap)36/3916/1910/112/2
Gemini 3.5 Flash-Lite (cheap)35/3914/1910/112/2

Data: compare_pathways.json · stance_eval.json (every labelled comment, every pathway) · challenge labels