Conversational Extraction: Which LLMs actually capture structured data from real dialogue?

The wedding planner: one case that separates the top four models from everyone else

Before the methodology, look at the single most discriminative case in the benchmark. Priya books a wedding-planner vision meeting and attends alone (her fiancé cannot make it), but the conversation contains a separate "40" referencing estimated wedding guest count. The ground truth for party_size is 1. Only 4 of 11 models consistently get this right — gpt-5.4, gpt-4o, gpt-5-reasoning, and gemini-2.5-pro (2/3). The other 7 latch onto the 40-guest distractor. What separates the four that solve it is not model size or reasoning — it is context-carrying disambiguation: remembering "just me for this meeting" across 20 turns of realistic conversation. case-10 is still a better differentiator than every intentional multi_ref case combined.

A universal service_type bias

All 11 models share an identical signature failure: over-prediction of booking, under-prediction of general_inquiry and implicit follow_up. When customer intent is ambiguous, every model reaches for booking — the most common business-action label in service-business training data. The bias is structural, not stochastic: even gpt-5.4 drops 2.3 pp on service_type while hitting 100% on customer_name and party_size. Model capability shifts severity (gpt-5.4 at 97.7% → mistral-7b at 77.3%, a 20.4 pp spread) but never direction.

The v1.0 implicit headline was single-case noise

v1.0 reported a 25 pp gap on implicit inference as its headline finding. That gap was measured across 2 cases. In v1.1 we expanded implicit to 8 cases; the gap compressed to 6.3 pp (gpt-5.4 / gpt-4o / gpt-5.4-mini tied at 96.9% top; three models tied at 90.6% floor). The real, statistically durable capability signal is long-context retention (16.7 pp, n=6), not implicit inference. This is a vindication of statistical-robustness standards over dramatic but fragile findings.

Reasoning is overhead, not an asset

gpt-5-reasoning and o4-mini are both dominated by chat counterparts. gpt-5-reasoning burns roughly 306 reasoning tokens per call for a 2.3 pp accuracy penalty relative to the same-generation chat model (gpt-5.4), 3.5× slower at p50, and — with real pricing now locked — strictly Pareto-dominated by gemini-2.5-pro. o4-mini sits on the frontier but is narrowly beaten by gemini-2.5-pro on accuracy for only a 4% cost premium. Interpretation: conversational extraction is pattern-matching where the answer is usually visible in the input. There is nothing to reason about — the customer says their name, you extract their name.

gemini-2.5-pro is legitimately competitive

On its first appearance in the benchmark, Gemini 2.5 Pro lands at #3 overall (97.9%), effectively tied with gpt-4o (98.3%) and only 1.0 pp behind gpt-5.4. It is the only model besides gpt-5.4, gpt-4o, and gpt-5.4-mini to crack 95% on implicit inference. Its weakness is long (90.3%) — still stronger than all budget chat models and OSS models, weaker than the three frontier chat models above it. Latency is on the high side (p50 4,985 ms) but cost ($2.00 / 1,000 calls) puts it mid-frontier. For buyers needing vendor diversity away from OpenAI, Gemini 2.5 Pro is a credible drop-in for gpt-4o-class extraction.

Two categories produce zero signal

Both self_correction (11-way tie at 100%, 4 cases) and deep_ambiguity (8-way tie at 100%, 8 cases, 3.1 pp total range) fail to discriminate the roster. The 8 deep_ambiguity cases authored for v1.1 — oscillating party sizes, conditional dates, keyword-vs-intent mismatches, nested self-corrections — were designed to test whether reasoning models ever justify their overhead. Most models solve these cleanly. Both categories should be retired in v1.2 and the case budget redirected to more long and implicit variants, which is where real capability gaps surface.

Surprise: case-35 breaks most frontier models

case-35 (implicit date: "the Monday after Memorial Day weekend") requires resolving a US holiday + applying the "after" rule to a specific weekend. 9 of 11 models fail it consistently, including gpt-5.4, gpt-5-reasoning, gemini-2.5-pro, o4-mini, and all three OSS models. Only gpt-5.4-mini and gpt-4o solved it. The failure is not calendar arithmetic; it's that models extract "Memorial Day" literally, or the "Monday" keyword matches a wrong anchor. Implicit temporal inference — even with a pinned reference date — remains brittle across model tiers.

Agreement with prior art

Qualitative ranking here (gpt-5.4 > gpt-4o ≈ gemini-2.5-pro > budget > OSS) tracks LLMStructBench's composite-score ordering for overlapping models on static extraction. Absolute numbers are higher than MultiWOZ/SGD Joint Goal Accuracy (60–70% for frontier models), reflecting task-shape differences — 4 fields versus slot ontologies, whole-dialogue versus turn-incremental — not model improvements. ChatExtract's finding that "follow-up questions improve accuracy" is consistent with our observation that models fail on implicit + long-context cases where a follow-up question in a real system would disambiguate.