Change the model or fix the prompt? An LLM selection evaluation on 200 production records

The first question in a pipeline that extracts structured data from text is often, “Which model should we use?” We started there too. An influencer-data service had accumulated profile bios as strings, and we needed to extract activity categories, contact channels, and relationships between accounts.

The candidates were Amazon Nova 2 Lite, Claude Haiku 4.5, and Claude Sonnet 5. The answer turned out to be closer to prompt design than model selection. This article describes the evaluation and the measurements from August 5, 2026. The source database and raw results contain personal information, so we publish only aggregate numbers and anonymized examples.

Start with the most expensive mistake

We priced the consequences of each wrong decision before designing the evaluation.

• A wrong activity category makes a discovery filter less precise. It is easy to repair.
• A wrong SAME_AS relationship merges two accounts into one person. It contaminates follower totals, performance attribution, and settlement, and is difficult to reverse.

We therefore measured agreement on relationship-type decisions. It was the hardest decision and the most expensive one to get wrong. The metric must come before the model comparison. The same principle guided our RAG regression evaluation: an evaluation set is a decision contract, not a list of questions.

Define six relationships before asking the model

Account mentions in bios contained at least five distinct relationships. We added an explicit “unknown” outcome and fixed the result to six enum values. Each value triggers a different system action. The taxonomy is not merely a classification scheme; it controls downstream behavior.

Two boundaries matter most. Only SAME_AS can become a merge, and OPERATES never contributes followers to the parent account. If a magazine operates separate beauty and music accounts, adding all three as one audience inflates reach threefold.

MENTIONS is an intentional result. A classifier that cannot say “unknown” must force every mention into another type. Our first prompt did exactly that.

The task also requires context. In one bio, two handles appeared in the same grammatical position. One was the person's second account (SAME_AS); the other was their gym (AFFILIATED_WITH). Handle similarity alone could not separate them.

Use a proxy baseline when no labeled answer set exists

We did not have a human-labeled answer set. We used the strongest candidate, Sonnet 5, as an answer proxy and measured how closely each remaining configuration matched it.

The percentage is agreement with Sonnet 5, not accuracy. Manual review found Sonnet's decisions more consistent and defensible in disagreements, but it cannot replace labeled truth. We therefore avoid the word accuracy in the result table and chart.

Three implementation differences blocked the first run

Changing the model name was not enough to run the same experiment.

The v1:0 suffix in the model ID is an AWS-managed model revision. P1, P2, and P3 below describe our prompt stages: P1 is the initial prompt, P2 adds a decision tree and rules derived from observed errors, and P3 adds explicit input boundaries and rules for the remaining errors.

We used @aws-sdk/client-bedrock-runtime to call both Nova and Claude from the same runtime. An Anthropic-only SDK cannot invoke Nova. Without these adjustments, the experiment compares request and parser failures instead of model quality.

Prompt revision beat the model change

Moving from Nova 2 Lite + P1 to Claude Haiku 4.5 + P1 increased agreement from 50% to 64% and cost about 2.98 times as much. Revising the same Nova 2 Lite model from P1 to P3 reached 87% at about 1.36 times the cost. If model capability had been the bottleneck, Haiku should have led Nova by a wide margin. The bottleneck in this experiment was ambiguous decision criteria.

Inspect the decision distribution, not only the aggregate

Aggregate agreement did not reveal each model's error pattern. The distribution of predicted relationship types did.

Nova 2 Lite + P1 produced 19 COLLABORATES decisions and no MENTIONS decisions. It pushed weakly supported mentions into collaboration. The most dangerous type, SAME_AS, was also overproduced at 33 decisions. Manual review found 20 false merge candidates. In P3, the six OPERATES and one BRAND_OF decisions matched Sonnet exactly.

Do not trust confidence by default

Confidence is a self-reported value from zero to one returned with each decision. We store it beside the result and always preserve the source fragment (evidenceText) that supports the decision. A reviewer cannot judge the number without its evidence.

The initial confidence values were unusable for triage.

More than half of P1's decisions claimed maximum confidence. Sending confidence < 0.7 to a review queue would miss too many risky decisions. We added an anchor contract to the prompt: “Use 1.0 only when the relationship is stated literally in the source.” The share at 1.0 fell to 14%. Confidence was another output that needed a prompt-level definition.

We also chose not to display decimals such as 0.87 as if they were calibrated probabilities. There is no evidence that 0.87 has meaningfully more resolution than 0.85. The database stores a continuous value to three decimal places, but operations uses only a 0.7 review threshold and shows a band label with evidenceText. Confidence and confirmation are separate axes: the model owns confidence; a person moves status from PENDING to CONFIRMED or REJECTED. Re-inference never overwrites a human decision.

Turn observed errors into prompt rules

We counted 22 P1 errors by pattern and added one rule for each recurring class.

We fixed the order with a decision tree: test for personal ownership, a separate topical account, a brand or institution, a personal relationship, and finally evidence of actual joint work. A negative rule allowed COLLABORATES only when the source stated evidence that the parties created something together.

P3 also fixed an input-boundary bug. Nova extracted the profile owner's handle from the prompt header as if it appeared in the bio in 12 cases. Separating [metadata] from [bio text] reduced that error to two cases. Prompt improvement includes the structure of the input, not just better wording.

The anonymized examples show how the decisions changed.

"Beauty brand CEO. Maintained an 18 kg loss for three years @brand_official"
   P1: SAME_AS   ← merges a personal account and a brand account
   P3: BRAND_OF  ← records a relationship without merging

"Fashion brand ambassador, cohort 2 @fashion_kr"
   P1: COLLABORATES     P3: AFFILIATED_WITH

"Wife of actor A @actor_a"
   P1: COLLABORATES     P3: MENTIONS

The skeleton of the final prompt (P3)

The original prompt file is no longer available. The following excerpt reconstructs only the decision logic documented in the evaluation report.

[metadata]
Profile owner: @{handle}
(This handle is not a decision target. Evaluate only handles mentioned in the bio text.)

[bio text]
{cleaned bio text}

For each mentioned @handle, test in this order:
1. Is it explicitly the person's own secondary account?             → SAME_AS
2. Is it a separate topical account named as an "X account"?        → OPERATES
3. Is it a brand named with CEO, founder, or owner language?         → BRAND_OF
4. Is it a brand, institution, agency, ambassador, or gym?           → AFFILIATED_WITH
5. Is it a spouse, family member, or personal acquaintance?          → MENTIONS
6. Does the source literally establish that they created work?      → COLLABORATES
7. If no relationship has textual support                           → MENTIONS

Negative rules:
- Use COLLABORATES only when joint work has explicit textual evidence.
- Treat an unqualified brand handle as affiliation or sponsorship, not collaboration.
- Treat curation handles such as "_official" or "_pick" as OPERATES, not SAME_AS.
- A link-hub URL is not an account; classify it as contact channel type link_hub.

Confidence:
- Use 1.0 only when the relationship is stated literally in the source.

The decision tree reduced discretion between types. Negative rules moved weakly supported decisions toward the conservative MENTIONS result. Separating [metadata] and [bio text] made the input boundary explicit.

Select the configuration that fits the operating boundary

We selected Amazon Nova 2 Lite + P3.

The costs are USD estimates calculated from the displayed mean input and output tokens and the model prices recorded on August 5, 2026.³⁴ Because the mean token counts are rounded to whole numbers, the second decimal place may contain a small error. The Sonnet 5 estimate uses the introductory price available through August 31, 2026. We record the model, endpoint type, input and output tokens, applied rates, and calculation date.

The lightweight model does not own every decision. We split responsibility according to the asymmetric cost of errors.

Controls outside model evaluation keep production safe

The evaluation result does not complete the production design. Four controls came directly from the experiment.

1. Post-validate the schema. Four percent of P1 results left the allowed vocabulary, including tone values placed in a category field. Discard values outside the schema.
2. Record inferenceVersion. A prompt revision increments the version and defines which existing records need reprocessing.
3. Mark truncated input. Eighteen percent of the sample ended at a “more” expansion. Lower confidence or collect the complete text before deciding.
4. Keep a human review workflow. A person confirms contact data, affiliations, and account merges whose errors carry high cost.

The remaining 13% of disagreements concentrated in context-free handles and multilingual bios that Sonnet also treated with uncertainty. Adding more prompt rules would risk overfitting unusual cases. Sending the result and its evidence to a person was the safer boundary. This decision connects to the gates for moving an AI PoC into production.

Choose the evaluation contract before the model

“Which model should we use?” is not the first question. Ask which mistake costs the most, what serves as a baseline when labels do not exist, and how observed errors become rules. Once those answers are explicit, the model can be selected within cost, latency, and operational constraints. In this experiment, the least expensive model, Nova 2 Lite, met that contract when paired with P3.

To evaluate model selection and extraction pipelines on production data, start with Data & ML engineering and define the evidence, error boundaries, and review workflow together.

References