Catch fragile prompts before you ship or switch models

Hosho reads your prompt text the way teams review code: a score, named findings on exact lines, and concrete fixes you can accept or dismiss. Get that structural read before you ship a wording change or switch models - so you know why a case is fragile, not only that an eval failed later.

When to use MCP, Prompt Editor, or Git Bot

Same engine, three moments:

• MCP - while you draft in Cursor, Claude Code, or any MCP client. Ask for a Hosho review as you edit: score, line-anchored findings, rewritten weak sections, plus compare and skill-alignment tools when you need them.
• Prompt Editor - at prompts.hoshoai.com for a longer shared pass: score, fix, re-run, versions you can return to, and co-presence with your coding agent. Plan, usage, and billing live here too.
• Git Bot - a GitHub Action that comments on PRs touching matching prompt files: verdict, what got better or worse, suggested fixes with before/after snippets. Runs on GitHub's token - no Hosho GitHub App, no merge block. Humans still merge; your tests still run.

Pick the surface for the moment: MCP while drafting, the editor for a shared deep pass, Git Bot when a prompt change is about to land without either.

If a coding agent is doing setup, point it at the agent guide. Your human still mints a pfr_ API key at prompts.hoshoai.com (Google sign-in) - agents must not invent keys. The co-presence plugin already bundles MCP; installing both registers the same tools twice.

What to do before ship

Treat the prompt like code that can regress. Freeze a small set of real cases - high-stakes flows, known misses, awkward edges - with fixed inputs and a plain-language pass rule per case. Version that set with the prompt. Re-run it on every material edit, model or provider swap, or change to tools and skills the prompt depends on. A chat that worked once is not a test. A tight set you actually re-run beats a large set you ignore.

Then get a Hosho structural read on the prompt text itself - the score, line-anchored findings, and suggested fixes for gaps that make those cases fragile even when the model is capable: weak or missing output contracts, vague or overlapping tool descriptions, conflicting rules, overloaded scope, and phrases that read two ways. Format, tool choice, refusal edges, and task success fail for different reasons - do not collapse them into one vibe score.

Keep your eval harness and production tracing. You own the frozen set and the pass rules. Hosho's job is naming why the text is fragile. Evals prove whether outputs still pass on the frozen cases. Tracing shows what happened after ship. Structural review sits beside registries, Promptfoo, Braintrust, and similar - it does not replace them.

Related: The messy problem of prompting · So what matters in a prompt?

Example (coming)

A worked example - fragile prompt, Hosho findings, and the fix - will land here next. Until then we are not inventing customers or before/afters.

What this does not do

Hosho does not guarantee quality or promise that regressions will not happen. It does not run your tests or execute your eval suite. The Git Bot does not block merges unless you build that yourself. It does not replace evals, registries, or production monitoring, and it does not autonomously rewrite and ship prompts without you.

If a finding looks useful, apply it, then validate like any other change: same inputs, your harness, your judgment.

Sources

1. 1.Sclar et al., Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (FormatSpread), ICLR 2024. arXiv:2310.11324 ↑
2. 2.What Prompts Don't Say: Understanding and Managing Underspecified Prompts. arXiv:2505.13360 ↑
3. 3.Shi et al., Large Language Models Can Be Easily Distracted by Irrelevant Context, ICML 2023. arXiv:2302.00093 ↑
4. 4.ConInstruct: conflict detection and resolution under conflicting instructions. arXiv:2511.14342 ↑
5. 5.Liu et al., We're Afraid Language Models Aren't Modeling Ambiguity (AmbiEnt), EMNLP 2023. arXiv:2304.14399 ↑