One Bad Tuesday

Now fix it

What would have prevented this?

Add controls to the system. Each one changes the architecture. Controls should be proportional to consequence — a low-risk workflow can stay lightweight; a brief that shapes an investment decision cannot.

01

Freshness checks

Every source declares how current it is expected to be. If it is stale, do not silently use it.

  • · Last updated
  • · Last successful sync
  • · Expected update frequency

CRM

  • Last sync17 hours ago
  • Expected15 minutes
  • ActionFLAG OUTPUT
02

Retrieval coverage monitoring

Before generating a brief, check whether the expected source categories are actually available.

  • · Compare retrieved categories against an expected baseline
  • · Never proceed silently on partial coverage

Expected coverage

  • Founder notesPRESENT
  • Data roomPRESENT
  • Financial modelMISSING
  • ThesisPRESENT

Incomplete source coverage — generate a partial brief with warning, or require human confirmation.

03

Citation integrity

Do not only show a citation. Verify it.

  • · Source still exists
  • · Retrieved content supports the claim
  • · Source isn't stale
  • · User has permission

Verification chain

with control

CLAIM→SOURCE→SUPPORT VERIFIED
04

Regression evaluations

Before model changes reach production, test against representative tasks.

  • · Factual extraction
  • · Assumption separation
  • · Cross-company comparison
  • · Source citation
  • · Uncertainty behavior
  • · Financial data interpretation

Release path

instead of

MODEL UPDATE→PRODUCTION

with control

MODEL UPDATE→EVAL SUITE→PASS→PRODUCTION
05

Prompt change control

Every prompt edit gets an author, timestamp, change description, version, test results and a rollback path.

  • · Author
  • · Timestamp
  • · Change description
  • · Version
  • · Test results
  • · Rollback

Prompt v14

  • ChangedFriday 3:17 PM
  • Reason“Make outputs more decisive.”
  • Assumption-label accuracy94% → 71%
  • Suggested actionROLL BACK
06

Evidence thresholds

The system should not use model confidence as the sole reliability signal.

  • · Source count
  • · Source freshness
  • · Evidence agreement
  • · Coverage
  • · Known task type

When evidence is weak

  • OutputINSUFFICIENT EVIDENCE

Say “insufficient evidence” rather than guessing.

07

Escalation rules

Human review required when the system is operating outside the conditions it was trusted under.

  • · Important data is stale
  • · Expected sources are missing
  • · A novel request type arrives
  • · Sources disagree
  • · Financial claims lack support
  • · Confidence and evidence diverge

The lesson

  • NotHumans approve everything
  • ButHumans review the right things
08

AI observability

Monitoring software for AI behavior, not just infrastructure uptime.

  • · Data freshness
  • · Retrieval quality
  • · Model version
  • · Prompt version
  • · Error rate
  • · Evaluation score
  • · Escalation frequency
  • · Source availability
  • · Latency
  • · User corrections

Behavior dashboard

  • Data freshness
  • Retrieval quality
  • Model version
  • Prompt version
  • Error rate
  • Evaluation score
  • Escalation frequency
  • Source availability
  • Latency
  • User corrections
09

Who owns this system?

Ownership is the control that makes every other control fire.

  • · Nobody explicitly
  • · Operations
  • · Engineering
  • · Investment team
  • · Named cross-functional owner

Recommended

  • OwnerA clearly named owner with defined escalation paths

A workflow without an owner becomes everyone's problem when it fails.

Projected reliability

illustrative

63%

0 / 9 controls installed

Architecture

Founder notes
Data room
Investment memos
Market data
CRM
Ingestion
Search + retrieval
AI Diligence Associate
Partner brief
Replay Tuesday →Back to review