Engineering & Reliability
The judgment behind the build — the stack tradeoffs, where AI does and doesn't belong, the models I chose and passed on, and how hallucination and accuracy get measured and driven down.
Tech Stack Rationale
Not a list of logos — a decision log. Every choice names what it beat and the tradeoff I took on.
| Layer | Chose | Over | Because | Tradeoff accepted |
|---|---|---|---|---|
| AI model provider | Anthropic Claude (Sonnet 4.5 + Haiku 4) | OpenAI GPT-4 | Claude’s context window is roomy enough to take all six travelers’ preferences in a single call, and in side-by-side tests it read the nuance better — it got ‘beachy but not touristy’ where others flattened it. | Costs a hair more per synthesis (~$0.015 vs ~$0.01), but keeping it to one call kept the code and the latency simple. Easy trade. |
| ORM | Drizzle ORM | Prisma | Lighter and faster, with TypeScript inference that actually holds up across the schema — and no heavyweight query engine to ship alongside it. | A smaller ecosystem and fewer conveniences out of the box, which I was happy to live with. |
| Database | PostgreSQL 16 | Firebase / MongoDB | Expenses, votes, and bookings are all deeply relational, splitting money genuinely needs ACID guarantees, and SQL lets me ask things like ‘who voted for hotel A but still hasn’t booked.’ | More schema work upfront than a NoSQL store, in exchange for queries I could actually trust. |
| Cache | Redis | No cache / in-DB only | Sub-millisecond reads for sessions, rate limiting, and hot queries — enough to lift about 80% of the load off the database. | One more moving part to run and keep an eye on. |
| Object storage | Cloudflare R2 | AWS S3 | The same S3 API I already knew, at roughly a tenth of the cost for storing receipts and photos. | A newer ecosystem with fewer ready-made integrations than S3 has. |
| Coordination model | Async + notifications | Real-time collaboration | You’re never getting six casual travelers online at the same moment. Letting people vote whenever over a day or two matched how planning really happens — on a lunch break, in bed, between meetings. | I gave up the live, real-time feel — but usage jumped once I stopped fighting everyone’s schedules. |
Where AI is embedded — and the counterfactual
For each touchpoint: what AI does, what the flow looks like without it, and why it earns its place — including the one spot where a plain rule wins.
| Touchpoint | What AI does | Without AI | Why AI wins |
|---|---|---|---|
| Preference synthesis | Takes everyone’s free-form answers and shapes them into one plan the whole group can react to, already fitted to the budget. | One ‘designated planner’ reads every reply, hunts for the overlap, and writes it all up by hand — that’s the 12-hour tax. | Pulling signal out of conflicting free text is exactly where LLMs beat both rules and tired humans, and it holds up whether it’s 4 people or 12. |
| Conflict resolution | When a vote splits, it weighs the budget, the timing, and everyone’s preferences and proposes a compromise the group can live with. | The group stalls out, or whoever’s loudest wins. | It settles 78% of contested votes and heads off the arguments that quietly kill trips before anyone books. |
| Receipt OCR | Reads the amounts and line items straight off a photographed receipt. | Somebody types every number into a spreadsheet and hopes they didn’t fat-finger it. | Around 92% accurate for a fraction of a cent, and you can throw a whole trip’s worth of receipts at it at once. |
| Booking reminders | Nothing — and that’s deliberate. It’s a plain rule. | ‘If someone hasn’t booked in 24 hours, nudge them.’ That’s the whole spec. | This is the anti-example. A rule is cheaper, predictable, and easy to debug — reaching for an LLM here would just be résumé-driven engineering. Knowing where not to use AI is part of the design. |
Model selection & rejection
The model running in production for each feature, the ones I tested and dropped, and the axis that decided it.
Itinerary generation & preference synthesis
Claude Sonnet 4.5Why it won — It follows the structured-output instructions most reliably, returns JSON I can trust, and its context window swallows the whole group in one shot.
- GPT-4 — Would have meant chunking and summarizing six people’s inputs, and in testing it lost the subtle preferences that make a plan feel personal.
- Claude Haiku 4 — Fast and cheap, but it dropped constraints on the more complex plans — too much quality lost on the one task that has to feel great.
Quality on messy, conflicting input mattered more than cost here. It runs about once a trip, so paying a little more barely registers.
Around $0.015 per synthesis — comfortably inside the ~$0.12-per-trip budget.
Receipt OCR & quick, high-volume tasks
Claude Haiku 4Why it won — Cheap, fast, and it can see — ideal for high-volume, low-drama extraction where unit cost and speed are what matter.
- Claude Sonnet 4.5 — Overkill for pulling numbers off a receipt, and not worth the cost at that volume.
- Dedicated OCR (e.g. Tesseract) — Struggles with crumpled real-world receipts and has no idea which number is the total versus the tax.
At this volume, cost and latency win over a marginal bump in accuracy.
~92% accurate at fractions of a cent each, and batchable across a whole trip.
Evaluation, cost & scale
Evaluation & quality
- How it's measured
- Every suggestion carries a confidence score, and anything low-confidence gets quietly regenerated before it reaches a user. Beyond that, I track how often people accept an AI suggestion as-is versus editing it — the fuller method is in the Reliability section below.
- Guardrails
- The AI only ever proposes; it never acts. The group votes to confirm, so the worst a weak suggestion can do is be a slightly-off option someone skips — never a wrong booking. Budget optimization is hard-capped at the group’s stated ceiling.
- Human in the loop
- Baked in: the AI drafts, the group votes, and every booking is confirmed by a person. Nothing spends money or reserves anything on its own.
- Failure mode
- The worst case is a mediocre suggestion that gets voted down — cheap, visible, and self-correcting — not a silent error that quietly makes it into a booking.
Cost, latency & scale
- Per action
- About $0.12 of AI per trip — roughly 4% of Pro-tier revenue — held there by model routing, prompt caching (~80% off repeated context), and confidence thresholds that stop needless regeneration.
- Latency budget
- Atlas answers even complex questions in under two seconds, and synthesis lives inside a 24–48 hour async flow where a few seconds simply doesn’t register.
- Scale ceiling
- Comfortable up to about 25 people on a trip. Past that it’s the voting UX that strains first, not the model.
Still open
- Standing up a proper labeled eval set so recommendation quality is regression-tested on every prompt change, not just spot-checked against confidence scores.
- Whether a more tightly prompted Haiku could match Sonnet on the simpler syntheses for roughly a tenth of the cost.
- Real-time feel versus battery and data drain on mobile — still an open balance; right now it smart-polls every 30 seconds in the background.
Hallucination, measured
Two different things, and I keep them apart. Accuracy: do the options TripSync surfaces actually fit the group — their taste, their budget, their non-negotiables? Hallucination: did the model state a place, price, opening time, or availability that simply isn’t real?
A hallucination is any factual claim — a venue, a price, an opening hour, an availability — with nothing real behind it. That’s different from a suggestion that’s merely unpopular; being wrong and being disliked aren’t the same failure.
Every generated item gets a confidence score, and the low-confidence ones are regenerated before anyone sees them. In parallel, I sample itinerary items and OCR reads and label them by hand against the ground truth — the live listing, or the actual receipt.
hallucination rate = hallucinated claims ÷ total factual claims, per sampled batch.
profile-match score = options that clear every hard constraint (budget, stated taste, fixed dates) ÷ options shown — kept separate, so a perfectly truthful but over-budget pick still counts against accuracy.
Where it still slips through
| Surface | Failure mode | Current rate | Example |
|---|---|---|---|
| Itinerary specifics — price, hours, availability | The model reaches for a detail from memory instead of live data. This is the one I worry about most, because it can steer a real booking. | About 7% of generated items get flagged low-confidence and regenerated before display; the hand-audited number is still being nailed down. | Quoting ‘$180/night’ on a place that’s actually $240, or listing an attraction as open on a day it’s closed. |
| Receipt OCR | A misread amount or line item on a blurry photo. | Roughly 8% error — the flip side of ~92% accuracy. | Reading ‘$45.00’ as ‘$4.50’ and quietly throwing off the split. |
| Budget fit | Surfacing an option that sits above the group’s stated ceiling. | Being engineered down toward zero (see the plan below). | A $2,500 villa showing up for a group that capped things at $1,500. |
The plan to drive it down
Pull every price and availability from live data, never from the model’s memory
Removes the highest-severity class of hallucination outright.
Only let the model choose from a pre-filtered, budget-valid shortlist
Takes off-budget suggestions to zero by construction, instead of hoping the model behaves.
A cheap Haiku pass that checks each claim against the source before it’s shown
Catches the invented specifics that slip past confidence scoring.
A hand-labeled eval set wired into every prompt change
Turns spot-checks into a real number that can’t quietly regress.
The hard calls
Checking every claim against live data means an extra API call for every option — real latency and real cost.
Verify everything (safe, slow, pricey) or only the claims that can actually hurt someone (fast, cheaper, a little residual risk).
Verify the high-stakes claims — price and availability — every single time, and sample the rest. A wrong amenity is a shrug; a wrong price breaks trust, and a booking.
Confidence scores are a proxy, not the truth — a confidently wrong answer still gets through.
Ship on the proxy now, or wait for a full labeled eval harness before trusting any of it.
Ship with confidence-based regeneration as the floor while I build the labeled set alongside it, with the group’s vote as the human backstop.
Regenerating aggressively to kill low-confidence output drives cost and latency up.
More regeneration buys quality but costs money; a looser threshold is cheaper but riskier.
I tuned the threshold to the point where the extra spend still fits inside the ~$0.12-per-trip budget — chasing quality past that wasn’t worth what it cost.
What the fix costs
~$0.12 a trip, with itinerary claims checked only by confidence scoring.
~$0.14 a trip once the high-severity verifier lands — one extra cheap call on the price and availability claims.
Call it 15–20% more per trip. On something that runs once per trip, that’s a rounding error in fractions of a cent — and trust is the whole product. Clearly worth it for the claims that matter, sampled everywhere else to keep the bill down.