Apollo-1 One-Shots 83% of 111 Live Google-Flights Scenarios — A Controllable AI Showcase

Introduction

Generative AI excels when acting on behalf of a user, but it falters when an agent must act for an entity—whether it’s an insurance company, a brand, or a travel platform. That’s where controllable AI becomes essential, giving companies the levers to keep an agent aligned with their policies, guardrails, and desired behaviour. To demonstrate, we wired Apollo-1 into the same real-time Google Flights feed powering Gemini 2.5-Flash and ran 111 multi-turn conversations across both models.

Apollo-1 is engineered to operate on behalf of an entity, not an individual. Its neuro-symbolic core merges generative fluency with deterministic policy enforcement, giving organisations the control, traceability, and reliability that conventional LLM co-pilots lack. The head-to-head below shows how that design translates into real-world wins over Gemini when every query hits identical inventory, prices, and schedules.

Most leaderboards cherry-pick the best answer from a batch of generations. We do the opposite. Every prompt gets one shot—exactly what a live user would see—and that single response is what we score.

We reuse τ-Bench’s philosophy in miniature: no retries, binary checks that mirror real customer journeys. And because there’s no second chance, the outcome is a single, unforgiving line in the sand—a simple, reproducible, brutally honest snapshot of how an agent really behaves when someone is on the other side of the chat box.

Results in a Snapshot

How Controllable AI Keeps Agents on the Rails

For almost eight years we built toward one goal: agents that act on behalf of an entity, not a user.

• Neuro-symbolic foundation model — a Symbolic Reasoner replaces the transformer as the model’s decision-making core, enabling controllable, context-aware interactions every turn.
• Deterministic control on demand — symbolic guardrails guarantee brand and regulatory compliance before the answer leaves the model, while still leveraging generative fluency.

Real-time Control Panel — teams inspect reasoning traces, tweak context schemas, inject policies, and replay trajectories; continuous fine-tuning at the sub-interaction level instead of retraining the whole stack.

This is how an agent can operate on behalf of an airline, a bank, or a government office – not just with a user – and still sound natural.

Example Scenario

Scoring criteria — Scenario 28 “Same-day One-Way, Arrive Before Lunch” Scenario

Scoring: binary (1 = pass, 0 = fail) per step. Scenario score per run = R-1 + R-2 + R-3 (max 3).

Scenario Reward Logs

Benchmark Overview & Group Analysis 

The 111 scenarios are grouped around five distinct technical competencies that an enterprise-grade flight agent must master. Each group isolates one capability, so a perfect model would score 100 % inside every row:

Apollo-1 Strengths Across the Five Competency Buckets

• Context-aware dialogue control – in Core Search scenarios Apollo-1 clarifies missing slots (dates, O/W vs R/T, party size) before searching, then keeps that context intact through follow-up constraints.
  
• High-fidelity data retrieval – consistently surfaces the exact fare family, bag fee, upgrade delta or aircraft type demanded in Ancillary Costs and Cabin Precision scenarios.
  
• Multi-constraint filtering – satisfies arrival-before / depart-after windows, hub-routing, Wi-Fi-only, airline-specific and environmental (CO₂) filters in a single pass, dominating the Constrained Planning bucket.
• Complete itineraries – returns full outbound + return legs where required, a failure point that still plagues Gemini in every bucket except Baseline Retrieval.

Areas that Still Need Work

• True multi-city pricing flows – both models stumble on leg-by-leg bag or upgrade math in scenarios that chain three or more segments.
• Long-tail ancillaries – niche items such as carrier-published CO₂ offset add-ons or exotic pet-in-cabin fees remain patchy (Gemini far more so).

Gemini’s Persistent Gaps

• Fails slot clarification: often assumes round-trip, ignores supplied date windows, or answers with only the outbound leg.
• Zero full-conversation passes in Ancillary Costs and Cabin Precision buckets.

Bottom Line

Generative AI is fine for user chat; Controllable AI is mandatory when an agent operates on behalf of a business or an entity. Across 111 one-shot scenarios Apollo-1 delivers a full-conversation pass on 92 (82.9 %); Gemini 2.5-Flash manages 24 (21.6 %), proving the need for neuro-symbolic control. The advantage grows as queries add pricing arithmetic or routing constraints, signalling Apollo-1’s readiness for production settings where predictable, auditable behaviour is mandatory.

Appendix A: Reward Logs

View Full Scenario List

Appendix B : Trajectories

View Full Trajectories