We built a deterministic air traffic control simulator to evaluate LLM agents over 90-minute shifts. Here we report a two-scenario pilot.
November 2026
Aircraft paths on scenario 1. Both models controlled the same 51 aircraft. We reconstructed the paths by replaying their logged radio transmissions. The small rings mark losses of separation, where aircraft came closer than the required minimum spacing: one for Opus 5.5 and eight for GPT-6.1 Sol. The concentric distance rings are 10 nautical miles apart; dashed shapes mark high terrain.
Approach control manages aircraft near an airport. In our simulator, the controller directs arrivals to a single runway and schedules departures between landings, while keeping aircraft safely separated. Instructions are sent by radio to simulated pilots. The controller can change an aircraft's heading, meaning its direction of flight, or its altitude and speed, and can send it into a holding stack to wait.
We built ApproachBench to measure how LLM agents manage this traffic over a full shift. The controller has to keep track of earlier instructions as new aircraft arrive. Keeping an arrival in a holding stack leaves room for other traffic, but also adds delay and consumes fuel. We score the resulting delay and safety events over the whole shift.
Arrivals and departures share the runway, and pilots share a radio frequency. The agent chooses when to request another observation of the traffic. Aircraft move between observations; the simulator pauses while the model generates its response, so model latency does not affect the outcome.
The simulator enforces a minimum spacing of 3 nautical miles horizontally or 1,000 feet vertically. A loss of separation occurs when a pair is closer than both limits. Departures cannot begin takeoff when an arrival is inside 2 nautical miles of the runway. Spacing also depends on aircraft size because heavier aircraft leave more wake turbulence, the disturbed air behind an aircraft. The grader checks simulator state against these rules.
At each turn, the agent receives a table of radar observations, radio calls since its last observation, and the runway state. It returns structured instructions such as KTA123 heading 180, descend 4,000, hold at LELEP or cleared approach, which authorizes an aircraft to begin its approach to the runway. It also requests a time for its next observation.
The observation and action loop. Model inference takes no simulated time. Radio transmissions take time under the same rules for every controller. The scripted reference controller uses the same loop as the models.
Each shift contains 90 minutes of traffic at one of three traffic levels. The middle level has about 50 aircraft; both pilot scenarios had 51. The simulator is deterministic, so an episode can be replayed exactly from its logged transmissions.
For this pilot, we ran Claude Opus 5.5 and GPT-6.1 Sol at high reasoning effort on the same two scenarios at the middle traffic level. This gives four episodes in total.
Opus 5.5 averaged 16.23 minutes of cost per aircraft, compared with 5.88 for the best-known plan. On scenario 0, the costs were 22.46 and 7.51 respectively; on scenario 1, they were 9.99 and 4.24.
Cost in minutes per aircraft; lower is better. Dots show the two scenarios and bars show their means.
Costs at the middle traffic level. Grey rows show the reference points. Doing nothing costs 862 on scenario 0 and 856 on scenario 1, averaging 859, beyond the plotted range. This two-scenario pilot is too small to establish a model ranking.
The scripted reference controller averaged 20.56 minutes per aircraft: 22.81 on scenario 0 and 18.30 on scenario 1. It uses hand-written rules to schedule landings, manage holding stacks and fit departures between arrivals. Opus had a similar cost on scenario 0 and a lower cost on scenario 1.
GPT-6.1 Sol averaged 26.54 minutes per aircraft, with costs of 29.89 on scenario 0 and 23.20 on scenario 1. Its cost exceeded the scripted reference controller's on both scenarios.
Delay costs were similar for both models, at 10 to 12 minutes per aircraft. Safety penalties accounted for much of the difference between their total costs.
Cost components for each episode. On scenario 1, Opus incurred 502 minutes of delay and 7 minutes of safety penalties across all 51 aircraft. Sol's combined safety penalties exceeded its delay cost on both scenarios. Unserved aircraft are those still awaiting completion when the episode ends.
Opus 5.5 served all 51 aircraft on scenario 1 and recorded one loss of separation, with 9% of the required margin lost. Across its shifts, arrivals were 15 to 20 minutes late on average. It made heavy use of the holding stacks: aircraft entered a hold 28 times on scenario 0 and 15 times on scenario 1.
GPT-6.1 Sol recorded 12 losses of separation on scenario 0 and 8 on scenario 1. Its worst event lost 88% of the required margin. It rarely held aircraft, with two hold entries on scenario 0 and none on scenario 1.
Scenario 0 was more costly for both models. Nine aircraft under Opus's control made a go-around, abandoning a landing attempt and rejoining the arrival queue. Four aircraft remained unserved at the end of the episode. Sol had two fuel emergencies in the same scenario after keeping aircraft waiting too long.
Observation frequency also differed. Opus took 453 turns on scenario 0 and 328 on scenario 1. Sol took 217 and 138 turns respectively. The plot below shows when each model received an observation on scenario 1.
Observations during scenario 1. Each tick marks an observation; the number gives the total. Shaded bands mark the two unannounced runway closures.
Exploit checks. On an earlier benchmark version, we scanned 219 episodes for exploit behavior and searched 1,390 reasoning summaries for discussion of the grader. Models did discuss the grading rules, but the audit found no reward hacking. For the shift version, we tested twelve scripted agents that attempted exploits such as withholding departures or holding aircraft indefinitely. None beat the reference controller on average. This audit also found a simulator loophole involving aircraft redirected during a go-around, which we fixed.
We first evaluated agents on fixed, curved arrival routes, then on radar vectors, where the controller assigns headings to guide aircraft toward the runway. Both versions used pass-or-fail scoring.
Share of episodes passed, using each model's best effort setting. Opus passed 30 of 30.
Episodes passed out of 6 at the two hardest traffic levels.
Results from the earlier designs. Opus passed every episode in version 1. In version 2, no model passed every episode, but six episodes per model were insufficient to distinguish the four models.
In version 1, Opus inferred a policy from the briefing: keep arrivals on their curved routes and release one every 100 seconds. This matched the reference controller's policy. Because the permitted delay was defined relative to that controller, the scoring gave Opus little reason to improve on it.
We also tried increasing traffic until the first safety failure. Models and the reference controller failed at the same time on the same aircraft. On two of 20 scenarios, the generator introduced traffic with conflicts that a search over possible maneuvers could not resolve. We abandoned this evaluation because those failures could not be attributed to the controller.
For the shift design, we replaced the pass threshold with a continuous cost and added the independent bound and planner. We also put the reference controller and models on the same observation and action loop; earlier versions had allowed the reference to act more frequently.
The pilot covers two scenarios and four episodes. It establishes that the evaluation runs end to end with models from two providers, but cannot support a reliable ranking. No human has worked a shift in the interface yet, and no rated controller has reviewed the airspace or rules. The relative price of safety events and delay is our choice; changing it can change model rankings. The best-known plan is not a proven optimum. The simulator uses a single runway, structured instructions instead of voice, and scripted pilots. Performance here does not establish competence at real air traffic control.
Inference cost also limits the size of the evaluation. Each Opus shift cost $13.72 to $24.24 in the pilot, substantially more than our estimate before running models on the shift design.
The proposed full protocol uses 20 scenarios, each run twice per model at the middle traffic level. Our power analysis estimates that this can detect a difference of about 1.5 minutes per aircraft at 80% power. Given the pilot's inference costs, we need to reduce context per turn or use fewer repeats before running it.
We also plan to collect human results from novices and rated controllers. The interface uses the same scenarios, observations, clock and grader as the model evaluation; we have no human results to report yet.