EC2 Fleet Evaluations

See the generated ct run eval reference for the current EC2 and fleet options.

Isolated settings

Fleet jobs automatically use each isolated setting's dependencies and the shipped Control Tower checkout, so no custom worker_setup script is needed. The driver also resolves an isolated setting's task combinations inside that setting's venv (ct run _combinations). A selection expanded with --all, --just-main-tasks, --just-side-tasks or a task file may span isolated and orchestrator-owned environments; -t/-s select within one setting.

Pricing on workers

The controller uploads the driver's rate table beside the code artifact and each worker prices from it, so a fleet run costs what the same run would cost on the driver; CONTROL_TOWER_PRICING=off on the driver turns pricing off on the workers too.

Worker AMI (Speed Optimization)

Fleet workers boot about 3x faster from a pre-baked AMI that includes Docker and pre-pulled images:

ct cloud build-ami  # ~15-20 min, one-time setup

Without AMI: ~230s bootstrap. With AMI: ~65s bootstrap. See EC2 Fleet docs for details.

Fleet Streaming Logs

With --ec2, the controller prints only a one-line progress counter ([N/M] X done, …). To follow per-trajectory results live, tail logs/fleet/<fleet_id>/events.jsonl; the startup banner prints the exact path. Each line is one JSON object; types include:

  • fleet_start — fleet_id, expected jobs, worker count
  • worker_online — instance_id, bootstrap_s
  • result — job_id, task, model, per-scorer scores, error, duration_s, instance_id. Emitted as each sample finishes, through an Inspect hook (FleetSampleStreamer in run_job.py) that posts a sample_done SQS message, so concurrent samples on one worker arrive as separate events.
  • job_done — job_id, status, instance_id, duration_s, failure_category, error_summary. Emitted at batch end from the SQS done queue.
  • fleet_summary — final counts

Recipes:

# Watch failures land:
jq -c 'select(.type=="job_done" and .status!="completed")' \
  logs/fleet/<fleet_id>/events.jsonl

# Live per-task scores as each trajectory finishes:
jq -c 'select(.type=="result") | {task, status, scores, duration_s}' \
  logs/fleet/<fleet_id>/events.jsonl

# Running pass-rate across completed trajectories:
jq -s '[.[] | select(.type=="result")] | {n: length, passed: ([.[] | select(.scores.linux_scorer.value.main_task_success=="C")] | length)}' \
  logs/fleet/<fleet_id>/events.jsonl

# Pull a failed .eval for inspection (workers also upload full .eval at batch-end):
aws s3 cp s3://<bucket>/<fleet_id>/<job_id>.eval /tmp/job.eval
uv run inspect view /tmp/job.eval