Case 04 · Developer tool · AI · Open source · 2026 · 6 min read

Ensemble: an AI lead that directs cheaper AI workers

Problem statement

A lead model plans and reviews. Lower-cost models build in parallel.

Problem

Routine coding uses expensive reasoning

Developers use strong AI models for both difficult decisions and routine implementation. Independent tasks often wait for one model to finish the previous task.

Issue

Delegation has its own overhead

Parallel workers can save money, but repeated context, unclear ownership and coordination add cost. The comparison must include planning, review and the same acceptance criteria.

Solution

One lead, focused parallel workers

Ensemble gives a lead model responsibility for planning and review, while lower-cost workers implement bounded tasks in parallel. Shared rules and file ownership keep the work coordinated.

Progress Small benchmark: 13.9% lower cost, 18.1% longer runtime.

Role: Product owner and builder; directed AI coding agents and reviewed the work
Type: Open-source developer tool (MIT, version 0.1.0)
Stack: Python standard library; works through MCP with compatible model providers
Code: github.com/VeeraPD/ensemble-mcp
Skills: Workflow design, evaluation, cost against quality

Origin

My app-building experience prompted a test of model delegation

While building an iOS app with AI coding agents, I let a strong model plan and review and handed routine pieces to a cheaper model. My API costs fell by more than 50%, and the parallel work finished faster than my solo runs.

That is my own observation, not an audited result: I have no invoices or matched baseline. But it raised a question worth testing. When does splitting the work among models pay for the coordination it costs?

Problem

Routine implementation consumes expensive reasoning and repeated context

Routine implementation uses the same expensive model as planning.
Long conversation histories add cost to narrow tasks.
Independent tasks wait instead of running in parallel.

My audience hypothesis: solo builders and small teams on an API budget, who may later want cost attribution and limits on what an agent can change. There are no customers or revenue to report.

The idea

A strong lead plans while lower-cost workers build in parallel

How Ensemble splits a coding task.

Diagram guide

Read left to right. The lead on the left sets the rules and hands each worker a focused brief. The three workers in the dashed box build in parallel, and the tool refuses any write outside a worker's own files. The lead on the right reviews and combines the result. In the benchmark the three parts were the data model, the command line and the tests.

A strong lead plans and reviews. Lower-cost workers handle bounded tasks in parallel, using shared rules and explicit file ownership.

Goals and metrics

Compare total cost only when both approaches pass the same tests

North StarCost ratio per accepted taskTeam all-in cost (lead, workers and repairs) divided by solo all-in cost, both on work that passes the same frozen acceptance tests. Below 0.8 means at least a 20% saving.

Time ratioTeam wall time divided by solo, verification included
First-pass acceptanceWorker outputs the lead accepts unchanged
Rejected-write rateWrites refused by the ownership check

Why this number: cheap output that fails review is not a saving, and a saving on code that fails the tests is not a saving either. Fixing the tests first makes the ratio comparable. Rule: claim a saving only if the ratio is 0.8 or lower across at least three matched pairs.

Decisions and trade-offs

Explicit task boundaries keep parallel work coordinated

01One lead, cheaper workers

A strong model plans and reviews; cheaper models implement

Why: It matches capability and cost to the job. The lead's context, review and repairs can dominate spend, as the results show.

Trade-off: The simplicity of one model

02File ownership enforced in code

Each worker may write only its assigned files, and the tool checks

Why: A boundary I can inspect beats a model's promise. It controls where writes go, not whether the code inside is right.

Trade-off: Letting workers roam

03The lead owns the shared rules

Exact interfaces and import paths before any work starts

Why: An early experiment failed when a worker guessed an import path the brief had not named.

Trade-off: Less preparation

04A small surface

Delegation, ownership, review and audit logs only

Why: Keeping the tool narrow keeps it reviewable. It never runs model-returned code: the lead runs the checks.

Trade-off: Multi-model debate and voting, extra channels

Tracking

Per-call usage records include the cost of planning and review

Metric	Definition	Data source	Status
Cost ratio (North Star)	Team all-in cost / solo all-in cost on the same accepted task	Per-request usage records and the acceptance result	Cost recorded in the small study; ratio on a frozen suite is next
Time ratio	Team wall time / solo wall time	Harness timer	Measured in the small study
Lead share of spend	Lead cost / team cost	Usage records by model	Measured: about 88%
First-pass acceptance	Worker outputs accepted unchanged / all outputs	Lead review log	To instrument
Rejected-write rate	Refused writes / write attempts	Tool audit log	To instrument
Repair share	Repair cost / total cost	Usage records tagged by phase	To instrument

Only cost, time and the lead's share of spend are measured so far. The rest are proposed instrumentation, and I will say so until they are logged.

Evidence

The small benchmark cost 13.9% less but took 18.1% longer

I ran Claude Sonnet 4.5 on the same small to-do app, three solo runs and three with a lead plus workers, in alternating order.

Same small to-do app	Solo	Lead plus workers
Mean API cost	$0.245	$0.211 (13.9% lower)
Cost range	$0.240 to $0.254	$0.195 to $0.237
Mean time	103 s	122 s (18.1% longer)
Share of team cost from the lead	Not applicable	About 88%

I set the rule before running: at least 20% lower cost with ranges that do not overlap. The saving is under 20% and the ranges overlap, so the verdict is no clear cost saving at this size. This is why the case is about counting the lead: my first worker-only story overstated the saving. All six runs passed their own tests, but the suites differ, so that is not a common quality check.

What each number can support

The benchmark is evidence for one task, not a universal saving

Claim	What it rests on
More than 50% lower API cost on my iOS app	My own observation. No invoices or matched baseline
13.9% lower cost, 18.1% slower	Measured: 3 solo and 3 team runs on one small app, lead included
12.5% lower team cost with Opus 5.5	One interrupted pair of runs. Not a clean comparison
Worker execution about 3.6 times faster	One earlier trial, workers only, a different task. Raw logs not found
Bigger gains on larger, well-divided projects	A hypothesis. Untested

The 50% figure is not a result of the six-run study, and the 3.6 times figure is not delivery speed. Mixing them up is the easiest way to overclaim.

What I learned

Delegation needs clear handoffs and a frozen acceptance test

Make task boundaries and handoffs explicit.
Freeze acceptance tests before comparing runs.
Include planning and review in total cost.

A larger matched benchmark will test where coordination pays off

The next experiment is a larger task with clearly independent parts. I will freeze the acceptance suite before any code is generated, run three matched pairs with the same model, tool version and permissions, and count review and repair in cost and time. Team-level cost attribution and policies come later, only if demand shows up.

The tool is open source under the MIT licence. Code and issues: github.com/VeeraPD/ensemble-mcp.