Measure the work between agents

One agent updates a parser. Another implements the endpoint that consumes its output. Each produces plausible code, and most tests pass. The endpoint still expects the old return shape. Counting completed files makes the team look productive; checking the dependency reveals the work that remains.

If we want to know whether real-time shared state helps agents collaborate, we need an evaluation that can see this difference. Final task success matters, but it cannot explain where the collaboration worked or failed.

Make the connection an executable check

Start with a producer and a consumer that genuinely depend on one another. Write a check for the behavior they must preserve together. Decide that relationship before running the agents, so an attractive result cannot change what counts as collaboration after the fact.

AsynCodeBench makes this idea concrete with 19 tasks from 16 repositories and 52 directed dependencies. ADPR measures final dependency satisfaction. DRS records the first passing integration checkpoint, with a sentinel for dependencies that never pass. Its experiments find that stronger standalone coding performance does not uniformly translate into stronger collaboration.

First success is only part of the trajectory

A dependency can pass midway and fail after a later edit. The first passing checkpoint is useful, but the final state and the intervening trajectory still matter. For a shared workspace, record the sequence of relevant changes and run the same behavioral check against the retained states.

Checkpoint number is also different from elapsed time. One team may reach its second integration step in a minute; another may take twenty. The paper defines DRS over logical checkpoints rather than wall-clock time and notes that normalization sharpens some aggregate patterns. A product comparison should report time and cost alongside dependency progress.

Do not let the manager change the experiment

A manager that can directly repair production code has different authority from one that can only send feedback. If one system gives its manager write access and another does not, the result mixes coordination quality with who was allowed to solve the task. Keep those conditions explicit.

The same applies to model choice, tool access, retry budgets, and the amount of context available. A useful comparison can change one coordination mechanism while keeping the rest of the setup matched. Run several attempts where variability matters, and retain failed integrations rather than hiding them behind a final successful patch.

What we would measure for Ablo and Lobby

For Ablo, a focused experiment could introduce a deliberate stale dependency and measure whether a guarded write rejects it, what recomputation costs, and whether the invariant survives. For Lobby, we could compare a shared workspace with isolated work and later integration, using the same task, models, and checks. These are proposed evaluations; the cited benchmark does not establish an Ablo performance gain.

The practical target is work that composes correctly at a reasonable cost. Measuring dependency closure, regressions, elapsed time, and wasted retries gives a team a better basis for choosing its coordination design than agent count alone.