Benchmark: Python async race condition diagnosis

All three models correctly diagnosed the async cache race condition and offered valid fixes, but they differ in explanation depth and implementation trade-offs. Claude Sonnet 4.6 delivered the clearest, most comprehensive answer with the best cost efficiency.

Task

The following Python async function intermittently returns stale data. Identify the bug, explain why it happens, and provide a minimal corrected version.

import asyncio

cache = {}

async def get_or_fetch(key, fetcher):
    if key in cache:
        return cache[key]
    value = await fetcher(key)
    cache[key] = value
    return value

Answer format: three sections (“Bug”, “Why it happens”, “Fix” with code).

Results

Model	Latency (ms)	Input tokens	Output tokens	Cost (USD)	Verdict
Claude Haiku 4.5	4704	121	447	0.00236	Correct but incomplete error handling
Claude Sonnet 4.6	17925	121	839	0.01295	Complete, accurate, best trade-off
Claude Opus 4.7	10517	159	635	0.05001	Correct, concise, highest cost

Analysis

Claude Haiku 4.5 identified the race condition accurately and explained the suspension point clearly. The fix uses an asyncio.Lock with a double-check pattern: acquire the lock, check cache again, fetch if still missing, and write the result. This approach is correct for single-threaded async safety. However, Haiku does not address error handling; if the fetcher raises an exception, the lock is held during the exception path but the code does not show cleanup or whether failed fetches should be retried. The explanation is direct and concise, making it suitable for readers seeking a quick answer. The output token count (447) is the most economical but also the least thorough.

Claude Sonnet 4.6 provided the most comprehensive response. It explained the race condition with a detailed scenario walkthrough and emphasized that await is the critical suspension point. Crucially, Sonnet recognized the fundamental limitation of the lock approach: it serializes all calls, even for different keys, reducing concurrency. Instead, Sonnet proposed the in-flight task pattern, storing an asyncio.Task in a second dictionary and coalescing concurrent requests for the same key onto that single task. The answer included a comparison table highlighting four key properties: no duplicate fetches, no stale overwrites, error safety with explicit finally cleanup, and single-threaded safety. Sonnet also noted that thread-safety (if needed with ThreadPoolExecutor) would require an asyncio.Lock per key. At 839 output tokens, this response is more verbose but substantively richer and more actionable for production use.

Claude Opus 4.7 also correctly diagnosed the race condition and named it explicitly as a “cache stampede” or “missing single-flight” problem. The fix uses the in-flight pattern with asyncio.Future objects, nearly identical in substance to Sonnet’s approach but slightly more compact (635 output tokens). Opus correctly cleaned up failed fetches by popping the cache entry and setting the exception on the future, allowing retries. The explanation was accurate but less detailed than Sonnet’s; it did not provide a table or discuss concurrency properties. Opus also cost 0.05001 USD, approximately 4x higher than Sonnet, despite producing fewer output tokens, likely due to prompt pricing differences.

Winner and why

Claude Sonnet 4.6 is the clear winner for this task. It achieved the highest quality (correctness, comprehensiveness, pedagogical value) at a reasonable cost (0.01295 USD, roughly 5.5x cheaper than Opus). Sonnet correctly identified the in-flight task pattern as superior to per-key locking, explained the trade-off explicitly, and included a structured comparison table that clinches the reasoning. The detailed scenario and error-handling discussion make the answer immediately applicable to production code.

Haiku is the budget option: correct but incomplete on error handling and less prescriptive on concurrency trade-offs. Opus is the most premium option: technically sound and concise, but offers no new insight beyond Sonnet and costs significantly more. Sonnet hits the value sweet spot: depth, accuracy, and efficiency in balance.

Takeaways

1. In-flight task coalescing beats per-key locks for async cache design. Both Sonnet and Opus recognized that storing a shared Future/Task for in-flight requests is more efficient than acquiring a lock per key; lock-based approaches serialize all calls for a key, whereas in-flight tracking allows concurrent calls for different keys. Haiku’s lock solution is safe but leaves performance on the table.

2. Error handling must be explicit in async cache patterns. Sonnet and Opus both clear the in-flight entry on failure so retries can proceed; Haiku does not address exceptions. In production, failed fetches must not poison the cache indefinitely. This distinction separates toy solutions from production-ready code.

3. Cost per correct answer varies sharply, but not by token count alone. Sonnet delivered the most value (comprehensive, actionable, correct) at 0.01295 USD; Opus, despite producing fewer tokens, cost 3.9x more. For benchmarks, token economy is less important than explanation quality and technical depth per dollar spent.

4. Scenario walkthroughs and structured comparisons build confidence. Sonnet’s table and detailed timeline of execution order made the problem and solution crystal clear. Explanation format matters as much as correctness for developer trust and adoption.

Further reading

• asyncio.Task, Python standard library documentation provides the foundation for understanding Future and Task objects in async code.
• Cache stampede, Wikipedia explains the class of concurrency problem this code exhibits.
• asyncio.Lock, Python standard library documentation covers the locking primitive Haiku used as an alternative approach.