Designing async pipelines for Claude.

Most teams meter their entire Claude bill at the real time rate because that is how the first integration was built. A user types, the application calls the model, the answer comes back while they wait, and every workload after that gets bolted onto the same synchronous path out of habit. The result is that a great deal of work that has no real time requirement at all is paying the real time price. The Batch API exists precisely for that work, and it returns the result at half the per token cost in exchange for relaxing the deadline from seconds to up to a day. Designing an asynchronous pipeline is the discipline of finding the work that can tolerate that delay and moving it off the expensive path. This piece sets out how to identify async candidates and how to build a pipeline that captures the fifty percent saving cleanly.

The core trade you are making

Batch processing on Claude is a simple bargain. You submit a set of requests as a job rather than one at a time, you accept that results arrive within a completion window rather than immediately, and in return you pay half the standard input and output token rate. Nothing about the model changes. You get the same Opus, Sonnet, or Haiku output you would get synchronously, at the same quality, for half the money. The only thing you give up is the guarantee of an instant response. So the entire design question reduces to one thing: which of your workloads genuinely need the instant response, and which only appear to because they were built that way. Almost every enterprise has more of the second kind than it realizes.

How to recognize an async candidate

The test for whether a workload belongs in an async pipeline is whether anyone is actually waiting on the result. If a human is sitting in front of a screen expecting an answer to appear, the work is real time and stays on the synchronous path. If the result feeds a report, a queue, a downstream system, or a scheduled process, no one is waiting and the work is a batch candidate. The following patterns are almost always asynchronous in nature even when they have been built synchronously.

• Bulk document processing. Summarizing, classifying, extracting, or tagging a corpus of documents is a job, not a conversation. It runs against a set, it produces a result set, and the deadline is when the downstream consumer needs it, which is rarely the next second.
• Overnight and scheduled work. Anything that runs on a cron, a nightly job, or a periodic refresh has a deadline measured in hours. That is the natural home of batch.
• Backfills and reprocessing. When you reprocess historical data after a prompt change or a model upgrade, the volume is large and the urgency is low. Running it as batch halves the cost of the backfill.
• Enrichment and analytics. Generating embeddings inputs, scoring records, producing internal analytics, or preparing data for a dashboard are pipeline steps, not user interactions.
• Content generation that is reviewed before use. If a human edits or approves the output before it ships, the few minutes saved by synchronous processing buys nothing, because the work waits on the reviewer anyway.

The shape of a good async pipeline

An asynchronous pipeline has a different architecture from a synchronous call, and getting the shape right is what makes the saving reliable rather than fragile. The pipeline collects work into a set rather than handling each item as it arrives. It submits that set as a batch job and records the job so it can be tracked. It waits for the completion window without blocking anything a user can see. When the results return, it processes them, handles any items that failed, and delivers the output to whatever consumes it. The key architectural shift is that the pipeline is built around a job and its lifecycle rather than around a single request and its immediate response. Once the system is organized that way, adding more batch workloads is straightforward, because the machinery to submit, track, and collect jobs already exists.

Decoupling submission from consumption

The most important design principle in an async pipeline is to separate the moment work is submitted from the moment its result is needed. In a synchronous system those two are the same instant, which is exactly what forces the real time price. In an async system you submit early and consume later, which gives the batch window room to run inside time you were not using anyway. A nightly report that is read at nine in the morning can submit its batch at midnight and have nine hours of slack. A document set uploaded during the day can be queued and submitted as a batch that completes well before anyone opens the results. Designing the pipeline so submission happens as early as possible and consumption happens as late as it can widens the window and makes the completion time a non issue.

Handling partial results and failures

A batch job is a set, and sets can come back partially complete or with individual items that failed. A robust async pipeline is built to expect this rather than to assume every job returns perfectly. That means recording which items were submitted, reconciling them against what came back, retrying the failures, and only marking the job done when the set is whole. This is not difficult, but it is different from synchronous error handling, where a single call either succeeds or fails and you deal with it on the spot. In a batch pipeline the unit of error handling is the job and its items, so the design needs to track state at that level. Teams that skip this find that occasional missing items erode trust in the pipeline, which pushes work back onto the synchronous path and quietly gives back the saving.