The Second Pattern: Bulkhead

Carrier B did not go down. Had they gone down, this would have been a twenty minute incident, somebody would have posted in the channel and we would all have got on with our afternoon.

What happened is that one resolver in their GraphQL schema got slow. Their trackingEvents field started taking eleven seconds while everything else in the response came back promptly, and since you ask for tracking events in the same query as everything else, every call you made took eleven seconds and returned a perfectly valid 200.

$response = $this->http->post(
    uri: 'https://api.carrier-b.example/graphql',
    headers: ['Content-Type' => 'application/json'],
    body: json_encode([
        'query' => $query,
        'variables' => ['reference' => $reference],
    ]),
);

There was no exception, no error status, and nothing in the response that any reasonable piece of code would treat as a failure, because by every definition available to you it had succeeded. It had simply taken eleven seconds about it.

Multiply that by the requests in flight and every worker you had was sitting inside that call, waiting politely. Carriers A and C were both completely healthy, which did not help anybody, because a customer looking up a Carrier A consignment got a timeout from your application anyway. There was nothing left to serve them with.

So one slow dependency took the whole thing down without ever once failing, which is a difficult sentence to put in an incident report and an even harder one to design against.

The obvious move

The standard answer is a circuit breaker, and it is a good pattern that deserves its reputation.

It would not have helped here, though, and why it would not is most of what makes bulkheads a separate idea rather than a footnote to breakers.

A breaker watches for failures and opens when it sees too many, so that subsequent calls fail immediately rather than piling up against something that is already broken. The question it asks is whether the dependency is failing. Carrier B was not failing. Carrier B was succeeding, slowly, which is the one condition a failure counter cannot see.

You can address that specific gap by treating latency above a threshold as a failure and feeding it to the breaker, which is a real improvement and worth doing regardless of anything else in this article.

But there is a deeper problem that no breaker addresses. Imagine the breaker works perfectly and trips after ten slow calls. Those ten calls each occupied a worker for eleven seconds. If you have twenty workers, you were already at half capacity before the breaker had enough information to make a decision, and during the seconds it takes to reach that threshold your ability to serve unrelated traffic has already gone.

A breaker limits how long you spend in a bad state. It does not limit how much of your capacity a single dependency can consume while getting there. Those are different problems, and conflating them is why people are surprised when the breaker is in place and the site still falls over.

The pattern

Michael Nygard put bulkheads in Release It!, and the metaphor is precise enough to be worth taking literally. A ship’s hull is divided into sealed compartments so that a breach floods one of them rather than the whole vessel, which does nothing whatsoever to stop the hull being holed and everything to change what happens next.

For software, the compartments are resources: workers, connections, threads, memory. The pattern says to partition them per dependency, so that a dependency which misbehaves can consume only its own share and cannot reach into everyone else’s.

Put another way, a bulkhead is a decision about what you are willing to lose. Carrier B being slow should degrade Carrier B lookups and nothing else. Stated that way, the design question stops being “how do I stop this happening” and becomes “when this happens, what specifically breaks”, which is a question you can actually answer and then verify.

There is a second direction that gets almost no coverage. Everything above is about protecting yourself from things you call. The same reasoning applies to things that call you, and in an API that receives webhooks from three carriers it matters just as much.

When this is worth it

Any time one dependency can consume a resource that unrelated work needs. That is the whole test, and it is worth applying literally rather than by feel, because the shared resource is often not the obvious one. Two carriers can look properly separate in your code and share a connection pool, a worker pool, or a rate limit on an outbound proxy.

It matters more the more dependencies you have. With one external service, partitioning is meaningless: it is the only thing using the resource, and if it is slow you are slow. With three, isolation is the difference between one broken feature and a broken product.

And it becomes urgent when a dependency’s failures are slow rather than fast. Fast failures are self-limiting, because the worker is released almost immediately. Slow ones accumulate, and accumulation is the mechanism by which a partial outage becomes a total one.

When it is not

A single dependency does not need it, for the reason above: it is the only thing using the resource, so partitioning it against itself achieves nothing.

Neither does work you can afford to lose entirely. If a slow dependency degrading everything is genuinely acceptable for some background job nobody is waiting on, then partitioning is effort for no gain.

The awkward case is when isolation costs you more capacity than it saves. Partitioned resources are, by definition, not shareable, so five slots reserved for Carrier B are five slots Carrier A cannot use when Carrier B is quiet. On a small deployment that inefficiency can be worse than the failure you are guarding against. Bulkheads suit systems with enough capacity to give some of it away.

Building it in Tempest

The most important control here is not a pattern, it is a number, and Tempest does not give it to you.

I went looking for timeout configuration in tempest/http-client and there is none. Not a default, not an option, nothing. The driver is a thin adapter over a PSR-18 ClientInterface:

public function sendRequest(RequestInterface $request): ResponseInterface
{
    return $this->client->sendRequest($request);
}

Whatever timeout you get is whatever the client you registered happens to default to, which for some clients is no timeout at all. If you take one action after reading this, make it going and checking what your PSR-18 client is configured with, because an eleven second call that should have been cut off at two is the difference between a degraded feature and an outage, and no amount of architecture compensates for waiting forever.

None of which is a fault in the framework. Timeouts belong to the transport, and the abstraction is honest about being a thin adapter rather than pretending otherwise. It is only that most of us assume a framework’s HTTP client has opinions here, and this one deliberately does not.

While you are in there, one inconsistency that will trip you up: the convenience methods take ?string $body, but Request::body is an array that the driver JSON-encodes for you. So post() needs json_encode() and a content type of your own, and sendRequest() does not. I lost a few minutes to that.

Getting the work off the request path

The first structural bulkhead you already have. Anything marked #[Async] is stored and handled elsewhere, which means a slow carrier consumes relay capacity rather than the workers that serve your customers.

That is a genuine compartment, and for the original incident it is most of the fix. Carrier B lookups being slow no longer costs you the ability to serve anything else, because the thing being exhausted is a background process, not your web tier.

It is also where the framework’s help currently stops, and the reason is something we found back in article three. The monitor holds five child processes, hardcoded, and there is no way to run a second monitor safely because pending commands are not claimed atomically. So all three carriers share one pool of five slots.

Which means the compartment exists, and it has all three carriers in it. Carrier B saturating those five slots delays Carrier A’s work just as surely as it used to delay HTTP requests. You have moved the problem somewhere less damaging without partitioning it.

Building the compartment yourself

You can cap a carrier’s share of those slots without touching the framework, using the cache locks that tempest/cache already exposes:

public function lock(
    Stringable|string $key,
    Duration|DateTimeInterface|null $duration = null,
    Stringable|string|null $owner = null,
): Lock;

Named locks with a duration are enough to build a counting semaphore. Give each carrier a fixed number of slots, and to do work you have to hold one:

final readonly class CarrierSlots
{
    public function __construct(
        private Cache $cache,
    ) {}

    public function acquire(CarrierCode $carrier, int $slots): ?Lock
    {
        foreach (range(1, $slots) as $slot) {
            $lock = $this->cache->lock(
                key: "carrier-slot:{$carrier->value}:{$slot}",
                duration: Duration::seconds(30),
            );

            if ($lock->acquire()) {
                return $lock;
            }
        }

        return null;
    }
}

Two slots for Carrier B out of the five means Carrier B can never take more than two, whatever it does. The other three stay available for carriers that are behaving. That is the bulkhead, and it is about thirty lines.

Get the duration right. It is what releases a slot when the process holding it dies, so it wants to be comfortably longer than your slowest legitimate operation and comfortably shorter than the point where you would rather give up. Set it too short and you release a slot while the work is still running, at which point your cap is not a cap.

The finally is safe even so. Locks carry an owner, and a plain release() verifies it, so a handler that overruns its duration and releases late cannot take a slot away from whoever picked it up in the meantime.

What to do when there is no slot

Here is where the last few articles come due.

The instinct is to throw, and throwing is the one thing you must not do here. Recall what happens to a handler that raises: the relay stamps it failed and excludes it from every future pass. Throwing because a carrier was momentarily busy would therefore discard the work outright, which is a great deal worse than the slowness you set out to manage.

Re-dispatch instead:

#[CommandHandler]
public function handle(FetchCarrierBTracking $command): void
{
    $lock = $this->slots->acquire(CarrierCode::B, slots: 2);

    if ($lock === null) {
        command($command->deferred());

        return;
    }

    try {
        // the actual call
    } finally {
        $lock->release();
    }
}

Put the command back and return normally. The relay picks it up on a later pass, by which point a slot may have freed. Add an attempt counter to the command so this cannot loop forever, and give up loudly rather than silently once it is exhausted.

There is an interaction with idempotency here that will catch you if you have followed article four. A command carrying no explicit key is deduplicated on a fingerprint of its contents, so a re-dispatched command looks identical to the original and gets dropped as a duplicate before it ever reaches a handler. Deferral has to change something the fingerprint can see, which the attempt counter does anyway, but only if you remember to add it.

The other direction

Everything so far has been about protecting yourself from carriers you call. There is nothing yet about the volume they send you, which is the same problem pointed the other way.

Carrier C delivers booking confirmations by webhook, and when their delivery backlog drains after an incident on their side, those arrive as a burst. Every webhook you accept turns into work, and that work goes into the same five slots everything else uses. Carrier C catching up on four hours of confirmations will starve Carrier A’s live traffic just as effectively as a slow query did.

The same semaphore applies, keyed on the inbound side, and the accept-then-defer shape above is what you want: take the webhook, acknowledge it quickly, and let the slot limit control how fast you actually process it. Acknowledging fast is also what stops the carrier retrying, which would otherwise turn a burst into a larger burst.

Most bulkhead writing I have come across only covers the outbound half, which seems odd given the mechanism is the same in both directions and an API taking webhooks from three providers is hardly an exotic setup.

What it costs

You have given up capacity on purpose. Reserved slots sit idle when their carrier is quiet, and on a small deployment that waste can genuinely exceed the value of the isolation. Do the arithmetic before you commit, rather than assuming partitioning is free.

Then there are the numbers, none of which you can derive from first principles. How many slots per carrier, what lock duration, how many deferrals before you give up. All empirical, all subject to change as traffic does, and all exactly the sort of configuration that gets picked during an incident and never looked at again.

The failure mode also moves rather than disappearing. Before, everything was slow. Now Carrier B work queues up and eventually ages out while everything else is fine, which is better, and is still a customer not getting tracking information. Bulkheads convert total failures into partial ones, which is the entire benefit on offer, and partial is not the same as none.

Worst of all, it is invisible when it works, because nobody ever notices the incident that failed to spread. In about eighteen months somebody perfectly reasonable will look at this code, fail to see what it is for, and propose removing it, and you will find you have very little to point at.

Where this leaves us

A circuit breaker is asking whether a dependency is broken, which is a fair question and not the one that mattered in this incident. What you needed to know was how much of your system Carrier B was allowed to occupy while it was neither broken nor working properly, and slow is both the more common state and the more damaging one.

On Tempest today you get one compartment for free, which is the async command bus keeping slow work off your request path, and it holds all your dependencies together in a pool of five. Partitioning inside it is yours to build, and cache locks are enough to do it.

Next in the series: a status column with eleven values and a chain of if statements deciding what happens next. Process Managers, and how they differ from the Sagas everyone calls them.