The Second Pattern: Idempotent Receiver

Carrier C books asynchronously. You POST a consignment, get back a 202 and a reference, and some minutes later they call your webhook to tell you whether it actually worked. Everybody’s integration guide has the same line buried in it somewhere: deliveries may be repeated, so your endpoint should handle that.

Which everyone does, roughly like this.

#[Post('/webhooks/carrier-c')]
public function __invoke(CarrierCWebhookRequest $request): Response
{
    $seen = query(ProcessedWebhook::class)
        ->select()
        ->whereField('event_id', $request->eventId)
        ->first();

    if ($seen !== null) {
        return new Ok();
    }

    $this->bookings->confirm($request->reference, $request->carrierBookingId);

    query(ProcessedWebhook::class)
        ->insert(event_id: $request->eventId)
        ->execute();

    return new Ok();
}

Read it, and it is obviously right. Have we seen this event before? No? Then do the work and write it down. There is a table, there is a check, there is a record, and the reviewer who approved it was not being careless.

The trouble is the gap between the select and the insert, which is where two simultaneous deliveries both find nothing and both go on to confirm the booking. Carrier C’s retry policy makes this considerably more likely than it sounds, because a delivery that times out on their side gets retried immediately, and “times out on their side” often means your handler is still running. Their retry arrives while the original is mid-flight, which is the exact window the check does not cover.

You will not reproduce it locally. It needs concurrency you do not have on your laptop, and it fails maybe one time in several thousand, which in webhook volumes is once or twice a week, forever.

The obvious move

Put a unique constraint on event_id and let the database arbitrate. That is a real fix and I would take it over the version above without hesitating.

try {
    query(ProcessedWebhook::class)
        ->insert(event_id: $request->eventId)
        ->execute();
} catch (UniqueConstraintViolation) {
    return new Ok();
}

$this->bookings->confirm($request->reference, $request->carrierBookingId);

Insert first, work second, and let the second delivery bounce off the index. The race is gone.

What is left is more awkward. You have made the check atomic without making the outcome consistent, because the insert commits and then the confirmation happens separately, so a crash in between leaves a row saying you have processed an event you have not processed. Every future delivery of it now returns Ok and does nothing, and the booking stays unconfirmed with nothing anywhere suggesting anybody should look at it.

Wrap both in a transaction and you have swapped that for a different problem, since the confirmation involves work you would rather not hold a transaction open across, and if any of it is an outbound call you are back in the dual write territory from the last article.

And nobody asks what the second delivery should actually return until a support ticket forces the issue. The version above says Ok with an empty body. But the first delivery returned a body, presumably one Carrier C parsed, and now the same request produces two different answers depending on timing. That is not idempotent in any sense that would satisfy the person who wrote the integration guide. It is merely harmless.

The pattern

Idempotent Receiver comes from Hohpe and Woolf’s Enterprise Integration Patterns, where it sits alongside the messaging patterns it exists to support. The framing there is the useful part: if a channel can only promise at-least-once delivery, and most can, then duplicate detection is not the sender’s job to solve, because the sender cannot solve it. It belongs to the receiver.

Doing it properly means three things, and the homegrown versions above only ever manage the first.

You need to identify the message, which means a key that is stable across retries. You need to remember the outcome rather than the fact, so a repeat gets the same answer the original did rather than a polite acknowledgement. And you need to handle the case where the duplicate arrives while the original is still running, which is the case that produced the bug in the first place and the one that homegrown implementations essentially never cover.

That third requirement is what makes this properly difficult, because now you are holding a lock across a request, deciding what a concurrent caller sees, working out what happens if the process holding it dies, and choosing a TTL. None of that is webhook logic. All of it has to be right.

When this is worth it

Any time retries are automatic, which is the same as saying any time the sender is a machine. Carrier C retries because their integration guide says they do. Your own relay from the last article retries on crash. A load balancer will happily replay a request it thinks failed. In none of those cases is a duplicate unlikely, so treating it as an edge case is a scheduling decision rather than a technical one.

It matters more the more the operation costs. Confirming a booking twice is annoying. Charging twice, dispatching a courier twice, or emailing a customer twice are the sort of thing that ends up in a postmortem.

When it is not

If the operation is naturally idempotent, leave it alone. Setting a status to delivered twice lands you in the same place, and adding key tracking to it is work you will maintain for no benefit.

Reads do not need it either, however tempting the symmetry.

And if you genuinely cannot get a stable key from the sender, this pattern will not save you. Some webhooks arrive with nothing but a payload and a timestamp, and hashing the body is a guess rather than an identifier, because two legitimate identical events are indistinguishable from one duplicated event. You should go and ask them for an event ID, and in the meantime you are reconciling, not deduplicating.

Building it in Tempest

We have spent three articles finding that Tempest ships most of a pattern and few of the guarantees. This one is different. Somebody sat down and did this properly.

There is a first-party tempest/idempotency package. On the HTTP side it is an attribute:

use Tempest\Idempotency\Attributes\Idempotent;

final readonly class CarrierCWebhookController
{
    #[Idempotent]
    #[Post('/webhooks/carrier-c')]
    public function __invoke(CarrierCWebhookRequest $request): Response
    {
        $this->bookings->confirm($request->reference, $request->carrierBookingId);

        return new Ok(['status' => 'confirmed']);
    }
}

The controller no longer knows it is being deduplicated, and the third requirement above, the hard one, is handled. IdempotencyMiddleware takes a cache lock on the key before doing anything, and a concurrent request that cannot get the lock is told so:

if (! $lock->acquire()) {
    return $this->inProgressResponse();
}

That response is a 409 carrying retry-after: 1. Not a queue, not a wait, just an honest statement that this is in flight and the caller should come back. Compare that to the homegrown version, where the concurrent caller silently performs the work a second time.

A completed record replays the original response, including its body and status, with an idempotency-replayed: true header added so the caller can tell. That is the second requirement, remembering the outcome rather than the fact, and it is the part homegrown implementations skip.

Reuse a key with a different payload and you get a 422, because the middleware fingerprints the request and compares. Somebody thought about the case where a client’s retry logic reuses keys after mutating the body, which is a failure mode most people do not discover until it happens to them.

There is also a liveness mechanism I did not expect to find. A pending record carries an owner and a heartbeat that a background renewer keeps ticking, and if the process that took the lock dies, the record goes stale and the next request with a matching fingerprint takes over rather than being locked out until the TTL expires. That is a genuinely thoughtful piece of engineering and not something I would have built myself on a Tuesday.

The consumer side, which closes last article’s loop

The outbox from article three left you with a relay that redelivers on crash, and the honest answer to that was that consumers have to cope. They are covered too, by a separate middleware for the command bus:

use Tempest\Idempotency\HasIdempotencyKey;

#[Async]
final readonly class NotifyCustomerOfBooking implements HasIdempotencyKey
{
    public function __construct(
        public string $consignmentId,
        public string $reference,
    ) {}

    public function getIdempotencyKey(): string
    {
        return "booking-notification:{$this->consignmentId}";
    }
}

A redelivered command whose record already exists is dropped, quietly, with no handler invocation. Quiet is correct here, since there is no caller waiting for a response, but it does mean the observable behaviour of a duplicate is nothing at all, which is worth knowing when you are staring at logs wondering why a command appears to have vanished.

The part that will bite you

HasIdempotencyKey is optional, and the fallback is where the trap is:

private function resolveKey(object $command, string $fingerprint): string
{
    if (! $command instanceof HasIdempotencyKey) {
        return $fingerprint;
    }
    // …
}

No key means the key becomes a fingerprint of the command’s contents. For a redelivery that is exactly right, because the redelivered command is byte-for-byte the same. For two genuinely separate commands that happen to be identical it is exactly wrong, and they are indistinguishable.

Dispatch AddParcelToConsignment('01JQ…', weight: 2000) twice because the customer really is sending two identical parcels, and the second one is silently discarded. No error, no log line at your level, no parcel. Any command whose meaning depends on how many times it happened needs an explicit key, and the framework cannot work out which of yours those are.

The defaults deserve reading too. Keys live for 24 hours, which is a decision about how late a duplicate can arrive and still be caught, and Carrier C’s retry schedule may well outlast it. The header is Idempotency-Key and requireKey defaults to true, so a webhook that does not send one gets a 400 rather than being processed unprotected, which is the right default and will still surprise you the first time. Only POST and PATCH are supported, and anything else throws rather than passing through. Windows throws outright.

The one I would look hardest at is the store. It defaults to CacheIdempotencyStore, and a cache is a thing that evicts. If your cache is under memory pressure, or gets flushed on deploy, or is an in-memory driver that does not survive a restart, then your idempotency records go with it and the duplicates you were protected from become duplicates you process. For webhook handling I would want that backed by something durable, and storeClass in the config is there for exactly that.

What it costs

The cheapest cost is that you now have state with a lifetime, and lifetimes need choosing. Twenty-four hours is a guess about your senders, and the way you find out it was the wrong guess is a duplicate getting through weeks later.

More interesting is that idempotency only applies to completion. Look at the catch: on a Throwable the pending record is deleted and the exception rethrown, so a handler that blows up frees the key and a retry runs the work again from scratch. That is almost certainly what you want, and it also means a partially completed operation that throws halfway is not protected by any of this. If your handler writes three rows and dies after the second, idempotency will not save you. Transactions will. They are different tools and it is easy to assume the newer one subsumes the older.

Then there is the debugging. A duplicate on the HTTP side is at least visible in the response header. On the command side it is silence, and silence is a hard thing to notice when the question is why something did not happen.

Where this leaves us

Duplicate delivery is not an edge case, it is the normal operating condition of every automated sender you integrate with, and the receiver is the only party in a position to do anything about it. The homegrown select-then-insert everybody writes handles the easy half and leaves the concurrent case, which is the half that actually fires.

This is also the first pattern in the series where my advice comes out as just use what is already there, which after three articles of finding the guarantees missing was not what I expected to be writing. Read the config defaults properly, move the store somewhere that does not evict, and give your commands explicit keys unless you are certain that two identical ones really do mean the same thing.

Next in the series: one carrier sends a 40KB manifest and another sends 12MB, and both go through the same queue. The Claim Check, and what it costs to move the same bytes twice.