The Second Pattern: Claim Check

Labels are the first thing in this domain that gets big. A domestic parcel comes back as a 40KB PDF and nobody thinks about it again. A twelve-piece international consignment comes back with a label per parcel, a commercial invoice and a customs declaration, and now you are holding eleven megabytes of paperwork that has to get from the carrier’s API to your storage to the customer.

The command that does it looks entirely sensible when you write it.

#[Async]
final readonly class StoreConsignmentPaperwork
{
    public function __construct(
        public string $consignmentId,
        public string $labelPdf,
        public string $commercialInvoicePdf,
        public string $customsDeclarationPdf,
    ) {}
}

Dispatch it, the handler writes the files somewhere, everyone goes home. For a while this is genuinely fine, because most consignments are one parcel going to Swansea and the payload is small enough that nobody has cause to look at what is happening to it.

What is happening is that AsyncCommandMiddleware hands the object to the repository, which does this:

public function store(string $uuid, object $command): void
{
    query(StoredCommand::class)
        ->insert(
            id: $uuid,
            payload: serialize($command),
        )
        ->execute();
}

Your eleven megabytes of PDF gets serialize()d and written into a database column. Not a reference to it, the whole thing, as a string, in a row.

That on its own is unpleasant but survivable. What turns it into an outage is the relay loop, which runs twice a second:

$availableCommands = arr($this->repository->getPendingCommands())
    ->filter(fn (object $_, string $uuid) => ! array_key_exists($uuid, $processes));

getPendingCommands() selects every pending row and unserialises all of them into an array, so that the loop can pick one uuid to work on. Every payload, every pass. With a short backlog this costs nothing worth measuring, which is precisely why nobody notices until the afternoon a carrier goes slow and forty large consignments queue up behind each other. Then the relay is loading several hundred megabytes into memory, throwing it away, sleeping half a second and doing it again.

The relay dies, bookings stop being confirmed, and the stack trace you get, if you get one, points at a memory limit inside a deserialisation call rather than anywhere near the code that decided to put a PDF in a command.

The obvious move

Compress it. gzencode() on the way in, gzdecode() in the handler, and since PDFs compress well the numbers look much better straight away.

My only objection is that it is not a fix. You have bought a factor of three or four on a quantity that varies by two orders of magnitude between consignments, so the same failure arrives later with a worse stack trace, and every payload now costs CPU at both ends for the privilege.

Raising the memory limit is the same trade with even less to show for it.

The more interesting wrong answer is splitting the command into one per document, because that feels like it addresses the size directly. It does not. Three commands carrying four megabytes each still put twelve megabytes through getPendingCommands(), and you have introduced the possibility of them failing independently, leaving a consignment with a label and no customs declaration.

What all of these share is that they treat the size of the payload as the problem. The problem is that the payload is moving through a channel that was never meant to carry it, and making it smaller does not change that.

The pattern

Hohpe and Woolf call this the Claim Check, after the theatre cloakroom. You hand in your coat, you get a numbered ticket, and the ticket goes in your pocket rather than the coat. The coat has not gone anywhere. It is simply not being carried around all evening by somebody who has no use for it.

Applied here: write the large payload to storage once, put its location in the message, and let the consumer fetch the bytes if and when it actually needs them. The message goes back to being small and cheap to move, and the system that is good at holding large objects holds the large object.

It took me a while to stop thinking of this as an optimisation. It is really a claim about what belongs in a message. A message says what happened and what needs doing, and eleven megabytes of PDF is neither, it is an attachment to them. Once you see it that way the pattern stops being a workaround for a queue limitation and starts being the obviously correct modelling, which it usually was all along.

When this is worth it

The signal I trust is variance rather than size. A payload that is reliably two megabytes is one you can size a system around, whereas a payload that is 40KB almost always and 12MB occasionally is not, because you end up provisioning every stage for a worst case that hardly ever happens and the stages you forget are the ones that fall over.

Multiple consumers push it over the line on their own. Without a claim check each of them receives its own copy of the same bytes, moved independently, to no purpose at all.

The case that gets overlooked is when most consumers never open the payload. Four handlers react to paperwork being stored, one of them actually reads the PDF, and the other three have been shipped several megabytes apiece so they can look at a consignment ID.

When it is not

Small, predictable payloads do not need any of this. Everything under a few kilobytes gains nothing from a storage round trip at each end and acquires a whole new failure mode in exchange, which is a poor trade made with confidence.

It is also wrong when the consumer always needs the payload immediately and your storage is further away than your queue. Nothing has been removed, it has been moved and a hop added, so if the consumer’s first action is invariably to fetch the thing you just wrote, it is fair to ask what the indirection bought.

Lifetime is the one that catches people out. A message that can sit in a queue for a week, and storage with a seven day lifecycle rule on it, gives you something that works beautifully right up until it quietly does not. Reference and referent have to outlive one another properly, and that is a coordination problem rather than a coding one.

Building it in Tempest

The Storage contract has the two methods that matter, which are the streaming ones:

public function writeStream(string $location, mixed $contents): static;

public function readStream(string $location): mixed;

Reach for those rather than write() and read(). A claim check that pulls the whole object into a PHP string in order to store it has solved the message size problem and kept the memory problem, which is not much of a trade.

The command shrinks to the ticket:

#[Async]
final readonly class StoreConsignmentPaperwork
{
    public function __construct(
        public string $consignmentId,
        public string $paperworkLocation,
        public string $checksum,
    ) {}
}

Three strings, a couple of hundred bytes once serialised into that column, and the relay can now load ten thousand of them without noticing.

Writing the ticket, on the producing side:

use Tempest\Storage\Storage;
use function Tempest\Support\Random\uuid;

final readonly class CheckInPaperwork
{
    public function __construct(
        private Storage $storage,
    ) {}

    public function handle(string $consignmentId, mixed $stream): PaperworkTicket
    {
        $location = "paperwork/{$consignmentId}/" . uuid() . '.zip';

        $this->storage->writeStream($location, $stream);
        // see "Naming the coat" below before you copy that uuid

        return new PaperworkTicket(
            location: $location,
            checksum: $this->storage->checksum($location),
        );
    }
}

That $stream is worth a note, because it is doing more work than the signature suggests. Tempest’s HttpClient hands you a Response whose body is a string, an array or a Generator, which is a sensible shape for the API calls you make ninety-nine percent of the time and the wrong shape for pulling down eleven megabytes of PDF. If you fetch the paperwork through it and then pass the body to writeStream(), the whole thing has already been in memory and you have gained nothing over the version you started with.

For downloads at this size, go under the abstraction. A PSR-18 client configured to stream, or a plain fopen() against a signed URL the carrier gave you, hands writeStream() a real stream and the bytes never fully land in your process. The claim check only pays off if both ends stream.

The checksum is not decoration. Message and payload now live in two systems that are capable of disagreeing, and carrying a checksum means the consumer can find that out rather than processing whatever happens to be sitting at that path. Storage exposes checksum() directly, so it costs one line.

The handler collects the coat:

#[CommandHandler]
public function handle(StoreConsignmentPaperwork $command): void
{
    if ($this->storage->checksum($command->paperworkLocation) !== $command->checksum) {
        throw new PaperworkDidNotMatchTicket($command->paperworkLocation);
    }

    $stream = $this->storage->readStream($command->paperworkLocation);

    // …
}

There is one more method worth knowing about, because it removes a whole category of work. Storage has temporaryUrl(), so when the consumer is the customer rather than one of your handlers, you need not proxy the bytes through your application at all. Hand them a signed URL that expires and let them fetch from storage directly, and your API stops being a file server, which it should never have been in the first place.

Naming the coat

There is a decision hiding in that uuid() call, and it is more consequential than it looks.

A random location means every write produces a new object. Retry the producing step, whether because the carrier call failed halfway or because your own code threw after the write, and you get a second object holding the same bytes with nothing pointing at the first. Storage fills up with near-duplicates that no sweeper can safely distinguish from live paperwork, because from the outside they look identical.

A deterministic location, derived from something stable about the consignment, makes the write idempotent instead. Retry it and you overwrite the same object with the same content, which is a no-op you can perform as many times as you like.

$location = sprintf(
    'paperwork/%s/%s.zip',
    $consignmentId,
    hash('xxh128', $carrierBookingReference),
);

That is the same argument as the last article, arriving from a different direction. Article four was about not doing the work twice. This is about the work being safe to do twice, which is the better property when you can get it, and here you can.

The exception is when the payload legitimately changes for the same consignment, which happens if a carrier reissues a label. Then a deterministic location overwrites history, and if anyone needs the old label you have destroyed it. Add a version or a timestamp to the path at that point, and accept that you are back to sweeping.

Where this collides with the outbox

Article three established that dispatching an async command inside a transaction is safe, because the command insert rolls back along with everything else. Storage does not take part in that transaction and never will, so ordering matters here, and only one order is safe.

Write the payload to storage first, then open the transaction that commits the business row and dispatches the command carrying the reference. If that transaction rolls back you are left with an object in storage that nobody holds a ticket for, which is litter: annoying, cheap, and sweepable by a lifecycle rule or a weekly job.

Do it the other way around, committing first and storing afterwards, and a failure in between leaves a command in your outbox pointing at a location that does not exist. The relay picks it up, the handler fails to read it, and under the semantics from article three that command is marked failed and never retried, so the paperwork is gone and nothing tells you.

Orphaned objects are housekeeping. Dangling references are data loss. Store first, always.

What it costs

You have taken one thing that could fail and made it three, since the write, the message and the read now fail independently, at different times, for unrelated reasons. The failure you will actually meet is the one where the message is perfectly fine and the payload is not, which is also the one that is hardest to read from a log.

Cleanup becomes yours. Nothing in the pattern deletes anything, so unless you write the lifecycle rules or the sweeper yourself, storage grows forever and the bill turns up eventually. It is boring work that nobody schedules, and it is much the most common way I have seen claim checks rot.

Local development gets worse too. A payload inside a message is visible in a row you can select. A payload behind a reference is somewhere else entirely, and reproducing a bug now means having the right object in the right bucket. InMemoryStorageConfig exists for tests and helps a great deal, but the gap between what you test and what runs does widen.

The subtler cost is traceability. Once message and payload are separated, the question of what exactly was in the thing that failed needs both halves, and if your sweeper got there first you can no longer answer it. A retention policy longer than your incident response time is not a detail.

Where this leaves us

When payload size varies by orders of magnitude you cannot size a pipeline for it, and the answer is not to make the payload smaller. Send the ticket and keep the coat somewhere that is good at holding coats.

The reason to care sooner on Tempest specifically is getPendingCommands(), which deserialises every pending payload on every pass of the relay loop. That is entirely fine while commands are small and entirely unforgiving once they are not, and the distance between those two states is one busy afternoon.

Next in the series: the carrier’s API is not down, it is slow, and every one of your workers is waiting politely for it. Bulkheads, on both sides of the door.