Last week I wrote about the reaper: the part of APDocs that picks up a job a worker has abandoned and finishes it, rather than starting again. I said you would never see it, and that it was the part of the platform I would least like to run without. Since then it has been given a second job, and the way it got that job is worth a post of its own, because the fault it now covers was one we had designed in ourselves.
A document that quietly knew less
When a document is filed into APDocs it is indexed twice over. The metadata (title, reference, folder, author) is indexed immediately. The content is read by Amazon Textract, which takes the file apart and gives us back every word on every page, and those words become the search tokens that let someone find a warranty by the name of the roofing contractor rather than by remembering what the file was called.
Textract is asynchronous. You hand it a file, it gives you a job id, and you ask it later whether it has finished. Our indexing worker asked every five seconds for up to four minutes, then stopped asking.
Last Tuesday a customer was being set up on our test environment, and a few dozen documents were filed in a few minutes. One of them, a one-page PDF, did not come back inside the four minutes. The worker did what it had always done: caught its own timeout, sent us an alert, wrote the index with the metadata tokens only, and reported the job a success. A few minutes later Textract finished the page. The result sat on Amazon's side for seven days, as it always does. Nobody collected it.
Three things were wrong with that, and I want to put them in order of harm.
The document was less findable than its neighbours and nothing said so. Not the index row, not the document's history, not the customer's screen. Support could not have listed "documents whose content was never read" even if they had thought to ask.
The alert was the only trace, and it arrived before anything had tried to recover. That is precisely backwards from how the reaper is meant to work: it runs first, and people hear about what it could not finish.
And the four minutes was our ceiling, not Amazon's. The function it runs in is allowed fifteen. We stopped listening with eleven minutes on the clock because four was a default somebody had typed in years ago and nobody had had a reason to question it until a burst of filings put small jobs in a queue behind each other.
The one-line fix, and why it was not enough
The first change took an afternoon to agree and a minute to make: wait twelve minutes instead of four. The time was already paid for. That alone will remove most of these incidents, and it was in test by the end of the day.
But it does not change the shape of the failure. A job can still outlast the wait, and when it does we would still be throwing away a result that Amazon is holding for us for a week. The proper fix is to stop forgetting.
The reaper's second job
The reaper already exists. It already runs on a schedule through the working day, and it already reads a table of jobs in flight. So the change was to give it one more kind of row to look at.
When the indexing worker runs out of time now, it writes the index with the metadata tokens as before, marks the row as having content still to come, and records a small pending row holding the Textract job id and where everything is. No alert. The document is filed, versioned, and findable by its title and reference, which is what it would have been anyway.
On its next pass the reaper finds the pending row and sends a message back to the indexing worker: go and collect this one. The worker asks Textract for the job by id. If it is finished, the words are merged into the index, the pending flag is cleared, and a line is written into the document's history saying that its content was indexed later. If it is still running, the reaper counts the attempt and waits for the next pass. Only if Textract reports a failure, or three passes go by with the job still running, or the result has aged out, does anyone get told, once, with the document, the job id and the reason, which is everything you need to look.
Two properties from last time carry over exactly. The reaper finishes; it never restarts. It would be easy to resubmit the file and start a second Textract job, but that pays twice for the same page and puts the second job in the same queue that delayed the first. And a document is never in a state the system cannot describe: its content is indexed, or pending, or failed and reported. There is no fourth state where it simply knows less and keeps quiet.
The thing the spec got wrong
The design note for this change said: delete the temporary copy of the file as before, the result lives in Textract now, not in our bucket. That was wrong, and we found out at build time rather than in production, which is the better of the two ways to find out.
Textract reads the file when the job runs, not when it is submitted. Under a busy queue a job can still be waiting when our twelve minutes are up, which means it has not read the file yet. Delete the copy at that point and the job we are about to collect will fail for want of its input. So the copy now lives until the collection pass closes the job, and the pending row remembers where it is. A few kilobytes for an hour or so. The spec has a line in it now that says why.
What you will see
Nothing, mostly. A document you file is findable by its title straight away, and by its contents within the hour even on the worst afternoon Amazon's queue has to offer. If you open its history you may occasionally see two index lines instead of one, a few minutes or an hour apart, and now you know what the second one means.
The honest footnote this time is on us. Last time the lesson was that the reaper needed to be patient. This time it is that an alert is not a recovery, and a default is not a decision. The change went live this week.
It is still not a premium feature. Seatbelts come as standard. Every tier gets every feature, and customers choose usage and support hours, not features. The one exception is storage: customers on Growth and Scale can choose AWS S3 instead of Wasabi, which opts out of the Covert Copy. Recovery is a seatbelt, and this week it got a second strap.