One of the jobs APDocs does for a customer sounds simple when you say it quickly: set up a new product in Smartsheet. In practice a new product means a folder in the right place in the filing structure, seven sheets copied from a template, each one renamed for the product, cross-sheet references remapped, a summary field stamped with the product code, and the new Document Register wired into the filing system so documents start landing in it straight away. Call it forty API calls, spread across three stages, for every product.
That workflow has been live for a customer for about a month now. It runs on the hour and half hour through the working day, and it is earning its keep. This post is about the part of it nobody sees, and why that part is the one I am most pleased with.
What goes wrong in the middle
When you split work across queues and serverless functions, you accept a bargain. The queue promises to deliver every message at least once. Not exactly once: at least once. A worker can be cut off part way through a job by a timeout, a throttle from the API it is calling, or simply a lost response to a call that the far end went on to complete.
Smartsheet's copy operations make that last case interesting. There is no idempotency key. If you ask for a copy twice, you get two copies. Early in testing we saw precisely that: two translation sheets, identical names, in one product folder, because a retry had fired after a slow copy that had actually succeeded.
So the question is not whether a worker will ever die mid-job. It will. The question is what happens next.
The job row is the truth
The first decision was that the messages on the queue carry nothing but keys. Everything that matters about a job lives in one database row: which stage it is at, which sheets have already been copied, which have already been renamed, the folder it landed in, when the copy started, and how many attempts it has had. Every step writes its progress to that row before moving on.
That changes the meaning of a failure. If a worker dies after copying four of seven sheets, the row says four of seven. Nothing is lost except time.
Meet the reaper
Every scheduled sweep that looks for new work also sends a second message: go and check on the old work.
The reaper picks up any job that has sat "in progress" longer than any honest worker would need, and finishes it. The important word is finishes. It does not start again. It looks inside the product folder, adopts whatever a previous attempt already created by name, copies only what is missing, and then hands the job on to the next stage exactly as the original worker would have. Because every name it composes is deterministic, the recovery run and the first run always agree about what things are called.
If a job has been recovered too many times, the reaper stops and parks it, and tells us. A job that keeps failing is information, not something to loop on forever.
The lesson in the window
The first version of the reaper had no sense of time. It recovered any job marked in progress, including ones a live worker was still working on. Two runners on one job meant two copies of the same sheet, which was the duplicate we had been trying to design out in the first place.
The fix was patience. A job is only reaped once it has been in progress for longer than a worker is allowed to run. Fifteen minutes, in our case. That window is the whole design: the reaper is deliberately slow, so that it only ever takes over from something that has genuinely stopped.
Earning its keep
This week it paid for itself in one afternoon.
Two products arrived with names at the long end of what the customer uses. The copy stage got the folder and the template sheets in, then failed on an auxiliary sheet part way through the copy stage, because the composed name ran a couple of characters over a limit the API enforces. Both jobs were parked. The folders sat there, half-provisioned.
We changed the naming rule so that every name is composed to fit, tested it, and shipped it the same day. Then the recovery was one field change on each of the two job rows: put them back to "in progress", with a start time of midnight. The next scheduled sweep did the rest. The reaper found both jobs, saw the folders already existed, copied the one sheet that was missing, and handed them on to be renamed and registered. No folders deleted, nothing copied twice, nothing for the customer to redo.
The honest footnote is that the customer's own sheet should have told them a job had stalled, and it did not. That is the next change, and it is small, because the job row already knows which row on their sheet it came from.
Why this matters to you
You will never see the reaper. That is the point. What you see is that a product you ordered at ten past the hour exists, complete, before the next one. The machinery that makes that true when a cloud API has a slow afternoon is the part of the platform I would least like to run without.
It is also not a premium feature. Seatbelts come as standard. Every tier gets every feature, and customers choose usage and support hours, not features. The one exception is storage: customers on Growth and Scale can choose AWS S3 instead of Wasabi, which opts out of the Covert Copy. Recovery is a seatbelt.