Most AI automation projects are judged on the model and fail on the plumbing. The interesting part is a small share of the work; the part that determines whether it runs reliably next month is the integration layer, and it is consistently under-designed.
Assume delivery will fail
Webhooks get lost. Networks blip, receivers restart, providers have partial outages. Any system that assumes an event delivered once is an event processed once will silently drop work, and the gap is invisible because nothing errored anywhere.
Design for at-least-once delivery: acknowledge fast, process from a queue, and reconcile periodically against the source of truth so a missing record is discovered by the system rather than by a customer.
Idempotency is not optional
Because delivery is at-least-once, the same event will arrive twice. Processing must be safe to repeat, which usually means keying on an identifier from the source and checking whether that identifier has already been handled before acting.
Without it, the visible symptoms are duplicate records, duplicate emails to customers and double-counted figures, and unwinding those is far more work than preventing them.
Order is not guaranteed
Events frequently arrive out of sequence, so an update can land before the creation it depends on. Systems that assume order produce confusing states that are hard to reproduce and easy to dismiss as one-offs until they happen during something important.
Handle it by making each event carry enough context to be processed independently, or by holding events that arrive early until their prerequisite appears.
Nearly every automation we are called in to stabilise works fine in testing. It breaks because the same event arrived twice, or arrived before the record it referenced existed.
Lena Fischer, Solutions Architect, Engineered With AI
Retry with backoff, and give up eventually
- Retry transient failures with increasing delay rather than immediately and repeatedly.
- Do not retry a genuine rejection; a malformed payload will fail identically forever.
- Cap the attempts and move the item to a dead letter queue somebody actually reviews.
- Alert on the queue depth, not on individual failures, or the alert becomes noise.
A retry loop with no ceiling is how one bad record consumes a rate limit and takes the rest of the pipeline down with it.
Decide what happens when the model is unavailable
Language model providers rate limit and occasionally degrade, and the automation needs a defined behaviour when that happens: queue and process later, fall back to a smaller model, or stop and surface it. All three are legitimate; having no answer is not.
Whichever you choose, make it visible. A system that silently skips the work it could not process is worse than one that fails loudly.
Log enough to reconstruct
When something goes wrong, you need to know what came in, what the model returned and what was written. Logging the inputs and outputs with a correlation identifier turns a mystery into a five-minute investigation, and its absence turns a small bug into a week.
Secure the endpoints
An unauthenticated webhook endpoint is an open door into your systems. Verify signatures where the provider offers them, restrict by source where you can, and treat the payload as untrusted input rather than as data from a system you trust.
Build the boring parts first
Queue, idempotency, retries, dead letter handling and logging before any of the interesting logic. Retrofitting them into a working prototype is substantially harder than starting with them, and the prototype always goes to production. Our automation index covers where this applies by sector.
Rate limits are shared, not per-feature
A quota applies across everything using the credential, so a background job processing a backlog can exhaust the allowance a customer-facing feature depends on. The user-visible symptom is a feature failing for reasons entirely unconnected to what the user did.
Separate credentials by workload where the provider allows it, and give interactive traffic priority over batch. Where separation is not possible, throttle the background work deliberately rather than letting it run as fast as the API permits.
Timeouts need to be shorter than the caller's patience
Model calls can take considerably longer than a typical HTTP client default, and a chain of services each with generous timeouts produces a request that hangs for a minute before failing. Set the timeout from what the caller will actually wait for, and fail early enough to do something useful with the remaining time.
Where the work genuinely takes longer than a request should, accept it, return an identifier and let the caller poll or receive a callback. That is a design decision worth making early rather than a slow endpoint nobody wants to own.
Automation failing in the joins?
We build the integration layer to survive the ordinary failures, so the automation keeps working when an upstream system does not.