Integrating ai into existing software is mostly an architecture problem rather than a model problem. You are adding a component that is slow, occasionally wrong and priced per use into a system built on the assumption that functions return quickly, correctly and for free.
Where you place it decides how much of that assumption you break.
Start Where Being Wrong Is Cheap
The first integration should be somewhere a mistake is visible and reversible. Suggesting a category a user confirms. Drafting text somebody edits. Summarising a document that remains available. Ranking results a person still chooses from.
What all of these share is a human between the output and the consequence. That is not a limitation to design away; it is what makes the first version safe to ship while you learn how the thing actually behaves on your real data.
The integrations that go badly are the ones placed immediately in an unattended path: automatically categorising financial records, sending customer emails, or deciding eligibility. Those can come later, once you have evidence rather than optimism.
Put It Behind Your Own Interface
Do not scatter provider SDK calls through the codebase. Create one service or module that owns every model interaction, and have the rest of the application call that.
This costs a day and buys three things: swapping provider becomes one change, all logging and cost tracking sit in one place, and you can substitute a stub in tests so your test suite does not depend on a paid non-deterministic external service.
That last one matters more than it sounds. Teams that skip it end up with a test suite that is slow, flaky and expensive, and they respond by not testing that path at all.
Design For Latency You Do Not Control
A model call can take two seconds or forty for identical input. If it sits inside a synchronous request a user is waiting on, you have made your application’s responsiveness a function of somebody else’s load.
- Move it out of the request path where you can: queue the work, return immediately, notify when it is done.
- Stream the response where the interface allows, so the user sees progress rather than a spinner.
- Set a timeout you can actually live with, and decide what happens when it fires. Falling back to the old behaviour is usually better than an error.
- Never let it block a page load or a save. Users will forgive a slow suggestion and will not forgive a slow save.
Keep The Old Path Working
The most valuable thing you can build alongside the first integration is the switch that turns it off.
A feature flag, defaulting off, per customer or per environment. When quality drops after a provider update, when costs spike, or when something behaves oddly on a Friday afternoon, you want a switch rather than a deployment.
This also lets you run it for ten percent of traffic and compare, which is the only honest way to know whether it improved anything.
Ship the off switch in the same release as the feature. Teams that add it later always add it during an incident.
Ethan Caldwell, CTO and Co-Founder, Engineered With AI
Decide What Data Leaves
This needs settling before the first call, not after legal notices it.
Establish what is being sent, whether it contains personal or confidential data, what the provider’s retention and training policy is on your tier, and whether that is compatible with what you have told your own customers.
Frequently the answer is to send less: redact identifiers, send the paragraph rather than the document, or send a reference rather than the record. That is usually easy at the start and awkward to retrofit once the integration is embedded.
Measure Something Other Than Enthusiasm
Decide before launch what would make this worth keeping. Fewer support escalations. Less time per document. Higher acceptance of suggestions. Then measure it against the period before.
Without that, the feature gets judged on how impressive the demo was, which is how organisations end up maintaining components that cost money every month and improve nothing measurable.
Track cost per useful outcome alongside the quality measure. A feature that works well and costs more than the problem it solves is still a feature to remove.
A Reasonable First Release
One well-chosen assistive feature, behind your own service interface, out of the synchronous path, behind a flag defaulting off, with full logging, a spend cap, and one metric that says whether it helped.
That is a couple of weeks of work for most teams and it teaches you more about what your systems and data can actually support than any amount of planning.
If you are deciding where to start, describe the system and what you are hoping it will do. The useful conversation is usually about which part of the workflow to touch first, not which model to use.
Adding AI to something that already works?
Describe the system and what you want it to do. We will tell you where to put it so a wrong answer stays cheap, and what to build before you ship it.
