Generative AI integrations that do useful work

What does a generative AI integration actually cover?

Four things, usually in this order. First, the task: what the system should answer, what it should refuse, and what it should hand to a person. That decision drives everything after it, and getting it wrong is the most expensive mistake available, because it is the one you only discover in production.

Second, the knowledge. If the assistant needs your material to be useful, that material has to be ingested, chunked, retrieved and attributed, with access boundaries that match the ones already in your product. A retrieval layer that ignores permissions is a data-leak waiting for its first curious user.

Third, the integration itself: prompts and responses, structured outputs your code can rely on, tool actions the model is allowed to call, and an interface that makes it obvious which parts of an answer came from a source and which did not.

Fourth, evaluation and maintenance. A set of representative requests, the refusal cases, the citation checks, and a way to notice when a new model version or an edited prompt quietly changes behaviour. Without that last part you have a demo, not a feature.

When is generative AI the wrong tool?

When a rule would do. If the task is deterministic — routing on a field, validating a format, calculating a total — a language model adds cost, latency and a failure mode you did not previously have. We will say so before proposing one.

When nobody owns the data question. If it is unclear who may see which documents, or what the retention terms of your chosen provider are, retrieval will inherit that ambiguity and amplify it.

When there is no review path and the action has consequences. A model that drafts a reply is very different from a model that sends one. The second needs approvals, limits and an audit trail before it is worth building, not after.

And when the brief is "AI everywhere" with no acceptance criteria. An assistant expected to handle every request from every department cannot be evaluated, so it cannot be improved. Scoped assistants tied to a clear task are easier to build, easier to trust, and far easier to extend once they have earned it.

How does a bounded task become a production release?

We start with real requests, not imagined ones. Twenty or thirty things people actually ask, sorted into answer, refuse, and escalate. That list becomes the evaluation set, and it is the single most useful artefact of the whole engagement — it outlives the prompt, the model and often the vendor.

A prototype then tests the approach against your own material rather than a public benchmark. This is where retrieval quality gets settled: whether chunking suits your documents, whether citations point somewhere a reader can verify, and whether the honest answer to a large share of questions is "I don't have that".

Production adds what a prototype can skip. Permissions that mirror the host application, observability that lets you see what was retrieved as well as what was said, rate and cost controls, and a documented procedure for changing models, prompts or sources without shipping a regression.

We have built the pieces this depends on. MarketSync parses trading signals with a fast-path parser and a language-model fallback, because most messages do not need a model and the ones that do should not block the ones that do not. ProTrader's coach is given a member's own computed numbers and guardrails that refuse trade recommendations outright. Axon Wallet lets a model propose a typed intent and nothing else — the intent then passes the same schema validation, policy limits and two-factor gate as any other API call. In each case the interesting engineering is the boundary, not the prompt.

What drives the cost of an AI integration?

Source quality first. Clean, current, well-structured material makes retrieval look easy; a decade of inconsistent documents makes it the bulk of the project. An honest estimate needs a look at the real corpus, not a description of it.

Then provider terms — pricing per token, data-retention policy, rate limits, and whether the model you want is available in the region you need. These belong in the architecture conversation on day one, because a retention clause can rule out an approach entirely.

Then the action surface. Reading is cheap to make safe. Writing is not. Every tool the model can call needs a schema, a limit, and a decision about whether a person approves it, and that work scales with the number of actions rather than the number of prompts.

Finally, prompt injection. Any system that reads untrusted text and can then act on it needs to assume that text is trying to steer it. Where a model can move money or change records, we design for that assumption rather than patching it later.

Who owns the result?

You do. Prompts, evaluation sets, retrieval configuration and operating runbooks are handed over, and the proposal says so explicitly rather than leaving it to be discovered at the end of an engagement.

We prefer arrangements you can operate without us. That means model access under your own account where practical, documented procedures for the routine changes, and an evaluation set your team can run. Provider abstraction can reduce the cost of switching later, but it does not remove it — models differ in behaviour, tool support, latency and output shape, so any switch still needs testing against your acceptance criteria.