The layer that decides whether it still works in month six.
AI features do not usually fail at launch. They fail three months later, when the index is stale, the source system changed shape and nobody owns the refresh job. This is the unglamorous half of the build, and it is where the renewal is won or lost.
Scope this with us“Why is it giving answers from the old policy document?”
A production ingestion and refresh pipeline for one or two sources fits a 14-day sprint, and often runs in parallel with a RAG build.
Reliable ingestion
Connectors to the client's real sources — file stores, databases, ticketing systems — that handle the formats they actually have.
Incremental refresh
Changed documents get re-processed on a schedule or on event, so the index does not drift away from reality.
Monitoring and alerting
You find out a sync failed from an alert, not from your client forwarding a complaint.
Backfill and replay
When a parsing bug is found, the pipeline can reprocess history rather than requiring a rebuild from zero.
The parts that decide whether it survives contact with users.
- Content hashing so unchanged documents are not re-embedded, which is where most of the wasted spend sits
- Idempotent jobs safe to re-run after a partial failure
- Dead-letter handling for documents that fail parsing, surfaced for review instead of silently dropped
- Schema and drift checks on source systems, so an upstream change is caught early
- Embedding version tracking, so a model upgrade can be rolled out without corrupting the index
- Cost instrumentation per run, because embedding spend scales quietly with corpus growth
- Re-embed the entire corpus nightly and bill the client for it
- Run ingestion as a script on someone's laptop
- Drop documents that fail to parse and never tell anyone
- Leave the client with a pipeline only we know how to operate
Before you scope it.
Can this run on our client's existing infrastructure?
Usually yes. We prefer to deploy inside their cloud account or yours rather than adding another vendor relationship for the client to worry about.
Who operates it after handover?
Your team, with our documentation and dashboards — and we stay reachable for build issues. Nothing is designed to require us permanently.
How much does embedding actually cost at scale?
It depends on corpus size and refresh rate, and we will give you real numbers from the build rather than an estimate. The hashing and incremental logic exist specifically to keep that bill flat as the corpus grows.
Want to stop turning down AI projects?
Bring the scope you’re unsure about — an RFP, a client request, a half-quoted project. We’ll tell you what’s buildable, what it takes, and whether it fits in one sprint. No charge for the call.