Build vs. buy: evaluating AI vendors without the hype

Almost every vendor pitch in this space now includes the word "agentic" somewhere in the first slide. That's not a reason to distrust the product, but it is a reason to have a fixed set of questions you ask regardless of how good the demo looked — because the demo is the part every vendor has optimized hardest.

This isn't a build-always or buy-always argument. It's the set of questions we actually walk clients through before a vendor contract gets signed, organized around the three places these deals tend to go wrong later.

Start with what you're actually deciding

"Build vs. buy" is usually the wrong framing for the first conversation. The real question is narrower: does this specific capability need to be a durable, differentiating part of your product, or is it commodity infrastructure everyone needs and nobody should be maintaining themselves? Commodity capabilities — document parsing, a base LLM, vector search — lean toward buy almost by default. Anything that touches your actual competitive edge deserves more scrutiny before you hand it to a vendor's roadmap.

Data handling: the questions that matter more than the pricing page

• Where does our data go, specifically? Not "we take security seriously" — which subprocessors touch it, is it used to train models beyond your own instance, and can you get a straight answer about retention.
• What happens to embeddings and derived data — the vectors, summaries, and intermediate artifacts a system generates from your source data — when the contract ends? These often aren't covered by the same deletion clause as the raw data.
• Can you audit what the system actually did with a given request, after the fact? If the answer is a shrug, that's a governance problem you're importing, not solving.

Lock-in: harder to see than the contract term

The multi-year contract is the lock-in everyone negotiates over. The lock-in that actually costs you later is usually invisible at signing time.

• Proprietary formats and workflows. If your team's prompts, fine-tuning data, or workflow logic live entirely inside the vendor's proprietary format, migrating later means rebuilding, not exporting.
• Integration depth. The deeper a vendor's system reaches into your internal tools, the more a future migration looks like a second implementation project rather than a vendor swap.
• Ask for the exit path at signing, not at renewal. A vendor confident in their product will answer "how do we leave, and what do we take with us" without treating it as an adversarial question.

Evaluation: don't accept their benchmark

Every vendor has a benchmark where they win. That benchmark is rarely built on your data or your task.

• Test on your own data, before committing. A pilot on real, messy, internal data — not the vendor's curated demo set — is the only evaluation that tells you what production will look like.
• Define what "good enough" means before the pilot starts, with the metric your business actually cares about — not the vendor's default accuracy number.
• Check how failure is handled, not just how often it happens. A system that fails rarely but silently is often riskier in production than one that fails more often but tells you when it doesn't know.

The vendor's demo tells you what the system can do under ideal conditions. Your pilot needs to tell you what it does under yours.

Where "build" earns its cost

Building makes sense when the capability is close to your actual differentiation, when the data involved is too sensitive to hand to a third party regardless of their assurances, or when the commodity option genuinely doesn't fit your workflow and the gap is wide enough to justify the maintenance burden. Outside of those cases, buying and integrating well is usually the faster path to the same outcome.

Mid-evaluation on a vendor and want a second set of eyes on the contract or the pilot design? Talk to us before you sign.