Choosing an AI development partner is a strange purchase: the quotes differ by 5x, every website says “cutting-edge,” and by definition you cannot fully evaluate the expertise you are buying — if you could, you would not need to buy it. We are a vendor in this market, so read this knowing that — but the checklist below is the one we would use ourselves, and it disqualifies plenty of vendors who would love your budget. Sometimes it will disqualify us.
Start with what you’re actually buying
“AI development company” covers at least four different products, and mismatches cause most of the horror stories:
- Strategy — figuring out what is worth building at all.
- A project — a defined system, built and handed over.
- Capacity — senior engineers embedded in your team.
- A product build — an MVP taken to launch.
Know which one you need before comparing anyone. A firm that is excellent at enterprise projects may be a poor fit for a scrappy MVP, and vice versa.
The questions that separate builders from resellers
Ask every candidate these, and listen for specifics:
“Show me something you’ve shipped that’s still running.” Not a demo, not a slide — a system with real users, and what its accuracy or uptime looks like today. Teams that ship their own products (we keep ours here) or can walk through live client systems pass instantly. Teams that can only show mockups are selling their first attempt — to you.
“What happens in week two when it gives a wrong answer to a real customer?” The answer reveals whether they think in demos or in production. You want to hear: evaluation suites, guardrails, monitoring, escalation paths, rollback. You do not want to hear: “the model is very accurate.”
“How will we measure whether this worked?” Good vendors push you to define success as a number before they quote. A vendor who accepts “make it smart” as a spec will also accept it as a deliverable.
“What would you NOT build here?” The best predictor of a good partner is hearing “honestly, that part doesn’t need AI” in the sales conversation. A vendor with no opinion about what to skip is a vendor optimising for scope.
“Who exactly will do the work?” Agencies routinely sell the senior team and staff the junior one. Names, not roles — and meet them before you sign.
Reading the proposals
Quotes that differ wildly are usually scoping differences in disguise. One includes evaluation, monitoring and a handover; the other is a demo priced as a product. Force the comparison: line up what each quote includes against what production AI actually requires, and re-ask the cheap vendor about the missing lines. Sometimes the expensive quote is the cheap one.
Red flags, in rough order of severity: guaranteed accuracy figures before anyone has seen your data; “AI” proposals with no mention of your data at all; IP terms where the vendor keeps rights to what you fund; a proposal that starts with a platform instead of a pilot; timelines with no working-software milestone in the first month; and case studies that are all logos, no numbers.
Green flags: a prototype phase with an explicit kill decision built in; a definition of done you both signed; named engineers; weekly demos; your repositories from day one; and references you may actually call.
The trial that beats every reference call
If the engagement is significant, start small on purpose: a 2–6 week paid prototype against your real data (typically $10k–$35k at market rates). You learn how they scope, how they communicate, and whether they hit numbers — for a twentieth of the platform budget. Any vendor who resists starting small is telling you something; believe them. This is how our own engagements start, precisely because it keeps both sides honest.
The decision, compressed
Match the vendor type to what you are buying. Demand shipped evidence, not decks. Define success numerically before quoting. Prefer the partner who tries to shrink the project. Start with a prototype. And weigh the in-house vs agency question honestly — sometimes the right answer is a hire, and a good vendor will tell you so.
Evaluating partners right now? Send us what you’re trying to build — worst case, you get a free second opinion and a sharper checklist for whoever you pick.