R&D operations

Vendor and architecture review for AI projects: what to look for

What to examine in an AI vendor proposal, the architecture questions worth asking before you commit, and the red flags that reliably signal delivery risk.

Most AI vendor proposals are written to pass review, not to enable it. They are detailed where detail is comfortable and vague where the real decisions are. Reviewing one well requires knowing what the proposal is not saying and where to push.

What to look for in a vendor proposal

A clear technical approach, not a capabilities list. A good proposal describes what the vendor intends to build and why, not what their platform can do in general. If the proposal reads more like marketing material than an engineering plan, it is because nobody has done the design work yet. That work gets done later, on your time and budget.

Specific performance claims with specific conditions. “Our model achieves 94% accuracy” is meaningless without knowing on what data, with what preprocessing, against what baseline. Ask: on what evaluation set was this measured, who labelled it, and is that data representative of our use case? If the answer is vague, the claim is vague.

Explicit data requirements. The proposal should say what data is needed, in what volume, in what format, and what the vendor’s plan is if the data is less clean or less plentiful than expected. AI projects most commonly fail at the data layer. A proposal that treats data as a given has not thought through execution.

A realistic timeline with dependencies stated. Timelines that compress the hard parts (model development, evaluation, iteration) and expand the easy parts (meetings, documentation, integration) are a red flag. Ask the vendor to walk through the timeline week by week and explain what would cause each phase to slip.

A clear handover plan. What happens when the engagement ends? Who maintains the model? What does the company own? A vendor who cannot answer this clearly is either planning to create dependency or has not thought about it.

Architecture questions worth asking

Before committing to a technical direction, the following questions are worth getting written answers to:

Build vs buy: Are we using a commercial API, fine-tuning an open-source model, or training from scratch? Each has different cost, performance, ownership, and lock-in profiles. Has the vendor compared these options for our specific use case, or are they proposing their default?

Data governance: Where does our data go during training and inference? Which third-party providers does it pass through? What are the retention and deletion policies? For regulated industries, this is non-negotiable.

Evaluation methodology: How will we know if the model is working? What is the evaluation dataset, who controls it, and how was it constructed? A vendor who is also responsible for constructing the evaluation set has an obvious conflict of interest.

Latency and infrastructure: At what scale does the system need to operate, and has the architecture been validated at that scale? A model that performs well in a demo environment often performs differently in production under concurrent load.

Fallback and error handling: What happens when the model is wrong? How are low-confidence outputs handled? An AI system without a well-designed fallback path is a liability in production.

Red flags that signal delivery risk

References that all describe project types different from yours. A vendor with strong references for document classification may have never successfully delivered a real-time inference system. References only count when the project type matches.

“We will adapt our existing platform.” Adaptation is frequently shorthand for “we will discover, mid-project, that our platform does not support your requirements and will spend two months building it.” Ask what has been built before versus what will be built new, and for whom.

Success metrics defined by the vendor alone. If the vendor is both delivering the work and defining what success means, there is no accountability surface. Insist on jointly defined acceptance criteria before work begins.

No prior failures mentioned in the conversation. Every AI project encounters setbacks. A vendor who cannot describe a project that went differently than planned, and what they learned from it, either has no real track record or is not being candid. Either is a problem.

Pricing that does not reflect the unknowns. A fixed-price proposal for genuinely uncertain technical work is an optimism statement, not a plan. Either the vendor has under-priced the risk and will renegotiate later, or they have priced in a large contingency you cannot see. Both are worse than an honest time-and-materials or milestone-based structure.