The Enterprise Inference Vendor Checklist

Every vendor evaluation we have watched go wrong made the same mistake early: it started with a benchmark and ended with a surprise. Benchmarks measure what models can do; procurement is about what vendors will do — with your data, your bill, and your exit. Those are different questions, and only one of them is on a leaderboard.

This is the checklist we would bring to any inference vendor call, including a call with us. It is organized as five areas, each with the questions to ask and what a good answer sounds like. Bring it to every finalist — the value is in comparing the answers, not in any single vendor's performance on it.

One note before the list: everything below is verifiable in documents or in a trial. If an answer only exists as a sentence said on a call, it does not exist. Ask for the page, the policy, or the status history.

The questions: Are prompts and completions retained, and for how long? Is any of it used to train models — yours or theirs? Where is data processed, and can you pin a region? What does deletion look like contractually?

Good sounds like: a written policy page that answers each of these in plain language, a default of not training on customer data, and a named contact who responds when legal sends the hard version of these questions. Vague sounds like: "industry-standard practices" and a link to a marketing page.

Our answers live where they should: the privacy policy is the document of record, and we would rather you read it than take this post's word for it.

The questions: Are rates published per model, or do you need a sales call to learn them? When the vendor claims savings, is the figure dated, sourced, and quoted model by model — or is it one blended average? Can you attribute spend by key, workspace, or team without exporting to a spreadsheet? Is there a committed-use path once your baseline is stable?

Good sounds like: a public price list you found before anyone contacted you, comparison claims with an as-of date and links to the other side's pricing, and billing exports that map to how your org chart actually looks. Our pricing page and dated comparison snapshot are our answer to the first two; judge them the same way.

On savings claims specifically: when a vendor quotes one, ask for the range and the conditions, not the hero number. We publish a forty-five to seventy percent reduction range for adaptive routing on our homepage, with the mechanism attached — a range with a mechanism is a claim you can test, while a single average is a lobby poster. Our invoice teardown shows where the waste that range feeds on usually hides.

The questions: Is there a public status page with real incident history, not a permanently green badge? What happens to an in-flight request when capacity fails — retry internally, or return an error to your code? What are the rate limits in writing, and what happens at the boundary? Can you get latency numbers measured on your workload during the trial, rather than theirs?

Good sounds like: failover described as architecture, with the boring specifics of what a caller sees; status history that includes incidents, because a vendor with no visible incidents is a vendor with no visible status page. Everyone has failures; only some vendors let you watch how they handle them.

Reliability is also a cost line, though it never appears on one: every provider error your code has to absorb is a retry you pay for. An endpoint that fails over internally is quietly cheaper than an identical one that returns errors honestly.

The questions: How broad is the model catalog, and how fast do new models arrive after release? Is the API OpenAI-compatible, so your SDK code survives a switch? And the one nobody asks on the first call: what does leaving look like?

The catalog question matters because model choice is a cost lever that renews monthly — a vendor whose catalog tracks the frontier lets you re-decide without re-platforming. Compatibility matters because it caps your switching cost at an afternoon, which we covered in detail in what "OpenAI-compatible" actually buys you.

The exit question has a sharp edge for fine-tuning: if you train a model on a vendor's infrastructure, do you own the weights, or does the vendor? A fine-tune you cannot take with you is the deepest lock-in in the industry. Our answer is structural rather than rhetorical: fine-tuned weights belong to the customer, which is why our fine-tuning docs spend their effort on training, not on explaining export restrictions.

The questions: What channels exist, and what response expectations are written down rather than implied? Who answers — engineers who can read a stack trace, or a tier whose job is to apologize? During the trial, open a real ticket with a real question; the response is the most honest demo you will get.

Good sounds like: response expectations in writing, support staff who can discuss rate limits and model behavior without escalating every question, and documentation that answers the common cases without a ticket at all. Our support docs state our channels and expectations; again, the checklist's rule applies to us — written down beats said warmly.

Support is the area where trial behavior and production behavior diverge most. If the trial answers arrive in minutes, ask whether that tier continues after signature, and get the answer in the contract.

The questions: How long is the commitment, and what happens at renewal — a conversation, or an auto-rollover with a quiet price change? How much notice does the vendor give before changing rates, and do existing workloads get grandfathered? Do credits expire? If you prepay, what happens to the balance when you leave?

Good sounds like: rate changes announced in advance, in writing, with the current rates always visible on a public page so you can diff them yourself. Bad sounds like: a discount that is contingent on not asking about the list price, and a renewal clause your legal team finds in section nineteen.

The quietest commercial trap is the expired credit. Generous trial credits with a short fuse create a migration decision under time pressure — precisely when evaluation discipline slips. Treat credits as a way to run the trial, never as a reason to shorten it, and note the expiry date on the scorecard next to everything else.

Weight evidence over adjectives. "Enterprise-grade" is an adjective; a dated price list, a status page with scars, and a written retention policy are evidence. Every area above has a document that settles it, and a vendor evaluation is mostly the exercise of collecting those documents and noticing who hesitates.

Then run the trial properly: the same real workload against every finalist, graded on the acceptance checks your product already uses, with cost per completed task as the scoreboard rather than per-token price. The vendor who encourages exactly this test is telling you something, and so is the vendor who steers you toward their benchmark instead.

Keep the scorecard after the decision. The same checklist is how you review the incumbent a year later, and incumbents should not be exempt from evidence either.

If the workload is a prototype, a hackathon, or a side project, pick anything OpenAI-compatible with a generous free tier and go build. The checklist is procurement infrastructure; it earns its keep when the bill is material, the data is sensitive, or the dependency is becoming structural.

If your usage is genuinely single-model and single-purpose, several areas above collapse to one line — you do not need a catalog strategy for a workload with one model in it. Checklist discipline scales with stakes.

And if you are evaluating us: we would rather be scored by this list than skipped for lacking a salesperson. The answers that matter are already published — pricing, privacy, support expectations — and the trial is an API key away.