Written for Procurement, security, operations and the sponsors of an AI purchase or a build.
What's in this scorecard
This 25-page scorecard is a way to choose that is not a reaction to a demo. Most AI purchases are decided in a room where a product has just been shown at its best. The questions that should have come first — what the job is, whether your data will be used to train a model, whether you can leave — are asked later, or not at all. This scorecard is the work that should happen before, during and after that room.
It is fifty-two criteria, in six dimensions, with a default weight for each, summing to 100 percent; and six must-pass gates that a total is not allowed to outvote. Evidence first: a demo, a logo or a brochure sentence is not evidence of production quality. Build is an option, scored on the same card, on the same three-year basis, alongside a reasoned “do nothing.” Changing a weight after a demo needs a reason, written down.
It is written for procurement, security, operations, and the sponsors of a purchase or a build. It assumes a defined job, not a shopping trip. We are vendor-neutral: a recommendation can be a tool you already own, a new product, a custom build, or no build at all, and we explain why. A companion workbook holds the 52 criteria, the weights, the gates, the questions and an illustrative evaluation of four hypothetical options. Yellow cells are inputs. Dimension and overall scores, and a gate result, are formulas. A completed workbook is a recommendation tool, not a certificate.
Gates first, then the total
Apply the six gates first. They live in the security and data-handling dimension (V19–V28): training use, agreed in the contract or forbidden; encryption at rest and in transit, on the path the data actually takes; identity — single sign-on or equivalent, least privilege, no shared vendor logins, vendor access named and revocable; an audit trail you can export; export and deletion of prompts, outputs, embeddings, logs, configurations and fine-tunes, in documented formats; and residency and subprocessors — a list, a jurisdiction, a notice of change, and a path if a location is not acceptable.
A failed gate is “Do not proceed,” unless a named executive accepts residual risk in writing, with the residual risk named and a date to look again. It is not a note in the margin. A pattern of accepted fails is a different risk appetite, and should be described as such, not as a successful evaluation.
The remaining dimensions cover strategic fit, capability and evidence, integration depth, support and operations, and total cost, commercial terms and exit — including a three-year view, lock-in named, and Build on the same card. How to read the result is a recommendation, not a ranking: the number serves the recommendation. The LLM security paper and the offline guide in this library are companions for the path and the hosting decision. This scorecard will not stand in for them.
Six must-pass gates (V19–V24)
These are the ones a total is not allowed to outvote. A vendor default is not an agreement.
V19
Training use. Agreed in the contract, or forbidden. A vendor default is not an agreement.
V20
Encryption. At rest and in transit, on the path the data actually takes, including logs, backups and subprocessors.
V21
Identity. Single sign-on or equivalent; least privilege; no shared vendor logins; vendor access named and revocable.
V22
Audit trail. Access and automated or model actions, in a form you can export.
V23
Export and deletion. Prompts, outputs, embeddings, logs, configurations and fine-tunes, in documented formats, with a deletion path.
V24
Residency and subprocessors. A list, a jurisdiction, a notice of change, and a path if a location is not acceptable.
If a gate is failed and someone still wants to proceed, that is an executive decision, in writing. It is not a courtesy.
How the card is meant to be used
Fifty-two criteria, weighted 100 percent. Build is on the same card. Evidence first.
A defined job
If the job is not frozen, later scores are a reaction to a presentation. Strategic fit asks whether the product, the build or the choice to leave things as they are is judged against a named outcome.
Evidence, not a demo
Every score has evidence. A demo, a logo or a brochure sentence is not evidence of production quality.
Build, and do nothing
Score a Build option and a reasoned “do nothing” on the same card, on the same three-year basis. A purchase that cannot be compared to building, or to stopping, is not finished.
Get the guide and workbook
Enter your name, work email and company. The files appear on this page immediately. A backup download link is also emailed when email delivery is available.