The most useful decision in an AI-First build is deciding where the AI does not go.
Post 5 in a weekly series focused on building MyAcumen from the ground up using AI-First principles. The decisions we made, why we made them, and how we went about it. Aimed wider than my usual sales audience.
Ask the same public AI model the same question twice and you can get two different answers. That’s charming in a chat window. It’s useless when somebody has built a development plan on the first one.
So scoring never touches a model. Fixed rules, held as named constants, applied identically every time. Two people who answer the same way get the same result, and if your score moves next time out, it moved because you did.
Interpretation is where the AI is allowed in, and that’s fenced carefully.
One instruction governs how every finding gets written. What earns a place in the report, what has to be quoted, and a hard limit on length. A finding always quotes the real score, never the band, because 41 and 69 sit in the same band and tell completely different stories.
A list of banned phrases is built into the prompt itself. That’s how the writing stays in a coaching frame rather than sliding into the language of a warning letter. “Work with those two” passes. “Address these underperformers” doesn’t.
A second pass then reads that output back against the voice and evidence rules, and flags what it breaks, ranked by how badly.
Faster models do the sorting and the structuring. The strongest model is kept for the writing a customer actually reads.
Then a checklist runs before release. My favourite test on it: take a finding, and ask whether it would read the same if the number were at the other end of the same band. If it would, the engine has slipped back into reading bands instead of scores, and it goes back.
None of those rules arrived right first time, and one run in particular was miserable. Eight sections came back failing quality control and we spent a long time assuming the model had gone off. It hadn’t. Our own instructions were arguing with each other, because one page banned a phrase while another used that same phrase as a worked example of good writing, so the model could not win either way. We read every failure line by line and hardened the rules against what came back.
What all of it offers the person using it is simple. Come back in a month and you get the same reading you built your plan on, because the write-up is tied to the exact answers behind it rather than made fresh each time you open it. Hand it to somebody else and it says the same thing to them. Act on it this quarter and it will still stand up next.
Worth asking of any AI tool being sold to you at the moment. What happens if you run it twice? Do you get the same answer, or just a different one?
