StackCraft.by Chris Jack
Three slots, Q3 2026
Practice, 25 August 2025, 6 min read

Why I'd rather run a four-week pilot than write you a roadmap

A roadmap assumes you already know what's worth building. With AI you usually don't, and you find out much faster by building one small thing.

CTOs and operations leads ask me for AI roadmaps fairly regularly. I don't write them, and I'd gently push back on anyone who does. The companies I've watched actually get somewhere with this didn't follow a roadmap. They ran a small scoped pilot, learned something, and then ran the next one.

A roadmap assumes you know what's worth building before you've built anything. With AI you usually don't. The model behaves differently to a deterministic system, your data has gaps you'll only find by running into them, and your team has skill gaps you'll only see once they're using the thing. So plan a pilot instead.

The pilot I run has the same shape every time. Pick one workflow. Build something working in two weeks. Get it in front of a real user in week three. Measure it in week four and decide. That decision is one of three things: put more into it, change direction, or stop. All three are fine outcomes.

Picking the workflow is the call that matters most. A good pilot has high volume so you'll actually see it working, low judgement so the model has a fair shot, and a real owner so somebody cares about the result. If the user is everyone and the metric is engagement, you haven't got a pilot yet.

Two weeks of building isn't much, so scope it hard. One model call, one interface, one type of user. Resist adding a second model or a second user or a second feature. The job of the pilot is to find out whether the model can be useful here at all.

Week three matters more than the build does. A pilot nobody uses is just a demo. Real users find the failure modes you didn't think of and the useful cases you didn't expect, and both of those are worth more than anything you learn while building.

The measurement in week four needs to be honest. Not whether people enjoyed it, because they always say yes when the founder is in the room. Measure time saved, or errors avoided, or how many tasks got finished without someone stepping in. Pick the one that maps to what the workflow costs today and set the threshold before you start, not after.

If the pilot works, do another one. Different workflow, same shape, and it'll be cheaper the second time because everyone knows the rhythm. Most clients I've worked with are on their third or fourth before they consider anything bigger.

If it doesn't work, write down why. Was the model wrong, was the data thinner than you thought, was the workflow less routine than it looked, was the user not actually motivated? Each of those tells you something different about where AI fits your business, and that's a much better roadmap than one written in advance.

The biggest mistake I've seen is treating this like an enterprise software rollout. You don't need a steering committee or a maturity model or a forty-slide strategy deck. You need one small pilot, four weeks, and the honesty to keep going or stop based on what actually happened.

All articlesStart a scoping call
Three slots open for Q3 2026

Have a feature that needs shipping?

Start a projectHow I work