AI Is Not a Hammer: Knowing When to Use Code, AI, and Humans
In many organizations, AI has become the new hammer, and now every problem looks like a nail. In reality, it serves as a powerful reasoning layer that still depends on the quality, structure, and scope of the data you give it.
If you don't give it well-structured, well-scoped data, it falls back on a mix of training data and whatever it can pull from the public web. That view is often shallow or incomplete. In many cases, AI systems are working off search results or indexed snapshots of pages rather than a deeper, up-to-date view of the underlying content. And this content issue is becoming more severe as more and more websites and CDNs (Cloudflare in particular) block AI crawlers by default.
The Context Window Problem
There is also a context window issue. If you dump a lot of unstructured text into a prompt (including website content), the model will hit its context limit and only see a limited set of messy data. Without clean structure and a focused question, it spends most of its "attention" on essentially noise.
When AI is used alone, without enough relevant data, that's when you see hallucinations and very surface-level analysis. The model is forced to guess across a narrow slice of information instead of reasoning over a well-defined problem with enough signal.
When AI Works Best
From my experience as a developer, AI is most powerful when you pair it with a well-scoped question and high-quality structured data. But it is not a solution for every problem. Sometimes a script or algorithm is better because it is deterministic, faster, and cheaper. Other times you need a human-in-the-loop process because the problem is too ambiguous, the data is too messy to parse reliably, or the outcome is too high risk.
The real skill isn't "using AI for everything." It's knowing when to use code, AI, and humans, and how to ground them in the right data.
A Framework for Choosing
I sketched the table below to capture this thinking:
| Task Type | Code (scripts / algorithms) | AI models (reasoning layer) | Humans-in-the-loop |
|---|---|---|---|
| Clear, deterministic rules | Best | — | — |
| Pattern recognition / reasoning | — | Best at scale | Helpful for validation |
| Messy or unstructured data | Helpful for preprocessing | Helpful for structuring | Best |
| Ambiguous / subjective judgment | — | Helpful for framing | Best |
| High-stakes decisions | Helpful | Helpful for decision support | Required |
| Complex, real-world workflows | Each plays a distinct role | Each plays a distinct role | Each plays a distinct role |
The Bottom Line
AI is a powerful tool, but it's not the only tool. The organizations that get the most value from AI are the ones that understand its limitations and pair it with the right data, the right guardrails, and the right human oversight.
I'd love to hear others' reactions. Feel free to with your thoughts.