The Hard Part of AI-Native Apps Is Not the AI

When you run a business for more than twenty years, you collect a long list of workflows that are necessary, repetitive, and more painful than they should be.

Certificates of insurance, or COIs, are one of those workflows.

They are not glamorous. But they matter. They show up in vendor onboarding, client relationships, compliance reviews, contract requirements, renewals, and risk management. When they are missing, expired, incomplete, or incorrect, someone has to stop what they are doing and chase the issue down.

The idea for COI Dash started in a practical place. I was talking with friends in the insurance industry about the COI process, and they confirmed what I had already experienced from the business side: it is painful.

There is manual review. There is back-and-forth. There is document handling, status tracking, and interpretation of details that live inside PDFs and email threads. There are rules, exceptions, dates, limits, endorsements, and missing fields.

This was not a vague “let’s add AI” opportunity. It was a focused operational problem: repetitive review, unstructured documents, structured data, human judgment, and workflow.

In other words, it was exactly the kind of problem where AI could help if it was applied carefully.

A demo is not a product

One of the biggest lessons from building COI Dash is that AI is not a magic bullet. It can feel like one at first. You can open a model, write a strong prompt, upload a document, and get something useful back.

That first moment is powerful. It can extract fields, summarize a certificate, identify obvious issues, and make the workflow feel solved before the product is actually ready.

A demo is not a product. A good answer from a model is not the same thing as a reliable workflow. A prompt that works on the clean example is not the same thing as a system people can trust with messy documents, edge cases, and real business consequences.

Production software is judged by messy cases, not the easy ones.

Narrow problems, messy inputs

COIs are a good example because the problem is both narrow and messy. The domain is specific enough that you can define what matters: policyholder, certificate holder, coverage types, effective dates, expiration dates, limits, endorsements, missing fields, and compliance requirements.

But the inputs are not always clean. Documents vary. Language varies. Formatting varies. Some certificates are straightforward. Others require judgment. Sometimes the system can confidently flag a problem. Other times it should slow down and ask for human review.

That is where AI-native applications get real. AI is very good at automation and document parsing, but it still needs structure around it: a product frame, rules, evaluation, and a clear understanding of where the system should act, assist, or defer.

That is why requirements matter more than ever.

Requirements matter more, not less

There is a common misconception that AI reduces the need for detailed requirements. I have come to believe the opposite. The more AI you put into a workflow, the more important the requirements become.

If you are building a traditional form or dashboard, weak requirements are already a problem. If you are building an AI-native workflow, weak requirements become a multiplier for confusion. The model may interpret vague instructions differently than the product team intended. Developers may build around assumptions that were never stated. Reviewers may disagree about whether the output is acceptable.

The antidote is not bigger documentation for its own sake.

The antidote is better definition.

For COI Dash, that means granular user stories. Not giant stories like “analyze COIs with AI.” That is too broad to build, test, or trust.

A better story describes the user, the moment, the expected behavior, and the acceptance criteria.

Who is using this? What are they trying to accomplish? What information should the system extract? What counts as missing or valid? When should the system flag an issue? When should a human review it? What should the system never decide on its own?

Those questions are not busywork. They are the foundation of a reliable AI-native product.

Strong acceptance criteria force the team to move from “the AI should understand the document” to something more specific: “Given this type of COI, when this field is missing or expired, the system should flag it in this way and explain why.”

That level of clarity helps the product team, the engineering team, and the AI itself.

It also helps avoid one of the easiest traps in AI product development: trying to automate too much too soon.

Start smaller than you want to

When you see AI do something impressive, the natural instinct is to expand the scope. If it can parse the certificate, maybe it can handle renewals. If it can handle renewals, maybe it can email vendors. If it can email vendors, maybe it can manage the whole compliance process.

But trying to do it all at once is where teams get into trouble.

My advice is to start smaller.

Pick the highest-value part of the workflow first. Find the place where automation creates immediate value and where success can be clearly defined. Build that. Test it. Watch how users respond. Learn where the edge cases are. Improve the workflow. Then expand.

The same principle applies to the team. When you are moving fast on an AI-native application, a small, tight-knit team can be a major advantage. The requirements are evolving. The workflow is being discovered. The AI behavior is being tested. The edge cases are showing up in real time.

In that environment, coordination matters as much as capacity. A small team with clear ownership, strong communication, and a focused scope can move faster than a larger team with overlapping responsibilities.

If I were starting COI Dash again today, I would spend even more time up front on planning and requirements.

Not months of theory. Not a bloated specification. Just more discipline around defining the first version: the specific user, the highest-value workflow, the core document types, the required outputs, the human review points, and the acceptance criteria that would tell us whether the system was working.

AI makes it tempting to widen the ambition early. But the better path is to earn expansion. Start with the feature set that matters most. Build a cadence. Learn from real usage. Improve the system. Then add the next layer.

AI-native products are still products

COI Dash reinforced something I already believed, but now believe more strongly: AI-native products are still products.

They still require judgment, clear requirements, thoughtful design, careful testing, and disciplined execution.

AI can make the work faster. It can make painful workflows easier. It can unlock automation that would have been too expensive or too slow to build a few years ago. But it does not remove the need for product thinking. In many ways, it raises the bar.

The teams that win with AI-native applications will not be the teams that throw the biggest prompts at the broadest problems. They will be the teams that choose focused problems, define them clearly, start small, test carefully, and expand with discipline.

That is the lesson I am taking from COI Dash.

The hard part is not getting AI to do something impressive.

The hard part is turning that impressive moment into a reliable product people can actually use.