
Before You Sign Off on AI: The Real Cost and Security Questions to Ask
Originally published at https://benlobaugh.medium.com/before-you-sign-off-on-ai-the-real-cost-and-security-questions-to-ask-4e2b62db7579
What leaders should settle before AI reaches their product
Imagine a product leader watching an impressive AI demonstration. The feature works beautifully, everyone sees the potential, and funding is approved before the meeting ends.
Three months after launch, two messages arrive in the same week. The first is a usage bill far higher than expected. The second is a customer asking where their sensitive data goes when the feature runs.
Could you answer both questions today?
I have led teams through AI adoption, and one lesson has stuck with me: getting the technology to work is often the easy part. The harder work is deciding how to use it responsibly, sustainably, and with confidence.
A demo shows what is possible under ideal conditions. It does not tell you what the feature will cost at scale, how it will behave when things go wrong, or what risks you are accepting. That is not a criticism of demos. It is simply a reminder that what is possible and what is ready for your customers are two different things.
I am not suggesting we hold back on AI. I want companies to find meaningful ways to use it. But before you approve that next AI feature, there are a few questions worth working through.
1. Start with the problem, not the demo
What problem does this feature solve for your customers, and how will you know it is helping? Just as importantly, what happens when it gets something wrong, becomes unavailable, or needs to be withdrawn?
AI can produce answers that sound convincing and are still incorrect. Decide how you will detect errors, how users can report them, and when a human needs to review an answer before it reaches a customer.
Plan for outages, too. If a vendor's service fails or reaches a usage limit, your product needs a fallback. That might be a manual process, a simpler non-AI experience, or a queue that resumes when service returns.
Finally, make sure you can retire the feature if costs, quality, or risk make it impractical to continue. Vendors change models and terms, too, so do not assume a replacement will behave the same way.
Once customers depend on a feature, it becomes a commitment. Make sure you understand that commitment before you make it.
2. Understand the economics before you scale
Traditional software can be expensive to build but relatively inexpensive to serve. AI can change that equation.
Every interaction with an AI service can carry a cost. As usage grows, so can your bill. We have seen AI usage drive development costs sharply upward, with some organizations facing bills in the hundreds of thousands of dollars. Now imagine those costs coming not just from developers experimenting with AI, but from customers using your product every day.
Before launch, estimate the cost of a typical interaction and a more demanding one. Model expected usage at pilot scale, normal growth, and peak demand. Then decide what you are willing to spend, how you will monitor usage, and what should happen when you approach your budget limit.
Do not assume today's pricing or model behavior will remain unchanged. Vendor pricing and tokenization methods can change, affecting your costs even when your product has not changed.
If you are charging a flat subscription fee, consider whether that price still makes sense for customers who use the AI feature heavily.
A forecast with one number is a wish with a decimal point.
Build realistic scenarios, understand the assumptions behind them, and assign someone to monitor the actual costs after launch.
3. Treat security as a design decision
Security is not a phase that follows the build. It determines what you are permitted to build.
Start by mapping the data flow. What information leaves your environment, where does it go, who can access it, and how long is it retained? Decide what data the feature is allowed to handle, and enforce those boundaries in the system itself rather than relying on a policy document.
In one of my own rollouts, the first rule was that no customer data could reach a public AI service. Establishing that boundary early gave the team a clear constraint to work within. It is much harder to impose a rule after a project has been built around a different assumption.
Review vendor terms carefully, including data retention, model training, subprocessors, and storage locations. Confirm your contractual and regulatory obligations with the appropriate experts.
You will also need to account for risks specific to AI. Prompt injection, for example, occurs when crafted text tries to manipulate an AI system into following an attacker's instructions. A web page or email the feature is asked to analyze could contain such instructions. Data leakage and excessive permissions granted to tools or integrations are other concerns.
Apply the principle of least privilege: each integration should receive only the access it needs. Test the feature, seek independent security expertise where appropriate, and revisit the review as the system changes.
Useful starting points include the NIST AI Risk Management Framework and the OWASP Top 10 for LLM Applications, though neither replaces a security review tailored to your product.
For a closer look at third-party connector risks, see my article, Dear Model, Don't Tell the User: What MCP Servers Whisper Into Your Agent's System Prompt.
4. Keep a human accountable
AI can generate answers, make recommendations, and carry out tasks, but people remain responsible for the product they ship.
Define who owns quality, cost, and security. Decide when human approval is required, who responds to incidents, and who has the authority to disable the feature.
Make the review process practical. If it adds so much friction that people routinely skip it, it will not provide the protection you intended. Accountability means giving people both responsibility and the authority to act.
5. Roll out gradually, and prepare your people
A sailor does not cast off the dock lines simply because the morning looks promising. You check the forecast, inspect the rigging, and make sure you have a way back to harbor if conditions change. An AI launch deserves the same care.
In one rollout I led, we started with AI-assisted analysis that had no access to code. From there, we moved to small, contained tasks and expanded only as trust was earned.
That approach gave the team room to learn and helped us surface problems while they were still small. Each stage should have clear criteria for moving forward, including acceptable error rates, costs, and compliance with data-handling rules.
Prepare your people, too. Developers need to understand what information they can share. Reviewers need to know how to check AI-generated work. Support and sales teams need to understand the feature's limitations and what they can promise customers.
Give people training, clear expectations, and room to ask questions. Teams that feel informed make better decisions than teams that feel pressured.
The goal is not to move slowly for the sake of caution. It is to build the confidence and controls that let you move forward reliably.
6. Ten questions for the executive table
Before signing off, make sure your team can answer these questions:
- What customer problem does this solve, and how will we measure success?
- What does a typical interaction cost?
- What will the feature cost at pilot scale, expected growth, and peak demand?
- What is our budget ceiling, and what happens when we reach it?
- What data does the feature access, and where does it go?
- What do the vendor's terms say about retention and training on our data?
- Have we reviewed risks such as prompt injection, data leakage, and excessive permissions?
- Who is accountable for cost, security, and incident response?
- What happens when the feature is wrong or unavailable, or if we need to withdraw it?
- What must be true before we move from one release stage to the next?
If the room goes quiet at any question, that is useful information. It tells you where you need a better answer before committing to the launch.
Before you sign off
AI offers real opportunities to build better products and solve problems in new ways. Those opportunities are worth pursuing.
But the best outcomes do not come from adopting AI simply because the technology is impressive or the competition is moving. They come from understanding what you are building, what it will cost, what risks you are accepting, and how you will take care of the people who depend on your product.
You do not need to predict every problem. You do need a plan for recognizing problems, responding to them, and changing course when the evidence calls for it.
Before you approve the feature, understand its costs, address its risks, and name the people accountable. Then give your team the room to build something genuinely useful.
------------------------------------------------------------------------
Sources and further reading
- NIST: AI Risk Management Framework
- OWASP: Top 10 for LLM Applications, 2026
- European Commission: AI Act implementation and enforcement
- Ben Lobaugh: Transitioning to an AI-First Engineering Organization: A Practical Overview
- Ben Lobaugh: AI Is Not a Magic Wand: Thoughtful Use of AI to Enhance Your Organization