I am pro-AI in finance work. I use it every day and have written about the exact prompts I run. But there is a version of “AI first” that gets people in trouble, and it usually shows up as the CFO asking the model a question the model is structurally bad at answering, then acting on the answer.

Here are the six categories where I have watched it go wrong, or nearly go wrong. Learn from other people’s mistakes so you do not need to make your own.

1. PII, PHI, or wage detail without a private tenant

Failure mode: Someone in FP&A pastes a payroll register into a personal ChatGPT to build a comp analysis. Individual salaries by name are now in a consumer product’s logs. In a regulated industry (healthcare, financial services, government contracting) this is a reportable incident.

Do this instead: Only paste PII, PHI, or individual wage data into a tenant that is contractually covered by a DPA and marked “do not train on customer data.” Enterprise tenants of Claude, ChatGPT, and Copilot all meet this bar in 2026. Personal accounts do not. See the CFO LLM Privacy Guide for the checklist.

An even safer pattern: rename before you paste. If you are looking at comp bands you do not actually need names. Employee_001, Employee_002 is enough.

2. Anything unverifiable where you are treating “confidence” as accuracy

Failure mode: “What was the average EBITDA multiple for {niche software subvertical} deals in 2024?” The model returns a specific-sounding answer. You put it in a memo. Two weeks later a partner asks for the source and there isn’t one. The answer was plausible but fabricated.

Do this instead: For any factual claim that will land in a memo, deck, or email, either (a) provide the source in the prompt yourself, or (b) require the model to cite a URL you can open. If it cannot cite, treat the answer as a hypothesis, not data.

The tell: the more specific and confident the number, the more you should verify. A hedged “roughly 8-12x for that segment historically” is more trustworthy than “the median was 10.4x in Q3 2024.”

3. Legal interpretation of a contract

Failure mode: Paste an MSA in, ask “is this indemnification clause standard?” Get an answer that reads like a first-year associate summary. Sign or push back based on that answer. Miss a carve-out that changes your exposure.

Do this instead: Use the LLM to spot flags and generate questions for counsel. Not to make the call. A useful prompt: “Read this MSA. List the 5 clauses that most commonly get negotiated in a similar deal and my current position on each. Then list the 3 clauses where I should ask outside counsel a specific question, and write the question.” That output goes to your lawyer. Their answer goes into the negotiation.

The line: the model is fine at pattern recognition. It is not fine at interpretation with legal consequences.

4. Revenue recognition judgment on ASC 606 edge cases

Failure mode: “For this multi-element contract with variable consideration and a customer option, how should I recognize revenue?” The model writes a confident memo. Your auditor rejects it in October. Now you are restating.

Do this instead: The model can help you organize the facts (what are the performance obligations, when does control transfer, what is the transaction price, what is variable). It can help you draft the memo structure. It cannot replace the accounting judgment that has to sit with you or your Controller, ideally reviewed by your audit firm’s technical group for anything non-standard.

Rule of thumb: if the answer might get challenged by your auditor, it should have a human’s name on it, not the model’s.

5. Board-facing forecasts without a human owning the numbers

Failure mode: “Extend my forecast through 2027 with a reasonable ramp.” The model produces a curve. It ends up in the board deck. A director asks “why did you pick 22% growth in H2 2027” and the answer is “the model suggested it.” That is the last time the director trusts your forecast.

Do this instead: Every number in a board-facing forecast should have a person and a rationale behind it. AI is fine for the mechanics of building out the schedule. It is not fine for the assumption layer. Your CRO owns bookings. Your COO owns delivery. You own the composite and the sensitivity. The LLM owns nothing.

Practical test: for every line item, could you defend the assumption without referencing “the AI recommended it”? If not, do not put it in front of a board.

6. Anything where memory could leak across sessions or users

Failure mode: You share a Custom GPT with your team that has “memory” enabled and personal-tier access. One analyst asks a question that includes a customer name and pricing. A second analyst on a different account asks a general pricing question the next week. The model surfaces the first analyst’s customer detail because memory is user-scoped but the shared tool config is not.

Do this instead: Turn memory off on any shared tool. Use isolated Projects or workspace-scoped Custom GPTs. Assume that whatever any user pastes is potentially visible to any other user of the same tool unless the platform explicitly says otherwise. When in doubt, check the admin controls and read the settings, not the marketing page.

Summary table

The one question that outed a candidate CFO in an interview

A partner I know was interviewing a CFO candidate who claimed to be “AI-native.” He asked one question: “Tell me something you asked an LLM last week that you wouldn’t want a board director to know you asked.”

The candidate paused and said, “honestly, everything I ask it is board-appropriate.” The partner passed on him the next day. Not because the answer was suspicious. Because the answer meant the candidate wasn’t actually using AI for real work. Anyone using LLMs seriously has asked something clumsy, something exploratory, something they’d rather not project on a screen. Denying it is a tell.

The corollary for you: use AI enough that you have a “wouldn’t want to project” answer. Just make sure it is not one of the six things above.

FAQ

Is Claude better than ChatGPT for sensitive data? Their enterprise tenants have similar guarantees. Both do not train on your data. Both offer DPAs. See our comparison for the details that matter.

What about running a model locally? Possible for smaller open-weight models. Not practical for CFO-quality reasoning yet. If you have a hard requirement to keep data on-prem, Copilot inside your M365 tenant is the closest to that in 2026.

Can I use AI for tax positions? Same rule as ASC 606. Model organizes facts, drafts memo structure. Human tax preparer or tax attorney owns the position.

What about audit prep? Fine for organizing PBC lists, drafting workpaper narratives, cross-referencing supporting documentation. Not fine for making the accounting call in the workpaper.

How do I train my team on the do-not-ask list? Print the table above. Put it on the wall in the finance team room. Every new hire gets the walkthrough in week 1. This is a culture problem, not a policy problem.

Related reading

• The CFO’s Guide to Uploading Financials to LLMs Without Privacy Risk
• Custom GPT vs Claude Project vs Copilot Agent
• How to Screen an AI-Native CFO Resume
• Interviewing an AI-Native FP&A Analyst
• The Prompt Library Every CFO Should Steal

Sources

• Anthropic, trust and safety documentation, docs.anthropic.com
• OpenAI, enterprise privacy commitments, help.openai.com
• Microsoft, Copilot for M365 data protection, learn.microsoft.com
• AICPA, technical guidance on ASC 606, aicpa-cima.com

Written by The Pragmatic CFO. 15+ years running P&Ls and building AI-native finance workflows across portfolio companies.