AI chatbots can't do your taxes - here's what actually works

No, a general AI chatbot can't reliably do your taxes. When the New York Times tested ChatGPT, Gemini, Claude and Grok on eight tax scenarios, they miscalculated refunds by more than $2,000 on average. What works is purpose-built tax AI that reads the source documents, checks every number against IRS rules with a deterministic validation layer, and hands the return to a tax professional who reviews it and signs off.

Last week, the New York Times published a piece titled "A Word to the Wise: Don't Trust A.I. to File Your Taxes." They tested ChatGPT, Gemini, Claude, and Grok on eight fictional tax scenarios. The chatbots miscalculated refunds by an average of more than $2,000.

I run an AI tax company and I completely agree with the headline. Here's why that's not a contradiction, and what it tells you about how purpose-built AI actually differs from general models.

What did the New York Times test actually measure?

The NYT tested whether consumers can use general AI chatbots to file their own taxes. The conclusion: they can't. That's correct.

But the test was designed around a question nobody building AI for tax professionals is actually trying to answer. Nobody in this industry is trying to replace the tax professional with a chatbot. That would be like handing a medical student a textbook, leaving them alone with a patient, and then reporting that medical students aren't ready to practice medicine independently. The finding is technically true and completely beside the point. Medicine works because of systems, supervision, and layered review. Tax preparation works the same way.

The real question isn't "can AI do your taxes?" It's "can AI make the people who do your taxes faster, more accurate, and more focused on the work that requires their judgment?" The answer to that is yes. And it requires a fundamentally different architecture than a general chatbot.

Why does general AI fail at tax?

Because chatbots are prediction engines built to produce plausible answers, and tax needs exact, rule-bound answers where one early error cascades through the whole return.

The expert commentary in the NYT piece is more useful than the test itself.

Benedict Evans told the Times that AI gives you "roughly the right answer, and that's not what you want." He's right. Roughly right is dangerous in tax because errors cascade. A wrong cost basis on an early form produces wrong numbers on downstream forms. By the time it reaches e-filing, the return looks plausible but is wrong throughout.

Erik Brynjolfsson from Stanford described the "tax-code paradox": traditional tax tools follow procedural if-then logic built for precision, while AI models are prediction engines optimised for plausible output, not correct output. He's describing a real limitation. The failure modes the chatbots produced in the NYT test are predictable from this: bracket calculations instead of the IRS-mandated Tax Table, structural errors on complex forms, cascading mistakes from one early wrong input.

These aren't random errors. They're what happens when you use a prediction engine to do a rules-enforcement job.

What does purpose-built tax AI do differently?

Filed uses multi-agent orchestration, where specialised AI components handle different parts of the tax preparation process and a deterministic validation layer runs independently of the AI output at every step. The system doesn't guess and hope. It produces output, then checks it against IRS rules separately. When they don't agree, it flags the discrepancy before it reaches a human reviewer.

This isn't a theoretical design choice. Column Tax released TaxCalcBench in 2025, the first open-source benchmark for AI tax accuracy. We ran Filed against it alongside ChatGPT, Claude, and Gemini.

The generic chatbots scored between 23% and 42% on complete returns. Filed scored 72.5%. On a line-by-line basis, which maps to how tax professionals actually review returns, Filed reached 94% accuracy.

Since then, frontier models have improved and the latest generation is approaching that 94% threshold on their own. That's worth noting. But Filed's multi-agent architecture is built on top of these same frontier models. When they improve, our accuracy improves too. The 94% we published months ago is a floor, not a ceiling. What separates purpose-built systems is everything around the calculation.

What part of tax preparation did the test leave out?

There's one thing the NYT piece doesn't address, and it's the most important: none of the chatbots they tested do tax preparation.

Calculating a return from structured data is one step. In practice, tax preparation starts with a pile of messy source documents. PDFs. Handwritten notes. Clients who send ten K-1s in five different formats. The work involves extracting information from those documents, entering it into professional tax automation platforms like CCH Axcess, UltraTax, Lacerte, Drake, or ProConnect, navigating rolled-over data from prior-year returns, adding depreciation schedules on top of existing ones, and validating everything against the original sources before anything gets filed.

No chatbot does any of that. The NYT handed models pre-structured data and asked them to compute. The gap between that test and actual tax preparation doesn't close just because general AI gets better at arithmetic.

Should you trust AI to do your taxes?

Don't trust AI to file your taxes. Trust your tax professional. Then give them a tax automation platform purpose-built for the job, one with document-first architecture, deterministic validation, and humans in the loop for the decisions that actually require judgment.

That's not a hedge. It's the design philosophy that produces 94% line-by-line accuracy instead of 23%. To see it on your own returns, start with Filed Reviewer, which compares the source documents against the draft return line by line.

Frequently asked questions

Can ChatGPT or Claude do my taxes?

Not reliably on their own. In the New York Times test, general chatbots miscalculated refunds by more than $2,000 on average, and on TaxCalcBench they scored between 23% and 42% on complete returns. They also can't read your source documents or work inside professional tax software. More on that in can Claude or ChatGPT prepare tax returns.

What is TaxCalcBench?

TaxCalcBench is an open-source benchmark from Column Tax that measures how accurately AI models calculate US federal income tax returns. Results are reported for complete returns and line by line, and the line-by-line view is closer to how a reviewer checks work.

Does Filed replace the tax professional?

No. Filed does the analytical work so the tax professional can spend their time on judgment. Filed Prep reads the source documents and prepares the return across 1040, 1065 (with K-1), 1120, 1120-S, 1041, and 990 returns, inside CCH Axcess, UltraTax, Lacerte, Drake, and ProConnect. A named reviewer at the firm signs off on every return, and firms using Filed report 64% shorter review cycles.