AI confidently gave the wrong analysis???
I gave Claude Code my real data and it missed what any decent analyst would catch
A lot of people think AI is going to replace Data Analysts.
Personally, I don’t think so. I see AI as a tool Data Analysts use.
Not as a replacement.
I wanted to give you a quick explanation of where AI is in terms of analysis.
So I gave Claude Code my entire YouTube channel data.
And said “analyze this”.
AI Analysis is Insanely Fast
The goal of this analysis was to learn what videos actually drives subscribers, watch time, and growth.
If I did this analysis on my own - even as a Senior Data Analyst - it would take me hours.
Claude Code explored the data, cleaned it, analyzed it, and visualized it in 7 minutes.
7 friggin’ minutes. Crazy.
But fast….doesn’t mean good.
So how was the analysis?
Fast Does Not Mean Smart
Most of the analysis was fine to be honest. Nothing super wrong. But also nothing super impressive. However, it did make two big mistakes.
Not really number crunching errors (although I’m sure those exist somewhere - I just didn’t have time to check it thoroughly).
But domain knowledge errors.
At the end of the day, Claude (and any AI) still lacks in actual domain experience & knowledge.
Mistake #1: It mixed Shorts and long-form videos together
Two totally different mediums. They really shouldn’t be compared.
It’s is like comparing TikTok videos to movies. Shorts get quick views. Long videos get loyal subscribers. You cannot put them on the same chart.
It was like, “your watch time for these videos really sucked!”
Well, no duh. They’re Shorts. Shorts are less than 60 seconds and served to viewers who love to watch 3 second and swipe. Of course they have low average view time. That’s expected.
But Claude didn’t know that. Claude doesn’t know the in’s and out’s of being a YouTuber.
It doesn’t know the actual business of making videos.
Mistake #2: It Picked The Wrong Winner
Claude looked at my data and confidently said my biggest problem was low click-through rate, or CTR.
Which, it’s not wrong. I do have low CTR compared to industry standards - need to step up my title & thumbnail game for sure.
BUT it over-indexed on this big time
It kept pointing to a video that had a massive 15.6% CTR and said, “This is your best video! Make more like this!”
But once again, Claude does not know how YouTube really works.
Deep down, I know why this CTR is so high. Why? Because I know YouTube. I know my videos. I’ve spent thousands of hours working on this. And I guess, I know things AI can’t know.
That video has such a high CTR because it ranks #1 on Google.
Seriously. Go search “how to export tableau public” and check if you see my face :)
People search for that exact problem, find my video, and click it.
Okay, great! Make more videos like that because CTR is really high…
But what does that video actually do for me?
It’s made $135 total over four years. That is just 9 cents a day.
It got only 73 subscribers. On YouTube, that is almost nothing.
And really, it’s just people who are stuck watching - who don’t really stick around for other videos.
Meanwhile, Claude told me my actual best videos were bad.
My videos on ‘How to become a data analyst’ have a low CTR of around 2.8%. If you only looked at that number, you’d think they were losers.
But those “loser” videos drove 12,000 new subscribers and made thousands of dollars.
Claude somehow couldn’t wrap it’s head around that.
The Part of Your Job AI Cannot Touch
This is exactly why AI will not replace great data analysts.
AI can do the messy work fast. It can write the Python code. It can build the charts.
But it completely misses the most important part: judgment.
AI often sees a high number and assumes it is good. It cannot always tell the difference between a nice-looking number and a number that ACTUALLY helps the business.
In fact, a lot of the time, it doesn’t know the business. It does not know the domain.
It may think it does - but there’s knowledge it does not have - and will not have.
And you have it.
As a data analyst, you are not paid only to write SQL or build dashboards. You are paid to use judgment.
You have to know which numbers matter.
You have to ask the right questions.
You have to notice when data is missing.
You have to catch AI when it guesses or makes things up.
The analysts who win in the future will use AI for speed, but they will use their own brains for judgment.
Three Things to Start Doing Right Now
If you want to stay valuable in data, you need to get better at the human side of the work.
Stop obsessing over code. AI can write the code. You need to focus on what the code actually means.
Learn the business. An analyst who understands how the company makes money will always beat the one who just writes SQL
Question the data. When AI hands you a chart, ask: “Is this a vanity metric? Does this actually drive business value?”
AI is just a tool. It still needs a human expert to tell it what matters.
Be that human.
Avery
P.S. If you want to go from “just learning data” to doing real analysis like this, the Data Analytics Accelerator is where that happens. We work on real projects, build strong portfolios, and help you get ready for the job. You can check it out.
This article was really insightful! Thanks for putting it out. Loved it.