The Three Levels of Data Support
How AI Can Transform Data Support at Every Level of Complexity
In my last post, I talked about how Klarna replaced 700 customer support agents by integrating Open AI’s models, and about the potential that large language models (LLMs) have to help automate support in data teams.
Now, how can this happen in practice?
To simplify and re-use the customer success comparison, let’s consider 3 levels of ticket complexity
Level 1 Support: Quick Wins
Level 1 support is where AI can make an immediate impact. Often, they are the most frustrating for teams because they can be resolved with basic knowledge and skills. These are the low-hanging fruit of data requests.
Typical Level 1 support requests include:
"Where can I find the dashboard about MRR?"
"What were the conversion rates for our latest email campaign?"
"What is the definition of the ‘churn rate’ metric?”
These queries are precise, well-defined, and usually involve simple data extraction or basic calculations. They often take up a significant chunk of a junior data analyst’s day.
These tasks mainly require technical skills and time. Large Language Models trained on company-specific data can now handle these Level 1 requests efficiently. At CastorDoc, we've developed an AI assistant that can effectively answer such questions accurately.
Level 2 Support: Junior Data Analyst Territory
Level 2 support tasks typically fall into the realm of junior data analysts, requiring a bit more analytical skill.
Some examples of Level 2 requests include:
"How does our customer retention rate in Europe compare to the US, and what might be driving the differences?"
"Can you analyze the impact of our recent pricing changes on sales across different product categories?"
"the numbers in my dashboard look odd, can you check?”
These questions aren't just about pulling data anymore. They require some interpretation, comparison, and the ability to identify key drivers of performance. It's the kind of work that often has junior analysts digging into multiple data sources.
AI is making strides in this area, but it's not quite there yet. To handle these Level 2 requests effectively, AI systems need more context, better fine-tuning, and a deeper understanding of the business. It's like teaching the AI to think more like an analyst and less like a database query tool.
Level 3 Support: Senior Analyst Insights
This is where we enter the domain of senior analysts, tackling the most complex data challenges. Level 3 requests might look like:
"What strategic initiatives should we prioritize to boost market share next year?"
"How might emerging trends and competitor moves affect our long-term growth strategy?"
"Analyze how customer lifetime value varies across segments and recommend strategies to increase it."
These questions demand a deep understanding of the business, industry, and the intricate relationships between various data points. They require transforming numbers into actionable insights and strategic recommendations.
Currently, AI systems aren't equipped to handle this level of complexity. These tasks require strategic thinking, business acumen, and the ability to connect seemingly unrelated dots - skills that senior analysts develop over years of experience.
The Expectation Gap: Why We Underestimate AI in Data Analytics
To climb the ladder from Level 1 to Level 3, AI systems need more than just raw computing power. They require richer context, finer tuning, more nuanced question analysis, and a deeper understanding of the person behind the question.
A key reason why current AI solutions sometimes fall short is that they often operate at a junior level, but their performance will be compared against the work of senior analysts.
The reality is that LLMs excel at handling the kind of work typically assigned to junior analysts. They can quickly extract data, perform basic analyses, and answer straightforward queries with impressive accuracy. These Level 1 tasks consume a significant portion of data teams' time. By automating these, we can already reduce the workload on analysts.
Another area where LLMs can save considerable time for data teams is in the automatic classification of incoming requests. By categorizing tickets into Levels 1, 2, or 3, organizations can:
Efficiently route simple queries to AI systems
Direct more complex requests to human analysts
Gradually expand AI capabilities to handle higher-level tasks
Even if AI can only reliably manage Level 1 and some Level 2 requests, this represents a significant improvement in efficiency for many data teams. Automating the simpler tasks frees up analysts to tackle more complex problems.