The dashboard is lying to you: How to measure the real impact of AI on development?

More tokens = more impact? Well, no. Not long ago, a large software company was sure it had finally cracked one of the burning questions of the last year: what is the real impact of AI tools on its R&D performance. The dashboards looked great, the graphs were climbing, adoption rates were up, and one metric kept appearing in management presentations: AI Code Percentage — a metric designed to show how much of the code entering the system was written with the help of artificial intelligence. On the surface, it sounds like an excellent KPI — if the number goes up, it's clear that adoption is increasing. But what did this usage actually change?

Here the problem begins. Development managers, VPs, and CTOs don't really want to know just how much code was written with AI. They need to understand if development has become faster, if code quality is maintained, if the testing load has changed, and if the investment in new tools is generating value for the organization. The AI Code Percentage metric reflects the extent of usage, but it cannot answer the question of impact on its own.

The perfect solution might be lying to you

When coming to solve this visibility problem, the initial instinct of most organizations is to rely on analytics data and off-the-shelf tools provided by the AI manufacturers themselves. It looks like the perfect solution — to take complex development activity and distill it into a single, unambiguous number of usage percentages. These dashboards present a detailed picture of tool telemetry: how many times a developer activated a plugin in the IDE, how many code suggestions were accepted, how many prompts were sent, and how long an active session lasted.

These dashboards can be very accurate in describing activity within the tool, but this information is limited to the question of how much AI was used. It does not necessarily explain what happened to delivery speed, review load, code quality, or the total cost of the work. AI Code Percentage is a useful metric for understanding the scope of adoption, but it cannot draw conclusions about productivity, quality, and business value.

So how do you measure code attribution in practice?

Even before approaching a solution, one must understand that today there are two completely separate worlds of information in the organization. On one side is that telemetry of AI tools and the various agents being released to the market at a relentless pace. On the other side are the organization's actual development data, those sitting inside classic development platforms like GitHub and Jira.

When the measurement system tries to estimate effectiveness without a direct connection between the worlds, it is forced to make rough estimates. It sees that a certain developer used Cursor or Claude Code that day, and it sees that they uploaded three Pull Requests. Without the ability to cross-reference directly, the system might attribute the AI's impact to all the activity performed in that time frame. Thus, an Attribution Gap is created, where we know there was AI usage and we know we got a result, but we mistakenly base the connection between them on indirect inference instead of reliable attribution to the final product.

So how do you measure correctly? The transition from measuring usage to code attribution is not a theoretical idea, and it requires completely different data engineering. A practical solution requires establishing an infrastructure that directly connects the organizational GenAI Data Lake to the developers' actual Git activity and analyzes the changes in the code in real time.

Instead of looking at the duration of activity in the tool, the system analyzes the product at the level of the individual line of code. For example, when a developer completes a complex task originating from a specific ticket in Jira, the system will identify that out of 120 new lines of code created in that Pull Request, the source of about 50 lines is direct code suggestions from the agent, and the remaining lines are the result of human architecture.

From the moment the system performs this association, it is possible to examine the impact of the AI models using three families of metrics: code stability, friction in the testing process, and quality after release:

Code Survival metric: The system checks how much of the code created with AI survives within the Codebase after two weeks or a month. If a high percentage of it is deleted or rewritten again and again by human developers in the days that follow, it is a sign that rapid adoption is creating technical debt rather than real productivity.
Review Friction metric: Instead of assuming that development has shortened, measure the time it takes for a Pull Request to be approved and the number of revision rounds it requires. This way, it is possible to identify cases where code created quickly with AI shortens the writing stage but increases the testing and verification load for other developers.
Escaped Bugs metric: Cross-referencing AI data with bug reports allows comparing the rate of bugs, regressions, and reopening of tickets in work influenced by AI to similar work that was not influenced by it.

Lessons and practical recommendations

Moving to attribution-based measurement surfaces complex organizational truths and forces development managers to make decisions based on reliable data. Here are three practical recommendations:

Don't measure activity, measure outputs: Tracking the number of active licenses or the volume of tokens in the organization gives an indication of the scope of financial expenditure, but not of productivity. Start linking AI signals directly to the code actually written at the Commit level.
Identify the new bottlenecks: AI tools do shorten the initial code writing time, but they may shift the load to the Code Review stage. If even small Pull Requests, which should be relatively simple to check, wait longer for review or require more revision rounds when they include AI contribution — it is possible that the team has shifted the bottleneck from the writing stage to the verification stage.
Adopt a trade-off of transparency versus convenience: Moving to accurate measurement requires a more complex data infrastructure and may reveal that expensive tools you purchased are not providing the desired impact. Development managers should be prepared to give up "green" and reassuring dashboards in favor of truth data that can completely change the organization's budget decisions and adoption plans.

The development world is in the midst of an accelerated transition to work that is AI-native. The central question is no longer how much developers use AI or what percentage of code was written with its help, but how this usage changes the entire development system: delivery speed, testing load, code quality, labor costs, and the ability of humans, tools, models, and agents to act together. As long as we continue to measure usage instead of impact, the dashboards may be completely accurate — but they will lead us to the wrong conclusion.