The benefits of AI change usually show only after a delay. Before euros, hours or lead times change, people\u2019s experience, skills and ways of working change. That is why measuring must start as soon as a new solution is first brought into everyday work.<\/p>\n\n\n\n
The feedback from participants in the AI workshops we ran for El\u00e4keliitto was good, with an NPS of 50. Participants praised the clear guidance and useful exercises. So we knew that participants found the workshop successful and its content necessary. That is valuable information, but it only tells us about the first stage.<\/p>\n\n\n\n
After the workshop, we still don\u2019t know whether the participant used artificial intelligence in two weeks, if any work phase changed or if time was saved. The same gap is visible in many organizations. Management sees licenses, sign-ins and those who participated in training, but the daily hustle, uncertainty and extra checking work may not be visible anywhere. If measurement starts only from business results, the problem is easily noticed too late.<\/p>\n\n\n\n
The introduction of artificial intelligence should be monitored from three perspectives: experience, action, and impact. These levels are interconnected, but they are not the same thing and the result of one measure does not automatically prove the other. A good experience can predict usage, but it does not guarantee permanent change. Conversely, a large amount of usage can occur without improving work quality or productivity.<\/p>\n\n\n\n
The experience metric shows how people perceive the new way of working. Good questions include: does AI help with the agreed task, do I know at which stage of the work it is used, can I check the output, and can I get help when there is a problem. It is also worth asking whether people have time to practise and whether they dare to raise mistakes. Usage logs rarely give a reliable picture of these things.<\/p>\n\n\n\n
Perceived usefulness and ease are long-used variables in research on the acceptance of technology. However, in the context of artificial intelligence transformation, they need to be accompanied by task-specific clarity, verification assurance, and psychological safety. Experience measurement serves as an early signal: it tells us what needs to be addressed in the implementation process before usage rates or results start to decline. Therefore, the pulse should be short, repetitive, and tied to a concrete work task.<\/p>\n\n\n\n
The activity metric shifts the focus from opinions to doing. Track, for example, recurring usage, the share of tasks that use AI, adherence to the agreed workflow and whether human review actually happens. The key is to define in which task usage is expected. The number of prompts, conversations or sign-ins alone does not tell you whether the process has changed as intended.<\/p>\n\n\n\n
Usage data should be supplemented with a small quality sample. Regularly reviewing as few as 20 cases can reveal whether agreed steps are skipped, where rework arises and whether human responsibility is exercised in practice. If the experience is positive but the activity does not change, the cause may be an unclear process, missing access rights or the fact that no time has been set aside for practice. In that case, more training is not necessarily the first solution.<\/p>\n\n\n\n
The impact metric tells whether the benefit for which the solution was purchased has been achieved. Examples of suitable metrics include lead time, proportion of outputs approved in a single instance, rework, quality deviations, cost per instance, and confirmed saved work time. Speed should not be monitored without quality, as local time savings can be passed on to the QC or to the next process step. Alongside the benefit, it is also necessary to monitor significant errors and near-misses.<\/p>\n\n\n\n
The impact can only be evaluated once the baseline has been measured in the same way before the change. If there is no information about the baseline, it is difficult to show afterwards what changed and why. The time saved should be evaluated using time stamps or sampling, and the result should be reported in terms of the workload performed. The user\u2019s own assessment of the time savings can complement the picture, but it should not be used as the sole benefit calculation.<\/p>\n\n\n\n
Before the pilot starts, you need two to four weeks of baseline data from the current process. At the same time, run the first experience pulse and agree target levels for 6\u201310 selected metrics. There is no need to use every possible metric: two to four per level is enough for most pilots. Most important is that each metric has an owner, a data source and an agreed decision that the result feeds into.<\/p>\n\n\n\n
Experience is worth measuring at weeks 2, 4, 8 and 12; initially more frequently and later less often. Use and workflow compliance can be monitored weekly if the data is generated automatically, or monthly with sampling. The first actual benefit comparison should be done around week 8\u201312, when enough cases have accumulated. During weeks 4\u20136, it will already be clearer whether the new method is becoming established or whether the pilot will remain a separate experiment.<\/p>\n\n\n\n
Once usage is established, the experience pulse can be run quarterly and the activity metrics automated into monthly tracking. Impacts are tracked monthly or quarterly depending on the metric. So the measurement interval is not decided once for all metrics. Early experience needs a fast feedback cycle, while verifying financial impact takes time and a large enough data set.<\/p>\n\n\n\n
A small pilot does not need a separate measurement platform. Microsoft Forms or Google Forms is enough for the experience pulse, the AI tool\u2019s own report for usage data, and Excel for tracking the process and benefits. Process data should primarily come from the system where the work is done anyway. That way, measuring does not create a new, heavy reporting process.<\/p>\n\n\n\n
When there are multiple teams, work tasks, or data sources, the same structure can be transferred to a Power BI view. For Microsoft 365 Copilot users, the Copilot Dashboard provides ready-made views on readiness, usage, impact, and user experience. That doesn\u2019t eliminate the need for your own process data either, because the general usage report doesn\u2019t know what the organization should have achieved. The tool\u2019s job is to connect the data, not to define the goal for us.<\/p>\n\n\n\n
The four-level model of Kirkpatrick is widely used in evaluation of training. The model divides evaluation into reaction, learning, behavior, and outcomes, and it was developed as early as the 1950s. In measuring artificial intelligence transformation, it is a useful reminder that successful training days should not be confused with changed activities or business benefits. However, the model does not imply that there is always a strong or automatic correlation between the levels.<\/p>\n\n\n\n
A good set of metrics helps you make decisions during the pilot, not just present results afterwards. Don\u2019t put too many metrics into the pilot. The point of measuring is to see if the experience weakens, so that the obstacle to use can be removed. If activity has not changed by week 8, review the process, skills and support. If activity changes but impact does not, reassess the task, the baseline or the goal.<\/p>\n\n\n\n
The real benefit of AI transformation only comes when people are able to use the solution; the workflow changes and the change is visible in quality, time, or costs. One metric cannot show the entire chain. Therefore, experience, action, and impact must be monitored in parallel, even if they appear at different times. The purpose of measurement is not to prove that the acquisition was successful, but to help make the implementation successful.<\/p>\n\n\n\n
In quality assurance of design, measurement issues are concrete: are deficiencies detected earlier, will inspection work decrease and how many erroneous alarms will be generated? The same questions must be asked of new artificial intelligence solutions as well. In the following article, I will review ODA\u2019s MCP solutions and the possibilities they open up for the AI-assisted inspection of CAD and BIM data.<\/p>\n\n\n\n
\ud83c\udfa5 Watch the YouTube Short: Measuring AI change
<\/p>\n\n\n\n