Learning Analytics and Evidence Informed Improvement

The central weakness in learning analytics is not a shortage of data. It is the frequency with which data is mistaken for evidence, prediction is mistaken for understanding and reporting is mistaken for improvement.

Educational institutions can measure attendance, submissions, assessment performance and virtual learning environment activity. Corporate Learning and Development teams can count registrations, completions, learning hours and digital credentials. Yet these measures largely describe participation in systems. They do not necessarily demonstrate learning, capability, behavioural change or improved performance.

The value of analytics therefore cannot be judged by the sophistication of a dashboard. It must be judged by whether the evidence changed a decision, whether that decision produced an intervention and whether the intervention improved an outcome.

The dashboard fallacy

Learning analytics frequently begins with what a platform can measure rather than with what an organisation needs to understand. This produces dashboards filled with convenient indicators whose educational significance is uncertain.

A login is not evidence of engagement. Time spent on a platform is not evidence of understanding. Completion is not evidence of competence.

Repeated access to a resource may indicate productive study, confusion or poor resource design. The same digital behaviour can therefore support several conflicting interpretations. Systematic research confirms that analytics interventions remain highly dependent on how data is interpreted and embedded within learning design. The existence of an analytical intervention does not, by itself, establish that it improves teaching or learning (Pan et al., 2024).

This is where analytics can become institutionally convenient but educationally weak. It makes learners increasingly visible while leaving the quality of the programme, assessment, teaching and support comparatively unexamined.

When a large proportion of learners struggles with the same activity, the problem may not be learner deficiency. It may be weak sequencing, ambiguous guidance, inaccessible content or an assessment that does not align with what was taught. Analytics that identifies “at risk learners” but does not question the design producing the risk reinforces an incomplete diagnosis.

Prediction does not constitute improvement

The Open University’s OU Analyse system uses machine learning to identify students considered at risk of failing their next assignment. The risk information is updated weekly and supplied to tutors and Student Support Teams to inform possible support (Open University, n.d.).

This is a credible use of analytics, but the prediction should not be confused with the intervention. The model can identify a statistical pattern; it cannot establish why an individual learner is struggling or what response will be effective. The educational value still depends on whether:

• The tutor receives the information in time.
• The risk indicator is sufficiently accurate.
• Staff have the capacity to respond.
• The learner accepts the support offered.
• The intervention addresses the actual difficulty.
• The institution evaluates what happened afterwards.

Without these conditions, an early warning system becomes an early labelling system.

Georgia State University presents predictive analytics as part of a wider student success strategy. Its approach uses institutional data and risk indicators to trigger proactive advising and address barriers to progression (Georgia State University, n.d.).

The results reported by Georgia State are impressive, but they should not be attributed to analytics in isolation. The institution combined analytics with advising, financial interventions, course planning, communication and organisational reform. The evidence therefore supports the effectiveness of an integrated student success model more strongly than it proves the independent effect of predictive analytics.

That distinction matters. Institutions may purchase an analytical platform expecting comparable outcomes without investing in the advisers, support structures and operational changes that made the Georgia State model possible.

The lesson is not that every institution needs the same algorithm. It is that analytics has limited value when the surrounding organisation lacks the capacity to act.

Corporate L&D has a measurement problem

Corporate L&D faces an even sharper difficulty. Learning functions routinely report participation measures because they are readily available and easy to defend. Completion rates, learning hours and learner satisfaction provide evidence that training occurred. They do not establish that workplace performance improved.

Google’s Project Oxygen is frequently cited as an example of evidence informed management development. Google analysed employee feedback and performance information to identify ten behaviours associated with effective managers and used those findings to shape manager development and feedback (Google re:Work, n.d.).

Its significance lies in beginning with an organisational question about managerial effectiveness rather than beginning with a leadership course. However, the findings remain rooted in Google’s culture, workforce and internal measures. Treating the ten behaviours as a universal management framework would ignore the context in which they were produced.

There is also a circularity risk. When organisations define effective behaviour through their own surveys and performance systems, they may reproduce what those systems already value. Analytics can strengthen organisational assumptions as easily as it can challenge them.

IBM’s learning ecosystem provides a more ambitious attempt to connect learning with workforce outcomes. Its Your Learning platform integrates learning resources, skills information, digital credentials and career development. A study of IBM technical sales employees found positive associations between learning activity and internal credentials, sales performance and career progression (Qin & Kochan, 2020).

The relationship is important, but it does not establish causation. Employees who undertake more learning may already be more motivated, better supported or more strongly positioned for advancement. Promotion and performance may also provide employees with greater access to learning rather than simply result from it.

The IBM report itself acknowledges that further investigation is required and does not guarantee that participation in its learning system will produce comparable outcomes for individual employees.

This limitation reflects a broader problem. A 2025 systematic review found no widely agreed method for evaluating the transfer of workplace e-learning and reported that self-assessment remained the most common measurement approach, despite its vulnerability to bias (O’Neill, 2025).

Corporate L&D should therefore be cautious about claims that learning “drove” performance when the evidence shows only that learning and performance occurred together.

Evidence informed improvement requires causal discipline

Analytics becomes credible when organisations distinguish between four different claims:

These claims require different evidence. A completion record may support the first. It cannot substantiate the remaining three.

For universities, improved retention cannot automatically be attributed to an engagement dashboard when advising, curriculum changes and financial support were introduced simultaneously. For corporate organisations, increased productivity cannot automatically be attributed to training when technology, staffing, incentives or market conditions also changed.

Evidence informed improvement does not require perfect experimental control. It does, however, require greater discipline than presenting a positive trend after training and implying causation.

Organisations should establish the problem and intended outcome before selecting the data. They should identify the decision owner, define the possible intervention and determine how change will be evaluated. Where feasible, this should include baseline measures, comparison groups, performance evidence, delayed follow up and qualitative investigation of why an intervention worked or failed.

Analytics can also distort decisions

Metrics do not simply represent organisational priorities. They can reshape them.

When engagement scores become performance indicators, staff may focus on increasing visible platform activity rather than improving learning. When course completion becomes the primary L&D measure, programmes may be shortened or simplified to increase completion without increasing capability. When learners know that every click contributes to a risk score, their behaviour may become performative rather than authentic.

There are also significant questions of power. Institutions determine which behaviours are normal, which patterns indicate risk and who has access to the resulting classifications. Learners and employees may have little opportunity to challenge an inaccurate label or understand how a decision was reached.

Jisc’s code of practice therefore emphasises transparency, validity, data minimisation, human oversight, positive intervention and the avoidance of spurious correlations. It also states that analytics should be used for learners’ benefit rather than as an undisclosed mechanism of monitoring or control (Jisc, 2023).

Analytics that individuals cannot understand, question or correct should not be presented as supportive. It is closer to administrative surveillance.

The value lies in the decision

Learning analytics creates no inherent value. Its contribution depends on the quality of the decisions and interventions surrounding it.

The Open University demonstrates that prediction must connect with human support. Georgia State demonstrates that analytics is most powerful when embedded within wider institutional reform. Google shows the value of beginning with a performance question rather than a training product. IBM illustrates both the potential of connecting learning with workforce data and the difficulty of establishing causation.

These examples should not be treated as templates to copy uncritically. They reveal the organisational conditions under which analytics may become useful, as well as the evidential weaknesses that remain.

A credible analytics system must be able to answer four demanding questions:

Without convincing answers, learning analytics remains measurement without improvement and visibility without value.

The real test is not whether an organisation has become more data driven. It is whether it has become more capable of questioning its assumptions, acting on evidence and learning from the consequences of its decisions.