Learning from Learning Traces: The Promises and Blind Spots of Learning Data

During a workshop with a teaching team, I suggested that an instructor and I review together the results grid from a quiz given weekly to their class. The page appeared: three students in red on the same question, twelve entirely in green, the rest in orange. Their first reaction was: “Oh, I hadn’t noticed that.” The second, a few seconds later: “But what do I do with this?”

This double reaction, surprise followed by uncertainty, captures quite well the current state of learning data in educational practice. The instrumentation is there, abundant, often well designed and easy to access. The pedagogical use of it, however, remains an open challenge. This article reflects on four dynamics commonly observed in the field: two opportunities that digital tools now make accessible even to teams without pedagogical or data engineering expertise, and two pitfalls that recent research continues to highlight.

Real-time feedback loops: a textbook case of “closing the loop”

The theory is well known. Clow (2012) describes Learning Analytics as a cycle: learners generate traces, these traces are analyzed, teaching practices are adjusted, and learners benefit from those adjustments. Until the loop is closed, data serves little purpose beyond existing. This is perhaps the simplest criterion for evaluating a tool: does it actually allow educators to “close the loop”?

Student Response Systems (SRS), interactive questions, polls, word clouds, projected quizzes, embody this cycle synchronously. When instructors see the distribution of responses to a question in real time, they are not consulting a report. They are seeing, live, where the class lost track, and can redirect the session within minutes. Tools such as Mentimeter, Kahoot!, Wooclap, or polls integrated into Microsoft Teams operate on this principle. In asynchronous settings, platforms such as Wooflash, Anki, or quiz modules within Moodle extend the loop, sometimes using spaced repetition algorithms that adapt question frequency to individual performance.

The recent review published by Serrada-Sotil, Huertas Martinez, and Granado-Peinado (2025) confirms a central point: the ability to adjust teaching in real time is indeed a significant mediator of pedagogical effectiveness, provided that educators possess the skills required to interpret and act upon the data. Moreno-Medina et al. (2023) also report that 93% of learners using an SRS perceived an improved understanding of course content.

The first point of friction appears here: tools do not teach people how to read the tool itself. A results grid remains silent if pedagogical interpretation has not been developed. Co-constructing this interpretation between instructional designers, teachers, students, assistants, or trainers often leads to better understanding and easier use of the data. One well-used indicator can sometimes be more valuable than an intimidating or overly complex dashboard.

Early dropout detection: a “lightweight” Early Warning System

Institutional Early Warning Systems (EWS) have proven effective in many universities. Combining grades, LMS attendance, assignment submissions, and engagement indicators makes it possible to identify struggling students early. Yet implementing a “complete” EWS requires substantial infrastructure: LMS integrations, machine learning models, data governance, GDPR compliance, and staff training. For most faculties, schools, or organizations, the entry cost remains prohibitive.

Several everyday tools offer a more modest but immediately operational alternative. Reports from Wooflash, student-level views in Moodle, engagement reports in Microsoft Teams, dashboards in Anki, or analytics in Brightspace all allow educators to quickly see who is disengaging, on which concepts, and at what pace. The underlying principle remains the same everywhere: a color code, a threshold, a signal.

A systematic review published in 2025 confirms researchers’ caution: despite increasingly sophisticated predictive models, there is still limited causal evidence linking detection systems to actual improvements in learning outcomes, and real classroom deployment remains embryonic. Unsurprisingly, effectiveness depends elsewhere: detection only matters if it leads to coordinated human intervention. A red warning indicator that triggers no conversation, no meeting, and no instructional adjustment is not only useless, it can become discouraging for the educator watching it blink without knowing how to respond.

This leads to the second friction point, mirroring the first: the tool is not the intervention itself. It initiates the process, it sheds light on situations, but it does not replace human action. Reading and interpreting the data remains essential in order to act meaningfully.

Under the hood: what exactly is the algorithm doing?

When a spaced repetition tool decides to present flashcard X five times to learner A and only once to learner B, it is making a pedagogical decision. It applies that decision at scale, without consultation. Hariyanto, Kristianingsih, and Maharani (2025) identify model interpretability as a critical challenge in AI-driven adaptive education and explicitly call for Explainable AI approaches.

Three questions deserve to be asked, regardless of the tool:

• Can educators audit the algorithm’s decisions? Do they know why one card is postponed while another is prioritized?
• Is the optimization criterion explicit? Is the system optimizing correct recall, self-reported confidence, response time, or a combination of factors?
• Are learners informed about the logic shaping their learning pathway?

Gašević et al. (2015) had already identified the risk of a silent return to behaviorism: if an algorithm optimizes only for correct recall, it may neglect deeper understanding, transfer, and critical thinking. Hakimi, Eynon, and Murphy (2021) frame this as a learner autonomy issue. Choosing what to review is also part of learning how to learn.

These critiques do not invalidate spaced repetition, which rests on strong empirical foundations dating back to Ebbinghaus. Rather, they invite us to open the “black box”: to ask vendors for a minimum level of algorithmic transparency, to explain the logic of these systems to learners, and to preserve for educators a meaningful area of control, including which cards are included, which criteria define mastery, and what relative weight different indicators carry. Without this, we risk delegating pedagogical choices to opaque systems that should remain open to human judgment.

The reductionism trap: what data does not tell us

The fourth pitfall is perhaps the most subtle, because it lies in treating numbers as something they are not. The tools discussed here primarily measure behavioral data: participation rates, quiz scores, response times, login sequences. These are useful indicators. They are not direct indicators of learning.

Drugova et al. (2024) make this point explicitly: very few studies currently establish a causal relationship between learning dashboards and genuine improvements in pedagogical design. Research on SRS tools (Serrada-Sotil et al., 2025) points in the same direction: longitudinal evidence demonstrating durable gains remains limited. Participating actively in a quiz does not guarantee meaningful learning. Scoring 80% on an MCQ does not guarantee conceptual understanding. Clicking through flashcards does not guarantee the cognitive effort we may assume is taking place.

The danger lies in reducing pedagogy to what is measurable. Numbers alone rarely speak for themselves, and data must also be made understandable to learners. Dashboards do not replace educators; they complement them. Quantitative indicators such as scores, participation, and progression still need to be combined with qualitative judgment: the quality of classroom interactions, the depth of an open-ended response, or the reasoning articulated orally. Unsurprisingly, this qualitative dimension is also the hardest to automate.

Some directions moving forward

This overview argues neither for dataphobia nor for blind data enthusiasm. Instead, it advocates for a structured and thoughtful approach. Several simple principles emerge:

• Close the loop. Data that is neither read, discussed, nor followed by adjustment serves little purpose. One meaningful indicator is often better than an overwhelming dashboard.
• Support interpretation. A tool without training is a warning light; a tool with training becomes a pedagogical device. Interpretive competence makes the difference.
• Open the black boxes. Asking vendors what their algorithms measure, optimize, and ignore is as much a pedagogical act as a technical one.
• Combine perspectives. Behavioral data gains meaning when confronted with qualitative evidence such as classroom observation, written exchanges, or open-ended productions.
• Make data explicit for learners. Data circulating without students understanding its origin or purpose deprives them of an essential part of their learning journey.

None of these recommendations are revolutionary. They align with a pedagogical tradition that understands that no tool, whether calculators, search engines, or generative AI, removes the need for human judgment. Learning data is no exception. When well framed, it equips educators. When poorly framed, it risks adding yet another layer of cognitive overload.

The real work to be done, whether at the scale of a course, a department, or an institution, is methodological: agreeing together on what should be monitored, why it matters, how often it should be reviewed, and what actions should follow. That is where the shift from available data to useful data truly happens.

Bibliography

Clow, D. 2012. The learning analytics cycle: closing the loop effectively. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12). Association for Computing Machinery, New York, NY, USA, 134–138. https://doi.org/10.1145/2330601.2330636

Serrada-Sotil, J., Huertas Martínez, J. A., & Granado-Peinado, M. (2025). Do audience response systems truly enhance learning and motivation in higher education? A systematic review. Humanities and Social Sciences Communications, 12(1), 1767. https://doi.org/10.1057/s41599-025-06042-w

Irene Moreno-Medina, Manuel Peñas-Garzón, Carolina Belver, Jorge Bedia, Wooclap for improving student achievement and motivation in the Chemical Engineering Degree, Education for Chemical Engineers, Volume 45, 2023, Pages 11-18, https://doi.org/10.1016/j.ece.2023.07.003

Cabral, L., Pinto, R., & Gonçalves, G. (2025). AI-powered learning analytics dashboards: a systematic review of applications, techniques, and research gaps. Discover Education, 4(1), 525. https://doi.org/10.1007/s44217-025-00964-y

Hariyanto, Kristianingsih, F. X. D., & Maharani, R. (2025). Artificial intelligence in adaptive education: a systematic review of techniques for personalized learning. Discover Education, 4(1), 458. https://doi.org/10.1007/s44217-025-00908-6

Gašević, D., Dawson, S. & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning.  TECH TRENDS 59, 64–7. https://doi.org/10.1007/s11528-014-0822-x

Hakimi, L., Eynon, R., & Murphy, V. A. (2021). The ethics of using digital trace data in education: A thematic review of the research landscape. Review of Educational Research, 91(5), 671–717. https://doi.org/10.3102/00346543211020116

Drugova, Elena & Zhuravleva, Irina & Zakharova, Ulyana & Latipov, Adel. (2023). Learning analytics driven improvements in learning design in higher education: A systematic literature review. Journal of Computer Assisted Learning. 40. 510-524. https://doi.org/10.1111/jcal.12894

Serrada-Sotil, J., Huertas Martínez, J. A., & Granado-Peinado, M. (2025). Op. Cit.

Bergdahl, N., Bond, M., Sjöberg, J., Dougherty, M., & Oxley, E. (2024). Unpacking student engagement in higher education learning analytics: A systematic review. International Journal of Educational Technology in Higher Education, 21(1), 63. https://doi.org/10.1186/s41239-024-00493-y