Technology-enhanced research: Educational ICT systems as research instruments

Richard Cox
Richard is a Reader in Computer Science and Artificial Intelligence in the Department of Informatics at the University of Sussex.

Aims of this resource

Educational ICT systems offer wide scope for new and detailed insights into learning processes through their potential for methodological innovation and for capturing new forms of rich process data. E-learning systems used as research instruments allow the processes as well as the products (or outcomes) of learning to be studied. They can be used (simultaneously) as interactive learning environments and as measurement instruments i.e. for technology-enhanced research (TER). Rich process data capture can reveal the effects of slips and errors upon learning, make apparent the range of individual differences between learners in their styles and strategies, and provide data that can inform the design of adaptive systems.  They facilitate the study of:

• the time-course of learning of individual learners or groups of learners
• how fine-grained, relatively short-duration interventions can impact learning over the longer term
• how individuals differ e.g. in terms of their reasoning strategies

Potentially useful software and other resources are linked to below – the aim is to provide an indication of some of the ways in which the technology used to provide or deliver e-learning systems can also be used as instruments for process oriented research.

ESRC TLRP Phase III project - VL-PATSy

In a recent TLRP Phase III project on Vicarious Learning (VL) in the health sciences, an established TEL system (PATSy – www.patsy.ac.uk - Cox & Lum, 2004) was used as s a research testbed for the evaluation of VL as an innovative mode of learning (Cox, 2005; Hoben, Varley & Cox, 2007; Lee, 2005; Hoben & Morris, 2005).  VL can be defined as  'learning by observing the learning of others' (www.tlrp.org/proj/phase111/cox.htm).

In the first phase of the project, we identified students' difficulties in clinical reasoning, where this was defined as an inability to apply formal knowledge in a clinical setting'.  Pairs of students (dyads) were studied as they diagnosed previously unseen patients using PATSy under exam conditions.  Three sources of rich process data were collected: student-system interactions (PATSy log data); video of the student dyads engaged in dialogue during collaborative diagnostic reasoning, and real-time video capture of the computer screen. The three data sources were integrated and synchronised at playback using NITE XML tools developed by colleagues at Edinburgh (www.ltg.ed.ac.uk/NITE/).

PATSy was designed for use in research from the outset  and has built-in learning data collection functionality.  It is recommended that developers of educational ICT should consider the potential of their systems for research at the design and development stage. However, e-learning systems without built-in data logging can also be exploited as research instruments. Researchers can use application-independent utilities for dynamic (video) screen capture and/or keystroke logging utilities. These include Camtasia Studio for Windows – (www.techsmith.com/camtasia.asp), SnapZ Pro for Mac OS X (http://www.ambrosiasw.com/utilities/snapzprox/),  RUI (ritter.ist.psu.edu/projects/RUI/) and (http://www.keylogger.org) (Kukreja, Stevenson & Ritter, 2006). The Kukreja et al. keystroke logging utility (RUI) has versions for both Mac OS X and Microsoft Windows and supports replay of learner-system interactions from the log file.

This kind of background learner-system interaction recording software has an established history of use in HCI research (Westerman, Hambly, Alder, Wyatt-Millington, Shryane, Crawshaw & Hockey, 1996). Such programs can be run with - these days - little impact on a computer's performance.  The use of logging software, of course, raises issues of research ethics.  As in all research, participants must give informed consent and codes of research conduct and ethical guidelines should be followed. In the case of technology-enhanced research, `informed consent' would include the participant's awareness of data recording systems.

Analysing rich data

It is relatively easy to collect voluminous amounts of process data but hypotheses and analyses should be thought through in advance of data collection. How can coding systems for systematically analysing large volumes of rich process data be developed? Chi (1997) provides a useful framework. Her procedure for developing verbal data (e.g. think and talk-aloud protocols) analysis protocols can also be applied to learner-TEL system interaction log analysis. Chi (1997) suggests that researchers should first decide upon an appropriate granularity for their units of analysis analysis. Chi (1997) also suggests checking whether the chosen grain size is appropriate for the type of research question being asked. She proposes several functional steps to follow when developing an analysis protocol:

1. Reduce or sample the protocols

2. Choose grain size & segment the sampled protocols

3. Develop or choose a coding scheme.

4. Operationalise evidence in the coded protocols - what events or units count as evidence?

5. Seek patterns, interpret patterns, maybe repeat cycle

A range of computer-based analysis support tools are available, useful links to many of them can be found on the Research Capacity Building Network's website at the University of Cardiff (www.psychology.heacademy.ac.uk). The ESRC e-social science centre is another useful source of information (www.ncess.ac.uk).

In the vicarious learning with PATSy (VL-PATSy) studies, the relatively new NITE XML tools for synchronised replay and analysis of multiple video sources were used (www.ltg.ed.ac.uk/NITE/), together with 'Transcriber', a speech segmentation tool (http://trans.sourceforge.net).  The NITE tools, were used to code significant clinical reasoning events in conjunction with an analysis protocol in which clinical reasoning skills in terms were partitioned into domain-specific and general-reasoning components (Hoben et al, 2007; Howarth et al, 2005; Cox, 2005).

Detailed recordings of learner-system interactions also afford another kind of research opportunity. They can replayed to learners for verbal protocol using a retrospective debriefing approach (Taylor & Dionne, 2000). They can also be re-used as educational resources in their own right (as in the case of VL-PATSy where logs are used as a focus for tutorial discussions). A particularly useful resource for interested readers is the Psychonomic Society's journal Behavior Research Methods  - a very useful source of resources and information for technology-enhanced e-learning research (www.psychonomic.org).

The proliferation of large data corpora from TEL systems has resulted in the recent emergence of new special-interest group in the area of educational data mining (www.educationaldatamining.org). Some such corpora are extremely large – for example, at Stanford, an online system called 'Grade Grinder' has, to-date, received approximately 1.8 million submissions of work by more than 38,000 individual students over the past eight years; this population is drawn from approximately a hundred institutions in more than a dozen countries (ggww2.stanford.edu/GUS/openproof/).

In work funded partly by an ESRC/SSRC Visiting Americas Fellowship, the author is collaborating with colleagues at Stanford and Macquarie Universities in large scale corpus in the area of formal logic education using innovative TEL systems such as Tarski's World ( ggww2.stanford.edu/GUS/tarskisworld/). We are using the very large GradeGrinder dataset to provide empirical evidence for specific characteristics of natural language problem statements that frequently lead to students making mistakes when they translate natural language statements into first-order logic.  We have developed a rich taxonomy of the types of errors that students make, and implemented tools for automatically classifying student errors into these categories (Barker-Plummer, Cox, Dale & Etchemendy, 2008) and we have extended this work to include cases where students also produce diagrammatic representations as well as logic translations  
(Cox, Dale, Etchemendy & Barker-Plummer, 2008).

Visual Attention monitoring

Tools are also available for tracking visual attention (without the need for an expensive eye-tracker). The Restricted Focus Viewer (RFV) is a program which takes visual stimuli, blurs them and displays them on a computer screen, allowing the participant to see only a small region of the stimulus in focus at any time (Blackwell, Jansen & Marriott, 2000; Jansen, Blackwell & Marriott, 2003; Romero, Cox, du Boulay, Lutz & Bryant, 2007a).

A sample recording can be viewed at www.informatics.sussex.ac.uk/projects/crusade/clips/subj26.mov

The region in focus can be configured to move such that the 'foveal spot' is centered on the mouse cursor. Thus the learner leaves a record of what he or she has chosen to look at and moment-by-moment shifts in the focus of visual attention can be captured. The software includes a replayer to play back screen activity after a session.  In recent research a modified version of the RFV has been used to study undergraduates as they acquire programming skills (Romero, du Boulay, Cox, Lutz & Bryant, 2007b). In these studies, the user's visual attention to parts of the screen was tracked and digital audio files of `think aloud' protocols were recorded directly to the computer's hard disk.

Conclusion

The examples and links above are intended to illustrate the nature of process-analytic research and to illustrate how technology-enhanced learning system can function as useful research instruments. Capturing rich process data – 'high density sensing' (Anderson, 2002) is now much easier to achieve with modern fast computer systems and gigabytes of storage readily to hand. Such data, when combined with powerful process-analysis support tools, and the degree of experimental control that is possible using TEL systems. For example, versions of TEL systems (e.g. with features switched on/off) can be experimentally compared in one type of between-group design ('ablative evaluation', Beck et al., 1999). Such methods offer exciting scope for the research-informed development of effective and innovative interactive environments. They are useful too in non-TEL teaching and learning contexts - where technology might be used to record tutor-learner dialogue or to record the ways in which individuals in a classroom move around a teaching space.

References

Anderson, J.R. (2002) Spanning seven orders of magnitude: a challenge for cognitive modelling, Cognitive Science, 26, 85-112.

Barker-Plummer, D., Cox, R., Dale, R. & Etchemendy, J. (2008) An Empirical Study of Errors in Translating Natural Language into Logic.  To be presented at the 30th Annual  Cognitive Science Society Conference to be held in Washington, DC, July. (ccc.utexas.edu/cogsci08/)

Beck, J. E., Arroyo, I., Woolf, B. P., & Beal, C. R. (1999) An ablative evaluation, in: S.P. Lajoie & M. Vivet (Eds) Artificial Intelligence in Education. (Amsterdam: IOS Press). Proceedings of the 9th Artificial Intelligence in Education (AI-ED99) conference, Le Mans, France, July, 1999, pp 181-188.

Blackwell, A., Jansen, A., & Marriott, K. (2000). Restricted focus viewer: a tool for tracking visual attention, in: M. Anderson, P. Cheng, & V. Haarslev (Eds) Theory and application of diagrams. Lecture Notes in Artificial Intelligence 1889. (Berlin, Springer-Verlag).

Chi, M. (1997) Quantifying qualitative analyses of verbal data: A practical guide, The Journal of the Learning Sciences, 6(3), 271-315

Cox, R. (2007) Technology-enhanced research: Educational ICT systems as instruments for research and development.Technology, Pedagogy & Education, 16(3), 337-356. http://dx.doi.org/10.1080/14759390701614470

Cox, R. (2005) Vicarious learning and case-based teaching of clinical reasoning skills, Pedagogy paper: Technology, learning relationships & networks seminar.  ESRC TLRP Thematic seminar series on `Contexts, communities, networks: Mobilising learners' resources and relationships in different domains.' TLRP & Centre for Research in Lifelong Learning, Open University, Milton Keynes. Paper available at http://www.tlrp.org/dspace/

Cox, R., Dale, R., Etchemendy, J. & Barker-Plummer, D. (2008). Graphical revelations: Comparing students' translation errors in graphics and logic. To be presented at Diagrams 2008 conference (http://www.cmis.brighton.ac.uk/diagrams2008/)

Cox, R. & Lum, C. (2004) Case-based teaching & clinical reasoning: Seeing how students think with PATSy, in: Brumfitt, S. (Ed.) Innovations in professional education for Speech and Language Therapists (London, Whurr).

Hoben, K. & Morris, J. (2005) PATSy: Innovations in learning for speech and language therapy, Bulletin of the Royal College of Speech and Language Therapists (London, Royal College of Speech and Language Therapists), June.

Hoben, K., Varley, R. &  Cox, R. (2007) The clinical reasoning skills of speech & language therapy students, International Journal of Language and Communication Disorders, 42(1), 123-135.

Howarth, B., Hoben, K., Morris, J., Varley, R., Lee, J. &  Cox, R. (2005) Investigating speech therapists' clinical reasoning: analysing think-aloud protocols and integrating multiple-source data, Proceedings of the 11th Biennial Conference of the European Association for Research in Learning & Instruction (EARLI), Nicosia, Cyprus, August.

Jansen, A. R., Blackwell, A. F., & Marriott, K. (2003). A tool for tracking visual attention: The restricted focus viewer, Behavior Research Methods, 35(1), 57–69.

Kukreja, U., Stevenson, W.E., & Ritter, F.E. (2006) RUI - Recording user input from interfaces under Windows and Mac OS X, Behavior Research Methods, 38(4), 656–659

Lee, J. (2005) Vicarious learning, in: C. Howard et al. (Eds) Encyclopedia of Distance Learning, (Hershey, PA, Idea Group).

Romero, P., Cox, R.,  du Boulay, B., Lutz, R. & Bryant, S. (2007a) A methodology for the capture and analysis of hybrid data, Behavior Research Methods, 29(2), 309-317.

Romero, P., du Boulay, B., Cox, R., Lutz, R. & Bryant, S. (2007b) Debugging strategies and tactics in a multi-representation software environment, International Journal of Human-Computer Systems, 65, 992--1009.

Taylor, K.L. & Dionne, J-P. (2000) Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing, Journal of Educational Psychology, 92(3), 413-425.

Westerman, S.J., Hambly, S., Alder, C., Wyatt-Millington, C.W., Shryane, N.M., Crawshaw, C.M., & Hockey, G.R.J. (1996) Investigating the human-computer interface using the Datalogger, Behavior Research Methods, Instruments & Computers, 28(4), 603-606.

TLRP Resources for Research in Education by Teaching and Learning Research Programme is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.0 UK: England & Wales License