The use of large scale data-sets in educational research

Introduction

In the last few decades, there has been an unprecedented increase in the availability and quality of large-scale data sets that are suitable for use in education research. Analyses of these data have the potential to radically improve the robustness and generalisability of educational research (See for example our project on Widening Participation in Higher Education: A Quantitative Analysis http://www.tlrp.org/proj/wphe/wp_vignoles.html ).

In an exploratory project funded by TLRP we hosted a series of workshops around the theme of using large-scale data sets in education research. The purpose of the workshops was to encourage and facilitate education researchers in their use of such data and to highlight any difficulties researchers have encountered when trying to access and use such data. The workshops focused on some specific data sets, such as the Millennium Cohort Study (MCS http://www.cls.ioe.ac.uk/studies.asp?section=000100020001 ), as well as particular methodological issues, such as the research potential of linked administrative and survey data. It also focused on some substantive research themes, such as adult learning.

In this resource we summarise the key messages that emerged from the workshops (key findings), as well as highlighting the particular strengths and weaknesses of a select number of specific data sets with regard to their use in educational research (important data sets). We then illustrate these themes in the context of research into adult learning (the case of adult learning). Lastly, we conclude with a series of recommendations about how large scale data sets might be more effectively used in education research, attempting to also identify the barriers that currently prevent researchers from using these data (next steps).

Key findings

Quantitative social research is being quietly transformed as better quality administrative data, often linked to richer survey data, becomes increasingly available to researchers. Quantitative social research has long relied on using survey data (both cross-sectional and longitudinal) for substantive and methodological work. In recent years however, policy-oriented research has turned to administrative data as a research resource, as access to government administrative datasets has opened up across a range of fields. In the UK , the Department for Children, Schools and Families (link) leds the field, in terms of making high quality administrative data available to academic researchers. Access to comprehensive administrative data on students in the education system has facilitated more robust, generalisable research into education issues, although not without problems. As a result of good access arrangements, the National Pupil Database (NPD – a user group can be found at http://www.bristol.ac.uk/Depts/CMPO/PLUG/whatisplug.htm ), which contains information on all school children in the UK , has increasingly been used by researchers to look at what determines children's outcomes in schools (e.g. Wilson et. al., 2005 and Steele et. al., 2007). Other government departments have also started to make administrative data available to researchers. For instance, in the UK, Department for Work and Pensions (DWP) administrative data have been used to evaluate some of its employment programmes such as the New Deal programmes (e.g. Blundell et. al., 2004). Access to these data may also prove useful to education researchers, as linking different sources of administrative data may “fill in the missing pieces” of any particular administrative data set.

Administrative datasets have many advantages, not least that they are effectively a census of individuals. However, one of the disadvantages of administrative data is that they typically do not contain very rich information. The most recent innovation has therefore been to link these comprehensive but sparse administrative data sets to richer survey data. For example, the NPD has very detailed information on children's educational outcomes but does not have any information on the labour market status or education of the child's parents. As it is well known that a child's educational attainment is influenced by their parent's educational attainment, this omission is serious. Linking the NPD to a rich survey, such as the Longitudinal Study of Young People in England (LSYPE - http://www.esds.ac.uk/longitudinal/access/lsype/L5545.asp ) is a means to marry the benefits of both types of data. Linking administrative and survey data enables the education researcher to have the best of both worlds; the large sample sizes and longitudinal elements from administrative data are linked with the richness of survey data. These linked data sets are a hugely important research resource.

Some linking has already occurred but more is planned. Survey data linking to administrative data is planned or has already taken place for the National Child Development Survey (NCDS), the British Cohort Study (BCS), the Avon Longitudinal Study of Parents and Children (ALSPAC) and the new Millennium Cohort Study (MCS). These, and other longitudinal surveys (such as the British Household Panel Survey (BHPS - http://www.iser.essex.ac.uk/ulsc/bhps/ ) and the Longitudinal Study of Young People in England (LSYPE)), unlike administrative data, have very rich information on individuals' personal characteristics, which is ideal for quantitative modelling to investigate many education research questions.

There are limitations to the potential for using administrative data. The exceptionally good range of administrative information held on all children in the education system, including personal characteristics, school characteristics and educational achievements, can be contrasted to the rather sparser administrative data held on adult learners. Partly the lack of good quantitative data on adult learning is a timing issue. When children today grow up into adults, we could potentially have a full administrative record of their educational achievements in childhood (from their NPD records). However, being able to access an adult's childhood records is not necessarily a foregone conclusion. A key recommendation from our discussions is for the government to adopt a unique learner number for all types of learning, or to make better use of existing unique identification numbers, such as individuals' National Insurance Number, to track the education and learning of individuals. Only then will we be able to improve our quantitative data on educational trajectories throughout the life course, and specifically beyond initial education. We understand that the Department for Children, Schools and Families is adopting a unique learner system but it is not clear how extensively this number will be used, particularly by other government departments.

Of course, data collection by governments, and in particular the linking of individuals' personal information across government departments, raises ethical issues about oversight and confidentiality. However, from a research perspective, it is clear that the benefits of having one unique identification number, with which all data on an individual could be linked, would be enormous. Ethical issues arising from such linking therefore need to be debated more widely, and with the involvement of all parties involved, including government, members of the public, those concerned with civil liberties and researchers.(Link to TLRP ethics resources here)

Another need is for better dissemination of information on data availability and guidance on the use of all kinds of data, but particularly administrative data. Whilst guidance notes for particular surveys, such as the National Child Development Study, are generally comprehensive, information on particular fields in administrative data sets, such as the Pupil Level Annual School Census (PLASC)/National Pupil Database (NPD), is often weak. Such data are primarily collected and cleaned for administrative purposes, not specifically for research. There would be advantages in involving the research community more directly in the preparation of administrative data for research purposes.

The UK has a huge range of secondary data sets available and of interest to education researchers. These include the National Child Development Study (NCDS), the British Cohort Study (BCS), the Labour Force Survey (LFS - http://www.data-archive.ac.uk/findingData/lfsTitles.asp ), the British Household Panel Survey (BHPS), the Longitudinal Survey of Young People in England (LSYPE a.k.a. Next Steps). Some of these survey data sets have a long history of use for education research, e.g. the NCDS. Others are not so widely used by education researchers (e.g. the BHPS), partly because these data sets are extremely complex, particularly for researchers who are developing their quantitative research skills. The data sets are not only large and complex, but they also have panel structures that make analysis challenging (although often more robust too). Education researchers are need better information on firstly, which data sets might hold education related information of interest and secondly, where they might get straightforward guidance on access and basic manipulation of such data. Guidance targeted specifically at education researchers, documenting specific variables and, for example, providing “cleaned” and usable versions of the data, has the potential to substantially increase the use made of these data sets in education research.

Important data sets

This section identifies key data sets that have been used successfully by education researchers, and other data sets that have the potential to be used in education research. We also discuss their strengths (and any weaknesses). The list is not meant to be comprehensive.

The three data sets discussed give a clear flavour of the enormous possibilities and indeed the limitations of such data for quantitative education research in the UK . However, it should be remembered that certain questions can only be answered using qualitative data. Using quantitative data as a sampling frame for qualitative research is an area of methodology that needs to be developed further in education research. Better quantitative analysis can be used to supplement the qualitative work currently being undertaken in education research, to set it within a wider national context, making the findings more generalisable and of greater interest to policy makers.

National Pupil Database (NPD)

The National Pupil Database (NPD) includes information on all pupils in England in State schools. The NPD consists of a Pupil Level Annual School Census (PLASC), containing information on each child's personal characteristics, as well as their attainment in Key Stage Tests at age 7 (levels attained in reading, writing and mathematics at Key Stage 1), age 11 and 14 (levels attained and the marks achieved in English, mathematics and science at Key Stage 2 and 3), age 16 (attainment in all subjects at GCSE or equivalent at Key Stage 4) and post 16 (attainment in AS/A levels or equivalent at Key Stage 5). The annual PLASC data has individual information on ethnicity; special educational needs; free school meals eligibility; absences, current address and the date the pupil first entered the school and the date the pupil left the school (if applicable).

The comprehensive nature of the NPD data is currently a real asset to educational researchers. Although the NPD has only been available to researchers for around five years, already numerous articles have been published using these data and the data are an internationally recognised research resource. Research topics that have been addressed using NPD include the following:

• School effectiveness and value added ( Taylor , 2007);
• School competition (Burgess and Slater, 2006; Gibbons, Machin and Silva, 2005);
• School segregation (Allen, 2007; Allen and Vignoles, 2007; Burgess and Briggs, 2006);
• The gender gap (Machin and McNally, 2005);
• Gaps in achievement of different ethnic groups (Burgess, Wilson and Briggs, 2005);
• The effects of school resourcing on achievement (Machin, McNally and Meghir, 2007; Steele, Vignoles and Jenkins, 2006; Levacic et al., 2005).

However, some concerns are expressed about the lack of information about these data and the construction of particular variables. The documentation that accompanies these data is perceived to be quite poor, although some useful guides are emerging (Ewens, 2005) and some user guides have been provided by the DCSF ( http://www.bris.ac.uk/Depts/CMPO/PLUG/userguide/guide.htm ). The lack of information on key variables, and confusions about the nature of some data fields, mean that many people may be deterred from using these data. The DCSF has responded to researcher concerns and has set up a user group (PLASC/NPD User Group, known as PLUG, whose website can be found at http://www.bris.ac.uk/Depts/CMPO/PLUG/index.htm ). PLUG is designed to help researchers using these data to come together to discuss both the data itself and the appropriate methods of analysis that might be used with the data. Further work is still needed however, to ensure better documentation of the data and clearer and, equally importantly, more rapid ways of securing access to these data.

Crucially important is the potential to linking together administrative data from different parts of the education and training system. The DCSF has already linked data from schools (NPD), Further Education (Individual Learner Record) and Higher Education (Higher Education Statistics Agency data), and these data have been made available to some researchers to look at issues around widening participation in higher education. These linked data will basically provide comprehensive data on a person's entire trajectory through the education system, including the nature of their compulsory schooling and school achievement, their FE participation (if any) and subsequently whether they go on to higher education. These linked data can be used to analyse education trajectories and educational achievement for an entire cohort. The one weakness of such linked data is that it can only cover young people, i.e. individuals who pass through the education system and go straight on into FE and HE. If an individual leaves the education system and returns to learning later in their life, currently they will not be picked up in these data (but see Case study of adult learning).

Longitudinal Study of Young People in England (LSYPE)

The DCFS's LSYPE survey is perhaps unique in the UK in that it was set up with the link to administrative data in mind. Specifically it was set up such that only children aged 13/14 in 2003 who consented to having their data linked to the NPD schools administrative data participated in the survey. This has ensured that linking to administrative data is automatic for all the individuals in LSYPE, sometimes also known as the Next Steps survey. The overall purpose of LSYPE is to provide a panel study of young people's education and training progress through the system, based on linked administrative and annually collected survey data. The survey starts with young people aged 13/14 and follows them through to the end of compulsory education, and then beyond into other education or training and into the labour market. There is some over sampling of deprived children and certain ethnic minority groups.

The LSYPE is seen as an important educational research resource, particularly for those interested in transitions; such as the transition at age 16, the transition into FE and HE, and later the transition into the labour market. The administrative linking of LSYPE is crucially important for two reasons. Firstly, the link to NPD ensures that we have an historical record of these young people's educational achievement earlier in the school system. Secondly, the administrative link can be used to attempt to deal with the attrition problems that normally plague such studies (for example the Youth Cohort Study). We have information on the entire cohort from which LSYPE is sampled, so, if responders disappear from the survey, we know their characteristics relative to the cohort as a whole and can therefore allow for non-random attrition that often produces biases in quantitative modeling.

Millenium Cohort Study (MCS)

Another important data resource for education researchers is the MCS. This is a longitudinal panel survey, run by a consortium headed by the Centre for Longitudinal Studies, and funded by the ESRC and a range of government departments. The first sweep of the survey took place in 2001-2004, collecting information from the parents of nearly 19,000 babies born in the UK over a 12 month period and living in the UK at age 9 months. Since then there have been two further sweeps at age 3 and age 6. The scope of MCS is much broader than the LSYPE for example, and very similar to that of the earlier cohort studies, such as the National Child Development Study (cohort born in 1958) and the British Cohort Study (cohort born in 1970). The remit of MCS therefore covers the social conditions surrounding the child's birth and early childhood, including the health and socio-economic circumstances of the child's parents, the child's own heath and the child's development more generally. It has been used extensively by academic researchers on a range of different topics (Bartington et al., 2006; Dearden et al., 2005; Goldthorpe and Jackson, 2007; Hansen et al., 2006).

The MCS has already been linked to administrative birth records and linkage to the educational records from the Foundation Stage Profile in the NPD is currently under negotiation. The linkage provides completely unique data for use by researchers interested in education. This linkage in the MCS will be expanded at the age of 7, when there will potentially be links to parental benefit data, DWP programme data, pension data, employment data and earnings data (held by the DWP in the Work and Pensions Longitudinal Study (WPLS)) and to parental and children's health administrative records as well as continued linkage to the NPD.

Research possibilities made possible by linking administrative data to the Millennium Cohort Study include:

• linking Key Stage achievement to background family characteristics and the home in which the child grew up;
• comparing early cognitive ability tests at age 3 and 5 with performance in later Key Stage exams;
• examining the associations between paths through the Key Stages; looking to see if teacher reports of the child's ability match performance at the Key Stages;
• using information at the school level to look at school context effects on the cohort child's ability (ie. percentage of FSM);
• seeing how well the cohort child does compared to his/her peers (in year at school, at school overall, and/or in their neighbourhood);
• examining the impact of siblings educational outcomes on the cohort child's educational outcomes and examining whether school choice is related to school performance for the cohort child.

Linking to NPD could have benefits for the MCS survey. If existing information about the school were matched to survey data such questions would not have to be asked in the survey. Cutting the length of the survey could reduce respondent burden and cost. Alternatively, similar information obtained from the administrative and survey data would allow researchers to gauge which is most accurate and reliable. In addition, the administrative records could be used to supply contextual information about the school not accurately recorded elsewhere – e.g. Absences, SEN, school moves.

The MCS has huge potential, nor just to study the pre-school and schooling experience of young children born at the turn of the century, but also to study their progress through the education system and beyond. The link to administrative data is also a potential strength of the MCS, although administrative data is currently only being negotiated (and funded) on a case-by-case basis. As data linking has not been built into the survey from inception (as is the case with LSYPE), the full potential of the MCS remains unknown. If full data linking to DWP and DCSF records takes place, the MCS will no doubt be one of the most important data resources for education researchers for many years to come. Even without links to administrative data, the MCS is a rich and useful survey, not least because it offers the opportunity to look at cross cohort changes in education and learning across three broadly comparable cohort panel studies (NCDS, BCS and MCS).

British Household Panel Survey

The British Household Panel Survey began in 1991 when around 10,000 adults in around 5,000 households were interviewed. These same individuals have been interviewed annually since then. Information on a wide range of topics is collected, including detailed questions on income, employment, household composition, education and housing. In addition, a number of topics are included on a less regular basis, such as wealth (every five years). Individuals who leave the original households to form new households are followed and all adult members of these new households are also interviewed. Similarly, new members joining sample households become eligible for interview and children are interviewed as they reach the age of 16. Since 1994, children aged 11-15 also complete a short interview. Extension samples of 1,500 households in each of Scotland and Wales were added to the main BHPS sample in 1999 to enable independent analysis of each country. In 2001 a sample of 2,000 households was also added in Northern Ireland . As a result, the most recent sample in 2006 (Sweep 14) contains around 10,000 households.

The BHPS has not been widely used in education research but has some useful features. It collects data on adult learning and education, and can therefore be used to look at learning that takes place after compulsory education. It is also a relatively rich data set and has been used to look at issues of intergenerational mobility and well being, for example. The main drawback of the BHPS, apart from modest sample sizes for some analyses, is that it is a complex data set. It was identified as a prime candidate for needing specific guidance so that it could be more widely used by education researchers.

The case of adult learning

Whilst there were numerous data sets that can be used to undertake education research on issues pertaining to the learning and education of children and young people, the situation is somewhat different for research on adult learning. Adult learning has been an under researched area of education, at least with regard to quantitative research methods, and that this has reflected the serious limitations of existing large scale data.

Existing research projects that have used quantitative methods to address issues around adult learning include research projects covering topics such as the determinants and nature of FE participation and the effectiveness of basic skills courses. However, there is a need for the research community of quantitative researchers to develop some sense of how particular issues pertaining to adult learning and education might be better researched using large-scale quantitative data, and the extent to which existing data resources could be utilised for this.

Peter Lavender, from National Institute for Adult and Continuing Education (NIACE), has made the case for a broad ranging research agenda into issues around adult learning and called for a public inquiry in order to develop a coherent strategy on adult learning for the modern world. Three main research questions are:

•  What are the current and recent education and labour market policies affecting potential and actual adult learners in post-compulsory and higher education in the three administrations of the UK , and how do they impact on participation and achievement for participants?

•  What is the place of, and potential for, adult learning in supporting other national, regional and local government policies in the UK ?

•  What is the performance on investment in adult learning and outcomes in economic performance in other OECD countries – measured by GDP, innovation and wider measures of human development and inclusion?

(Dr Peter Lavender, Deputy Director NIACE at a TLRP seminar on Adult Learning at the Institute of Education, 18 May 2006.)

Also identified was a range of specific topics that merited further research, from an adult learning perspective, including: the role of adult learning in a world of globalisation and technological change; the potential of adult learning in preventing poverty or helping individuals out of poverty; the potential for adult learning to improve well-being and happiness; and the way in which adult learning can contribute to overcoming the economic challenges posed by demographic changes, such as the aging population. In addition, the role of adult learning in promoting sustainability, civic participation, and community involvement remains under researched. Further research into the nature of provision of adult learning, including public and private sector provision, is also a pressing need. Any such public inquiry into adult learning would need to be backed by persuasive high quality evidence, and particularly quantitative evidence.

A major area of research interest is the role of adult education in improving outcomes for individuals and for raising productivity. Alison Wolf 's TLRP project Workplace learning and adult basic skills (link) tackles the thorny issue of basic skills and work-based provision. She is investigating whether work based basic skills provision is actually working in practice, i.e. whether learners' skills are really improving following basic skill training at work. She is also attempting to determine whether employers benefit from this provision, i.e. whether there is a productivity dividend from basic skill training at work. This TLRP project was based on a primary data collection process followed by both qualitative and quantitative analysis of the data. The research team faced many challenges, in particular:

• the problems posed in trying to secure an adequate enough sample size,
• the myriad issues arising when trying to quantify the impact of such small scale and diverse provision,
• the problems in interviewing hard to reach learners and the difficulty in really measuring progress in this type of learning.

Despite these problems, the project illustrates the benefits of primary data collection, as the data needed are specific and varied and the research questions may change during the course of the project, as preliminary evidence emerges. In the case of this project, it became apparent that many learners being reached in the survey were actually students learning English as a Second Language. Primary data collection has allowed the research team to respond appropriately.

Flora Macleod and Paul Lambe (University of Exeter) have taken a somewhat different approach to investigate the different pathways through adult learning, and in particular the role of occupation and family. They have opted to use secondary data, undertaking a cross cohort analysis using the British Household Panel Survey. A number of secondary data sources, including the BHPS, are currently under utilised for education research on adults. One reason for this is that a huge time investment is needed by the researcher to analyse such data, as the data sets are large and complex. Making available better guidance specific to education researchers would be helpful to encourage greater use of these data sets.

There is an important and growing role of administrative data in education research. Yet administrative data on education is more focused on young learners. Nonetheless administrative data on FE and HE is available in the form of the Individual Learner Record and the Higher Education Statistics Agency database respectively. Further Education data from the Individual Learner Record might be used to investigate adult learning, following on from a project undertaken by NFER for the Learning and Skills Council analysing patterns of post 16 provision and issues around participation, retention and achievement. The ILR contains information on learning undertaken in FE including work-based learning (WBL), adult and community learning (ACL) and contains information on learners i.e. their characteristics (date of birth, gender, ethnicity and disability) and their learning aims. There are separate (but now linked) data bases on learners, as well as on each of their learning aims, covering information such as their institution of study, area of study, start and end date, outcomes, level, guided hours and information on fees. In recent years these ILR data have been linked into both schools data ( Pupil Level Annual School Census/ National Pupil database) as well as for the cohort potentially entering HE in 2004, higher education data from HESA (Higher Education Statistics Agency). These linked data sets have potential to examine trajectories through the education system for younger students but do little to enhance the available data on older students.

The ILR is a comprehensive data set, in terms of providing the researcher with information on the nature of study being undertaken, although it is highly complex to use (this may have got easier in the latest years due to linkage with the National Pupil Database/Pupil Level Annual School Census). However, the data contain only quite minimal information on learners themselves and, for adult learners, the ILR often lacks good information on prior experiences of learning and educational achievement, which are central for many research projects on adult learning.

This highlights a number of issues. Firstly, there are common themes across all areas of education research, such as the increasing importance of administrative data in quantitative education research. However, administrative and survey data on adult learners is weaker and in particular, we know very little about the education background of adults entering FE and HE. The administrative data sets for FE and HE (i.e. the ILR and HESA data sets respectively) do not provide sufficient information on the prior learning of adults. A possibility for the future is that all individuals have a unique identification number that would enable researchers to match subjects' prior academic record in school with any subsequent adult learning that may take place later in life. Whilst this is ambitious, and may be ethically contentious, from a research perspective it will enable us to undertake robust and genuinely life course research on learning.

Next steps

The TLRP seminar series distilled a series of action points. These recommendations should provide the research community with an agenda for action, and are particularly aimed at funding bodies, such as the ESRC, and those controlling data sources, such as government departments. The recommendations are:

• Government departments need to provide better guidance on the contents of administrative data sets. This may require a resource contribution from the research community, recognising that the purpose of government data is not primarily research. Collaboration between funding bodies and government departments is needed to take this forward.
• Access to administrative data continues to improve but again a more transparent and better-publicised access system would increase use of these types of data. Currently access conditions vary hugely across government departments and a more co-ordinated approach is urgently required.
• Many government departments are currently considering the importance of data linking between different administrative data sets and between survey and administrative data. This would appear to be best achieved by a unique identification number for each individual but the ethical issues arising need open debate.
• Education researchers, particularly those at the beginning of their career in quantitative research, need better guidance on the basic manipulation of complex panel surveys and other appropriate data sources. Such guidance needs to be targeted at education researchers rather than too generic. Provision of better web materials and also short courses on quantitative analysis methods using specific data sets were two suggestions to tackle this problem.
• One possible way to address many of the recommendations above is to keep an online database of education data resources that could be added to by education researchers themselves, along the lines of Wikipedia. The content would of course require managing so some infrastructure to support such an endeavour would be needed. This is an issue for the research community itself to resolve, perhaps with assistance from interest parties such as the British Education Research Association.

Details can be found at http://www.cls.ioe.ac.uk/text.asp?section=000100010002

For those wishing to contact PLUG, their email is plug-plasc@bristol.ac.uk .

References

Burns, C.M. and Myhill, D.A. (2004) Inactive or Interactive ?  Cambridge Journal of Education, 34 , 1, 35-49.

Bloome, D., Carter, S. P., Christian, B. M., Otto, S., & Shuart-Faris, N. (2005 ) Discourse Analysis and the Study of Classroom Language and Literacy Events: a microethnographic perspective . Mahwah , NJ : Lawrence Erlbaum Associates

Daniels, H. (2001) Vygotsky and Pedagogy . London : Routledge/Falmer.

Edwards, A.D., Westgate, D.P.G. (1994) Investigating Classroom Talk (Second Edition) . London : The Falmer Press

Edwards, D, and Mercer, N. (1987) Common Knowledge: The development of understanding in the classroom . London : Methuen/Routledge.

Mercer, N., Littleton , K., Wegerif, R. (2004) Methods for studying the processes of interaction and collaborative activity in computer-based educational activities. Technology, Pedagogy and Education , 13, 2, 193-209.

Mercer, N. & Littleton, K. (2007) Dialogue and the Development of Children's Thinking. A sociocultural approach . London : Routledge.

Monaghan, F. (1999) Judging a word by the company its keeps: the use of concordancing software to explore aspects of the mathematics of register. Language and Education, 13 , 1, 59-70.

Seedhouse, P. (2004). The interactional architecture of the language classroom: A conversation analysis perspective . Oxford : Blackwell.

Wegerif , N., Mercer, N. and Dawes, L. (1999). From social interaction to individual reasoning: an empirical investigation of a possible socio-cultural model of cognitive development. Learning and Instruction , 9 , 6, 493-516.

TLRP Resources for Research in Education by Teaching and Learning Research Programme is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.0 UK: England & Wales License