The Data Revolution and education post-2015: Considering the promise and the risks

The blog post by Joshua Muskin was published in Brookings on March 31, 2015. Sumandro Chattapadhyay gave his inputs.

This group was charged with evaluating the global state of data and deriving recommendations that permit data to live up to their potential as “the lifeblood of decision-making and the raw material for accountability.” The resulting report, A World that Counts, posits that “Governments, companies, researchers and citizen groups are in a ferment of experimentation, innovation and adaptation to the new world of data, a world in which data are bigger, faster and more detailed than ever before.” This, they assert, “is the Data Revolution.”

Central to the group’s arguments is the assertion that data are and must be treated as a “public good.”  Implicit is the notion that all data are good and that more data are “gooder.” Certainly there is much to laud and anticipate in the report’s definition of the Data Revolution and its recommendations. Perhaps most deserving of interest are the call for data that:

come “from other sources, such as qualitative data, citizen-generated data and perceptions data;”are generated “from all parts of society;”are “more detailed, timely and relevant;”increase “usefulness...through a much greater degree of openness and transparency,” while “avoiding invasion of privacy and abuse of human rights;” andlead to “more empowered people, better policies, better decisions and greater participation and accountability.”

The blog entry by Joshua Muskin was published in Brookings on March 31, 2015. Sumandro Chattapadhyay was quoted.

What could go wrong?

Currently neck-deep in research on the use of student learning assessment data in education, I suggest that this implicit assumption and the promise of unwaveringly positive outcomes are not guaranteed. As we experience the emerging revolution, there is every reason to embrace this promise and to take up arms to ensure its fulfillment. But, as history shows, many a revolution has derailed, claiming the benefits of the rhetoric for a few while leaving many victims. While working towards the noble and indeed crucial aims laid out in the report, vigilance will be vital—as will very strategic action to ensure that the revolution does not veer away from its vision.

It is hardly the aim of this seemingly premature appeal to vigilance to unseat completely the objectives and recommendations of the Expert Advisory Group. (Indeed, the Data Revolution has barely launched, though its roots are already solidly planted, as evident in education in the surge of countries involved in international and national testing. See, for example, Benavot and Tanner, 2007.) Rather, in the spirit of Charles Tilly’s 1963 “Analysis of Counter-Revolution,” my goal is to remind all those ready to enlist in the revolution that this initiative does not necessarily “unite the people.” Regardless the document’s comprehensive and progressive appearance, there are many who take different positions, with some fearing what a reign of big data, irrespective of its origins and nature, will yield once it moves to implementation in its myriad forms and locations. While the document features the poor as a main beneficiary of the revolution, Neva Frecheville (2014), among others, notes the absence of low-income country representatives on the panel. Arguably more significantly, the emergence of citizen-led assessments in education may be perceived as a sort of reaction against the hegemony of centrally managed data regimes, taking control of the data by the local poor to ensure that its use fully serves this population. While the report embraces such initiatives, it would seem important to secure their independence, alert to the risk of a usurpation and hijacking of the mission, the program and the resulting information.

Sumandro Chattapadhyay, research director at the Centre for Internet and Society in New Delhi, has pointed in “An Open Data Agenda for post-2015 Sustainable Development Goals” to a few likely opportunities for the revolution to go awry once operating. For one, the modern technology thrust obvious in the report does not translate automatically into heightened generation, retrieval, or understanding of information for the world’s poor majority. Quite simply, this group does not have automatic access to either the modern modes of communication or the institutions by which the agents of the revolution will transmit the results. Two, come the revolution, there is on the one hand no obvious guarantee that data will supplant politics in policy and decision-making. Yet on the other, big data may provide an even bigger and potentially obfuscating weapon in the hands of politicians’ and other leaders, whether public or not. As one learns in Statistics 101, the same numbers can justify very different conclusions; and those who control the data usually control the story: information is power. Three, the push for standards and comparability in data risk negating or at least undervaluing the validity of non-standard idiosyncratic data and other information that may emerge at the local level and still have great worth; and sometimes be more meaningful, even if not “scientific.” Finally, Chattapadhyay asks “Who is empowered by using (opened up) data?,” warning that the “falling costs of collection and archiv[ing] of data... create strong attractions towards gathering as much data as possible without specific objectives for their collection.” Furthermore, the “...availability of data at a global scale has massive commercial value, [the] unlocking of which may [also] not necessarily lead to positive impacts.” The threats of such an outcome to privacy should be obvious and are widely documented (see, for example, this post from the MIT Big Data Initiative). The fact that in education we are dealing with children raises the danger of this prospect even higher (see, for example, the U.S. Student Data Privacy Act).

What does this mean for the education sector?

In a world where data are “bigger, faster and more detailed than ever,” what are the opportunities and vulnerabilities for the local actors to whom Chattapadhyay refers; in education, these comprise teachers, other “front-line” educators, and parents. This line of inquiry was also central to the analysis undertaken by the Learning Metrics Task Force Learning Champions at their February meeting in Kigali—the topic of a recent blog of mine.

There are many claims of data as a positive asset in education, and these are often matched by actual experience in many settings. Prominent are the impact of information on policy; on the allocation of resources (financial, material and human); on the validation, revision or termination of a particular strategy or initiative; and ultimately on strengthening quality and equity. At the school and classroom level, formative assessment (with feedback) is regarded by many—e.g., see Hattie (2011); Barber & Rivzi (2013), p.65; and Black & William, in Lucas, Claxton & Spencer (2009), p. 3—as the most robust factor in learning. It is a problem, though, that there is little evidence that system-level assessments have real “meaning for...teachers” (Long, Dunne & Mokoena, pg. 158) and that formative assessment is usually either poorly done or not at all (see Shiohata, 2015).

Three fears concerning the Data Revolution come quickly to mind, adding to those mentioned above like the privacy issue. One is that data on what is easily measurable in learning will overwhelm other aspects of information on education that are equally (or even more) essential, but are not easily measured, particularly in the classroom. Conversely, there is a strong risk of eschewing measures that can be both more accurate and more nuanced in the classroom, and therefore more useful in guiding learning. An example might be the assessment of personal competencies, such as creativity, curiosity, confidence, and collaboration. While possible to do, the methodological challenges of devising valid, standardized, and comparable assessment instruments for these are prodigious, particularly across cultures, and any measures would likely be difficult to interpret into pedagogic strategies. In contrast, a school-level inquiry by teachers into the characteristics, observation, and cultivation of these same traits can be very precise, strategic, and effective, especially when facilitated as captured in the OECD background paper by Lucas, Claxton and Spencer.

The third fear is that data will influence policy, planning, and practice to a degree that exceeds the reliability of the numbers and the “external validity” (see Rodrik) of their analysis—sometimes considerably. Such would be the case, as Rodrik explains, when applying the subject of a favorable randomized controlled trial from one setting to another. Related to this, an outsized faith in statistics or a reliance on overly sophisticated analyses—lured to the shoals by the siren song of what is possible—may simply undermine basic common sense; for example, can we not sometimes rely on a teacher to identify which students are struggling and need further help without imposing frequent, narrowly defined tests? (I am reminded of a pun shared by my Swiss friend, Pierre Jaccard, who reformulates the French phrase “panne d’essence”—a breakdown due to running out of gas—to indicate the all-too-frequent occurrence in policy circles of a “panne de sens,” or running out of common sense.)

Where I perceive “red flags” to arise and start flapping most frenetically around data regards its use. In particular, both the U.N. document and much of the discussion around student learning assessment highlight the value of data for central decision-making and policymaking: As stated in A World That Counts, “Without high-quality data providing the right information on the right things at the right time, designing, monitoring and evaluating effective policies becomes almost impossible.” The report hardly ignores local level actors and institutions—in education, the classroom, the school and parents—but this level is definitely under-represented and vaguer in terms of approach. But here is where the greatest clarity and strategic precision is required.

What the Data Revolution risks overlooking most are those education actors who are furthest from policy (even excellent policy), but upon whom positive outcomes ultimately depend. These are teachers, other educators, parents, and students. Even though the Advisory Group claims to speak on their behalf and to embrace them as part of the movement, are they really co-revolutionaries? While the citizen-led assessments clearly are eager to engage with the authorities, does their legitimacy and influence derive from their being fully within the system or, as I suggest above, does it exist because of their status as an honorable and powerful counter-balance? As politicians and policy leaders demand ever more and ever more sophisticated data, and as technology makes the collection, analysis, and dissemination of these data more elaborate and extensive, how great is the risk of overwhelming the capacity of classroom teachers, other local educators, parents, and other local institutions to absorb, let alone make sense of, the information and to use it appropriately and effectively? Are we running in the opposite direction of a solution to the oft-repeated issue of low-capacity with data, especially in the developing world? What does the Data Revolution furnish truly and directly to these critical actors in terms of information, resources, capacity, and authority to make decisions that improve teaching and learning for better education and to foster broader sustainable development outcomes?

The Data Revolution surely has much of vital importance to offer in gathering, organizing, and analyzing the vast array of information that relate to education and learning. As indicated above, the aim here is not to assail the expanded definition of data and their use per se. It is, however, an exhortation to gather, analyze, disseminate and use data on learning to inform decisions and actions that link directly to real education circumstances, challenges, and goals. The language of the Expert Group report claims this as its objective. The admonition addresses the need to be sure that this crucial dimension not get lost or overwhelmed in practice. So, as we hail the intent and elements of the Data Revolution and endeavor actively to attain its full promise, it will be vital to remain keenly vigilant and, if necessary, not to hesitate in summoning a counter-revolution in order to defend fully the ultimate goals: appropriate data in the service of all, including most critically the most marginalized of the world.