Name: Li Wenjian University No.

3030018336 Faculty of Education

Does PISA keep its promise? ------- Through analyzing the definition and components of scientific literacy in PISA
Introduction While pupils and teachers from Toronto to Wroclaw, London to Rome returning to classroom after their summer break, schools themselves in the passing September were caught up in a global battle of ideas with their countries education at the forefront of political debate and reformers being desperate to improve their nations performance by drawing examples of good practice from all over the world after the availability of another round of PISA results (The great school revolution, 2011). The Program for International Student Assessment (PISA) was created by the Organization for Economic Cooperation and Development (OECD) in 1997 as a collaborative effort by member countries and a number of non-member countries in order to provide national governments and others with policy-oriented and internationally comparable indicators of student achievement on a regular basis and in timely manner. Administered every three years, PISA is a series of survey that assess the 15-year olds knowledge and skills in the domain of reading, mathematics and

science literacy with its rotating prior focus on each domain: PISA 2000 on reading, PISA 2003 on mathematics, PISA 2006 on science and PISA 2009 on reading again. According to PISA 2009 Results: What Students Know and Can Do, till 2009, the number of the participating countries/economies arrived to 65 and altogether, which represents nearly 90% of the world economy (OECD, 2010). And according to Assessing Scientific, Reading and Mathematical Literacy: A framework for PISA 2006, together with the PISA 2000 and PISA 2003 surveys, PISA 2006 completes the first cycle of assessment in the three key subject areas. PISA is now conducting a second cycle of surveys, beginning in 2009 with reading as the major subject and continuing in 2012 mathematics and 2015 science (OECD, 2007). The availability of the results of PISA each time causes much public debate, at least in some countries, about the educational, institutional, economical and social reasons for the apparent ranking of some countries over others ( Dohn, 2007).

One of the PISAs unique features, according to PISA 2009 Results: What Students Know and Can Do, is its innovative concept of literacy , which refers to the

capacity of students to apply knowledge and skills in key subject areas and to analyze, reason and communicate effectively as they pose, interpret and solve problems in a variety of situations (OECD, 2010). This vision for scientific literacy that highlights learners use of science in real-life contexts, invites comparisons to progressive movements in science education and in turn, science educators with progressive goals have increasingly focused on socio-scientific issues (SSI) as learning contexts in last

decade (Sadler & Zdidler, 2009). In this article, I will first explore the conceptual meaning of scientific literacy and then through examining the definition and

components of the scientific literacy in PISAs framework, to see whether PISA fulfills the goal it promised.

A brief literature review on scientific literacy and Millers three dimensions According to DeBoer (2000), scientific literacy is a term that has been used since the late 1950s to describe a desired familiarity with science on the part of the general public and now has been an overarching aim of science education for decades. As a matter of fact, the publics interest in and concern about elements of the concept of scientific literacy can be traced back to the beginning of last century and became the subject of systematic study by a broader public around 1930s when Dewey charged how the responsibility of science can be fulfilled by pointing out the importance of scientific attitude (Miller, 1983). According to Miller, a number of science educators like Ira C. Davis, Victor H. Noll and A. G. Hoff at this period of time began to consider the formal definition and measurement of scientific attitude and virtually all of the empirical work done before the Second Would War had focused on the development of a scientific attitude. Given the important dual context of support for science and science education, numerous authors began to promote various aspects associated with scientific literacy and vast number of literature related to the concept of scientific literacy was found between 1974 and 1990 with majority being published after 1980. However, despite the great attention paid to the concept, like other

classical terms like happiness, liberty and justice that we assume to contain simple and desirable qualities but that under closer examination become vastly more complex and often elusive (Venezky, 1990), interpretations and definitions of scientific literacy have never arrived to a consensus agreement among scholars (Laugksch, 2000). Table 1 shows the history of conceptual development of scientific literacy (DeBoer, 2000; Laugksch, 2000). Table 1.The history of the conceptual development of scientific literacy Snow, (1962) Pella et al. (1966) sharp division between literary intellectuals and scientists (a) Interrelationships of science and society; (b) Ethics that control the scientist in his work; (c) Nature of science (d) Difference between science and technology; (e) Basic concepts in science (f)Interrelationships of science and the humanities I. The scientifically literate person understands the nature of scientific knowledge. II. The scientifically literate person accurately applies appropriate science concepts, principles, laws, and theories in interacting with his universe. III. The scientifically literate person uses processes of science in solving problems, making decisions, and furthering his own understanding of the universe. IV. The scientifically literate person interacts with the various aspects of his universe in a way that is consistent with the values that underlie science. V. The scientifically literate person understands and appreciates the joint enterprises of science and technology and the interrelationship of these with each and with other aspects of society. VI. The scientifically literate person has developed a richer, more satisfying, and more exciting view of the universe as a result of his science education and continues to extend this education throughout his life. VII. The scientifically literate person has developed numerous manipulative skills associated with science and technology Three categories of scientific literacy, namely practical, civic,

Showalter (1974)

Shen (1975)

Branscombs (1981)

and cultural scientific literacy. Eight different categories of scientific literacy: (a) Methodological science literacy. (b) Professional science literacy. (c) Universal science literacy; (d) Technological science literacy; (e) Amateur science literacy; (f) Journalistic science literacy; (g) Science policy literacy. (h) Public science policy literacy.

Miller (1983)

Three dimensions of scientific literacy: (a) An understanding of the norms and methods of science (i.e., the nature of science). (b)An understanding of key scientific terms and concepts (i.e., science content knowledge). (c) An awareness and understanding of the impact of science and technology on society. American Association for Scientific literacy attainable by all students: the Advancement of (a)Being familiar with the natural world and respecting its unity. (b)Being aware of some of the important ways in which Science(AAAS) (1989) mathematics, technology, and the sciences depend upon one another. (c)Understanding some of the key concepts and principles of science. (d)Having a capacity for scientific ways of thinking. (e)Knowing that science, mathematics, and technology are human enterprises, and knowing what that implies about their strengths and limitations. (f)Being able to use scientific knowledge and ways of thinking for personal and social purposes. Hazen and Trefil (1990) Believe that scientifically literate individuals should be able to place news of the day about science in a meaningful context. Shamos (1995) Three forms of scientific literacy: The first form: cultural scientific literacy. The second form: functional scientific literacy. The third form and level: true scientific literacy Layton, Davey, & Jenkins, 1986; Layton, Jenkins, Macgill, & Davey, 1993 Science for specific social purposes: a functional view of scientific literacy in which the meanings and social uses which science has for members of the adult public are explored

Three different interpretations and uses of literate are considered here: literate as

learned; literate as competent; and literate as able to function minimally in society and thus generally speaking, two features of the summary scheme of different definitions of scientific literacy can be identified: the first feature is related to the relative or absolute nature of the scientific literacy concept; the other one is the extent of involvement in and with society (DeBoer, 2000). As a matter of fact, from the above analyses, we may probably draw a conclusion that the three interpretations of literate and the two features summarized from the overview of the history of the development of the concept of scientific literacy overlap with Millers three dimensions of his concept of scientific literacy: the fist dimension is understanding the process of science; the second dimension is understanding basic scientific terms and concepts; the third dimension of scientific literacy concerns an understanding of the impact of science on society(Miller, 1989). And this is also illustrated by Chi (2009):
As illustrated above, a trend could be identified among different levels of scientific literacy from simply recognizing the scientific terms and questions in everyday lives, to understanding the scientific concepts and processes and, finally, to understanding the nature of science and the complex relationships between science, technology, and society. (p 1066).

The Analyses of the definition and components of scientific literacy of PISA The definition of scientific literacy for PISA 2006 and PISA 2009 refer to an individuals (OECD, 2006 &2009): Scientific knowledge and use of that knowledge to identify questions, acquire new knowledge, explain scientific phenomena and draw evidence-based conclusions about science-related issues. Understanding of the characteristic features of science as a form of human

knowledge and enquiry. Awareness of how science and technology shape our material, intellectual, and cultural environments. Willingness to engage in science-related issues and with the ideas of science, as a reflective citizen (OECD 2006, p. 23; OECD 2009, p128). Definitions of scientific literacy in PISA 2000 and 2003 are: Scientific literacy is the capacity to use scientific knowledge, to identify questions, and to draw evidence-based conclusions in order to understand and help make decisions about the natural world and the changes made to it through human activities. (OECD, 2000; 2003). Definitions of the scientific literacy of PISA listed above illustrate the new changes in its definition: the first point of the definition for PISA 2006 is similar to the earlier definition and the other three points appear to be newly added aspects. According to Lau (2009), the second point also refers to some aspects of the epistemological and sociological nature of science as described in OECD (2006): The collection and use of data: data collection is guided by ideas and concepts (sometimes stated as hypotheses) and includes issues of relevance, context and accuracy; the tentative nature of knowledge claims; an openness to skeptical review; the use of logical arguments; and the obligation to make connections to current and historical knowledge, and to report the methods and procedures used in obtaining evidence (OECD, 2006, p.24). The second and together with the third point in PISA 2006 and PISA 2009, particularly the nature of science, are not minor but significant components of scientific literacy, which, if had not been assessed in similar ways in PISA 2000 and

2003 as in 2006 and 2009, and it would have posed a serious threat to the comparability of the PISA results over time. Consequently, in turn this would have jeopardized PISAs claim that its results can be used to monitor the outcomes of education systems in terms of student achievement on a regular basis (Lau, 2009; OECD, 2000, p. 3). Here, through the comparison of the definitions of the scientific literacy, the consistency and comparability are challenged.

The analyses of the components of scientific literacy of PISA The components of scientific literacy of PISA can be found in table 2 (OECD 2000, 2003, 2006 &2009). Table 2: Components of the definition of scientific literacy for PISA

PISA 2000, 2003 Scientific concepts Scientific processes Context

PISA 2006, 2009 Scientific knowledge Scientific competencies Context Attitudes

PISA 2006 and 2009 Science situated its definition of scientific literacy and its science assessment questions within a framework that used the following components: scientific contexts (i.e., life situations involving science and technology); the scientific competencies (i.e., identifying scientific issues, explaining phenomena scientifically, and using scientific evidence); the domains of scientific knowledge (i.e., students understanding of scientific concepts as well as their understanding of the nature of science); and student attitudes toward science (i.e., interest in science,

support for scientific inquiry, and responsibility toward resources and environments) (OECD 2006&OECD 2009). These four aspects of the PISA 2006 and 2009 conception of scientific literacy are illustrated in table 2. Apparently, it seems that except of the dimension of context, some changes and one extra dimension of attitude are added to PISA 2006 and PISA 2009. As a matter of fact, The components of scientific knowledge and scientific competencies are as a matter of fact overlapped with the components of scientific concepts and scientific processes when we exam the definitions of scientific literacy from OECD 2000, 2003 2006 and 2009 closely. And this can be illustrated in table 3 (OECD 2000, 2003, 2006 &2009). Table 3: The comparison between definition of PISA 2000, 2003 and scientific

The definition of scientific literacy in PISA 2000&2003

Scientific knowledge in PISA 2006 &2009 What do you know: .about the natural world (knowledge of science .about science itself(knowledge about science

Scientific literacy is the capacity to use scientific knowledge, to identify questions, and to draw evidence-based conclusions in order to understand and help make decisions about the natural world and the changes made to it through human activities.

scientific competencies in PISA 2006 &2009 .Identifying science issues . Explaining phenomena Scientifically . Using scientific evidence

Meanwhile, if we try to compare the framework of the scientific literacy for PISA and the three dimensions advocated by Miller, a high degree of correspondence also can be
found although different wordings are employed: Millers first dimension is similar in some sense with scientific competencies of PISA; second dimension is interlinked with the scientific knowledge claimed in PISA ; the third dimension somewhat contains the scientific

context and attitude advocated in PISA framework ((Miller, 1989; OECD 2006&2009).This can be illustrated in table 4: Table 4: The comparison between Millers concept of scientific literacy and PISAs

Millers three dimensions of scientific Components of the definition of scientific literacy literacy for PISA Scientific knowledge (knowledge of (a) An understanding of the norms and science and knowledge about science) methods of science (i.e., the nature of Scientific competencies (Identifying science). science issues; Explaining phenomena (b)An understanding of key scientific Scientifically; Using scientific evidence) terms and concepts (i.e., science content Context (life situations involving knowledge). science and technology) (c) An awareness and understanding of Attitudes (interest in science, support the impact of science and technology on for scientific inquiry, and responsibility society. toward resources and environments) However, although Miller he himself designed a series of assessment to test the scientific competence of the public, he probably never fulfilled the goal of testing the three dimensions at the same time in one single design. According to Miller (1998), in the National Association of Science Writers (NASW) national survey in and later on in 1988 collaboration between him in the United Sates and Tomas and Durant in the United Kingdom, the 1992 Euro-barometer study and the 1995 United States study, the publics understanding of scientific concepts and constructs are the main goals(Miller, 1998). So, the question here may be raised naturally: does PISAs assessment test all the four aspects as it promised in its framework? According to Lau (2009):
the competencies/processes appeared to differ over the three cycles of assessment. When examined more closely, however, these competencies/processes were found to have a high degree of correspondence, though some processes were combined or described by

different wordings. By and large, the relative weights of these competencies/processes were consistent across the three assessments, with nearly half of the score points devoted to the assessment of explaining scientific phenomena, while using scientific evidence taking a slightly greater proportion than identifying scientific issues in the other half. Nonetheless, an increasing trend was observed as to the weight placed on explaining scientific phenomena, from 43% in 2000 to 49% in 2006, while its desired weight for 2006 should have been 3540% only (OECD, 2006, p.40). Knowledge of science, as mainly assessed through explaining scientific phenomena, was thus taking up an increasing proportion in the assessment. Apart from this, as will be discussed later, a substantial amount of knowledge of science has been found hidden in the assessment of the other two competencies that were claimed to be assessing mainly knowledge about science. This hidden knowledge of science sometimes constituted a hurdle that made the assessment for these competencies invalid. Therefore, the competency explaining scientific phenomena would have probably crept into the other two competencies with an unknown weighting. This raises concern on how scientific literacy is operationally defined in PISA in terms of the proportion of the three competencies and the kinds of knowledge it draws upon.

According to PISA 2009 Assessment Framework: Key Competencies in Reading, Mathematics and Science (OECD, 2010), the weight placed on explaining scientific phenomena is 41% , which means the trend mentioned above still prevailed in PISA 2009. And thus, how scientific literacy is operationally defined in PISA is still in question.

Summary So far, we can draw a conclusion as DeBoer (2000) pointed out that understanding of scientific literacy is open-ended and ever-changing. It is organic, not static. One aspect of the challenges of communicating the findings of PISA is to avoid simplistic statement assuming that what is being measured is the outcome of the school curriculum (Harlen, 2001). Policy-makers and education reformers should realize

that ``...the critical principle that must guide any assessment framework adopted for the science curriculum . . . must be that assessment should exert a positive and benign influence on the teaching and learning of science'' (Millar & Osborne, 1998, p. 25).It is also important for policy makers and education reformers to recognize that standards-based reform does not come without its own problems and controversies. As Kyle (1996) has said, the standards approach ``effectively strips each individual of the opportunity to be an active agent in the democratic processes through which educational policies and decisions are made''. Wood (1988) also found that standards testing ``...constrains and reutilizes the teachers' behavior, causing them to violate their own standards of good teaching. As a result, students' opportunity to express curiosity and inquiry ---central processes in scientific thinking are constrained. Therefore, suggestion is here to make that local school districts and individual teachers should have more flexibility to choose their own science content and teachers could teach to their own strengths and to the interests of their students.

Reference

Bybee, R., McCrae, B. and Laurie, R. (2009), PISA 2006: An assessment of scientific literacy. Journal of Research in Science Teaching, 46(8), 865-883. DeBoer, G. E. (2000).Scientific literacy: Another look at its historical and contemporary meanings and its relationship to science education reform. Journal of Research in Science Teaching, 37: 582601. doi: 10.1002/1098-2736(200008)37:6<582::AID-TEA5>3.0.CO;2-L Dohn, N. B. (2007). Knowledge and Skills for PISAAssessing the Assessment. Journal of Philosophy of Education, 41(1), 116.

Harlen, W. (2001). The assessment of scientific literacy in the OECD/PISA project. Studies in Science Education, Vol 36(1), 79-104 Kyle, W. (1996). Shifting ideologies and science education. Journal of Research in Science Teaching, 33 (10), 1043 - 1044. Lau, K.C.( 2009).A critical examination of PISA's assessment on scientific literacy. International Journal of Science and Mathematics Education ,7(6): 10611088. Laugksch, R.C. (2000). Science literacy: A conceptual overview. Science Education, 84(1), 7194. Miller, J. D. (1983). Scientific Literacy:A Conceptual and Empirical Review. Daedalus, 112 (2), 29-48. Miller, J. D. (1989). Scientific literacy. Paper presented at the annual meeting of the American Association for the Advancement of Science, San Francisco. Miller, J. D. (1998). The measurement of civic scientific literacy. Public Understanding of Science,7, 203223. Millar, R. & Osborne, J. (Eds.) (1998). Beyond 2000: Science education for the future. London: King's College, School of Education. OECD (2000). Measuring student knowledge and skills: The PISA 2000 assessment of reading, mathematical and scientific literacy. Paris: OECD Publications. OECD (2003). The PISA 2003 assessment framework: mathematics, reading, science and problem solving knowledge and skills. Paris: OECD Publications. OECD (2006). Assessing scientific, reading and mathematical literacy: a framework for PISA 2006. Paris: OECD Publications. OECD (2010), PISA 2009 Assessment Framework: Key Competencies in Reading, Mathematics and Science. Paris: OECD Publications. OECD (2010), PISA 2009 Results: What Students Know and Can Do Student Performance in Reading, Mathematics and Science (Volume I). http://dx.doi.org/10.1787/9789264091450-en. Sadler, T. D. & Zeidler, D. L. (2009). Scientific literacy, PISA, and socioscientific discourse: Assessment for progressive aims of science education. Journal of Research in Science Teaching, 46(8), 909921. doi: 10.1002/tea.20327 Venezky, R. L. (1990). Definitions of literacy. In R. L. Venezky, D. A. Wagner, & B. S. Ciliberti,(Eds.), Toward defining literacy. Newark, DE: International Reading Association.