An electronic newsletter for the science education community
New Assessment Resources for Science Educators
The recent launch of the AAAS Science Assessment website completes a seven-year effort by Project 2061 to develop test items for middle and early high school science. Supported by the National Science Foundation, the work began amidst growing concerns about the quality of assessments that were being mandated in the No Child Left Behind Act of 2001. Since then, the U.S. Department of Education’s Race to the Top program has clearly recognized the important role that high-quality assessment can play in education reform and has called for assessments that are “valid, support and inform instruction, provide accurate information about what students know and can do, and measure student achievement against standards.”
In response to these challenges, AAAS Project 2061’s work in science assessment has emphasized two critical aspects of quality: (1) the precise alignment of test items to science ideas that students are expected to know and (2) the use of assessment to identify students’ misconceptions so that they can be targeted by instruction. With these goals in mind, the Project 2061 research team, working closely with hundreds of scientists, science educators, and thousands of students, has developed a set of high-quality and carefully validated assessment resources that can be used by teachers, researchers, test developers, and others with an interest in science education. These well-designed items make it possible to find out what students have learned about important science ideas, diagnose with greater precision the learning difficulties students are having, and then use the results of assessment to modify and improve teaching. With the launch of the website, the items and other assessment resources developed by AAAS Project 2061 are now available for use in the classroom. They can also serve as models of what assessment can accomplish when done well.
Introducing assessment.aaas.org
The new website provides more than 2000 pages of material and data for science assessment, including free public access to some 600 items testing topics in earth, life, and physical sciences and the nature of science. For each topic, items have been developed to test a set of key ideas that tell a coherent and interrelated story. To make it easier to access all of the related resources for a topic, the website is organized around sets of these key ideas (e.g., “all matter is made up of atoms” is one of seven key ideas within the larger topic of “Atoms, Molecules, and States of Matter”). Resources include the key ideas and their associated sub-ideas, the assessment items targeting these ideas, students’ misconceptions that have been identified, and data on students’ performance on the items in national field tests. In addition to being able to view items and other resources on the website, users can also take advantage of a My Item Bank feature to select, save, and print items either with or without an answer key and in both PDF and HTML formats.
Key ideas and sub-ideas specify knowledge to be tested. An important first step in Project 2061’s item development process was to define for each topic the essential elements of the science story middle school students are expected to be familiar with and the level of sophistication that would be appropriate for them. Decisions were then made about details that should be included and which should be left out. These decisions about exactly what knowledge would and would not be assessed were generated through a multi-year process that included a review of the research literature; interviews with students; results from Project 2061’s own pilot testing of items; as well as feedback from experts in science, science education, and assessment. The website provides full access to the key ideas and sub-ideas that are targeted by the assessment items.
Multiple-choice items with a difference. The multiple-choice items included on the AAAS Science Assessment website probe students’ ability to demonstrate factual knowledge, explain or predict real-world phenomena, and analyze the reasons why a scientific claim might be true or not. The items also incorporate known misconceptions as incorrect answer choices so that they can be used diagnostically. As a result, the items have the advantage of focusing students’ attention on a particular aspect of the knowledge being tested in contrast to open-ended items, which often produce answers that are unrelated to the targeted knowledge, making it difficult to use the results diagnostically.
Users will find that the degree to which the items require complex thinking versus knowledge of basic facts is dependent on which topic is being tested and the knowledge that is being targeted under that topic. For the cells topic, for example, the key ideas and sub-ideas focus primarily on basic facts about cells, such as what cells are, where they are located in organisms, and what their functions are. The items reflect that basic knowledge. Other topics are more conceptual, and students are expected to use their knowledge to make judgments about a range of complex situations. Items aligned to ideas about energy, for example, ask students to analyze situations involving the behavior of physical objects under various conditions or the interactions between physical objects.
Misconceptions. Misconceptions may be strongly held incorrect beliefs about the world that are difficult to change, or they may be misunderstandings that are more easily corrected. Identifying these misconceptions and their relative prevalence for students at different grade levels was a particularly important aspect of this work. As noted earlier, each item includes common misconceptions as answer choices so that when students choose one of these misconceptions as their (incorrect) answer, teachers are better able to diagnose the nature of their students’ learning difficulty and target instruction to overcome it. By reporting on how many students selected each of the incorrect answers during field testing, the website provides a clear picture of the prevalence of the misconceptions embedded in the answer choices. Extensive references to the misconceptions research literature and to articles and papers published by the Project 2061 research team are also included on the site.
Field test data highlight students’ science knowledge. An extensive set of national field test data collected during the project provides a snapshot of what middle and high school students know about the science ideas that were tested. Data are reported by gender, grade level, and primary language of the students. Across all topics, field test results show male and female students performing at essentially the same level. Primary language makes a bigger difference in performance: students whose primary language is English score about seven percentage points higher than students whose primary language is not English, which is about the same difference as between high school students and middle school students. Typically, each item was taken by at least 2000 students in a wide range of schools and communities across the country.
Using the data. Because items were developed in clusters to test various aspects of each key idea, the best way to get a picture of what students know about a topic is to look at the data from the complete set of items for each key idea. Both for correct answers and misconceptions, the data are organized top to bottom from highest percent correct to lowest percent correct, or from the misconception that was most frequently chosen to the misconception that was the least frequently chosen. By looking at the range of scores and misconceptions for the set of items, site users can see, for example, that students are more familiar with the language of DNA than the language of chromosomes when it comes to hereditary/genetic information.
Clustering test items around a key idea and rank ordering the items by how many students in the national field testing answered correctly also makes it easy for a wide range of users to compare their own data sets with those presented on the website. Teachers can compare the national field test data with the performance of their own students, and researchers can compare how often misconceptions were selected both within each topic and across topics and with previously reported results in the research literature.
“Within minutes, I was able to select items for my bank pertaining to states of matter, create a document of items, and print (with an answer key) as a pdf. This was a very straightforward, user-friendly process. Having the item details - the misconceptions research that the item is targeting, and the student data is also useful…. this is an outstanding resource!” Joyce Tugel, Science Specialist, Maine Mathematics & Science Alliance
View additional site feedback.
View comments left on the site by visitors.
Site users are responding by email and offering comments on the new AAAS Science Assessment website.
“The work you all have done will be very useful. I have already started to build my test bank and plan to utilize many of the features on the site…. Congrats on a job well done!” Jeff Remington, teacher, Palmyra Middle School, PA
“…I have been using this [site] for a while, and it is simply amazing and has worked as the pre-post assessment for what I teach. Kids must think critically, and it is a wonderful way to start the units I teach and to end them. Thank you SO much for your wonderful work!” Dan Toomey, Teacher, Edgewood Campus School, Madison, WI
“Having a single resource linking key concepts and subtopics with common misconceptions about those topics will be invaluable in planning pre-assessments and helping to drive appropriate instruction…. I will certainly help my colleagues become familiar with the site.” Denise McCulley, teacher, Eagle School, Madison, WI
Here’s a sample of comments left on the site by visitors:
“This is an amazing resource. My district is currently in the process of developing district wide common assessments for our science program and this resource will be most helpful.”
“An outstanding initiative.”
“Great! I am just starting to dig in, and I am wishing we had data for adults too! Keep up the great work.”
“Well researched and communicated. Easy to use. Much needed resource.”
“WOW! I have a QUALITY quiz ready in 5 minutes!”
“I love this information. The test items with misconceptions are invaluable. Thank you so much!!”
“Smoothly done site; I like the simplicity.”
“Thank you for the time you put into this; it is a valuable tool for the classroom.”
“Awesome site! Keep up the great work.”
Feedback and Next Steps
Since its launch in early April, the assessment website has been recognized in a wide range of media, including articles and op-eds at MSNBC.com, Scientific American, Education Week, Digital Journal, The Oregonian, and more. The site has received nearly 32,000 visits and generated more than 184,000 page views. As site users begin to get a feel for the kinds of resources that are available, their feedback has been overwhelmingly positive. Some users have made suggestions for ways to improve the website, and many of those suggestions have already been implemented. For instance, the website’s list of Frequently Asked Questions includes answers to the questions that users have raised. As additional feedback comes in, the research team will continue to review it and make necessary changes to the site.
Over the next few months Horizon Research, Inc., will be conducting an independent evaluation of the website’s overall quality and usability. To get a better sense of how educators and others are taking advantage of the site and its resources, researchers at Horizon will be surveying site visitors and conducting interviews with a sample of users. Results from Horizon’s work will also contribute to ongoing efforts to improve the user experience.
For those who would like to explore Project 2061’s approach to science assessment more deeply and learn more about how to use the assessment website and its resources, a three-day workshop on “Developing and Using Assessments Aligned to Science Learning Goals” will take place October 12-14 at AAAS headquarters in Washington, DC. A description of the workshop, sample agenda, and registration information are available on the Project 2061 website.
Read the AAAS press release announcing the launch of the AAAS Science Assessment website, along with a list of some things middle and high school students know and some misconceptions they hold.
Read a feature story about the assessment website in AAAS News & Notes from Science [29 April 2011]: “AAAS Testing Web Site Probes Students’ Misconceptions About Science.”
Read an earlier article in Project 2061 Connections about our assessment development procedures [May/June 2009]: “National Sample Puts Assessment Items to the Test.”
For more information, please contact:
Deputy Director: Dr. George DeBoer, (202) 326-6624
Also in the May 2011 issue: