About the Evaluation
With Project 2061's Benchmarks for Science Literacy (1993) and the National Research Council's National Science Education Standards (1996) representing a strong national consensus among educators and scientists on what all K-12 students should know and be able to do in science, researchers and materials developers have begun analyzing how well curriculum, instruction, and assessment support student achievement of specific learning goals. As part of this effort and with funding from the Carnegie Corporation of New York, Project 2061 began in 1998 an evaluation of middle grades science textbooks using its rigorous curriculum-materials analysis procedure. The procedure was developed over several years with funding from the National Science Foundation and in consultation with hundreds of K-12 teachers, materials developers, scientists, teacher educators, and cognitive researchers nationwide.
The evaluation's purposes were to examine how well currently available middle grades curriculum materials help students to learn important scientific ideas and to identify typical strengths and weaknesses of these materials. The evaluation's design and methods, including how the reviewers were trained, the analysis was conducted, and the reports on each textbook were prepared, are described below.
Reviewers: The analysts who reviewed and rated the textbooks were evenly divided between experienced middle school classroom teachers and science education university faculty members who were knowledgeable about research on science learning and teaching. Based on their area of science content expertise, reviewers were assigned to participate in one of three topic areas: Earth science, life science, or physical science. The following is a list of the reviewers and their affiliations at the time of the evaluation:
Theron Blakeslee, Michigan Department of Education
Helen Doyle, University of California
Sally Duff, Baltimore City Public Schools (retired)
Danine Ezell, San Diego City Schools
Henry Heikkinen, University of Northern Colorado
Patricia Heller, University of Minnesota
Natalie Hiller, Philadelphia Public Schools
Susie Hix, Howard County Public Schools (MD)
Joe Krajcik, University of Michigan
Norm Lederman, Oregon State University
Jim Minstrel, ACT Systems for Education (WA)
Gary Nakagiri, San Mateo County Public Schools (CA)
Mike Piburn, Arizona State University
Harold Pratt, National Research Council
Eric Pyle, West Virginia University
Kathleen Roth, Michigan State University
Sharon Sinclair, Long Beach Public Schools (CA)
Edward Smith, Michigan State University
Mike Smith, American Geological Institute
R. Timothy Smith, Michigan State University
Rene Stofflett, University of Illinois
Greg Thomas, District of Columbia Public Schools
Kathleen Tunney, Howard County Public Schools (MD)
Jeff Turley, Mesa Public Schools (AZ)
Emily Van Zee, University of Maryland
Molly Hand Weinburgh, Georgia State University
Reg Wild, University of British Columbia
Paula Wilson, University of Utah
Training: To ensure that the reviewers understood the evaluation criteria and would consistently follow the evaluation procedure, the reviewers were extensively trained in the use of the Project 2061 curriculum-materials analysis procedure. Initially, they attended a week-long workshop discussing the evaluation criteria and applying them to a variety of examples. The reviewers had opportunities to practice using the analysis procedure and compare their results.
Materials Examined: Analysts examined nine sets of middle grades science curriculum materials. Only comprehensive middle grades science programs-that typically cover three years of instruction-were selected. The programs selected included those that were being widely used (or considered for use) in school districts or states and three programs funded by the National Science Foundation (BSCS Middle School Science & Technology, PRIME Science, and Science 2000). In addition, this evaluation reports on the analysis of one stand-alone unit (the physical science unit Matter and Molecules) that is not part of any textbook.
The time and rigor required for the analysis procedure make it impractical to evaluate every topic included in a yearlong or multi-year curriculum. Therefore, the analysis was based on learning goals (called "key ideas" in the evaluation reports) from three important topics-the kinetic molecular theory (physical science), flow of matter and energy in ecosystems (life science), and processes that shape the Earth (Earth science). These key ideas are examples of the core science content likely to appear in any middle grades material and are included in all national-and most state-science standards. In addition, a research base on student learning difficulties exists for these key ideas, as shown in Benchmarks for Science Literacy, Chapter 15: The Research Base. Furthermore, the topics selected are the basis for learning other, more complex ideas, but are among the most difficult for students to grasp. Although the evaluation's findings are limited to the three topics examined, the consistent format and features of the textbooks within each middle grades science program suggest that their strengths and weaknesses are also consistent across other topics.
Evaluation Procedure: Project 2061's procedure is unique in that it examines the curriculum materials for how likely they are to help students progress to specific learning goals, such as Project 2061's Benchmarks for Science Literacy and the National Research Council's National Science Education Standards. Rather than matching the curriculum material's content to general topic headings-such as "cells" or "structure of matter" (which are so general that most materials could meet the criteria), this procedure compares ideas in the curriculum material to the substance of specific learning goals. The reviewers carefully examined how completely each textbook addresses or aligns to the content of the key ideas and how well the instructional strategies in the student text and teacher's guide make use of the most effective methods for student learning. The procedure's instructional criteria are based on existing research on student learning and are organized in seven categories, each of which focuses on a specific aspect of instructional support. These categories include Providing a Sense of Purpose, Taking Account of Student Ideas, and Engaging Students with Relevant Phenomena.
Analysis: Each curriculum material was analyzed by two independent two-member review teams. Each team consisted of one experienced teacher and one science education researcher (whenever possible). After the teams completed their analysis of the curriculum materials independently, the teams then met to discuss and reconcile their findings, while getting feedback from Project 2061 staff. The reviewers cited specific evidence from both the student text and teacher's guide-such as text segments, activities, teacher notes, and assessment questions-to justify each of their ratings. These citations served as the basis for the process, coordinated by Project 2061, of reconciling those places where reviewers' ratings disagreed. Thus, the ratings in the evaluation reports represent a consensus of the review teams and Project 2061 staff.
Reports: Project 2061 staff used the analysis data and notes to prepare detailed reports on each textbook. To learn more about what the reviewers looked for in the content and instructional analyses, click on Project 2061 Analysis Procedure on the main menu. To read the full text of the evaluation reports or to view summary charts of the instructional analysis ratings, click on Browse the Evaluation Reports.
Report Preparation: The following Project 2061 staff members were responsible for supervising the development of the procedure, training analysts, compiling data, and/or preparing the reports:
Jo Ellen Roseman, Director
Ann Caldwell, Research Associate
Sofia Kesidou, Program Director
Lori Kurth, Senior Program Associate
Luli Stern, Consultant
Freelance Editor Paul Elliott completed final editing of the reports. Freelance Editor Audrey Pendergast performed early editing of the reports.
The following Project 2061 staff members prepared the reports for publication on the Web:
Mary Koppal, Communications Director
Francis Molina, Technology Director
Jonah Ben-Joseph, Writer
Barbara Goldstein, Administrative Coordinator
Ed Krafsur, Technology Specialist/Database-Multimedia
Brian Sweeney, Technology Specialist/Webmaster
Catherine Tramontana, Associate Editor
