High School Biology Textbooks: A Benchmarks-Based Evaluation

About the Evaluation

With Project 2061's Benchmarks for Science Literacy (1993) and the National Research Council's National Science Education Standards (1996) representing a strong national consensus among educators and scientists on what all K–12 students should know and be able to do in science, researchers and materials developers have begun analyzing how well curriculum, instruction, and assessment support student achievement of specific learning goals. As part of this effort and with funding from the Carnegie Corporation of New York, Project 2061 began in 1998 a series of four textbook evaluations using its rigorous curriculum-materials analysis procedure. Textbooks were evaluated in middle grades mathematics, middle grades science, algebra, and high school biology. The analysis procedure had been developed over several years with funding from the National Science Foundation and in consultation with hundreds of K–12 teachers, materials developers, scientists, teacher educators, and cognitive researchers nationwide (Roseman, Kesidou, & Stern, 1997; Kesidou & Roseman, 2002). The evaluations' purposes were to examine how well currently available curriculum materials help students to learn important ideas specified in national science and mathematics standards and to identify typical strengths and weaknesses of these materials.

The design and methods of the high school biology textbook evaluation—including a reliability study, the content and instructional analyses, reviewer training, and the preparation of detailed reports on each textbook—are described below.

Reliability Study: As part of the high school biology textbook evaluation, Project 2061 carried out a reliability study, in which we examined the consistency of reviewers' judgments about the textbooks' coherence for the topic Matter and Energy Transformations. Coherence was defined by the extent to which a textbook aligned with the complete set of key ideas for the topic and whether it made explicit connections among the key ideas. Reliability in evaluating textbook coherence is an important first step for designing coherent curriculum materials. If review teams can agree with and adhere to a way of operationalizing curriculum coherence, then it becomes possible to design materials by those same guidelines.

Of the textbooks in the evaluation, we chose four for the reliability study based on their having the most complete content coverage of the topic. To ensure that the study would provide information on the applicability of the methods across different approaches, we chose both National Science Foundation-funded and commercially developed textbooks. Because the study was designed to test the reliability of the coherence analysis procedure under well-defined conditions, we selected eight reviewers already familiar with the Project 2061 curriculum-materials analysis procedure and provided them with additional training relevant to this study.

Using a strand map adapted from Atlas of Science Literacy (see map "Matter and Energy Transformations: What the Reviewers Looked For"), we specified the connections among key ideas that would be used to define coherence and organized them into three categories: Connections among key ideas, Connections between key ideas and their prerequisites, and Connections between key ideas and related ideas. Reviewers used this map to keep track of their judgments and to facilitate discussion within and between review teams. The study findings indicate that the coherence of the high school biology textbooks can be consistently judged using these maps. Several factors probably contributed to the reliability, including (a) providing written clarification of the meaning of the key ideas and what "alignment" and "connections" mean for each one, (b) using knowledgeable and experienced review teams, and (c) providing training that included applying the written clarification to a practice textbook. Details about the design and results of the reliability study are forthcoming.

Evaluation Procedure: Project 2061's curriculum-materials analysis procedure is unique in that it examines curriculum materials for how likely they are to help students progress to specific learning goals, such as Project 2061's Benchmarks for Science Literacy and the National Research Council's National Science Education Standards. Rather than matching the curriculum material's content to general topic headings—such as "cells" or "structure of matter" (which are so general that most materials could meet the criteria)—this procedure compares ideas in the curriculum material to specific key ideas in the learning goals. Reviewers carefully examine how completely a curriculum material addresses the content of the key ideas and how well the instructional strategies in the student text and teacher's guide make use of the most effective methods for student learning. The procedure's instructional criteria are based on existing research on student learning and are organized in seven categories, each of which focuses on a specific aspect of instructional support: Providing a Sense of Purpose; Taking Account of Student Ideas; Engaging Students with Relevant Phenomena; Developing and Using Scientific Ideas; Promoting Students' Thinking about Phenomena, Experiences, and Knowledge; Assessing Progress; and Enhancing the Science Learning Environment.

  

Analysis: The textbook evaluation itself was conducted in two stages. First, content specialists (including three Ph.D.-biologists) examined the textbooks' student and teacher editions to identify places where specific key ideas within four important biology topics were addressed—in text segments, activities, assessments, and other components. The next stage involved teams of experienced biology teachers and university faculty—all with biology backgrounds, knowledge of the research on science learning and teaching, and extensive training in the Project 2061 curriculum-materials analysis procedure. These reviewers applied a set of research-based instructional criteria to the textbooks' treatments of the four biology topics.

The instruction in each textbook was examined meticulously by a total of eight instructional analysts who worked as four independent two-member teams. Two of the four teams examined each textbook on the topics Molecular Basis of Heredity and Natural Selection and Evolution and two other teams examined each textbook on the topics Cell Structure and Function and Matter and Energy Transformations. The teams analyzed the textbook's prescribed program—what instruction would be like if the teacher were to follow the suggestions in the teacher's guide literally. All analysts used the same criteria for evaluating each textbook and used the same methodology to rate the textbooks on the more than 100 indicators supporting the criteria. This analysis required hundreds of hours of careful consideration for each textbook.

After the teams completed their analysis of the curriculum materials independently, the teams then met to discuss and reconcile their findings, while getting feedback from Project 2061 staff. The reviewers cited specific evidence from both the student text and teacher's guide—such as text segments, activities, teacher notes, and assessment questions—to justify each of their ratings. These citations served as the basis for the process, coordinated by Project 2061, of reconciling those places where reviewers' ratings disagreed. Thus, the ratings in the evaluation reports represent a consensus of the review teams and Project 2061 staff.

Reviewers: The instructional analysts who reviewed and rated the textbooks were experienced high school biology teachers, science curriculum specialists, and science education university faculty members who were knowledgeable about research on science learning and teaching. The following is a list of the instructional analysts and their affiliations at the time of the evaluation:

Sharlene Argamaso-Hernan, Ph.D., Baltimore County Public Schools

Andrea Barton, University of Wisconsin—Madison

Angela Benjamin, District of Columbia Public Schools

Leah Bricker, Indiana Department of Education

Judy Capra, Jefferson County Public Schools (Colorado)

Jennifer Cartier, Ph.D., University of Wisconsin—Madison

Sam Donovan, University of Wisconsin—Madison

Sally Duff, Baltimore City Public Schools (retired)

Marlene Hilkowitz, Penn-Delco School District (Pennsylvania)

Susan Johnson, Ph.D., University of Wisconsin—Madison and Madison Public Schools

Page Keeley, Maine Department of Education

Karen Mesmer, University of Wisconsin—Madison

Barbara Neureither, Holt Public Schools (Michigan)

Cynthia Passmore, University of Wisconsin—Madison

Howard Putterman, Montgomery County Public Schools (Maryland)

Kathleen Roth, Ph.D., Michigan State University

Francine Rowe, Edgewood College (Wisconsin)

Teresa Shume, Moorhead State University (Montana)

Eileen Theissen, Baltimore City Public Schools

Molly Weinburgh, Ph.D., Georgia State University

The following is a list of members of the Project 2061 Biology Textbooks Evaluation Advisory Committee and their affiliations at the time of the evaluation:

Andrea Bowden, Baltimore City Public Schools

David Campbell, Ph.D., National Science Foundation

Wayne Carley, Ph.D., National Association of Biology Teachers

Delores Dalton, Virginia Department of Education

Clarissa Evans, Ph.D., Howard County Public Schools (Maryland)

Carolyn Kornegay, District of Columbia Public Schools

Sharon Lynch, Ph.D., George Washington University

Gerhard Salinger, Ph.D., National Science Foundation

Rochelle Slutskin, Anne Arundel County Public Schools (Maryland)

Gordon Uno, Ph.D., National Science Foundation

Gerald Wheeler, Ph.D., National Science Teachers Association

Training: To ensure that the reviewers understood the evaluation criteria and would consistently follow the evaluation procedure, the reviewers were extensively trained in the use of the Project 2061 curriculum-materials analysis procedure. Initially, they attended a week-long workshop discussing the evaluation criteria and applying them to a variety of examples. The reviewers had opportunities to practice using the analysis procedure and to compare their results.

Textbooks and Topics Examined: The evaluation analyzed the following ten high school biology textbooks (in alphabetical order):

Biology by Miller and Levine, Prentice Hall, 1998
Biology: A Community Context, South-Western Educational Publishing, 1998
Biology: Principles & Explorations, Holt, Rinehart and Winston, 1998
Biology: The Dynamics of Life, Glencoe/McGraw-Hill, 2000
Biology: Visualizing Life, Holt, Rinehart and Winston, 1998
BSCS Biology: A Human Approach, Kendall/Hunt Publishing Company, 1997
BSCS Biology: An Ecological Approach, Kendall/Hunt Publishing Company, 1998

Heath Biology, D.C. Heath and Company, 1991
Insights in Biology, Kendall/Hunt Publishing Company for Educational Development Center, Inc., 1998
Modern Biology, Holt, Rinehart and Winston, 1999

The textbooks selected included those that were being widely used (or considered for use) in school districts or states and four textbooks funded by the National Science Foundation (Biology: A Community Context, BSCS Biology: A Human Approach, BSCS Biology: An Ecological Approach, and Insights in Biology). (Even though Heath Biology was on state adoption lists when the evaluation began, by the time the evaluation was completed Heath had decided not to publish any new editions of that textbook. Because the reviewed 1991 edition was not likely to be considered for new adoptions, Project 2061 decided not to publish detailed evaluation reports on Heath Biology.)

The time and rigor required for the analysis procedure make it impractical to evaluate every topic included in a yearlong biology curriculum. Therefore, Project 2061 surveyed biology teachers attending a national meeting of the National Science Teachers Association and teachers in an urban school district, who listed the following four topics as top priority: Cell Structure and Function, Matter and Energy Transformations, Molecular Basis of Heredity, and Natural Selection and Evolution. The analysis was then based on specific key ideas found in learning goals for these four topics. The key ideas are examples of the core content likely to appear in any high school biology textbook and are included in all national—and most state—science standards. In addition, a research base on student learning difficulties exists for many of these key ideas, as shown in Benchmarks for Science Literacy, Chapter 15: The Research Base. Although the evaluation's findings are limited to the four topics examined, the consistent format and features of the biology textbooks suggest that their strengths and weaknesses are similar across other topics.

Reports: Project 2061 staff used the analysis data and notes to prepare detailed reports on each textbook. To learn more about what the reviewers looked for in the content and instructional analyses, see Project 2061 Analysis Procedure on the main menu. To read detailed evaluation reports on each textbook, view maps summarizing the content analysis findings, and view charts summarizing the instructional analysis ratings, see Browse the Evaluation Reports.

Report Preparation: The following Project 2061 staff members were responsible for supervising the development of the procedure, training analysts, compiling data, and/or preparing the reports:

Jo Ellen Roseman, Director

Ann Caldwell, Research Associate

Lori Kurth, Senior Program Associate

Luli Stern, Consultant

The following Project 2061 staff members prepared the reports for publication on the Web:

Mary Koppal, Communications Director

Francis Molina, Technology Director

Barbara Goldstein, Administrative Coordinator

Brian Sweeney, Technology Specialist/Webmaster

Catherine Tramontana, Associate Editor