Middle Grades Mathematics Textbooks

A Benchmarks-Based Evaluation

For many mathematics teachers, the textbook is their primary guide to implementing the curriculum. Consequently, Project 2061 began its evaluation of mathematics textbooks with three basic propositions: First, good textbooks can play a central role in improving mathematics education for all students; second, the quality of mathematics textbooks should be judged mainly on their effectiveness in helping students to achieve important mathematics learning goals for which there is a broad national consensus; and, third, an in-depth analysis of much more than a textbook’s content coverage would be required to evaluate whether there is potential for students' actually learning the desired subject matter.

Data on poor student performance from the Third International Mathematics and Science Study and other research indicate that the middle school mathematics curriculum requires urgent attention. It is in middle school that many students find themselves in mathematics programs that are repetitious and non-challenging. As a result, their achievement and interest in mathematics stalls, and they are unable to take advantage of the full range of academic and career options in the future. For these and other reasons, middle school is a critical leverage point for education reform efforts and offers a productive focus for Project 2061’s first evaluation effort.

With funding from the National Science Foundation, Project 2061 developed a unique approach to curriculum materials evaluation. The Project 2061 curriculum-materials analysis procedure uses selected learning goals from national or state standards or benchmarks as the basis for an evaluation. (Much more information on the development and application of this procedure will be available in Project 2061’s Resources for Science Literacy: Curriculum Materials Evaluation (AAAS, 2000), a forthcoming print/CD-ROM tool.)

For this evaluation of middle school mathematics materials, analysts used Project 2061’s Benchmarks for Science Literacy (AAAS, 1993) in the procedure to identify to what extent textbooks address six important mathematics concepts and skills and also provide instructional strategies that are likely to help students learn them. While there are other, more abbreviated methods for evaluating materials, the Project 2061 procedure has been specifically designed to assess whether a material is satisfactory for use in classrooms where literacy in mathematics and science is a goal for all students.

Late in 1997, the Carnegie Corporation of New York agreed to fund the first of a series of evaluations of textbooks in mathematics and science. Work began in early 1998 on middle school curriculum materials.

About Project 2061’s Evaluation

Project 2061’s textbook evaluation has several distinct features that set it apart from other evaluation efforts:

• A rigorous and uniformly applied process. Analysts of the middle-school mathematics textbooks were evenly divided between experienced, practicing classroom teachers and higher education faculty who were knowledgeable about research on mathematics learning and teaching. All were highly capable in mathematics content, and extensively trained in the Project 2061 analysis procedure. Six independent two-person teams evaluated each textbook. Each team was assigned two benchmarks to use in the review of a textbook series. After completing their analyses, the teams met to reconcile their ratings before proceeding to the next series of books. Continuing in this way, each team analyzed a total of six series of books, using the same two benchmarks.

• An evidence-based analysis procedure. Using custom-designed software to document what they found, analysts examined the student and teacher editions of the textbooks to identify lessons, activities, teacher notes, assessments, and other materials that addressed one or more ideas in each of six commonly taught benchmarks drawn from Project 2061’s Benchmarks for Science Literacy. Next, they analyzed that content using a set of research-based instructional criteria. All analysts looked for the same instructional criteria in each textbook and used the same methodology and scoring guidelines to arrive at their ratings. Analysts were required to provide specific evidence from the materials to justify each of their ratings. A complete record of that evidence is available for each textbook that was analyzed.

• Key mathematics benchmarks. In creating its analysis procedure, Project 2061 found that studying a material’s treatment of a relatively small but carefully chosen set of benchmarks can identify the strengths and weaknesses of the material as a whole in its instructional design and support. Six benchmarks representing three important mathematical strands—number, geometry, and algebra—were used to conduct the analysis. These benchmarks are examples of the core mathematics content likely to appear in any middle grades material. Specifically, they include a concept benchmark dealing with fractions and operations on them; a skill benchmark dealing with equivalent forms of numbers; a concept and a skill benchmark dealing with properties of shapes and computations of circumference, area, and volume; and two concept benchmarks dealing with graphing and equations.

• Research-based instructional criteria. The reviewers rated each textbook on 24 criteria derived from research on learning and teaching and on the craft knowledge of experienced teachers. For example, reviewers rated the extent to which textbooks informed teachers about prerequisite ideas or skills that students would need to achieve a benchmark. They also considered whether the textbook provided appropriate assessment items that focus on understanding benchmark ideas and whether it offered teachers advice on how to use the assessment results to guide their classroom activities.

Evaluating Content

Six middle school benchmarks were selected as content criteria for the evaluation. The content in these and other learning goals recommended in AAAS’s Benchmarks overlap substantially with the  Curriculum and Evaluation Standards for School Mathematics (National Council of Teachers of Mathematics [NCTM], 1989) and deal with concepts and skills that nearly everyone would agree are important for middle school students to achieve:

Evaluating Instruction

After identifying textbook activities that address a part or all of the content in a benchmark, analysts then rated each activity on how well it addresses 24 instructional criteria. Arranged in seven broad categories, these instructional criteria are not intended to espouse or reflect any particular theory or ideology of learning, beyond the principles and strategies that are supported by the available research evidence. Much of that evidence is summarized in Chapter 15 of Project 2061’s Benchmarks for Science Literacy. In addition, these instructional criteria are consistent with the principles for effective mathematics learning and teaching found in NCTM’s current revision of its mathematics standards (NCTM, 1998).