Part I
Comparing Middle Grades Mathematics Textbooks
About
Project 2061’s Evaluation
The Ratings
Conclusions
For many mathematics teachers, the textbook is their primary guide to implementing the
curriculum. Consequently, Project 2061 began its evaluation of mathematics textbooks with
three basic propositions: First, good textbooks can play a central role in improving
mathematics education for all students; second, the quality of mathematics textbooks
should be judged mainly on their effectiveness in helping students to achieve important
mathematics learning goals for which there is a broad national consensus; and, third, an
indepth analysis of much more than a textbook’s content coverage would be required
to evaluate whether there is potential for students' actually learning the desired subject
matter.
Data on poor student performance from the Third International Mathematics and Science
Study and other research indicate that the middle school mathematics curriculum requires
urgent attention. It is in middle school that many students find themselves in mathematics
programs that are repetitious and nonchallenging. As a result, their achievement and
interest in mathematics stalls, and they are unable to take advantage of the full range of
academic and career options in the future. For these and other reasons, middle school is a
critical leverage point for education reform efforts and offers a
productive focus for Project 2061’s first evaluation effort.
With funding from the National Science Foundation, Project 2061 developed a unique
approach to curriculum materials evaluation. The Project 2061 curriculummaterials
analysis procedure uses selected learning goals from national or state standards or
benchmarks as the basis for an evaluation. (Much more information on the development and
application of this procedure will be available in Project 2061’s Resources for
Science Literacy: Curriculum Materials Evaluation (AAAS, 2000), a forthcoming
print/CDROM tool.)
For this evaluation of middle school mathematics materials, analysts used Project
2061’s Benchmarks for Science Literacy (AAAS, 1993) in the procedure to
identify to what extent textbooks address six important mathematics concepts and skills
and also provide instructional strategies that are likely to help students learn them.
While there are other, more abbreviated methods for evaluating materials, the Project 2061
procedure has been specifically designed to assess whether a material is satisfactory for
use in classrooms where literacy in mathematics and science is a goal for all students.
Late in 1997, the Carnegie Corporation of New York agreed to fund the first of a series
of evaluations of textbooks in mathematics and science. Work began in early 1998 on middle
school curriculum materials.
About Project 2061’s
Evaluation
Project 2061’s textbook evaluation has several distinct features that set it apart
from other evaluation efforts:

A rigorous and uniformly applied process. Analysts of the
middleschool mathematics textbooks were evenly divided between experienced, practicing
classroom teachers and higher education faculty who were knowledgeable about research on
mathematics learning and teaching. All were highly capable in mathematics content, and
extensively trained in the Project 2061 analysis procedure. Six independent twoperson
teams evaluated each textbook. Each team was assigned two benchmarks
to use in the review of a textbook series. After completing their analyses, the teams met
to reconcile their ratings before proceeding to the next series of books. Continuing in
this way, each team analyzed a total of six series of books, using the same two
benchmarks.

An evidencebased analysis procedure. Using customdesigned software to document
what they found, analysts examined the student and teacher editions of the textbooks to
identify lessons, activities, teacher notes, assessments, and other materials that
addressed one or more ideas in each of six commonly taught benchmarks drawn from Project
2061’s Benchmarks for Science Literacy. Next, they analyzed that content using
a set of researchbased instructional criteria. All analysts looked for the same
instructional criteria in each textbook and used the same methodology and scoring
guidelines to arrive at their ratings. Analysts were required to provide specific evidence
from the materials to justify each of their ratings. A complete record of that evidence is
available for each textbook that was analyzed.

Key mathematics benchmarks. In creating its analysis procedure, Project 2061 found
that studying a material’s treatment of a relatively small but carefully chosen set
of benchmarks can identify the strengths and weaknesses of the material as a whole in its
instructional design and support. Six benchmarks representing three important mathematical
strands—number, geometry, and algebra—were used to conduct the analysis. These
benchmarks are examples of the core mathematics content likely to appear in any middle
grades material. Specifically, they include a concept benchmark dealing with fractions and
operations on them; a skill benchmark dealing with equivalent forms of numbers; a concept
and a skill benchmark dealing with properties of shapes and computations of circumference,
area, and volume; and two concept benchmarks dealing with graphing and equations.

Researchbased instructional criteria. The reviewers rated each textbook on 24
criteria derived from research on learning and teaching and on the craft knowledge of
experienced teachers. For example, reviewers rated the extent to which textbooks informed
teachers about prerequisite ideas or skills that students would need to achieve a
benchmark. They also considered whether the textbook provided appropriate assessment items
that focus on understanding benchmark ideas and whether it offered teachers advice on how
to use the assessment results to guide their classroom activities.
Evaluating Content
Six middle school benchmarks were selected as content criteria for the evaluation. The
content in these and other learning goals recommended in AAAS’s Benchmarks
overlap substantially with the Curriculum and Evaluation Standards for School
Mathematics (National Council of Teachers of Mathematics [NCTM], 1989) and deal with
concepts and skills that nearly everyone would agree are important for middle school
students to achieve:
Number Concepts
The expression a/b can mean different things: a parts of size 1/b
each, a divided by b, or a compared to b.
Number Skills
Use, interpret, and compare numbers in several equivalent forms such as integers,
fractions, decimals, and percents.
Geometry Concepts
Some shapes have special properties: Triangular shapes tend to make structures rigid, and
round shapes give the least possible boundary for a given amount of interior area. Shapes
can match exactly or have the same shape in different sizes.
Geometry Skills
Calculate the circumferences and areas of rectangles, triangles, and circles, and the
volumes of rectangular solids.
Algebra Graph Concepts
Graphs can show a variety of possible relationships between two variables. As one variable
increases uniformly, the other may do one of the following: increase or decrease steadily,
increase or decrease faster and faster, get closer and closer to some limiting value,
reach some intermediate maximum or minimum, alternately increase and decrease
indefinitely, increase or decrease in steps, or do something different from any of these.
Algebra Equation Concepts
Symbolic equations can be used to summarize how the quantity of something changes over
time or in response to other changes.
Evaluating Instruction
After identifying textbook activities that address a part or all of the content in a
benchmark, analysts then rated each activity on how well it addresses 24 instructional
criteria. Arranged in seven broad categories, these instructional criteria are not
intended to espouse or reflect any particular theory or ideology of learning, beyond the
principles and strategies that are supported by the available research evidence. Much of
that evidence is summarized in Chapter 15 of Project 2061’s Benchmarks for Science
Literacy. In addition, these instructional criteria are consistent with the principles
for effective mathematics learning and teaching found in NCTM’s current revision of
its mathematics standards (NCTM, 1998).
