Middle Grades Mathematics Textbooks

A Benchmarks-Based Evaluation

The standards-based reform movement in education has produced several widely accepted sets of learning goals in most of the major content areas. In mathematics, for example, AAAS's Science for All Americans, Benchmarks for Science Literacy, and NCTM's Curriculum and Evaluation Standards for School Mathematics represent widespread agreement on the knowledge and skills all students need to have. To be implemented fully, however, benchmarks or standards for student learning must be central to the development of textbooks and other curriculum materials and to the design of instruction and assessment.

Project 2061 has developed an effective and reliable procedure for analyzing mathematics textbooks that attends to both content and instructional design by asking these questions: Does the textbook focus on a coherent set of significant, age-appropriate student learning goals? Does the textbook’s instructional design effectively support the attainment of those specified learning goals? The Project 2061 curriculum-materials analysis yields critical information in both areas that can be useful in guiding teachers and schools as they make important decisions about adopting textbooks and selecting materials for their students.

Taking its name from the year when Halley’s Comet will next approach Earth, Project 2061 reminds us that today's education will shape the quality of life in the 21st century. Since its inception in 1985, Project 2061 of the American Association for the Advancement of Science has been in the forefront of the education reform movement:

• defining literacy in natural and social science, mathematics, and technology as a goal for all Americans
• developing K-12 benchmarks for student learning in science, mathematics, and technology
• producing a wide range of innovative tools for educators—books, CD-ROMs, and online resources—to guide their reform efforts
• creating unique professional development experiences to help educators improve teaching and learning.

From the start, Project 2061’s work has been inspired by the belief that literacy in science, mathematics, and technology is an essential requirement for all students, regardless of their educational or career aspirations. Building on this conviction, the project also believes that literacy in these domains requires specific knowledge and skills, along with scientific habits of mind and an understanding of the nature of science, mathematics, and technology and their impact on individuals and society. For students to be able to achieve such literacy, it appears that the material covered in today’s curriculum must be significantly reduced to allow time for them to learn the most important concepts and skills well—a more difficult task, even for good students, than has been believed. Project 2061’s approach to reform is systemic, that is, it recognizes that meaningful, long-lasting improvements in student learning will require changes in many areas of the nation’s complex and highly decentralized education system.

At the center of this system is the curriculum itself, which is defined largely by the textbooks students use. As a result, the quality of those textbooks is critical to any effort to improve student achievement. By focusing carefully on how textbooks present some key ideas in mathematics and on the instructional support and guidance they provide for those specific ideas and skills, Project 2061 hopes to improve both teaching and learning.

Project 2061 evaluated a variety of mathematics textbook series. Because the curriculum-materials analysis procedure requires a great deal of time and other resources, we decided to evaluate printed materials only and not to include software or other media. We also decided to focus on complete mathematics programs written specifically for the middle grades rather than on K-8 basal series or on supplementary or other materials that did not span grades 6, 7, and 8. With these requirements in mind, we evaluated 13 textbooks in all. Some of the books are well established and are likely to be on adoption lists or already in use in many middle school classrooms. Some are more recently published materials that are just entering the market.

In choosing the mathematics learning goals to use as the basis for the evaluation, two conditions had to be met: (1) the goals must reflect a national consensus on mathematics that all students should know and be able to do, and (2) the intent of the goals must be clear, specific, and unambiguous.

We prepared for this evaluation by considering the two major sets of national mathematics content goals: NCTM's Curriculum and Evaluation Standards for School Mathematics and AAAS's Benchmarks for Science Literacy. A comparison of the two documents revealed a close correspondence in their content, especially through the 8^th grade. Earlier work with the analysis procedure suggested that some NCTM standards statements were not specific enough for this purpose. Furthermore, the NCTM standards were undergoing revision, with the expectation of a new version to be published in final form in 2000. These considerations led to a decision to select mathematics learning goals from AAAS’s Benchmarks for Science Literacy for the analysis.

The benchmarks selected for the evaluation represent core mathematics concepts and skills that any middle grades textbook should cover. Project 2061’s experience with a variety of curriculum materials has shown that analyzing a small but carefully chosen number of benchmarks can provide a profile of the instructional strengths and weaknesses of a material as a whole. On the other hand, books may differ in their treatment of particular content strands (number, geometry, and algebra), or in their development of concept knowledge as well as the application of skills and procedures.

We selected the following six benchmarks, two each in the core strands of number, geometry, and algebra. For number and geometry, we selected both concept and skill benchmarks. For algebra, we focused only on concept benchmarks because developing equation-solving skills is not a central part of the middle grades curriculum for all students. The bullets point to the individual ideas and skills that were considered as we identified textbook activities that attempted to address all or part of a benchmark.

MIDDLE SCHOOL BENCHMARKS USED IN THE EVALUATION

 (Adobe PDF document)

Appendix A contains a comparison of the selected benchmarks with similar statements from NCTM Standards and the NCTM Standards 2000 draft, Principles and Standards for School Mathematics (NCTM, 1998).

While many textbook developers and publishers have been eager to claim that their materials are aligned with benchmarks and standards, verifying such claims has been difficult. Furthermore, the mere presence of content in a textbook does not ensure that students will learn that content. For real learning to take place, textbooks need to focus sound instructional strategies specifically on the ideas and skills that students are intended to learn.

The accuracy of a textbook’s content is equally important, but it is not always easy to judge whether the content is presented at the proper level of sophistication for a given grade level. That is one reason why very clear and explicit statements of the knowledge and skills students are expected to learn are essential to the Project 2061 curriculum-materials analysis procedure. Vague statements such as "students should be able to understand and apply fractions" are not adequate. Instead, consider one of the grade 6-8 benchmarks used in the current evaluation: "Students should know that the expression a/b can mean different things: a parts of size I/b each, a divided by b, or a compared to b." This statement is sufficiently specific, enabling analysts to determine the extent to which a textbook addresses it and to judge the quality of the instructional design that supports it. Similarly, the instructional support itself must be explicit, precise, and targeted to a specific concept or skill.

The criteria for making judgments about instruction were derived from research on learning and teaching and on the craft knowledge of experienced educators. Sources included Chapter 13, "Effective Learning and Teaching," of Science for All Americans; Chapter 15, "The Research Base," of Benchmarks for Science Literacy; Research Ideas for the Classroom: Middle Grades Mathematics, (Owens, 1993); and Handbook of Research on Mathematics Teaching and Learning, (Grouws, 1992). From these and other sources, 24 criteria—arranged into seven categories—were selected to serve as a basis for the instructional analysis. Appendix B provides a summary of research results that support the seven categories of instructional criteria, and Appendix C further clarifies the criteria and describes the indicators and scoring scheme analysts used to rate the textbook activities.

Category I: Identifying a Sense of Purpose: Part of planning a coherent curriculum involves deciding on its purposes and on what learning experiences will likely contribute to achieving those purposes. Three criteria are used to determine whether the material conveys a unit purpose and a lesson purpose and justifies the sequence of activities.

Category II: Building on Student Ideas about Mathematics. Fostering better understanding in students requires taking time to attend to the ideas they already have, both ideas that are incorrect and ideas that can serve as a foundation for subsequent learning. Four criteria are used to determine whether the material specifies prerequisite knowledge, alerts teachers to student ideas, assists teachers in identifying student ideas, and addresses misconceptions.

Category III: Engaging Students in Mathematics. For students to appreciate the power of mathematics, they need to have a sense of the range and complexity of ideas and applications that mathematics can explain or model. Two criteria are used to determine whether the material provides a variety of contexts and an appropriate number of firsthand experiences.

Category IV: Developing Mathematical Ideas. Mathematics literacy requires that students see the link between concepts and skills, see mathematics itself as logical and useful, and become skillful at using mathematics. Six criteria are used to determine whether the material justifies the importance of benchmark ideas, introduces terms and procedures only as needed, represents ideas accurately, connects benchmark ideas, demonstrates/models procedures, and provides practice.

Category V: Promoting Student Thinking about Mathematics. No matter how clearly materials may present ideas, students (like all people) will devise their own meaning, which may or may not correspond to targeted learning goals. Students need to make their ideas and reasoning explicit and to hold them up to scrutiny and recast them as needed. Three criteria are used to determine whether the material encourages students to explain their reasoning, guides students in their interpretation and reasoning, and encourages them to think about what they’ve learned.

Category VI: Assessing Student Progress in Mathematics. Assessments must address the range of skills, applications, and contexts that reflect what students are expected to learn. This is possible only if assessment takes place throughout instruction, not only at the end of a chapter or unit. Three criteria are used to determine whether the material aligns assessments with the benchmarks, assesses students through the application of benchmark ideas, and uses embedded assessments.

Category VII: Enhancing the Mathematics Learning Environment. Providing features that enhance the use and implementation of the textbook for all students is important. Three criteria are used to determine whether the material provides teacher content support, establishes a challenging classroom, and supports all students.