Middle Grades Mathematics Textbooks: A Benchmarks-Based Evaluation
Example Scenarios for Applying the Results
The following scenarios reflect how various groups or individuals might use Middle Grades Mathematics Textbooks: A Benchmarks-Based Evaluation (the report), including the background material, individual textbook summary reports, the appendices, and the data sets on CD-ROM.
Scenario 1: Thinking about Improving Mathematics Curriculum Materials
Motivated by the Third International Mathematics and Science Study (TIMSS) and state assessment results, and by a commitment to improve mathematics achievement for all students, a district committee composed of teachers and parent leaders has become interested in changing the mathematics curriculum materials being used in their middle grades. They have reviewed and studied the National Council of Teachers of Mathematics' (NCTM) Standards and AAAS's Benchmarks for Science Literacy, deciding that these documents would be useful in guiding their work. They wonder whether they should adopt the newest edition of their current commercial textbook, which claims to address NCTM Standards; consider some of the new materials they have heard about; or consider writing their own curriculum materials so they could adapt to their own students' needs and interests.
After visiting the Project 2061 Web site (http://www.project2061.org) and finding the Web version of the report, they do the following:
(a) Reading the Table of Contents of the report, the group discovers the Appendices, which provide details about the textbook analysis procedure, rating criteria, and background research. They use this material to study, reflect on, and discuss the key content learning goals in mathematics, the instructional criteria, and the research on teaching and learning relevant to these criteria.
(b) The group agrees that their students should understand concepts well by the end of the middle grades and that Category II (Building on Students Ideas about Mathematics) and Category V (Promoting Student Thinking about Mathematics) are especially important to them.
(c) They see the Typical Sightings charts in the report and decide to become more familiar with the Project 2061 ratings. They use the link to publishers' Web sites to order examination copies of textbook series that are rated in the high, medium, and low range on the chart ranking all the textbooks, which is located in the executive summary of the report. They focus on textbook series that vary on conceptual benchmarks and on instructional Categories II and V. They also order the print version of the report and accompanying CD-ROM in preparation for a more in-depth review.
(d) When the examination copies of the textbooks and the report and CD-ROM arrive, the group browses all of the files for each of the books they have decided to review. They use the Typical Sightings charts to find lessons in each of the books that address the instructional criteria, learning what characteristics contribute to high and low ratings, especially on the content and instructional criteria of most interest to them.
(e) After several weeks of study, the group is ready to make some recommendations to the district on the key criteria and strategies that should be followed in the coming textbook adoption.
Scenario 2: Narrowing the List of Textbook Adoption Choices
A district mathematics textbook adoption committee has been charged with selecting three top textbook series for the middle grades. The early part of the adoption process has produced a set of basic requirements for the books: strong development of number and algebra content, support for teachers, adaptability for diverse students, alignment with national standards, strong assessment, and a high level of student engagement.
The committee has reviewed the Project 2061 Web site and ordered the report and CD-ROM. Their work to narrow the list of candidate textbook series includes the following:
(a) The committee studies the individual summary reports for each of the 13 textbook series, using the In Brief charts and the Instruction Highlights graphs. They read and discuss the content and instructional summary paragraphs, with special focus on the number and algebra benchmarks and instructional Category III (Engaging Students in Mathematics), Category VI (Assessing Student Progress in Mathematics), and Category VII (Enhancing the Mathematics Learning Environment), in which they have the most interest. They note which textbook series have acceptable content for the number and algebra benchmarks and satisfactory ratings on instructional Categories III, VI, and VII.
(b) The preliminary review has identified four or five textbook series that have some strengths related to the committee's requirements. They decide to use the CD-ROM data sets to do a more in-depth review of the criteria. Using the Compare feature, the committee members do a comparison of the Content Analysis of number and algebra benchmarks for each of five textbooks. This results in the elimination of one of the series, which does not develop algebra equation concepts sufficiently across the grades.
(c) The committee members continue with the remaining four series, comparing instructional Categories III, VI, and VII using both the Instructional Analysis and the Graphs. This data, especially the graphs of the ratings for number and algebra strands, help them to select their three finalist textbook series.
(d) In addition to having a great deal of evidence to support their selections, the committee has developed a set of recommendations for selecting the best textbook series for their students, and has a tentative ranking of the three finalists just in case anyone is interested.
Scenario 3: Choosing a Middle Grades Mathematics Textbook Series
A school mathematics textbook adoption committee has been charged with guiding the selection of a middle grades textbook series. They have narrowed the list to three top candidates and plan to continue having wide engagement by the teachers and community. The committee has already reviewed the Project 2061 Web site and has copies of the report and the CD-ROM. They continue to focus on key benchmarks and instructional criteria but also decide to consider other strengths and weaknesses in key content benchmarks and in all 24 instructional criteria. Their work to make a final selection includes the following:
(a) In order to align the content with state and district standards, the committee adds standards statements from their state and district frameworks to the comparison document in Appendix A. They note that some of the ideas from the six AAAS benchmarks are more closely aligned than others with their own standards. They decide to pay close attention to these particular parts of the benchmarks when looking at the content summaries of the textbooks in the report.
(b) The committee decides to use the Sightings by Benchmark charts from the CD-ROM to guide an in-depth examination of the books. They print the sightings, then six two-person teams - one for each benchmark - volunteer to compare the three books on each benchmark. They agree to focus on the ideas in the benchmark that are best aligned with the district and state standards. They pay special attention to how the concepts are developed across the grades and make notes. Afterward, the teams meet and compare their notes and compare their own findings with the Content Analysis on the CD-ROM. They use the Compare utility, viewing the Content Analyses side by side for each of the three books. Finally, the committee prepares a ranking of the three books on the content of the ideas contained in their district and state standards. One of the books appears stronger than originally thought, because even though it does not address all of the graphing benchmark, those ideas are not included in the district framework. The committee does decide to recommend a study by the framework committee on whether these ideas should be included in the district math framework.
(c) Using the Compare utility on the CD-ROM, the committee compares the Instructional Analyses for the three textbook series, taking into account their content findings. They keep track of the ranking of the three series for each of the 24 criteria and make special note of instructional strength on key math content ideas. At the end of this process, two of the textbook series appear to be stronger.
(d) Using the Compare utility, the committee examines each of the 12 Graphs side by side for the top two series. They pay special attention to criteria that are important for their students and teachers, and to the graphs of ratings on individual benchmarks and content strands that are most important. They keep track of the ranking of the two series, accumulated across all of the comparisons.
(e) At a summary meeting, the committee discusses their data and evidence. A clear consensus is reached on the top-rated textbook series. The committee not only has made an evidence-based decision but also has compiled a great deal of information about the series that they are eager to share with their colleagues and the community. Further, they have a list of areas that they believe will require supplementing, even for the top-rated book. They also know there are some key areas for which teachers will need professional development in order to implement the series successfully.
Scenario 4: Planning Staff Development for Using a Textbook Series
A district has just adopted a middle grades mathematics series. The mathematics curriculum specialist is responsible for providing staff development for the middle school teachers who will be using the books, and for considering additional materials that might be needed both for staff development and for student use. She knows that some of the teachers may need work in mathematics content in order to implement the conceptual development approach and emphasis on applications in the new series. Other teachers need work on instructional strategies, especially taking account of student knowledge, using embedded assessments, and helping students to learn problem solving and to reflect on what they have learned. Working closely with the teachers, she does the following:
(a) To identify the main areas of content that require attention, she looks at Part 2 of the report and examines the summary report of the textbook series they have adopted, focusing on the content analysis. From the In Brief chart, she notes that the geometry concepts and the algebra equation concepts benchmarks are rated as having minimal content coverage. She decides to examine this by printing copies of the Sightings by Benchmark charts for these benchmarks from the files on the CD-ROM. With the aid of the information about the content analysis procedure from Appendix C of the report, she checks some other geometry and algebra standards, trying to find lessons that address them. She concludes that, indeed, there is a need for additional conceptual development of several geometry and algebra concepts. She sets up a meeting with the middle school math teachers to explore possible supplementary units that might be used, including units from some of the textbooks series that were rated highly by AAAS.
(b) Turning to the instructional criteria, she examines the Instruction Highlights graph in the summary report. She notes that many of the criteria are less than satisfactory, but is most concerned about the very low ratings for Criterion II.4 (Addressing Misconceptions), all the criteria in Category V (Promoting Student Thinking about Mathematics), and Criterion VI.3 (Using Embedded Assessment). She also notes a very low rating for Criterion VII.1 (Providing Teacher Content Support), confirming her resolve to organize a series of mathematics content workshops for the teachers.
(c) In order to understand just what is involved in the instructional criteria she identified, she studies the instructional analysis procedure in Appendix C, then uses the Typical Sightings chart to find examples of the criteria in the textbook. She also studies the Instructional Ratings files on the CD-ROM to see the indicators for the criteria and the explanations that the analysts gave for assigning the low scores.
(d) With all of this information, and a better understanding of the instructional weaknesses of the textbook series her district is using, she plans a series of workshops to prepare teachers to use the supplementary units that will be needed. She sets up a meeting with the assistant superintendent to present her findings and discuss support for her plan.
Scenario 5: A Unit for an Undergraduate or Graduate Mathematics Methods Course
A professor of middle grades mathematics methods has read the report and decides that the analysis procedure, especially the instructional analysis, would make an important unit in her course. It would replace the previous unit in which she had students look through and use checklists to review textbooks as a way to familiarize themselves with adoption procedures. The objectives would include learning about the content and instructional analysis, reviewing some of the relevant research, and learning to use the report to review math textbooks. She decides that a good strategy for having students understand the instructional criteria and the report would be to have them do at least some part of the analysis themselves. The following are some of the activities she does with the students:
(a) As background reading, she assigns selected pages from the NCTM Standards, Science for All Americans, and Benchmarks for Science Literacy that deal with three of the six benchmarks (number, geometry, and algebra concepts) used in the Project 2061 analysis. In class, she employs the Project 2061 benchmark clarification process, guiding the students in an examination and discussion of the three benchmarks. The comparison document from Appendix A of the report helps to identify related benchmarks and standards. She provides a chapter from a textbook and a printout of the Sightings by Benchmark for one of the three benchmarks they have clarified. As an assignment for the next class, she asks the students to find five sightings of the benchmark and write a brief justification of why the sightings align with the benchmark. She also asks them to review the analysis procedure in Appendix C, with special attention to instructional Categories II and V.
(b) At the next class meeting, the students discuss their justifications of content sightings. The professor focuses their attention on the substance of the content alignment and on the specific ideas of the benchmark that are addressed. She then assigns instructional Categories II through V to groups and asks the students in each group to identify the instructional criteria from their category that are addressed by the content sightings. When they are finished, the groups make brief presentations to the class, then answer questions to justify their decisions. They reconvene to consider this input and make changes. The professor assigns them the task of calculating the rating for each instructional criterion, based on the content sightings they have decided on, using the indicators and scoring guide in Appendix C.
(c) At the third class meeting, the professor asks the students to reconvene in their groups, then hands each group a printout for their instructional criterion from the Instructional Ratings files on the CD-ROM. She asks each group to compare their own ratings with one another and with the ratings on the printout, then reconcile the differences. At the conclusion, she leads a discussion focused on probing their understanding of the instructional criteria.
(d) As a final unit project, the professor assigns pairs of students to use two benchmarks to review a textbook on content and on instructional Categories II, III, IV and V.