Middle Grades Mathematics Textbooks: A Benchmarks-Based Evaluation
Example Scenarios for Applying the Results
The following scenarios reflect how various groups or individuals might use
Middle Grades Mathematics Textbooks: A Benchmarks-Based Evaluation
(the report), including the background material, individual textbook summary
reports, the appendices, and the data sets on CD-ROM.
Scenario 1: Thinking about Improving Mathematics Curriculum
Materials
Motivated by the Third International Mathematics and Science Study (TIMSS)
and state assessment results, and by a commitment to improve mathematics achievement
for all students, a district committee composed of teachers and parent leaders
has become interested in changing the mathematics curriculum materials being
used in their middle grades. They have reviewed and studied the National Council
of Teachers of Mathematics' (NCTM) Standards and AAAS's Benchmarks
for Science Literacy, deciding that these documents would be useful in
guiding their work. They wonder whether they should adopt the newest edition
of their current commercial textbook, which claims to address NCTM Standards;
consider some of the new materials they have heard about; or consider writing
their own curriculum materials so they could adapt to their own students'
needs and interests.
After visiting the Project 2061 Web site (http://www.project2061.org) and finding
the Web version of the report, they do the following:
(a) Reading the Table of Contents of the report, the group discovers the Appendices,
which provide details about the textbook analysis procedure, rating criteria,
and background research. They use this material to study, reflect on, and
discuss the key content learning goals in mathematics, the instructional criteria,
and the research on teaching and learning relevant to these criteria.
(b) The group agrees that their students should understand concepts well by
the end of the middle grades and that Category II (Building on Students Ideas
about Mathematics) and Category V (Promoting Student Thinking about Mathematics)
are especially important to them.
(c) They see the Typical Sightings charts in the report and decide to become
more familiar with the Project 2061 ratings. They use the link to publishers'
Web sites to order examination copies of textbook series that are rated in
the high, medium, and low range on the chart ranking all the textbooks, which
is located in the executive summary of the report. They focus on textbook
series that vary on conceptual benchmarks and on instructional Categories
II and V. They also order the print version of the report and accompanying
CD-ROM in preparation for a more in-depth review.
(d) When the examination copies of the textbooks and the report and CD-ROM
arrive, the group browses all of the files for each of the books they have
decided to review. They use the Typical Sightings charts to find lessons in
each of the books that address the instructional criteria, learning what characteristics
contribute to high and low ratings, especially on the content and instructional
criteria of most interest to them.
(e) After several weeks of study, the group is ready to make some recommendations
to the district on the key criteria and strategies that should be followed
in the coming textbook adoption.
Scenario 2: Narrowing the List of Textbook Adoption Choices
A district mathematics textbook adoption committee has been charged with selecting
three top textbook series for the middle grades. The early part of the adoption
process has produced a set of basic requirements for the books: strong development
of number and algebra content, support for teachers, adaptability for diverse
students, alignment with national standards, strong assessment, and a high
level of student engagement.
The committee has reviewed the Project 2061 Web site and ordered the report
and CD-ROM. Their work to narrow the list of candidate textbook series includes
the following:
(a) The committee studies the individual summary reports for each of the 13
textbook series, using the In Brief charts and the Instruction Highlights
graphs. They read and discuss the content and instructional summary paragraphs,
with special focus on the number and algebra benchmarks and instructional
Category III (Engaging Students in Mathematics), Category VI (Assessing Student
Progress in Mathematics), and Category VII (Enhancing the Mathematics Learning
Environment), in which they have the most interest. They note which textbook
series have acceptable content for the number and algebra benchmarks and satisfactory
ratings on instructional Categories III, VI, and VII.
(b) The preliminary review has identified four or five textbook series that
have some strengths related to the committee's requirements. They decide to
use the CD-ROM data sets to do a more in-depth review of the criteria. Using
the Compare feature, the committee members do a comparison of the Content
Analysis of number and algebra benchmarks for each of five textbooks. This
results in the elimination of one of the series, which does not develop algebra
equation concepts sufficiently across the grades.
(c) The committee members continue with the remaining four series, comparing
instructional Categories III, VI, and VII using both the Instructional Analysis
and the Graphs. This data, especially the graphs of the ratings for number
and algebra strands, help them to select their three finalist textbook series.
(d) In addition to having a great deal of evidence to support their selections,
the committee has developed a set of recommendations for selecting the best
textbook series for their students, and has a tentative ranking of the three
finalists just in case anyone is interested.
Scenario 3: Choosing a Middle Grades Mathematics Textbook
Series
A school mathematics textbook adoption committee has been charged with guiding
the selection of a middle grades textbook series. They have narrowed the list
to three top candidates and plan to continue having wide engagement by the
teachers and community. The committee has already reviewed the Project 2061
Web site and has copies of the report and the CD-ROM. They continue to focus
on key benchmarks and instructional criteria but also decide to consider other
strengths and weaknesses in key content benchmarks and in all 24 instructional
criteria. Their work to make a final selection includes the following:
(a) In order to align the content with state and district standards, the committee
adds standards statements from their state and district frameworks to the
comparison document in Appendix A. They note that some of the ideas from the
six AAAS benchmarks are more closely aligned than others with their own standards.
They decide to pay close attention to these particular parts of the benchmarks
when looking at the content summaries of the textbooks in the report.
(b) The committee decides to use the Sightings by Benchmark charts from the
CD-ROM to guide an in-depth examination of the books. They print the sightings,
then six two-person teams - one for each benchmark - volunteer to compare
the three books on each benchmark. They agree to focus on the ideas in the
benchmark that are best aligned with the district and state standards. They
pay special attention to how the concepts are developed across the grades
and make notes. Afterward, the teams meet and compare their notes and compare
their own findings with the Content Analysis on the CD-ROM. They use the Compare
utility, viewing the Content Analyses side by side for each of the three books.
Finally, the committee prepares a ranking of the three books on the content
of the ideas contained in their district and state standards. One of the books
appears stronger than originally thought, because even though it does not
address all of the graphing benchmark, those ideas are not included in the
district framework. The committee does decide to recommend a study by the
framework committee on whether these ideas should be included in the district
math framework.
(c) Using the Compare utility on the CD-ROM, the committee compares the Instructional
Analyses for the three textbook series, taking into account their content
findings. They keep track of the ranking of the three series for each of the
24 criteria and make special note of instructional strength on key math content
ideas. At the end of this process, two of the textbook series appear to be
stronger.
(d) Using the Compare utility, the committee examines each of the 12 Graphs
side by side for the top two series. They pay special attention to criteria
that are important for their students and teachers, and to the graphs of ratings
on individual benchmarks and content strands that are most important. They
keep track of the ranking of the two series, accumulated across all of the
comparisons.
(e) At a summary meeting, the committee discusses their data and evidence.
A clear consensus is reached on the top-rated textbook series. The committee
not only has made an evidence-based decision but also has compiled a great
deal of information about the series that they are eager to share with their
colleagues and the community. Further, they have a list of areas that they
believe will require supplementing, even for the top-rated book. They also
know there are some key areas for which teachers will need professional development
in order to implement the series successfully.
Scenario 4: Planning Staff Development for Using a Textbook
Series
A district has just adopted a middle grades mathematics series. The mathematics
curriculum specialist is responsible for providing staff development for the
middle school teachers who will be using the books, and for considering additional
materials that might be needed both for staff development and for student
use. She knows that some of the teachers may need work in mathematics content
in order to implement the conceptual development approach and emphasis on
applications in the new series. Other teachers need work on instructional
strategies, especially taking account of student knowledge, using embedded
assessments, and helping students to learn problem solving and to reflect
on what they have learned. Working closely with the teachers, she does the
following:
(a) To identify the main areas of content that require attention, she looks
at Part 2 of the report and examines the summary report of the textbook series
they have adopted, focusing on the content analysis. From the In Brief chart,
she notes that the geometry concepts and the algebra equation concepts benchmarks
are rated as having minimal content coverage. She decides to examine this
by printing copies of the Sightings by Benchmark charts for these benchmarks
from the files on the CD-ROM. With the aid of the information about the content
analysis procedure from Appendix C of the report, she checks some other geometry
and algebra standards, trying to find lessons that address them. She concludes
that, indeed, there is a need for additional conceptual development of several
geometry and algebra concepts. She sets up a meeting with the middle school
math teachers to explore possible supplementary units that might be used,
including units from some of the textbooks series that were rated highly by
AAAS.
(b) Turning to the instructional criteria, she examines the Instruction Highlights
graph in the summary report. She notes that many of the criteria are less
than satisfactory, but is most concerned about the very low ratings for Criterion
II.4 (Addressing Misconceptions), all the criteria in Category V (Promoting
Student Thinking about Mathematics), and Criterion VI.3 (Using Embedded Assessment).
She also notes a very low rating for Criterion VII.1 (Providing Teacher Content
Support), confirming her resolve to organize a series of mathematics content
workshops for the teachers.
(c) In order to understand just what is involved in the instructional criteria
she identified, she studies the instructional analysis procedure in Appendix
C, then uses the Typical Sightings chart to find examples of the criteria
in the textbook. She also studies the Instructional Ratings files on the CD-ROM
to see the indicators for the criteria and the explanations that the analysts
gave for assigning the low scores.
(d) With all of this information, and a better understanding of the instructional
weaknesses of the textbook series her district is using, she plans a series
of workshops to prepare teachers to use the supplementary units that will
be needed. She sets up a meeting with the assistant superintendent to present
her findings and discuss support for her plan.
Scenario 5: A Unit for an Undergraduate or Graduate Mathematics
Methods Course
A professor of middle grades mathematics methods has read the report and decides
that the analysis procedure, especially the instructional analysis, would
make an important unit in her course. It would replace the previous unit in
which she had students look through and use checklists to review textbooks
as a way to familiarize themselves with adoption procedures. The objectives
would include learning about the content and instructional analysis, reviewing
some of the relevant research, and learning to use the report to review math
textbooks. She decides that a good strategy for having students understand
the instructional criteria and the report would be to have them do at least
some part of the analysis themselves. The following are some of the activities
she does with the students:
(a) As background reading, she assigns selected pages from the NCTM Standards,
Science for All Americans, and Benchmarks for Science Literacy
that deal with three of the six benchmarks (number, geometry, and algebra
concepts) used in the Project 2061 analysis. In class, she employs the Project
2061 benchmark clarification process, guiding the students in an examination
and discussion of the three benchmarks. The comparison document from Appendix
A of the report helps to identify related benchmarks and standards. She provides
a chapter from a textbook and a printout of the Sightings by Benchmark for
one of the three benchmarks they have clarified. As an assignment for the
next class, she asks the students to find five sightings of the benchmark
and write a brief justification of why the sightings align with the benchmark.
She also asks them to review the analysis procedure in Appendix C, with special
attention to instructional Categories II and V.
(b) At the next class meeting, the students discuss their justifications of
content sightings. The professor focuses their attention on the substance
of the content alignment and on the specific ideas of the benchmark that are
addressed. She then assigns instructional Categories II through V to groups
and asks the students in each group to identify the instructional criteria
from their category that are addressed by the content sightings. When they
are finished, the groups make brief presentations to the class, then answer
questions to justify their decisions. They reconvene to consider this input
and make changes. The professor assigns them the task of calculating the rating
for each instructional criterion, based on the content sightings they have
decided on, using the indicators and scoring guide in Appendix C.
(c) At the third class meeting, the professor asks the students to reconvene
in their groups, then hands each group a printout for their instructional
criterion from the Instructional Ratings files on the CD-ROM. She asks each
group to compare their own ratings with one another and with the ratings on
the printout, then reconcile the differences. At the conclusion, she leads
a discussion focused on probing their understanding of the instructional criteria.
(d) As a final unit project, the professor assigns pairs of students to use
two benchmarks to review a textbook on content and on instructional Categories
II, III, IV and V.
