Middle Grades Mathematics Textbooks: A Benchmarks-Based Evaluation
Using the CD-ROM Data Sets
The data interface on the CD-ROM allows easy access to the information in two ways. Click on Browse Evaluations and data sets 1 through 8 can be browsed. Click on Compare Data and data sets 2, 3, and 5 can be compared on screen. For example, graphs of instructional quality on geometry benchmarks can be displayed side by side for two different textbook series. Or, the Content Analysis and Instructional Analysis for the same textbook can be viewed simultaneously. All of the files can be printed for closer study.
The following list of possible uses for each data set is not comprehensive, but suggests some of the ways to take advantage of the Browse and Compare options. (Technical information on how to use the CD-ROM can be accessed by clicking the "Using the CD-ROM" navigation button on the left).
- Background Information - contains relevant summary information about the textbooks. Browse the description to (a) see the authors and publisher, (b) find a variety of types of information, including the main mathematics topics, the typical lesson activities, the assessment design, and the cost of the books, (c) locate references for studies of student achievement or reviews that might have been completed, and (d) link to publisher's Web sites.
- Content Analysis - describes the extent to which the content of the textbook series attempts to address the substance and level of sophistication of the ideas in each benchmark. Browse these in-depth content summaries for each benchmark or examine summaries for two textbooks side by side and Compare (a) how many of the ideas in the benchmark are addressed substantively, (b) what kinds of activities and lessons are employed to address the ideas, and (c) how the benchmark ideas are developed across the grade levels.
- Instructional Analysis - describes the quality of instruction including the ratings for each criterion and benchmark and an in-depth discussion of the ratings. Browse the ratings and in-depth discussion for each instructional category or examine the instructional analysis for two textbooks side by side and Compare (a) the strengths and weaknesses across benchmarks in the ratings charts, (b) the strengths and weaknesses in individual criteria in the ratings charts, (c) examples that justify and clarify the reasons for the ratings.
- Instructional Ratings - shows how the ratings were tabulated by displaying the indicators and scoring scheme for each instructional criterion. In addition, notes by the analysis teams are provided. Browse the ratings on the 24 instructional criteria and their indicators for each benchmark to (a) see the rating guidelines and how each indicator contributed to the rating, (b) review the ratings of analysts and read their notes, and (c) examine which benchmarks and which indicators elicited the most and least agreement among the analysts.
- Graphs - depicts the ratings for the 24 instructional criteria averaged in several ways: across all six benchmarks (providing an overall average rating for the textbook series), across conceptual and skill benchmarks, across content strands (number, geometry, algebra), and across each of the six individual benchmarks. Browse the different graphs or Compare the graphs for two textbooks side by side in order to (a) see the patterns of strength or weakness across the 24 criteria, and (b) see differences in strengths or weaknesses in particular benchmarks, strands, or between concepts and skills.
- Rater Consistency - displays the percentage agreement between analysis teams for each of the six benchmarks, a list of criteria on which the analysis teams disagreed, and an overall average percentage agreement for the series. Browse the data to (a) see the rater consistency across benchmarks, (b) see the specific instructional criteria on which analysis teams disagreed, and (c) examine rater consistency across textbook series and benchmarks.
- Typical Sightings - provides an example of the evidence examined by the analysis teams. The location (book, page number, activity description) of a sighting that typifies the textbook's treatment of each criterion is provided along with the average rating for each of the 24 instructional criteria. Browse the cited page numbers and activities to (a) make a guided review of a textbook series to get an overall sense of its average quality on each of the 24 instructional criteria, and (b) make a guided review of several textbook series, examining their quality using the typical sightings.
- Sightings by Benchmark - lists the location (book, page number, activity description) of all the evidence examined by the analysis teams. Browse the number and types of lessons, activities, or exercises that address each benchmark to (a) get an estimate of the amount of coverage for key benchmarks, (b) see the balance of content coverage across grade levels, (c) see the balance of coverage in lessons, exercises, teacher notes, or other material, and (d) locate and examine actual activities in the textbooks that address the selected benchmark ideas.