| |
 |
Middle Grades Mathematics Textbooks: A Benchmarks-Based Evaluation
Using the CD-ROM Data Sets
The data interface on the CD-ROM allows easy access to the information in two
ways. Click on Browse Evaluations and data sets 1 through 8 can be browsed.
Click on Compare Data and data sets 2, 3, and 5 can be compared on screen.
For example, graphs of instructional quality on geometry benchmarks can be
displayed side by side for two different textbook series. Or, the Content
Analysis and Instructional Analysis for the same textbook can be viewed simultaneously.
All of the files can be printed for closer study.
The following list of possible uses for each data set is not comprehensive,
but suggests some of the ways to take advantage of the Browse and Compare
options. (Technical information on how to use the CD-ROM can be accessed by
clicking the "Using the CD-ROM" navigation button on the left).
- Background Information - contains relevant summary information
about the textbooks. Browse the description to (a) see the authors
and publisher, (b) find a variety of types of information, including the
main mathematics topics, the typical lesson activities, the assessment
design, and the cost of the books, (c) locate references for studies of
student achievement or reviews that might have been completed, and (d)
link to publisher's Web sites.
- Content Analysis - describes the extent to which the content of
the textbook series attempts to address the substance and level of sophistication
of the ideas in each benchmark. Browse these in-depth content summaries
for each benchmark or examine summaries for two textbooks side by side
and Compare (a) how many of the ideas in the benchmark are addressed
substantively, (b) what kinds of activities and lessons are employed to
address the ideas, and (c) how the benchmark ideas are developed across
the grade levels.
- Instructional Analysis - describes the quality of instruction including
the ratings for each criterion and benchmark and an in-depth discussion
of the ratings. Browse the ratings and in-depth discussion for
each instructional category or examine the instructional analysis for
two textbooks side by side and Compare (a) the strengths and weaknesses
across benchmarks in the ratings charts, (b) the strengths and weaknesses
in individual criteria in the ratings charts, (c) examples that justify
and clarify the reasons for the ratings.
- Instructional Ratings - shows how the ratings were tabulated by
displaying the indicators and scoring scheme for each instructional criterion.
In addition, notes by the analysis teams are provided. Browse the
ratings on the 24 instructional criteria and their indicators for each
benchmark to (a) see the rating guidelines and how each indicator contributed
to the rating, (b) review the ratings of analysts and read their notes,
and (c) examine which benchmarks and which indicators elicited the most
and least agreement among the analysts.
- Graphs - depicts the ratings for the 24 instructional criteria
averaged in several ways: across all six benchmarks (providing an overall
average rating for the textbook series), across conceptual and skill benchmarks,
across content strands (number, geometry, algebra), and across each of
the six individual benchmarks. Browse the different graphs or Compare
the graphs for two textbooks side by side in order to (a) see the patterns
of strength or weakness across the 24 criteria, and (b) see differences
in strengths or weaknesses in particular benchmarks, strands, or between
concepts and skills.
- Rater Consistency - displays the percentage agreement between analysis
teams for each of the six benchmarks, a list of criteria on which the
analysis teams disagreed, and an overall average percentage agreement
for the series. Browse the data to (a) see the rater consistency
across benchmarks, (b) see the specific instructional criteria on which
analysis teams disagreed, and (c) examine rater consistency across textbook
series and benchmarks.
- Typical Sightings - provides an example of the evidence examined
by the analysis teams. The location (book, page number, activity description)
of a sighting that typifies the textbook's treatment of each criterion
is provided along with the average rating for each of the 24 instructional
criteria. Browse the cited page numbers and activities to (a) make
a guided review of a textbook series to get an overall sense of its average
quality on each of the 24 instructional criteria, and (b) make a guided
review of several textbook series, examining their quality using the typical
sightings.
- Sightings by Benchmark - lists the location (book, page number,
activity description) of all the evidence examined by the analysis teams.
Browse the number and types of lessons, activities, or exercises
that address each benchmark to (a) get an estimate of the amount of coverage
for key benchmarks, (b) see the balance of content coverage across grade
levels, (c) see the balance of coverage in lessons, exercises, teacher
notes, or other material, and (d) locate and examine actual activities
in the textbooks that address the selected benchmark ideas.
Example Scenarios for Applying the Results
|
