the Data Sets
The data interface
allows easy access to the information in two ways. Click on
Browse Evaluations and data sets 1 through 8 can be browsed.
Click on Compare Data and data sets 2, 3, and 5 can be compared
on screen. For example, graphs of instructional quality
on the Functions Idea Set can be displayed side by side for
two different textbook series. Or, the Content Analysis and
Instructional Analysis for the same textbook can be viewed simultaneously.
All of the files can be printed for closer study.
The following list
of possible uses for each data set is not comprehensive, but
suggests some of the ways to take advantage of the Browse
and Compare options.
Information - contains relevant summary information about
the textbooks. Browse the description to (a) see the
authors and publisher, (b) find a variety of types of information,
including the main mathematics topics, the typical lesson
activities, the assessment design, and the cost of the books,
(c) locate references for studies of student achievement or
reviews that might have been completed, and (d) link to publisher's
and developer's Web sites.
- Content Analysis
- describes the extent to which the content of the textbook
series attempts to address the substance and level of sophistication
of the ideas in each idea set. Browse these in-depth
content summaries for each idea set or examine summaries for
two textbooks side by side and Compare (a) how many
of the ideas in the idea set are addressed substantively,
(b) what kinds of activities and lessons are employed to address
the ideas, and (c) how the ideas are developed.
- describes the quality of instruction including the ratings
for each criterion and idea set and an in-depth discussion
of the ratings. Browse the ratings and in-depth discussion
for each instructional category or examine the instructional
analysis for two textbooks side by side and Compare
(a) the strengths and weaknesses across idea sets in the ratings
charts, (b) the strengths and weaknesses in individual criteria
in the ratings charts, (c) examples that justify and clarify
the reasons for the ratings.
Ratings - shows how the ratings were tabulated by displaying
the indicators and scoring scheme for each instructional criterion.
In addition, notes by the analysis teams are provided. Browse
the ratings on the 21 instructional criteria and their indicators
for each idea set to (a) see the rating guidelines and how
each indicator contributed to the rating, (b) review the ratings
of analysts and read their notes, and (c) examine which idea
sets and which indicators elicited the most and least agreement
among the analysts. Please note that no numerical ratings
were assigned for criteria in Category VII.
- depicts the ratings for the 21 instructional criteria averaged
in two ways: across all three idea sets (providing an overall
average rating for the textbook series) and across each of
the three idea sets. Browse the different graphs or
Compare the graphs for two textbooks side by side in
order to (a) see the patterns of strength or weakness across
the 21 criteria, and (b) see differences in strengths or weaknesses
in particular idea sets, strands, or between concepts and
- Rater Consistency
- displays the percentage agreement between analysis teams
for each of the three idea sets, a list of criteria on which
the analysis teams disagreed, and an overall average percentage
agreement for the series. Browse the data to (a) see
the rater consistency across idea sets, (b) see the specific
instructional criteria on which analysis teams disagreed,
and (c) examine rater consistency across textbook series and
- Typical Sightings
- provides an example of the evidence examined by the analysis
teams. The location (book, page number, activity description)
of a sighting that typifies the textbook's treatment of each
criterion is provided along with the average rating for each
of 21 instructional criteria. Browse the cited page
numbers and activities to (a) make a guided review of a textbook
series to get an overall sense of its average quality on each
of the 24 instructional criteria, and (b) make a guided review
of several textbook series, examining their quality using
the typical sightings.
- Sightings by
- lists the location (book, page number, activity description)
of all the evidence examined by the analysis teams. Browse
the number and types of lessons, activities, or exercises
that address each idea set to (a) get an estimate of the amount
of coverage for key idea sets, (b) see the balance of content
coverage across grade levels, (c) see the balance of coverage
in lessons, exercises, teacher notes, or other material, and
(d) locate and examine actual activities in the textbooks
that address the selected ideas.
Scenarios for Applying the Results