 |
Project
2061 Algebra Textbooks A Standards-Based Evaluation |
 |
 |
 |
 |
 |
 |
 |
Evaluation Methods
| Report Preparation: |
The following Project
2061 staff members were responsible for supervising the development
of the procedure, training analysts, compiling data, and preparing
the reports. Gerald Kulm, Program Director Kathleen Morris, Senior Program Associate Laura Grier, Project Coordinator Freelance Editor Paul Elliott completed final editing of
the reports. |
| Reviewers: | The analysts who reviewed
and rated the texts were experienced mathematics teachers
and university mathematics and mathematics education faculty
trained in the use of the Project 2061 procedure. The reviewers
were evenly divided between experienced, practicing classroom
teachers and mathematics educators who were knowledgeable
about research on mathematics learning and teaching. All of
the reviewers were highly capable in mathematics content.
Reviewers were formed into 6 teams of two persons each, at
least one of whom was an experienced classroom teacher. The
reviewers were:
|
| Training: | More than half of the reviewers had one or more previous experiences with the Project 2061 analysis procedure in reviewing and rating middle grades mathematics materials. The persons who had not had previous reviewing experience attended an intensive three-day training session immediately before the analysis. In the session, they learned the analysis procedure and applied it to the analysis of a middle grades mathematics textbook that had been reviewed earlier as part of the reliability study. Their ratings on each criterion were discussed and compared with the ratings that had been assigned by experienced reviewers. After the training, each of these reviewers was teamed with an experienced reviewer. |
| Design: | Each team rated a set of algebra standards from the Principles and Standards for School Mathematics (NCTM, 1998/2000). Each set encompassed ideas from one area of algebra: Functions, Operations, and Variables. Each team used the same idea set to review a total of twelve textbooks or sets of textbooks over a period of 12 days. About half of the materials consisted of a single textbook intended for use in an algebra I course. The other materials consisted of a set of textbooks intended to cover four to six years of integrated secondary mathematics. Two teams completed a review for each textbook or textbook series independently. |
| Analysis: | Preliminary sightings for each standards idea set were identified by Project 2061 staff and independent content reviewers prior to the analysis. After clarifying each idea set, the reviewers checked the sightings in the textbook, making necessary additions, deletions, or revisions. Reviewers then aligned sightings with the relevant instructional criteria and used indicators and rating guides to rate how well the textbook addressed each criterion. After completing the analysis for an idea set, teams met to reconcile their ratings, and to receive necessary clarifications from Project 2061 staff if ratings differed significantly. Teams submitted (1) the content sightings for each idea set; (2) the sightings and justifications for each instructional criterion; (3) the rating of each instructional indicator [Met, Not Met, Unsure]; and, (4) the rating and justification for each instructional criterion [High, Medium, Low, None] including supporting notes. |
| Reports: | Project 2061 staff took these data plus notes derived from debriefing the teams and prepared detailed reports on each textbook. |