Appendix C: Methodology

Appendix C

METHODOLOGY

Evaluation Methods

Report Preparation:	The following Project 2061 staff members were responsible for supervising the development of the procedure, training analysts, compiling data, and preparing the summary reports. Gerald Kulm, Program Director Kathleen Morris, Senior Program Associate Laura Grier, Project Coordinator These staff members completed the final editing of the reports and prepared them for publication. Mary Koppal, Communications Director Susan Shuttleworth, Senior Editor The following staff members designed and produced the Web version of the report and the CD-ROM. Francis Molina, Electronic Resources Manager Samuel Kim, Technology Specialist Thelxi Proimaki, Technology Specialist Alan Stonebraker, Technology Specialist Brian Sweeney, Technology Specialist Christina Bautista, Intern Vinod Rawat, Technology Assistant/Intern Moojan Zare-Parsi, Intern
Reviewers:	The analysts who reviewed and rated the texts were experienced mathematics teachers and university mathematics and mathematics education faculty trained in the use of the Project 2061 procedure. The reviewers were evenly divided between experienced, practicing classroom teachers and mathematics educators who were knowledgeable about research on mathematics learning and teaching. All of the reviewers were highly capable in mathematics content. Reviewers were formed into 12 teams of two persons each, at least one of whom was an experienced classroom teacher. The reviewers were: Donna Cook, Crossett Brook Middle School (VT) Joe Cook, Paige Academy (NY) Michelle Crowley, Northern Kentucky University Peg Darcy, Louisville Schools (KY) Mark Deegan, Springfield (VA) High School Sassan Dehghan, University of Maryland Kay Dighans, Quaw Elementary School (MT) Florence Fasanelli, Washington, DC Ben Flora, Morehead State University (KY) Mary Ellen Foley, Louisiana State University – Shreveport Marshall Gordon, The Park School (MD) Linda Hackett, American University, Washingtin, DC Steve Hays, Largo High School (MD) Willis Johnson, University of Kentucky Bill Kunnecke, Marshall County (KY) Schools Jan McDowell, Noe Middle School (KY) Alice Mikovch, Western Kentucky University Joan Montgomery, Pilgrim Christian Academy (NY) Linda Montgomery, Berea County (KY) Schools Sue Reehm, Eastern Kentucky University Kay Reinke, Oklahoma State University Patty Schumacher, Telford County (PA) Schools Faye Stevens, Cadiz County (KY) Schools Diane Surati, Crossett Brook Middle School (VT) James Telese, University of Texas – Brownsville Sri Viswanathan, Fairfax County (VA) Schools Connie Widmer, Northern Kentucky University Grace Wood, Fort Peck Community College (MT)
Training:	About half of the reviewers had one or more previous experiences with the Project 2061 analysis procedure in reviewing and rating middle grades mathematics materials. The persons who had not had previous reviewing experience attended an intensive three-day training session immediately before the analysis. In the session, they learned the analysis procedure and applied it to the analysis of a middle grades mathematics textbook that had been reviewed earlier as part of the reliability study. Their ratings on each criterion were discussed and compared with the ratings that had been assigned by experienced reviewers. After the training, each of these reviewers was teamed with an experienced reviewer.
Design:	Each team rated two mathematics benchmarks, one conceptual and one skill, from one of the three mathematics strands: Number, Geometry, and Algebra. Each team reviewed a total of five sets of books, using the same two benchmarks to review the textbooks over a period of two days. Most of the materials consisted of three textbooks, one for 6^th, 7^th, and 8^th grades. Two of the sets of materials consisted of a number of units, usually three or four per grade, that were relevant to a particular pair of benchmarks. Two teams completed a review for each textbook independently.
Analysis:	Preliminary sightings for each benchmark were identified by Project 2061 staff prior to the analysis. After clarifying each benchmark, the reviewers checked the sightings in the textbook, making necessary additions, deletions, or revisions. Reviewers then aligned sightings with the relevant instructional criteria and used indicators and rating guides to rate how well the textbook addressed each criterion. After completing the analysis for a benchmark, teams met to reconcile their ratings and receive necessary clarifications from Project 2061 staff if ratings differed significantly. Teams submitted: (1) the content sightings for each benchmark; (2) the sightings and justifications for each instructional criterion; (3) the rating of each instructional indicator [Met, Not Met, Unsure]; and, (4) the rating for each instructional criterion [High, Medium, Low, None] including supporting notes.
Reports	Project 2061 staff took these data plus notes derived from debriefing the teams and prepared detailed reports on each textbook. Summaries of these reports were published on Project 2061's Web site and in book form. The detailed reports and all the analysis data were made available on CD-ROM.

The Project 2061 Analysis Procedure for Mathematics Curriculum Materials

Introduction

Deciding which curriculum materials to use is one of the most important professional judgments that educators make. Textbook adoption committees make recommendations that influence instruction for years to come, and the daily decisions that teachers make about which teaching units or chapters to use and how to use them largely determine what and how students will be expected to learn.

Such important decisions require a valid and reliable method for evaluating the quality of curriculum materials. Even an in-depth review of the topics covered by a textbook or a teaching unit may not be sufficient to determine whether the material will actually help students learn that content. What is needed is a manageable process for examining curriculum materials that gets below the surface by focusing intensely on the appropriateness of content and the utility of instructional design.

With funding from the National Science Foundation and in collaboration with hundreds of K-12 teachers, curriculum specialists, teacher educators, scientists, and materials developers, Project 2061 of the American Association for the Advancement of Science (AAAS) has been developing a process for analyzing curriculum materials. Field tests suggest that Project 2061's curriculum-analysis procedure will not only serve the materials adoption needs of the schools but also help teachers revise existing materials to increase their effectiveness, guide developers in the creation of new materials, and contribute to the professional development of those who use it.

Specific Learning Goals Are Key

Until recently, there was nothing against which to judge appropriateness of content and utility of instructional design. Now, as a result of the standards-based reform movement in education, these judgments can be made with a high degree of confidence. In mathematics, for example, the appearance of Science for All Americans (AAAS, 1989), Curriculum and Evaluation Standards for School Mathematics (NCTM, 1989), and Benchmarks for Science Literacy (AAAS, 1993) has made it possible to make more thoughtful decisions about curriculum materials than ever before.

Although the Project 2061 curriculum-analysis procedure was developed using the learning goals in Benchmarks and the mathematics and science standards, subsequent work has indicated that some state education frameworks also can be used. Indeed, the process would seem to apply to any K-12 school subject for which specific learning goals have been agreed upon. These goals must be explicit statements of what knowledge and skills students are expected to learn, and they must be precise. Vague statements such as "students should understand fractions" are not adequate. Instead, consider this benchmark dealing with the meanings of fractions that students should know by the end of the eighth grade:

Students should know that:
"The expression a/b can mean different things: a parts of size 1/b each, a divided by b, or a compared to b."

At its simplest level, the Project 2061 curriculum-analysis procedure involves the following five steps:

Identify specific learning goals to serve as the intellectual basis for the analysis. This is done before beginning to examine any curriculum materials. The source for appropriate goals can be national standards or documents such as those mentioned above, state or local standards and curriculum frameworks, or sources like them. To be useful, the goals must be precise in describing the knowledge or skills they intend students to have. If the set of goals is large, a representative sample of them should be selected for purposes of analysis.

Make a preliminary inspection of the curriculum materials to see whether they are likely to address the targeted learning goals. If there appears to be little or no correspondence, the materials can be rejected without further analysis. If the outlook is more positive, go on to a content analysis.

Analyze the curriculum materials for alignment between content and the selected learning goals. The purpose here is to determine, citing evidence from the materials, whether the content in the material matches specific learning goals, not just whether the topic headings are similar. At the topic level, alignment is never difficult, since most topics, proportions, equations, graphing, and so forth, lack specificity making them easy to match. If the results of this analysis are positive, then reviewers can take the next step.

Analyze the curriculum materials for alignment between instruction and the selected learning goals. This involves estimating the degree to which the materials (including their accompanying teacher's guides) reflect what is known generally about student learning and effective teaching and, more important, the degree to which they support student learning of the specific knowledge and skills for which a content match has been found. Again, evidence from the materials must be shown.

Summarize the relationship between the curriculum materials being evaluated and the selected learning goals. The summary can take the form of a profile of the selected goals in terms of the content and instruction criteria, or a profile of the criteria in terms of the selected goals. In either case, a statement of strengths and weaknesses should be included. With this information in hand, reviewers can make more knowledgeable adoption decisions and suggest ways for improving the examined materials.

In addition to its careful focus on matching content and instruction to very specific learning goals, the Project 2061 procedure has other features that set it apart. For example, its emphasis on collecting explicit evidence (citing page numbers and other references) of a material's alignment with learning goals adds rigor and reliability to decisions about curriculum materials. Similarly, the Project 2061 procedure calls for a team approach to the analytical task, thus providing opportunities for reviewers to defend their own judgments about materials and to question those of other reviewers. These and other characteristics help make participation in the analytical process itself a powerful professional development experience.

The Project 2061 Curriculum-Analysis Procedure in Detail

To provide a better sense of how the procedure works, the following describes in more detail each step in the procedure. The description pays particular attention to the various criteria used to evaluate the instructional effectiveness of materials.

Identify specific learning goals to serve as the intellectual basis for the analysis. After reviewers have agreed upon a set of learning goals as a framework for the analysis, the task is then to choose specific learning goals that will serve as the focus of further study.

When evaluating stand-alone curriculum units that cover a relatively short period of time, it might be possible and worthwhile to analyze all of the learning goals that appear to be targeted by the material. However, in the evaluation of yearlong courses or multi-year programs, this becomes impractical. Therefore, a crucial step in the analysis procedure is the sampling of a few learning goals that will lead to valid and reliable generalizations about the material.

Sampling of standards should be representative of the whole set of goals specified in the framework or standards being applied and should reflect the reviewers' needs. For example, if the review committee's task is to select a course in high school Algebra that is aligned with a state mathematics framework or NCTM Standards, it might identify a sample of learning goals from important topic areas (e.g., number systems, equations, graphs, functions) and include learning goals that reflect different types of knowledge (e.g., skills, conceptual understanding, problem solving). When examining elementary or middle school mathematics materials, one would probably want to broaden the range of learning goal statements examined to include important strands in mathematics (e.g., number, geometry, algebra, and statistics).

Make a preliminary inspection of the curriculum materials to see whether they are likely to address the targeted learning goals. Once learning goal statements have been selected, the next step is to make a first pass at the materials to identify those whose content appears to correspond reasonably well to the learning goals. Materials that do not meet these initial criteria are not analyzed further.

Reviewers then examine materials on the shortened list more carefully to locate and record places where each selected learning goal seems to be targeted (e.g., particular readings, experiments, discussion questions). If several sightings are found for some or all of the sample learning goals in the material, then these sightings will be looked at more carefully in subsequent steps of the analysis. If, on the other hand, sightings cannot be found for a significant number of the sample learning goals, then the material is dropped from the list.

Analyze the curriculum materials for alignment between content and the selected learning goals. This analysis is a more rigorous examination of the link between the subject material and the selected learning goals and involves giving precise attention to both ends of the match; the precise meaning of the learning goal on one end, and the precise intention of the material on the other.

With respect to each of the sampled learning goals, the material is examined using such questions as:

Does the content called for in the material address the substance of a specific learning goal or only the learning goal's general "topic"?

Does the content reflect the level of sophistication of the specific learning goal, or are the activities more appropriate for targeting learning goals at an earlier or later grade level?

Does the content address all parts of a specific learning goal or only some? (While it is not necessary that any particular unit would address all of the ideas in a learning goal or standard, the K-12 curriculum as a whole should do so. The purpose of this question is to provide an account of precisely what ideas are treated.)

In addition, an attempt is made to estimate the degree of overlap between the material's content and the set of learning goals of interest. Thus, this step in the analysis is designed to answer questions regarding the material's inclusion of content that is not required for reaching mathematics literacy and the extent to which the material distinguishes between essential and non-essential content. (While distinguishing content essential for literacy from non-essential content in material might seem to be a luxury, it assists teachers in determining the range of students for which the material can be used. Identifying the non-essential material makes it easier for the teacher to direct better students to enrichment activities and allows students themselves to avoid overload from ideas that go beyond what is vital.)

Analyze the curriculum materials for alignment between instruction and the selected learning goals. The purpose here is to estimate how well material addresses targeted learning goals from the perspective of what is known about student learning and effective teaching. The criteria for making the judgments in the instructional analysis are derived from research on learning and teaching and on the craft knowledge of experienced educators. In the context of mathematics literacy, these are summarized in Chapter 13, "Effective Learning and Teaching," of Science for All Americans and in Chapter 15, "The Research Base," of Benchmarks for Science Literacy. From these and other sources, seven criteria categories (shown below) have been identified to serve as a basis for the instructional analysis.

Category I: Identifying a Sense of Purpose

Part of planning a coherent curriculum involves deciding on its purposes and on what learning experiences will likely contribute to achieving those purposes. But while coherence from the designers’ point of view is important, it may be inadequate to give students the same sense of what they are doing and why. This category includes criteria to determine whether the material attempts to make its purposes clear and meaningful to the student and genuinely relates lessons to the unit purpose.

I.1 Conveying Unit Purpose: Does the material convey an overall sense of purpose and direction that is understandable and motivating to students?

Clarification:
This criterion involves examining whether the material begins with (or early on presents) an over-arching question or problem to be addressed by the unit (e.g., How can a graph help to make predictions?), or a representation of what will be learned (e.g., a concept map of the main ideas that will be explored), or otherwise identifies a purpose for the unit or chapter for the students (e.g., a clear statement of objectives, using know terms). The problem, question, representation or purpose provided by the material should be explicit and comprehensible by the students, and it should be plausible that it would be interesting and/or motivating to them. A material that begins with abstractions or phenomena outside students’ range of perception, understanding, or knowledge does not adequately meet the criterion. However, a material that starts with an unfamiliar but highly interesting phenomenon that is likely to motivate students may meet the criterion.

Providing students with a sense of purpose for a whole unit or chapter is not always possible (for example, there may not be a single question or problem that is broad enough to foreshadow all learning goals in the unit) or even desirable (for example, providing a purpose on a large scale can lead to a complex sequence of activities that is too demanding on the memory of younger students). In such cases, it may be sufficient for the material to frame sections within a unit rather than the whole unit or chapter.

Indicators of meeting Criterion I.1:
1. The purpose is presented to students explicitly (or implicitly through a problem, question, or representation).
2. The purpose is likely to be comprehensible to students.
3. The purpose is likely to be interesting and/or motivating to students.
4. Students are given an opportunity to think about and discuss the purpose.
5. Most activities or lessons are consistent with the stated purpose.
6. The material returns to the stated purpose at the end of the unit or chapter.

Scoring Scheme:
High: The material meets all 6 indicators.
Medium: The material meets indicator 1, along with 3 of the other 5 indicators.
Low: The material meets indicator 1, along with 1 of the other 5 indicators.
None: The material does not meet any of the indicators.

I.2 Conveying Lesson Purpose: Does the material convey the purpose of each activity or lesson and its relationship to others?

Clarification:
The question is whether the purpose of individual activities or lessons (as opposed to the whole unit) is made apparent to the students and whether there are logical transitions and connections between activities or lessons. If a classroom visitor asked students what they were doing and why, is there reason to think they would know?

The purpose of individual activities or lessons could be brought out through the text, teacher comments (suggested in the material), and/or student responses to questions. For example, the purpose of gathering data about the heights of students in the class (to learn about dispersion of data) might be brought out by text explanation, teacher explanation, or by the students coming up with a description of the spread of data.

Indicators of meeting Criterion I.2:
1. The material conveys or prompts teachers to convey the purpose of each activity or lesson to students.
2. Each activity encourages each student to think about the purpose of the activity or lesson.
3. The material conveys or prompts teachers to convey to students how each activity or lesson relates to the other activities.
4. The material periodically engages students in thinking about what they have learned so far and what they need to learn/do next.

Scoring Scheme:
High: The material meets all 4 indicators.
Medium: The material meets 3 of the 4 indicators.
Low: The material meets 1 of the 4 indicators.
None: The material does not meet any of the indicators.

I.3 Justifying Sequence of Activities: Does the material involve students in a logical or strategic sequence of activities (versus a collection of activities) that build toward understanding of the ideas in the unit or chapter purpose?

Clarification:
The issue here is whether there is a logical or strategic sequence of activities in the material and whether this logic or strategy is made explicit to the teacher or just inferred by the reviewer. A rationale or implicit reason for the sequence of activities should be clear, providing the teacher and students with a sense of making progress toward the purpose of the unit or chapter.

Indicators of meeting Criterion I.3:
1. The material provides a rationale for the overall sequence of activities or lessons.
2. If no rationale for the overall sequence of activities or lessons is provided, the reviewer can identify one.
3. The sequence of activities reflects the stated or inferred rationale or purpose.

Scoring Scheme:
High: The material meets indicators 1 and 3.
Medium: The material meets indicators 2 and 3.
Low: The material meets indicator 1 or 2.
None: The material does not meet any of the indicators.

Category II: Building on Student Ideas about Mathematics

Fostering better understanding in students requires taking time to attend to the ideas they already have, both ideas that are incorrect and ideas that can serve as a foundation for subsequent learning. Such attention requires that teachers be informed about prerequisite ideas/skills needed for understanding a benchmark and what their students’ initial ideas are—in particular, the ideas that may interfere with learning mathematics. Moreover, teachers can help address students’ ideas if they know what is likely to work. This category examines whether the material contains specific suggestions for identifying and addressing student ideas.

II.1 Specifying Prerequisite Knowledge. Does the material specify and address prerequisite knowledge/skills that are necessary to the learning of the benchmark?

Clarification:
This criterion refers to (a) prerequisites to concepts or skills in the benchmark examined, and (b) prerequisites to activities used in the material to teach the concepts or skills in the benchmark examined.

(a) Understanding the ideas in benchmarks often requires that students first understand some other "prerequisite" concepts or skills. For example, knowing what a prime factor is is prerequisite to learning how to find the common denominator of fractions.

(b) In addition to prerequisites to specific ideas in benchmarks, additional prerequisites may arise from the specific activities used to teach them. For example, consider the benchmark: "Spreading data on a number line helps to see what the extremes are, where they pile up, and where the gaps are. A summary of the data includes where the middle is and how much spread is around it." There are no benchmarks that are prerequisite to the ideas in this benchmark. However, as the students work through a curriculum material, they may be engaged in investigations that target this benchmark and involve measurement. In these cases, the material should not take for granted that students will have developed measuring skills. Teachers should be alerted to this prerequisite (measurement skills) and encouraged to support their students in developing measurement skills.

Responding to the "prerequisites" criterion involves (a) making a list of prerequisite concepts and/or skills, (b) examining whether the material alerts to any prerequisite ideas on the list and if so which ones, (c) examining whether the material has in fact adequately addressed the prerequisites in the same or earlier units, and (d) examining whether the material helps students make connections between benchmarks and their prerequisites. While a stand-alone unit should not be faulted for not addressing prerequisite ideas or skills, it should be expected to make connections between benchmarks and their prerequisites.

Indicators of meeting Criterion II.1:
1. The material makes explicit what the specific prerequisite ideas or skills are, if any exist.
2. The material addresses the identified prerequisites in the same unit or in earlier units.
3. The material makes connections between benchmark ideas and their prerequisites (even if the prerequisites are addressed elsewhere).

Scoring Scheme:
High: The material meets all 3 indicators.
Medium: The material meets 2 out of the 3 indicators.
Low: The material meets 1 of the 3 indicators.
None: The material does not meet any of the indicators.

II.2 Alerting Teacher to Student Ideas. Does the material alert teachers to commonly held student ideas (both troublesome and helpful) such as those described in Benchmarks for Science Literacy Chapter 15: The Research Base?

Clarification:
Researchers have identified ideas that students have in several content areas. The issue here is whether the material informs teachers about students’ commonly held ideas in the topic areas the material addresses. This information can help teachers (a) understand better their own students’ ideas, (b) decide what ideas to build on and what changes to promote, or (c) if the material is already designed in ways that build on or attempt to change students’ commonly held ideas, to better understand the rationale and purpose behind designed strategies and activities.

Responding to this question involves examining (1) whether there is research on commonly held student ideas in the topic area/s that the material addresses, (2) whether the material alerts teachers to such ideas, and (3) whether the material accurately represents research findings. Summaries of research on students ideas in mathematics (such as those included in Benchmarks Chapter 15: The Research Base or the NCTM Research Ideas for the Classroom series) will be helpful to reviewers who will want to know what ideas students typically have about the topics that the curriculum material they are examining addresses. If there is no research on student ideas in the topic area/s that the material addresses, the material should not be faulted for not addressing this criterion.

Indicators of meeting Criterion II.2:
1. The material lists, conveys, or identifies specific commonly held ideas that are relevant to the benchmark (rather than just to relevant difficult topics).
2. The material clarifies/explains commonly held ideas.
3. The material explains or refers to commonly held ideas in an accurate way.

Scoring Scheme:
High: The material meets all 3 indicators.
Medium: The material meets indicators 1 and 2.
Low: The material meets indicator 1.
None: The material does not meet any of the indicators.

II.3 Assisting Teacher in Identifying Ideas. Does the material include suggestions for teachers to find out what their students think about familiar situations related to a benchmark before the mathematical ideas are introduced?

Clarification:
Teachers need guidance in identifying students’ ideas in these unresearched areas. But even in areas in which there is research on commonly held student ideas, teachers need help in identifying what proportion of their own students hold these ideas as well as other more idiosyncratic ideas.

Responding to this criterion involves examining not only whether the material encourages teachers to find out students’ ideas but also whether it provides specific suggestions for how to do so. Suggestions may include providing (a) tasks in which students make predictions and give their own descriptions and explanations of concepts or skills; (b) tasks in which students are asked to represent their understandings in drawings; (c) tasks in which students are asked to interpret information (for example the solution to an algorithm related to ideas in the benchmark), discuss connections with related topics, or discuss alternative solutions, or justifications; or (d) tasks which ask students about the meaning of specific terms and/or probe for understanding of important relationships between concepts.

Responding to the criterion also involves examining the quality of the suggestions provided. Tasks should not focus exclusively on identifying students’ meaning for terms. While including such tasks is useful, it is important to look for tasks in which students make predictions and/or give explanations of concepts or procedures. It is important that tasks make sense to students who have never studied the topic and include questions posed in ways meaningful to the students who are not familiar with the mathematical vocabulary. It is also important that the material encourages teachers to use probing questions to clarify what students mean or to get more information about students’ thinking.

Indicators of meeting Criterion II.3:
1. The material includes specific questions or tasks to assist the teacher in identifying the ideas students have before they study the benchmark.
2. The questions or tasks are likely to be comprehensible by students before they become familiar with the concepts, procedures, or vocabulary.
3. The material includes questions or tasks that ask students to make predictions or give explanations of concepts or procedures (vs. focus primarily on identifying students’ meaning for terms).
4. The material suggests how teachers can use questions or tasks to understand students' thinking and level of understanding.

Scoring Scheme:
High: The material meets all 4 indicators.
Medium: The material meets indicators 1-3.
Low: The material meets indicator 1.
None: The material does not meet any of the indicators.

II.4 Addressing Misconceptions. Does the material explicitly address commonly held student ideas?

Clarification:
The issue here is whether the material includes questions or activities that address students’ commonly held ideas (both concepts or skills that are incorrect and those that can serve as a foundation for subsequent learning). For example, the material may include experiences that help students change their ideas by providing activities that challenge students’ predictions or explanations, or prompt students to react to commonly held misconceptions and resolve differences between these misconceptions and the correct ideas. Alternatively, the material may include experiences that extend common student concepts or skills that have limited scope. Pointing out misconceptions and telling students that they should avoid them does not adequately address this criterion. Serious difficulties, either with concepts or with skills, are not generally successfully addressed by telling students they are wrong and providing them with the "right answer."

In addition to providing specific suggestions to teachers about how to address commonly held student ideas reported in the research literature, materials can be helpful by including suggestions to teachers about how to take into account their own students’ ideas. Addressing this aspect of the criterion may involve suggesting general strategies that teachers can use to build on or change students’ ideas, and providing examples of how these strategies can be implemented in the classroom. For example, teachers can be encouraged to probe students’ ideas further, juxtapose them with other students’ ideas, encourage students to compare their ideas on a topic before and after instruction on the topic, etc.

Indicators of meeting Criterion II.4:
1. The material explicitly addresses commonly held ideas related to the benchmark (if there is research on these ideas).
2. The material includes questions, tasks, or activities that are likely to help students progress from their initial ideas, for example, by:

(a) explicitly challenging students’ ideas, for example, by comparing their predictions to what actually happens.

(b) prompting students to contrast commonly held ideas and the correct concept or procedure, and resolve differences between them.

3. The material suggests general strategies for addressing student ideas related to the benchmark.

Scoring Scheme:
High: The material meets all 3 indicators.
Medium: The material meets indicators 1 and 2.
Low: The material meets any one indicator.
None: The material does not meet any of the indicators.

Category III: Engaging Students in Mathematics

Much of the point of mathematics is finding patterns and modeling ideas and relationships in terms of a small number of generalizations or ideas. For students to appreciate the power of mathematics, they need to have a sense of the range and complexity of ideas and applications that mathematics can explain or model. "Students need to get acquainted with the things around them—including devices, organisms, materials, shapes, and numbers—and to observe them, collect them, handle them, describe them, become puzzled by them, ask questions about them, argue about them, and then try to find answers to their questions." (Science for All Americans, p. 201)

III.1 Providing Variety of Contexts. Does the material provide experiences with mathematics in multiple, different contexts?

Clarification:
Mathematicians and others construct and use mathematical knowledge to describe, explain, predict, and design real-world objects, systems, or events as well as abstract relationships. Therefore, mathematical ideas need to be connected to meaningful problems, situations, and the real world. The question is whether the material provides a sufficient number of problems, experiences, or events in a variety of contexts to support the ideas put forth in benchmarks. The material can provide experiences with problems, situations, systems, or events directly through hands-on activities or demonstrations (firsthand experiences) or indirectly, through the use of text, graphs, diagrams, computer screens, videos, pictures, models, etc.

Indicators of meeting Criterion III.1:
1. The experiences with objects, applications, and materials are "right on target" in addressing the content of the benchmark.
2. The material provides an appropriate variety of experiences with objects, applications, and materials.

Scoring scheme:
High: The material meets indicators 1 and 2.
Medium: The material meets indicator 1.
Low: The material meets indicator 2.
None: The material does not meet any of the indicators.

III.2 Providing Firsthand Experiences. Does the material include activities that promote firsthand experiences with the benchmark ideas, when practical?

Clarification:
Students can learn more readily about things that are tangible and accessible to their senses; thus students, especially younger ones, will benefit most from firsthand experiences with the objects, problems, or events to which the mathematical knowledge in a benchmark refers. Providing students with some firsthand experiences (e.g., hands-on activities, problem solving, or measuring) is important, provided such experiences are practical. When such experiences are not practical (for example, providing firsthand experiences with measuring the height of a mountain), students can encounter objects and events indirectly, through the use of videos, pictures, models, etc.

However, it is neither necessary nor optimal that all experiences provided are firsthand. (For example, once students have had some firsthand experience with flipping coins to find probabilities, providing them with examples of other events that have finite outcomes would likely be adequate.) If all experiences provided to students were firsthand, it would limit the number of examples that could be provided (see previous criterion, Providing Variety of Contexts). Moreover, students should not be asked to reason only about ideas they see firsthand, when in real life they will also encounter problems indirectly.

Indicators of meeting Criterion III.2:
1. The activities, whether firsthand or not, provide experiences (e.g., text, pictures, video) that give students meaningful connections of the concept or skill to their knowledge.
2. An appropriate number of experiences with ideas are firsthand experiences.
(The number of firsthand experiences that are appropriate depends on the age level of the students and the difficulty of the benchmark.)
3. The firsthand experiences are efficient when compared to other firsthand experiences that could be used. (Efficiency of an experience is judged by the time and cost of the experience in relation to its value.)

Scoring Scheme:
High: The material meets all 3 indicators.
Medium: The material meets indicator 1, along with indicator 2 or 3.
Low: The material meets indicator 1.
None: The material does not meet any of the indicators.

Category IV: Developing Mathematical Ideas

Science for All Americans includes in its definition of mathematics literacy a number of important yet quite abstract ideas—e.g., symbolic representation, patterns and relationships, summarizing data. Such ideas cannot be readily discovered in the real world; the ideas themselves were developed over many hundreds of years as a result of considerable discussion and debate about the existence and logic of laws of mathematics and proofs of theorems. Mathematics literacy requires that students see the link between concepts and skills, see mathematics itself as logical and useful, and become skillful at using mathematics. This category includes criteria to determine whether the material expresses and develops ideas in ways that are accessible and intelligible to students, and to demonstrate the usefulness of the concepts and skills in varied contexts.

IV.1 Justifying Importance of Benchmark Ideas. Does the material suggest ways to help students develop a sense of the importance and validity of mathematical concepts or procedures?

Clarification:
This criterion highlights the importance of including some instances in the curriculum where an argument is developed in support of the concepts, skills, or strategies presented. There are both logical and psychological reasons for expecting a material to provide students with a sense of why ideas make sense and of why mathematicians are interested in them.

First, an understanding of the link between hypotheses and argument is itself a literacy goal. Science for All Americans includes in its definition of science literacy a basic knowledge of the nature of mathematics—both its logic and creativity—and its central role in human endeavor. Learning about the nature of mathematics can be studied in the context of learning about nearly any discipline, including students’ own discoveries.

Given the time it takes to properly develop an argument for ideas and the increased level of sophistication required for understanding both the evidence and the arguments, there is a limited number of ideas for which an evidence-based argument is required for literacy. However, it is possible that some concepts or procedures themselves are sufficiently difficult for students to understand—e.g., long division, infinite decimals—that a case needs to be made for students to find them plausible. The case might involve examining whether the concept fits well with other concepts, explains several relationships, and predicts new observations, and how it compares to other explanations of the same observations. If such a case is likely to be too difficult for most students to understand, then students should at least be informed that they are being asked to take an idea on faith. When a material does not attempt to make a case, reviewers should comment on (a) whether or not a case ought to have been made and, if so, why and (b) whether or not the material makes explicit that a case is not being built.

Indicators of meeting Criterion IV.1:
1. The material builds a case for the mathematical importance of the benchmark.
2. The material builds a case for the validity of the mathematical ideas.
3. The material builds a case for the benchmark that is likely to be comprehensible to students.
4. The material engages students in considering a case for the validity and importance of benchmark concepts or skills.

Scoring Scheme:
High: The material meets all 4 indicators.
Medium: The material meets indicator 1 or 2, along with indicator 3 or 4.
Low: The material meets indicator 1 or 2.
None: The material does not meet any of the indicators.

IV.2 Introducing Terms and Procedures. Does the material introduce terms and procedures only in conjunction with experience with them and only as needed to facilitate thinking and promote effective communication?

Clarification:
Understanding, rather than simply memorizing vocabulary or algorithms, should be the main purpose of mathematics teaching. In mathematics, many terms refer to concepts. For students to understand a concept, they should be able to describe its properties, give examples and non-examples of it, and eventually give a definition. Algorithms are important in themselves, as well as providing efficient ways to solve problems. Students should have opportunities to apply the concepts or procedures in problems and reasoning.

Indicators of meeting Criterion IV.2:
1. The material limits the use of terms and procedures.
2. The material introduces mathematical vocabulary or procedures in conjunction with experiences, rather than having students simply memorize definitions or procedures.
3. The material provides appropriate examples or meaningful applications of the terms or procedures.

IV.3 Representing Ideas Accurately. Does the material include accurate and comprehensible representations of mathematical concepts, procedures, and relationships?

Clarification:
This question highlights the importance of using accurate representations to make (abstract) ideas intelligible to all students. Different representations highlight different aspects of an idea and provide a variety of opportunities for the idea to connect to other students’ ideas and become embedded in a student’s knowledge system. Possible modes of representation include drawings, diagrams, graphs, images, analogies and metaphors, models and simulations, and role-playing. Representations need to be clear so that students can understand fairly quickly what ideas are being represented and how. In addition, because representations typically highlight only some aspects of an idea, care must be taken that they represent the real thing as accurately as possible (or they involve students in considering which aspects of the real thing are represented by the model and which are not).

Indicators of meeting Criterion IV.3:
1. The material includes accurate representations.
2. The material includes comprehensible representations (depending on the students’ grade level and the difficulty of the benchmark).
3. The material includes an appropriate number and variety of accurate and comprehensible representations.

IV.4 Connecting Benchmark Ideas. Does the material explicitly draw attention to appropriate connections among benchmark ideas?

Clarification:
This criterion emphasizes connections among benchmark ideas, concepts, and skills (rather than connections between activities, which are examined in Category I). Some kinds of connections can be classified as belonging to one of the following general types:

(a) One concept or skill may be an instance of a generalization (e.g., the sum of the angles of a rectangle illustrates the general sum of the angles of a polygon).

(b) One concept or skill might be analogous to another idea (e.g., adding rational fractions is like adding rational expressions in that they both require a common denominator).

(c) A concept or skill may show up in several fields or contexts (e.g., the number pi can be the ratio of the circumference to diameter, the radian measure of a semicircle, or the sum of a series).

Other connections are more unique to particular ideas (e.g., linking slope of a line to the tangent of a curve).

Responding to this criterion involves looking to see whether any general or unique connections are essential, which requires identifying what such connections might be. Growth of Understanding maps provide a rich source of potential connections. A set of maps on about 50 topics will be available in Atlas of Science Literacy. When a map is not available for a topic, the "Also-See" box in Benchmarks for Science Literacy may be helpful in identifying conceptual connections among ideas. Reading Science for All Americans may also be helpful in identifying conceptual connections, since several important connections among ideas are made in the prose.

Indicators of meeting Criterion IV.4:
1. The material notes connections among specific benchmark ideas (rather than just among general topics).
2. The material adequately explains or develops the identified connections.
3. The material engages students in making and/or explaining the identified connections.

Scoring Scheme:
High: The material meets all 3 indicators.
Medium: The material meets indicators 1 and 2.
Low: The material meets 1 of the 3 indicators.
None: The material does not meet any of the indicators.

IV.5 Demonstrating/Modeling Procedures. Does the material demonstrate/model (or include suggestions for teachers on how to demonstrate/model) skills or the use of knowledge?

Clarification:
Among the ways literate adults use their knowledge and skills are to describe and explain phenomena, to solve practical problems, and to consider alternative positions on issues. Hence students should learn to use their knowledge and skills in these ways. In order for students to know the type and level of performance expected for a skill or an application of conceptual knowledge, they need to see examples. This is particularly important for complex behaviors such as explaining how to solve problems, developing a generalization, argument, or proof, or carrying out a complex procedure or algorithm. Demonstrating or Modeling a skill involves (a) an expert's demonstrating or modeling the skill, (b) providing running commentary about important aspects to note about the performance or demonstration, and (c) providing criteria for judging a good performance. Demonstrating or Modeling how knowledge might be used, for example, to solve problems or construct a proof, is similar.

Responding to this criterion involves examining whether (a) demonstrating/modeling is carried out by the text or other accompanying materials (e.g., software, video), or (b) the material includes suggestions to teachers about how to go about demonstrating/modeling skills or use of knowledge in their classrooms.

Indicators of meeting Criterion IV.5:
1. The material demonstrates (or instructs teachers how to demonstrate) the expected procedure or performance.
2. The demonstration is clear and comprehensible.
3. The material provides commentary that points to particular aspects of the demonstration and/or provides justifications or explanations for the steps taken.

IV.6 Providing Practice. Does the material provide tasks or questions for students to practice skills or use knowledge in a variety of situations?

Clarification:
An important part of learning mathematics consists of giving students many opportunities to use their skill or knowledge, in particular giving them opportunities to practice using mathematical knowledge and skills in describing objects, relationships, and events, solving problems, and applying knowledge in new situations or contexts. Moreover, literacy means that people will be able to draw upon and use their understanding of mathematics when they encounter events that do not come with labels such as "algebra," "geometry, " or "statistics" but in political arguments, discussions of literature, and walks on the beach. Therefore, students will need practice in making connections to new situations. Providing students with opportunities to practice only calculating answers to predictable exercises does not adequately address this criterion.

Indicators of meeting Criterion IV.6:
1. The material includes appropriate practice exercises and tasks on the benchmark.
2. The material provides an appropriate number of practice exercises and tasks.
3. The material includes a variety of contexts, including everyday tasks and contexts and novel as well as familiar practice tasks.

Scoring Scheme:
High: The material meets all 3 indicators.
Medium: The material meets indicator 1, along with either 2 or 3.
Low: The material meets indicator 1.
None: The material does not meet any of the indicators.

Category V: Promoting Student Thinking about Mathematics

No matter how clearly materials may present ideas, students (like all people) will devise their own meaning, which may or may not correspond to targeted learning goals. Students need to make their ideas and reasoning explicit, hold them up to scrutiny, and recast them as needed. This category includes criteria for whether the material suggests how to help students express, think about, and reshape their ideas to make better sense of the world.

V.1 Encouraging Students to Explain Their Reasoning. Does the material routinely include suggestions for having each student express, clarify, justify, and represent his/her ideas and how to get feedback from peers and the teacher?

Clarification:
It is important to provide opportunities for students’ thinking to become overt to themselves, the teacher, and other students. By stating (clarifying, justifying, and representing) their ideas, in writing, drawing, or speaking, students become more aware of what they think. This may stimulate making explicit connections between their ideas and the ideas presented by the text or the teacher, and/or questioning of their ideas (if relevant). Exchange of ideas in small groups or a large group discussion may make students aware of the range of ideas that exist and may provoke students to reconsider their own ideas in light of others. Feedback from the teacher or other peers is necessary to help students understand their mistakes and improve the quality of their descriptions, explanations, or designs.

Responding to the first part of the criterion involves examining whether the material prompts (or encourages teachers to prompt) students to express their ideas either orally or in writing. It also involves examining whether the material has opportunities for each student to express his or her ideas. Responding to the second part of the criterion includes examining whether the material includes specific suggestions to help the teacher provide explicit feedback, includes text that directly provides students with feedback on the adequacy of their ideas, or provides teachers with strategies they can use to ensure that each student in the class receives feedback.

Indicators of meeting Criterion.V.1:
1. The material encourages students to express their ideas about the benchmark.
2. The material encourages students not only to express but also to clarify, justify, interpret, or represent benchmark ideas.
3. The material provides (or includes suggestions to help the teacher to provide) explicit feedback to students about their ideas.
4. The material includes suggestions to the students or teacher on how to use student responses to diagnose errors or difficulties, address errors or difficulties, or further develop students’ ideas about the benchmark.

Scoring Scheme:
High: The material meets all 4 indicators.
Medium: The material meets indicator 1, along with 2 of the other 3 indicators.
Low: The material meets at least 1 of the indicators.
None: The material does not meet any of the indicators.

V.2 Guiding Interpretation and Reasoning. Does the material include tasks and/or question sequences that guide student interpretation and reasoning about benchmark concepts, skills, and relationships?

Clarification:
Experiences with hands-on materials, problems, and examples of mathematical ideas are useful but not sufficient. Students need time, opportunities, and guidance to make sense of these experiences. If the students are turned loose to do exercises or problems on their own, very little happens except for a small number of students. The activities need to be guided with sequences of questions that lead students to make relevant generalizations and understand relationships. Similarly, students need time, opportunities, and guidance to make sense of things they read and ideas they are introduced to.

Responding to this criterion involves examining whether the material includes (in the teacher’s guide or student books) specific, carefully chosen and sequenced tasks or questions that are likely to support students’ thinking about exercises, problems, and investigations. Good tasks and questions frame important issues, help students relate their previous experiences to the mathematical ideas, anticipate common student difficulties or misconceptions, and focus on important generalizations and procedures.

Indicators of meeting Criterion V.2:
1. The material includes specific and relevant tasks and/or questions for activities related to the benchmark.
2. The material includes connected sequences (rather than only collections) of questions or tasks.
3. The questions or tasks guide student interpretation and reasoning through approaches such as:

(a) framing, introducing, or developing important ideas,
(b) helping students to relate their own experiences to mathematical ideas, or
(c) anticipating or eliciting common difficulties or student misconceptions.

Scoring Scheme:
High: The material meets all 3 indicators.
Medium: The material meets 2 of the 3 indicators.
Low: The material meets 1 of the 3 indicators.
None: The material does not meet any of the indicators.

V.3 Encouraging Students to Think about What They’ve Learned. Does the material suggest ways to have students check their own progress?

Clarification:
This criterion highlights the importance of having students look back at the progress of their thinking and learning. Monitoring one’s understanding and realizing which ideas one does not understand can shift some of the responsibility for learning to the students and may elicit their attempts to understand as a result.

Responding to this criterion involves examining whether the material includes questions or tasks that prompt students to monitor their understanding, or includes suggestions to teachers on how to encourage students to do so. For example, What was confusing to you today? How does the new knowledge compare with what you used to think? What do you think you understand and where do you need to work more? Encouraging students to monitor their understanding should also include (when appropriate) questions on how and why students changed their ideas. For example, Did you change any of your ideas today? What evidence convinced you to do so?

Indicators of meeting Criterion V.3:
1. The material engages (or provides specific suggestions for teachers to engage) students in monitoring their progress toward the benchmark.
2. The material asks students to think about how their ideas have developed or changed.
3. The material gives students an opportunity to revise their initial ideas about the benchmark based on what they have learned.

Category VI: Assessing Student Progress in Mathematics

Assessment provides information to students about what is important and to teachers about what has been learned. Just as important, assessment provides information to both the students and the teacher about adjustments that should be made in learning and instruction. Because assessment is so important to the teaching-learning process, it must match the mathematics learning goal of the curriculum materials. Further, assessments must address the range of skills, applications, and contexts that reflect what students are expected to learn. All of this is possible only if assessment takes place throughout instruction, not only at the end of a chapter or unit.

VI.1 Aligning Assessment. Are assessment items or tasks included that match the ideas, concepts, or skills of the benchmark?

Clarification:
This criterion highlights the necessity of including assessment items for each benchmark that is important in the material. To judge whether the items provided match the standard, the same procedure should be followed as in examining the content match between an activity and a benchmark. That is, examine whether the assessment item addresses the substance rather than only the topic of the benchmark, the level of sophistication of the benchmark, and what part of the benchmark is assessed.

Indicators of meeting Criterion VI.1:

1. The material provides at least one assessment task that addresses the specific ideas of the benchmark (assessment item should not be answerable by reading comprehension, general intelligence, or test-wiseness alone).
2. The assessment items that do address the benchmark require no other, more sophisticated, ideas.
3. If the material provides a test that is given to the students, an appropriate number of assessment items are content-matched to the benchmark. If the material provides a bank of assessment items that teachers select from, an appropriate proportion of assessment items are content-matched to the benchmark.

VI.2 Assessment through Applications. Does the material include assessment tasks that require application of benchmark ideas, concepts, or skills and avoid allowing students a trivial way out, like using a formula or repeating a memorized term or rule without understanding?

Clarification:
Rather than checking whether students have memorized certain bits of information, assessment needs to test students’ mathematical understanding, reasoning, and the application of knowledge. In addition, it needs to include tasks that engage students in activities similar to those they will engage in in their lives outside the classroom. Literate persons use mathematical knowledge to describe, explain, and predict real world phenomena, solve a practical problem, or discuss issues. Accordingly, assessment tasks need to engage students in descriptions, explanations, predictions, design, and discussion of issues. This, however, does not necessarily dictate the format that the assessment should include. For example, assessments of students’ use of knowledge to explain a concept could include a "multiple choice" or "constructed response" format.

Indicators of meeting Criterion VI.2:
1. The material provides assessment tasks that focus on application of benchmark ideas.
2. The material includes assessment tasks that are familiar as well as tasks that are novel or non-routine.
3. If the material provides a test that is given to the students, an appropriate number of assessment items focus on application. If the material provides a bank of assessment items that teachers select from, an appropriate proportion of items or tasks focus on application.

VI.3 Using Embedded Assessment. Are some assessments embedded in the curriculum along the way, with advice to teachers as to how they might use the results to choose or modify activities?

Clarification:
This criterion highlights the need for assessment to be in the service of instruction to guide teaching and learning. The criterion requires that materials include assessments that can be used as diagnostic or formative instruments, which help determine learners’ needs, rather than largely as instruments for grading students at the end of a unit or chapter.

Responding to this question involves examining whether the material (a) provides assessment tasks only at the end of a unit of study to help grade student achievement, or also along the way to help monitor student progress, (b) encourages and provides guidance to teachers about how to probe beyond students’ first response to clarify and further understand student answers, and (c) encourages teachers to use the information from the assessments to make instructional decisions about what ideas need to be addressed by further activities with the whole group or smaller groups of students.

Indicators of meeting Criterion VI.3:
1. The material uses embedded assessment as a part of the instructional strategy or design.
2. The material includes assessments that provide opportunities, encouragement, or guidance for students on how to further understand benchmark ideas.
3. The material includes suggestions or guidance for teachers on how to probe students’ understanding of benchmark ideas.
4. The material provides specific suggestions to teachers about how to use the information from the embedded assessments to make instructional decisions about what ideas need to be addressed by further activities.

Scoring Scheme:
High: The material meets all 4 indicators.
Medium: The material meets indicator 1, along with 2 of the other 3 indicators.
Low: The material meets indicator 1.
None: The material does not meet any of the indicators.

Category VII. Enhancing the Mathematics Learning Environment

Several other important considerations are involved in the selection of curriculum materials—for example, the help they provide teachers in encouraging student curiosity and creating a classroom community where all can succeed, or attractiveness. These can influence student learning or even whether the materials are used appropriately by the teacher and students. The criteria listed in this category provide reviewers with the opportunity to comment on these and other important features.

VII.1 Providing Teacher Content Support. Does the material help teachers improve their understanding of mathematics and its applications?

Clarification:
The issue here is whether the material includes a "content background" section or other features that help teachers develop their understanding of the mathematical knowledge addressed in the material. Responding to this question involves commenting on the quality of the support, not merely on whether such support is included. Just providing teachers with a list of resources that may enhance their understanding of mathematics does not adequately address this question. A minimum requirement is that such lists are annotated to describe the resources and specify what can be learned from them.

Indicators of meeting Criterion VII.1:
1. The material provides content information or recommends resources for improving specific skills or understanding of particular ideas.
2. The material provides content information that is in the form that is useful and appropriate for teachers, no matter what their background knowledge.
3. The material indicates how the ideas or skills are relevant and important to teaching the material to students.

VII.2 Establishing a Challenging Classroom. Does the material help teachers to create a classroom environment that welcomes student curiosity, rewards creativity, encourages a spirit of healthy questioning, and avoids rigidity?

Clarification:
Responding to this criterion involves examining whether teachers are given guidance to (a) encourage students to raise questions about the material being studied and suggest productive ways for finding answers, (b) use activities in which students’ creativity and imagination will pay off, (c) respect and value students’ ideas, and (d) avoid conveying the impression that they themselves or the textbooks are absolute authorities whose conclusions are always correct. In addition, the criterion involves examining whether the materials give a vision of what the curriculum might look like in action (i.e., teacher hints and suggestions, dialogue boxes, vignettes, or video clips that show desirable student teacher interactions).

Indicators of meeting Criterion VII.2:
1. The material provides opportunities for students to express their curiosity or creativity.
2. The material provides occasions for students to take risks and ask questions.
3. The material suggests how to encourage students to weigh and challenge their own and others’ ideas.
4. The material avoids sending a message that mathematics consists only of rules and single correct answers.

Scoring Scheme:
High: The material meets all 4 indicators.
Medium: The material meets 2 of the 4 indicators.
Low: The material meets 1 of the 4 indicators.
None: The material does not meet any of the indicators.

VII.3 Supporting All Students. Does the material help teachers to create a classroom that encourages high expectations for all students, enables all students to experience success, and provides all students a feeling of belonging in the mathematics classroom?

Clarification:
Several pedagogical criteria presented in previous categories highlight the need for materials to incorporate principles of teaching and learning that are likely to promote mathematics understanding for all students. This question highlights the importance of reviewing curriculum materials for features that might distract or impede the progress of females, minorities, students whose first language is not English, students with disabilities, or others from the intended work. Further, the criterion requires that materials provide specific suggestions and resources for encouraging all students to be able to learn mathematics and express their competence and performances during instruction and in assessment tasks.

Indicators of meeting Criterion VII.3:
1. The material avoids stereotypes or language that might be offensive to a particular group.
2. The material illustrates the contribution or participation of women, minorities, and persons with disabilities to mathematics-related fields.
3. The material suggests alternative formats for students to develop or express their mathematics knowledge during instruction and assessment.
4. The material includes specific suggestions on how teachers can modify activities for students with special needs, interests, or abilities.