## Algebra Textbooks Evaluation

### The Project 2061 Curriculum-Analysis Procedure in Detail

To provide a better sense of how the procedure works, the following describes in more detail each step in the procedure. The description pays particular attention to the various criteria used to evaluate the instructional effectiveness of materials.

**Identify specific learning goals to serve as the intellectual basis for
the analysis.** After reviewers have agreed upon a set of learning goals
as a framework for the analysis, the task is then to choose specific learning
goals that will serve as the focus of further study.

When evaluating stand-alone curriculum units that cover a relatively short period of time, it might be possible and worthwhile to analyze all of the learning goals that appear to be targeted by the material. However, in the evaluation of yearlong courses or multi-year programs, this becomes impractical. Therefore, a crucial step in the analysis procedure is the sampling of a few learning goals that will lead to valid and reliable generalizations about the material.

Sampling of standards should be representative of the whole set of goals specified
in the framework or standards being applied and should reflect the reviewers'
needs. For example, if the review committee's task is to select a course in
high school Algebra that is aligned with a state mathematics framework or
NCTM *Standards,* it might identify a sample of learning goals from important
topic areas (e.g., number systems, equations, graphs, functions) and include
learning goals that reflect different types of knowledge (e.g., skills, conceptual
understanding, problem solving). When examining elementary or middle school
mathematics materials, one would probably want to broaden the range of learning
goal statements examined to include important strands in mathematics (e.g.,
number, geometry, algebra, and statistics).

**Make a preliminary inspection of the curriculum materials to see whether
they are likely to address the targeted learning goals.** Once learning
goal statements have been selected, the next step is to make a first pass
at the materials to identify those whose content appears to correspond reasonably
well to the learning goals. Materials that do not meet these initial criteria
are not analyzed further.

Reviewers then examine materials on the shortened list more carefully to locate and record places where each selected learning goal seems to be targeted (e.g., particular readings, experiments, discussion questions). If several sightings are found for some or all of the sample learning goals in the material, then these sightings will be looked at more carefully in subsequent steps of the analysis. If, on the other hand, sightings cannot be found for a significant number of the sample learning goals, then the material is dropped from the list.

**Analyze the curriculum materials for alignment between content and the selected
learning goals.** This analysis is a more rigorous examination of the link
between the subject material and the selected learning goals and involves
giving precise attention to both ends of the match; the precise meaning of
the learning goal on one end, and the precise intention of the material on
the other.

With respect to each of the sampled learning goals, the material is examined using such questions as:

- Does the content called for in the material address the substance of a specific learning goal or only the learning goal's general "topic"?

- Does the content reflect the level of sophistication of the specific learning goal, or are the activities more appropriate for targeting learning goals at an earlier or later grade level?

- Does the content address all parts of a specific learning goal or only some? (While it is not necessary that any particular unit would address all of the ideas in a learning goal or standard, the K-12 curriculum as a whole should do so. The purpose of this question is to provide an account of precisely what ideas are treated.)

In addition, an attempt is made to estimate the degree of overlap between the material's content and the set of learning goals of interest. Thus, this step in the analysis is designed to answer questions regarding the material's inclusion of content that is not required for reaching mathematics literacy and the extent to which the material distinguishes between essential and non-essential content. (While distinguishing content essential for literacy from non-essential content in material might seem to be a luxury, it assists teachers in determining the range of students for which the material can be used. Identifying the non-essential material makes it easier for the teacher to direct better students to enrichment activities and allows students themselves to avoid overload from ideas that go beyond what is vital.)

**Analyze the curriculum materials for alignment between instruction and the
selected learning goals.** The purpose here is to estimate how well material
addresses targeted learning goals from the perspective of what is known about
student learning and effective teaching. The criteria for making the judgments
in the instructional analysis are derived from research on learning and teaching
and on the craft knowledge of experienced educators. In the context of mathematics
literacy, these are summarized in Chapter 13, "Effective Learning and
Teaching," of *Science for All Americans* and in Chapter 15, "The
Research Base," of *Benchmarks for Science Literacy*. From these
and other sources, seven criteria categories (shown below) have been identified
to serve as a basis for the instructional analysis.

**Category I: Identifying a Sense of Purpose**

Part of planning a coherent curriculum involves deciding on its purposes and on what learning experiences will likely contribute to achieving those purposes. But while coherence from the designers’ point of view is important, it may be inadequate to give students the same sense of what they are doing and why. This category includes criteria to determine whether the material attempts to make its purposes clear and meaningful to the student and genuinely relates lessons to the unit purpose.

I.1 Conveying Unit Purpose: Does the material convey
an overall sense of purpose and direction that is understandable
and motivating to students? |

**Clarification:
**This criterion involves examining whether the material begins with (or
early on presents) an over-arching question or problem to be addressed by
the unit (e.g., How can a graph help to make predictions?), or a representation
of what will be learned (e.g., a concept map of the main ideas that will be
explored), or otherwise identifies a purpose for the unit or chapter for the
students (e.g., a clear statement of objectives, using know terms). The problem,
question, representation or purpose provided by the material should be explicit
and comprehensible by the students, and it should be plausible that it would
be interesting and/or motivating to them. A material that begins with abstractions
or phenomena outside students’ range of perception, understanding, or
knowledge does not adequately meet the criterion. However, a material that
starts with an unfamiliar but highly interesting phenomenon that is likely
to motivate students may meet the criterion.

Providing students with a sense of purpose for a whole unit or chapter is not always possible (for example, there may not be a single question or problem that is broad enough to foreshadow all learning goals in the unit) or even desirable (for example, providing a purpose on a large scale can lead to a complex sequence of activities that is too demanding on the memory of younger students). In such cases, it may be sufficient for the material to frame sections within a unit rather than the whole unit or chapter.

**Indicators of meeting Criterion I.1:
**1. The purpose is presented to students explicitly (or implicitly through
a problem, question, or representation).

2. The purpose is likely to be comprehensible to students.

3. The purpose is likely to be interesting and/or motivating to students.

4. Students are given an opportunity to think about and discuss the purpose.

5. Most activities or lessons are consistent with the stated purpose.

6. The material returns to the stated purpose at the end of the unit or chapter.

**Scoring Scheme:
**High: The material meets all 6 indicators.

Medium: The material meets indicator 1, along with 3 of the other 5 indicators.

Low: The material meets indicator 1, along with 1 of the other 5 indicators.

None: The material does not meet any of the indicators.

I.2 Conveying Lesson Purpose: Does the material convey
the purpose of each activity or lesson and its relationship to
others? |

**Clarification:
**The question is whether the purpose of individual activities or lessons
(as opposed to the whole unit) is made apparent to the students and whether
there are logical transitions and connections between activities or lessons.
If a classroom visitor asked students what they were doing and why, is there
reason to think they would know?

The purpose of individual activities or lessons could be brought out through the text, teacher comments (suggested in the material), and/or student responses to questions. For example, the purpose of gathering data about the heights of students in the class (to learn about dispersion of data) might be brought out by text explanation, teacher explanation, or by the students coming up with a description of the spread of data.

**Indicators of meeting Criterion I.2:
**1. The material conveys or prompts teachers to convey the purpose of each
activity or lesson to students.

2. Each activity encourages each student to think about the purpose of the activity or lesson.

3. The material conveys or prompts teachers to convey to students how each activity or lesson relates to the other activities.

4. The material periodically engages students in thinking about what they have learned so far and what they need to learn/do next.

**Scoring Scheme:
**High: The material meets all 4 indicators.

Medium: The material meets 3 of the 4 indicators.

Low: The material meets 1 of the 4 indicators.

None: The material does not meet any of the indicators.

I.3 Justifying Sequence of Activities: Does the material
involve students in a logical or strategic sequence of activities
(versus a collection of activities) that build toward understanding
of the ideas in the unit or chapter purpose? |

**Clarification:
**The issue here is whether there is a logical or strategic sequence of
activities in the material and whether this logic or strategy is made explicit
to the teacher or just inferred by the reviewer. A rationale or implicit reason
for the sequence of activities should be clear, providing the teacher and
students with a sense of making progress toward the purpose of the unit or
chapter.

**Indicators of meeting Criterion I.3:
**1. The material provides a rationale for the overall sequence of activities
or lessons.

2. If no rationale for the overall sequence of activities or lessons is provided, the reviewer can identify one.

3. The sequence of activities reflects the stated or inferred rationale or purpose.

**Scoring Scheme:
**High: The material meets indicators 1 and 3.

Medium: The material meets indicators 2 and 3.

Low: The material meets indicator 1 or 2.

None: The material does not meet any of the indicators.

**Category II: Building on Student Ideas about Mathematics**

Fostering better understanding in students requires taking time to attend to the ideas they already have, both ideas that are incorrect and ideas that can serve as a foundation for subsequent learning. Such attention requires that teachers be informed about prerequisite ideas/skills needed for understanding a benchmark and what their students’ initial ideas are—in particular, the ideas that may interfere with learning mathematics. Moreover, teachers can help address students’ ideas if they know what is likely to work. This category examines whether the material contains specific suggestions for identifying and addressing student ideas.

II.1 Specifying Prerequisite Knowledge. Does the material
specify and address prerequisite knowledge/skills that are necessary
to the learning of the benchmark? |

**Clarification:
**This criterion refers to (a) prerequisites to concepts or skills in the
benchmark examined, and (b) prerequisites to activities used in the material
to teach the concepts or skills in the benchmark examined.

(a) Understanding the ideas in benchmarks often requires that students first understand some other "prerequisite" concepts or skills. For example, knowing what a prime factor is is prerequisite to learning how to find the common denominator of fractions.

(b) In addition to prerequisites to specific ideas in benchmarks, additional prerequisites may arise from the specific activities used to teach them. For example, consider the benchmark: "Spreading data on a number line helps to see what the extremes are, where they pile up, and where the gaps are. A summary of the data includes where the middle is and how much spread is around it." There are no benchmarks that are prerequisite to the ideas in this benchmark. However, as the students work through a curriculum material, they may be engaged in investigations that target this benchmark and involve measurement. In these cases, the material should not take for granted that students will have developed measuring skills. Teachers should be alerted to this prerequisite (measurement skills) and encouraged to support their students in developing measurement skills.

Responding to the "prerequisites" criterion involves (a) making a list of prerequisite concepts and/or skills, (b) examining whether the material alerts to any prerequisite ideas on the list and if so which ones, (c) examining whether the material has in fact adequately addressed the prerequisites in the same or earlier units, and (d) examining whether the material helps students make connections between benchmarks and their prerequisites. While a stand-alone unit should not be faulted for not addressing prerequisite ideas or skills, it should be expected to make connections between benchmarks and their prerequisites.

**Indicators of meeting Criterion II.1:
**1. The material makes explicit what the specific prerequisite ideas or
skills are, if any exist.

2. The material addresses the identified prerequisites in the same unit or in earlier units.

3. The material makes connections between benchmark ideas and their prerequisites (even if the prerequisites are addressed elsewhere).

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets 2 out of the 3 indicators.

Low: The material meets 1 of the 3 indicators.

None: The material does not meet any of the indicators.

II.2 Alerting Teacher to Student Ideas. Does the material
alert teachers to commonly held student ideas (both troublesome
and helpful) such as those described in Benchmarks for Science
Literacy Chapter 15: The Research Base? |

**Clarification:
**Researchers have identified ideas that students have in several content
areas. The issue here is whether the material informs teachers about students’
commonly held ideas in the topic areas the material addresses. This information
can help teachers (a) understand better their own students’ ideas, (b)
decide what ideas to build on and what changes to promote, or (c) if the material
is already designed in ways that build on or attempt to change students’
commonly held ideas, to better understand the rationale and purpose behind
designed strategies and activities.

Responding to this question involves examining (1) whether there is research
on commonly held student ideas in the topic area/s that the material addresses,
(2) whether the material alerts teachers to such ideas, and (3) whether the
material accurately represents research findings. Summaries of research on
students ideas in mathematics (such as those included in *Benchmarks*
Chapter 15: The Research Base or the NCTM *Research Ideas for the Classroom*
series) will be helpful to reviewers who will want to know what ideas students
typically have about the topics that the curriculum material they are examining
addresses. If there is no research on student ideas in the topic area/s that
the material addresses, the material should not be faulted for not addressing
this criterion.

**Indicators of meeting Criterion II.2:
**1. The material lists, conveys, or identifies specific commonly held ideas
that are relevant to the benchmark (rather than just to relevant difficult
topics).

2. The material clarifies/explains commonly held ideas.

3. The material explains or refers to commonly held ideas in an accurate way.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets indicators 1 and 2.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.

II.3 Assisting Teacher in Identifying Ideas. Does the
material include suggestions for teachers to find out what their
students think about familiar situations related to a benchmark
before the mathematical ideas are introduced? |

**Clarification:
**Teachers need guidance in identifying students’ ideas in these unresearched
areas. But even in areas in which there is research on commonly held student
ideas, teachers need help in identifying what proportion of their own students
hold these ideas as well as other more idiosyncratic ideas.

Responding to this criterion involves examining not only whether the material encourages teachers to find out students’ ideas but also whether it provides specific suggestions for how to do so. Suggestions may include providing (a) tasks in which students make predictions and give their own descriptions and explanations of concepts or skills; (b) tasks in which students are asked to represent their understandings in drawings; (c) tasks in which students are asked to interpret information (for example the solution to an algorithm related to ideas in the benchmark), discuss connections with related topics, or discuss alternative solutions, or justifications; or (d) tasks which ask students about the meaning of specific terms and/or probe for understanding of important relationships between concepts.

Responding to the criterion also involves examining the quality of the suggestions provided. Tasks should not focus exclusively on identifying students’ meaning for terms. While including such tasks is useful, it is important to look for tasks in which students make predictions and/or give explanations of concepts or procedures. It is important that tasks make sense to students who have never studied the topic and include questions posed in ways meaningful to the students who are not familiar with the mathematical vocabulary. It is also important that the material encourages teachers to use probing questions to clarify what students mean or to get more information about students’ thinking.

**Indicators of meeting Criterion II.3:
**1. The material includes specific questions or tasks to assist the teacher
in identifying the ideas students have before they study the benchmark.

2. The questions or tasks are likely to be comprehensible by students before they become familiar with the concepts, procedures, or vocabulary.

3. The material includes questions or tasks that ask students to make predictions or give explanations of concepts or procedures (vs. focus primarily on identifying students’ meaning for terms).

4. The material suggests how teachers can use questions or tasks to understand students' thinking and level of understanding.

**Scoring Scheme:
**High: The material meets all 4 indicators.

Medium: The material meets indicators 1-3.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.

II.4 Addressing Misconceptions. Does the material explicitly
address commonly held student ideas? |

**Clarification:
**The issue here is whether the material includes questions or activities
that address students’ commonly held ideas (both concepts or skills that
are incorrect and those that can serve as a foundation for subsequent learning).
For example, the material may include experiences that help students change
their ideas by providing activities that challenge students’ predictions
or explanations, or prompt students to react to commonly held misconceptions
and resolve differences between these misconceptions and the correct ideas.
Alternatively, the material may include experiences that extend common student
concepts or skills that have limited scope. Pointing out misconceptions and
telling students that they should avoid them does not adequately address this
criterion. Serious difficulties, either with concepts or with skills, are
not generally successfully addressed by telling students they are wrong and
providing them with the "right answer."

In addition to providing specific suggestions to teachers about how to address commonly held student ideas reported in the research literature, materials can be helpful by including suggestions to teachers about how to take into account their own students’ ideas. Addressing this aspect of the criterion may involve suggesting general strategies that teachers can use to build on or change students’ ideas, and providing examples of how these strategies can be implemented in the classroom. For example, teachers can be encouraged to probe students’ ideas further, juxtapose them with other students’ ideas, encourage students to compare their ideas on a topic before and after instruction on the topic, etc.

**Indicators of meeting Criterion II.4:
**1. The material explicitly addresses commonly held ideas related to the
benchmark (if there is research on these ideas).

2. The material includes questions, tasks, or activities that are likely to help students progress from their initial ideas, for example, by:

(a) explicitly challenging students’ ideas, for example, by comparing their predictions to what actually happens.

(b) prompting students to contrast commonly held ideas and the correct concept or procedure, and resolve differences between them.

(c) extending correct commonly held ideas that have limited scope.

3. The material suggests general strategies for addressing student ideas related to the benchmark.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets indicators 1 and 2.

Low: The material meets any one indicator.

None: The material does not meet any of the indicators.

**Category III: Engaging Students in Mathematics**

Much of the point of mathematics is finding patterns and modeling ideas and
relationships in terms of a small number of generalizations or ideas. For
students to appreciate the power of mathematics, they need to have a sense
of the range and complexity of ideas and applications that mathematics can
explain or model. "Students need to get acquainted with the things around
them—including devices, organisms, materials, shapes, and numbers—and
to observe them, collect them, handle them, describe them, become puzzled
by them, ask questions about them, argue about them, and then try to find
answers to their questions." (*Science for All Americans*, p. 201)

III.1 Providing Variety of Contexts. Does the material
provide experiences with mathematics in multiple, different contexts? |

**Clarification:
**Mathematicians and others construct and use mathematical knowledge to
describe, explain, predict, and design real-world objects, systems, or events
as well as abstract relationships. Therefore, mathematical ideas need to be
connected to meaningful problems, situations, and the real world. The question
is whether the material provides a sufficient number of problems, experiences,
or events in a variety of contexts to support the ideas put forth in benchmarks.
The material can provide experiences with problems, situations, systems, or
events directly through hands-on activities or demonstrations (firsthand experiences)
or indirectly, through the use of text, graphs, diagrams, computer screens,
videos, pictures, models, etc.

**Indicators of meeting Criterion III.1:
**1. The experiences with objects, applications, and materials are "right
on target" in addressing the content of the benchmark.

2. The material provides an appropriate variety of experiences with objects, applications, and materials.

**Scoring scheme:
**High: The material meets indicators 1 and 2.

Medium: The material meets indicator 1.

Low: The material meets indicator 2.

None: The material does not meet any of the indicators.

III.2 Providing Firsthand Experiences. Does the
material include activities that promote firsthand experiences
with the benchmark ideas, when practical? |

**Clarification:
**Students can learn more readily about things that are tangible and accessible
to their senses; thus students, especially younger ones, will benefit most
from firsthand experiences with the objects, problems, or events to which
the mathematical knowledge in a benchmark refers. Providing students with
some firsthand experiences (e.g., hands-on activities, problem solving, or
measuring) is important, provided such experiences are practical. When such
experiences are not practical (for example, providing firsthand experiences
with measuring the height of a mountain), students can encounter objects and
events indirectly, through the use of videos, pictures, models, etc.

However, it is neither necessary nor optimal that all experiences provided are firsthand. (For example, once students have had some firsthand experience with flipping coins to find probabilities, providing them with examples of other events that have finite outcomes would likely be adequate.) If all experiences provided to students were firsthand, it would limit the number of examples that could be provided (see previous criterion, Providing Variety of Contexts). Moreover, students should not be asked to reason only about ideas they see firsthand, when in real life they will also encounter problems indirectly.

**Indicators of meeting Criterion III.2:
**1. The activities, whether firsthand or not, provide experiences (e.g.,
text, pictures, video) that give students meaningful connections of the concept
or skill to their knowledge.

2. An appropriate number of experiences with ideas are firsthand experiences.

(The number of firsthand experiences that are appropriate depends on the age level of the students and the difficulty of the benchmark.)

3. The firsthand experiences are efficient when compared to other firsthand experiences that could be used. (Efficiency of an experience is judged by the time and cost of the experience in relation to its value.)

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets indicator 1, along with indicator 2 or 3.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.

**Category IV: Developing Mathematical Ideas**

*Science for All Americans* includes in its definition of mathematics
literacy a number of important yet quite abstract ideas—e.g., symbolic
representation, patterns and relationships, summarizing data. Such ideas cannot
be readily discovered in the real world; the ideas themselves were developed
over many hundreds of years as a result of considerable discussion and debate
about the existence and logic of laws of mathematics and proofs of theorems.
Mathematics literacy requires that students see the link between concepts
and skills, see mathematics itself as logical and useful, and become skillful
at using mathematics. This category includes criteria to determine whether
the material expresses and develops ideas in ways that are accessible and
intelligible to students, and to demonstrate the usefulness of the concepts
and skills in varied contexts.

IV.1 Justifying Importance of Benchmark Ideas. Does the
material suggest ways to help students develop a sense of the
importance and validity of mathematical concepts or procedures? |

**Clarification:****
**This criterion highlights the importance of including some instances in
the curriculum where an argument is developed in support of the concepts,
skills, or strategies presented. There are both logical and psychological
reasons for expecting a material to provide students with a sense of why ideas
make sense and of why mathematicians are interested in them.

First, an understanding of the link between hypotheses and argument is itself
a literacy goal. *Science for All Americans* includes in its definition
of science literacy a basic knowledge of the nature of mathematics—both
its logic and creativity—and its central role in human endeavor. Learning
about the nature of mathematics can be studied in the context of learning
about nearly any discipline, including students’ own discoveries.

Given the time it takes to properly develop an argument for ideas and the increased level of sophistication required for understanding both the evidence and the arguments, there is a limited number of ideas for which an evidence-based argument is required for literacy. However, it is possible that some concepts or procedures themselves are sufficiently difficult for students to understand—e.g., long division, infinite decimals—that a case needs to be made for students to find them plausible. The case might involve examining whether the concept fits well with other concepts, explains several relationships, and predicts new observations, and how it compares to other explanations of the same observations. If such a case is likely to be too difficult for most students to understand, then students should at least be informed that they are being asked to take an idea on faith. When a material does not attempt to make a case, reviewers should comment on (a) whether or not a case ought to have been made and, if so, why and (b) whether or not the material makes explicit that a case is not being built.

**Indicators of meeting Criterion IV.1:
**1. The material builds a case for the mathematical importance of the benchmark.

2. The material builds a case for the validity of the mathematical ideas.

3. The material builds a case for the benchmark that is likely to be comprehensible to students.

4. The material engages students in considering a case for the validity and importance of benchmark concepts or skills.

**Scoring Scheme:
**High: The material meets all 4 indicators.

Medium: The material meets indicator 1 or 2, along with indicator 3 or 4.

Low: The material meets indicator 1or 2.

None: The material does not meet any of the indicators.

IV.2 Introducing Terms and Procedures. Does the material
introduce terms and procedures only in conjunction with experience
with them and only as needed to facilitate thinking and promote
effective communication? |

**Clarification:
**Understanding, rather than simply memorizing vocabulary or algorithms,
should be the main purpose of mathematics teaching. In mathematics, many terms
refer to concepts. For students to understand a concept, they should be able
to describe its properties, give examples and non-examples of it, and eventually
give a definition. Algorithms are important in themselves, as well as providing
efficient ways to solve problems. Students should have opportunities to apply
the concepts or procedures in problems and reasoning.

**Indicators of meeting Criterion IV.2:
**1. The material limits the use of terms and procedures.

2. The material introduces mathematical vocabulary or procedures in conjunction with experiences, rather than having students simply memorize definitions or procedures.

3. The material provides appropriate examples or meaningful applications of the terms or procedures.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets indicator 1, along with indicator 2 or 3.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.

IV.3 Representing Ideas Accurately. Does the material
include accurate and comprehensible representations of mathematical
concepts, procedures, and relationships? |

**Clarification:
**This question highlights the importance of using accurate representations
to make (abstract) ideas intelligible to all students. Different representations
highlight different aspects of an idea and provide a variety of opportunities
for the idea to connect to other students’ ideas and become embedded
in a student’s knowledge system. Possible modes of representation include
drawings, diagrams, graphs, images, analogies and metaphors, models and simulations,
and role-playing. Representations need to be clear so that students can understand
fairly quickly what ideas are being represented and how. In addition, because
representations typically highlight only some aspects of an idea, care must
be taken that they represent the real thing as accurately as possible (or
they involve students in considering which aspects of the real thing are represented
by the model and which are not).

**Indicators of meeting Criterion IV.3:
**1. The material includes accurate representations.

2. The material includes comprehensible representations (depending on the students’ grade level and the difficulty of the benchmark).

3. The material includes an appropriate number and variety of accurate and comprehensible representations.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets indicators 1 and 2.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.

IV.4 Connecting Benchmark Ideas. Does the material explicitly
draw attention to appropriate connections among benchmark ideas? |

**Clarification:
**This criterion emphasizes connections among benchmark ideas, concepts,
and skills (rather than connections between activities, which are examined
in Category I). Some kinds of connections can be classified as belonging to
one of the following general types:

(a) One concept or skill may be an instance of a generalization (e.g., the sum of the angles of a rectangle illustrates the general sum of the angles of a polygon).

(b) One concept or skill might be analogous to another idea (e.g., adding rational fractions is like adding rational expressions in that they both require a common denominator).

(c) A concept or skill may show up in several fields or contexts (e.g., the number pi can be the ratio of the circumference to diameter, the radian measure of a semicircle, or the sum of a series).

Other connections are more unique to particular ideas (e.g., linking slope of a line to the tangent of a curve).

Responding to this criterion involves looking to see whether any general or
unique connections are essential, which requires identifying what such connections
might be. Growth of Understanding maps provide a rich source of potential
connections. A set of maps on about 50 topics will be available in *Atlas
of Science Literacy*. When a map is not available for a topic, the "Also-See"
box in *Benchmarks for Science Literacy* may be helpful in identifying
conceptual connections among ideas. Reading *Science for All Americans*
may also be helpful in identifying conceptual connections, since several important
connections among ideas are made in the prose.

**Indicators of meeting Criterion IV.4:
**1. The material notes connections among specific benchmark ideas (rather
than just among general topics).

2. The material adequately explains or develops the identified connections.

3. The material engages students in making and/or explaining the identified connections.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets indicators 1 and 2.

Low: The material meets 1 of the 3 indicators.

None: The material does not meet any of the indicators.

IV.5 Demonstrating/Modeling Procedures. Does the material
demonstrate/model (or include suggestions for teachers on how
to demonstrate/model) skills or the use of knowledge? |

**Clarification:
**Among the ways literate adults use their knowledge and skills are to describe
and explain phenomena, to solve practical problems, and to consider alternative
positions on issues. Hence students should learn to use their knowledge and
skills in these ways. In order for students to know the type and level of
performance expected for a skill or an application of conceptual knowledge,
they need to see examples. This is particularly important for complex behaviors
such as explaining how to solve problems, developing a generalization, argument,
or proof, or carrying out a complex procedure or algorithm. Demonstrating
or Modeling a skill involves (a) an expert's demonstrating or modeling the
skill, (b) providing running commentary about important aspects to note about
the performance or demonstration, and (c) providing criteria for judging a
good performance. Demonstrating or Modeling how knowledge might be used, for
example, to solve problems or construct a proof, is similar.

Responding to this criterion involves examining whether (a) demonstrating/modeling is carried out by the text or other accompanying materials (e.g., software, video), or (b) the material includes suggestions to teachers about how to go about demonstrating/modeling skills or use of knowledge in their classrooms.

**Indicators of meeting Criterion IV.5:
**1. The material demonstrates (or instructs teachers how to demonstrate)
the expected procedure or performance.

2. The demonstration is clear and comprehensible.

3. The material provides commentary that points to particular aspects of the demonstration and/or provides justifications or explanations for the steps taken.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets indicators 1 and 2.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.

IV.6 Providing Practice. Does the material provide tasks
or questions for students to practice skills or use knowledge
in a variety of situations? |

**Clarification:
**An important part of learning mathematics consists of giving students
many opportunities to use their skill or knowledge, in particular giving them
opportunities to practice using mathematical knowledge and skills in describing
objects, relationships, and events, solving problems, and applying knowledge
in new situations or contexts. Moreover, literacy means that people will be
able to draw upon and use their understanding of mathematics when they encounter
events that do not come with labels such as "algebra," "geometry,
" or "statistics" but in political arguments, discussions of
literature, and walks on the beach. Therefore, students will need practice
in making connections to new situations. Providing students with opportunities
to practice only calculating answers to predictable exercises does not adequately
address this criterion.

**Indicators of meeting Criterion IV.6:
**1. The material includes appropriate practice exercises and tasks on the
benchmark.

2. The material provides an appropriate number of practice exercises and tasks.

3. The material includes a variety of contexts, including everyday tasks and contexts and novel as well as familiar practice tasks.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets indicator 1, along with either 2 or 3.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.

**Category V: Promoting Student Thinking about Mathematics**

No matter how clearly materials may present ideas, students (like all people) will devise their own meaning, which may or may not correspond to targeted learning goals. Students need to make their ideas and reasoning explicit, hold them up to scrutiny, and recast them as needed. This category includes criteria for whether the material suggests how to help students express, think about, and reshape their ideas to make better sense of the world.

V.1 Encouraging Students to Explain Their Reasoning.
Does the material routinely include suggestions for having each
student express, clarify, justify, and represent his/her ideas
and how to get feedback from peers and the teacher? |

**Clarification:
**It is important to provide opportunities for students’ thinking to
become overt to themselves, the teacher, and other students. By stating (clarifying,
justifying, and representing) their ideas, in writing, drawing, or speaking,
students become more aware of what they think. This may stimulate making explicit
connections between their ideas and the ideas presented by the text or the
teacher, and/or questioning of their ideas (if relevant). Exchange of ideas
in small groups or a large group discussion may make students aware of the
range of ideas that exist and may provoke students to reconsider their own
ideas in light of others. Feedback from the teacher or other peers is necessary
to help students understand their mistakes and improve the quality of their
descriptions, explanations, or designs.

Responding to the first part of the criterion involves examining whether the material prompts (or encourages teachers to prompt) students to express their ideas either orally or in writing. It also involves examining whether the material has opportunities for each student to express his or her ideas. Responding to the second part of the criterion includes examining whether the material includes specific suggestions to help the teacher provide explicit feedback, includes text that directly provides students with feedback on the adequacy of their ideas, or provides teachers with strategies they can use to ensure that each student in the class receives feedback.

**Indicators of meeting Criterion V.1:
**1. The material encourages students to express their ideas about the benchmark.

2. The material encourages students not only to express but also to clarify, justify, interpret, or represent benchmark ideas.

3. The material provides (or includes suggestions to help the teacher to provide) explicit feedback to students about their ideas.

4. The material includes suggestions to the students or teacher on how to use student responses to diagnose errors or difficulties, address errors or difficulties, or further develop students’ ideas about the benchmark.

**Scoring Scheme:
**High: The material meets all 4 indicators.

Medium: The material meets indicator 1, along with 2 of the other 3 indicators.

Low: The material meets at least 1 of the indicators.

None: The material does not meet any of the indicators.

V.2 Guiding Interpretation and Reasoning. Does the material
include tasks and/or question sequences that guide student interpretation
and reasoning about benchmark concepts, skills, and relationships? |

**Clarification:
**Experiences with hands-on materials, problems, and examples of mathematical
ideas are useful but not sufficient. Students need time, opportunities, and
guidance to make sense of these experiences. If the students are turned loose
to do exercises or problems on their own, very little happens except for a
small number of students. The activities need to be guided with sequences
of questions that lead students to make relevant generalizations and understand
relationships. Similarly, students need time, opportunities, and guidance
to make sense of things they read and ideas they are introduced to.

Responding to this criterion involves examining whether the material includes (in the teacher’s guide or student books) specific, carefully chosen and sequenced tasks or questions that are likely to support students’ thinking about exercises, problems, and investigations. Good tasks and questions frame important issues, help students relate their previous experiences to the mathematical ideas, anticipate common student difficulties or misconceptions, and focus on important generalizations and procedures.

**Indicators of meeting Criterion V.2:
**1. The material includes specific and relevant tasks and/or questions
for activities related to the benchmark.

2. The material includes connected sequences (rather than only collections) of questions or tasks.

3. The questions or tasks guide student interpretation and reasoning through approaches such as:

(a) framing, introducing, or developing important ideas,

(b) helping students to relate their own experiences to mathematical ideas,
or

(c) anticipating or eliciting common difficulties or student misconceptions.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets 2 of the 3 indicators.

Low: The material meets 1 of the 3 indicators.

None: The material does not meet any of the indicators.

V.3 Encouraging Students to Think about What They’ve
Learned. Does the material suggest ways to have students check
their own progress? |

**Clarification: ****
**This criterion highlights the importance of having students look back
at the progress of their thinking and learning. Monitoring one’s understanding
and realizing which ideas one does not understand can shift some of the responsibility
for learning to the students and may elicit their attempts to understand as
a result.

Responding to this criterion involves examining whether the material includes questions or tasks that prompt students to monitor their understanding, or includes suggestions to teachers on how to encourage students to do so. For example, What was confusing to you today? How does the new knowledge compare with what you used to think? What do you think you understand and where do you need to work more? Encouraging students to monitor their understanding should also include (when appropriate) questions on how and why students changed their ideas. For example, Did you change any of your ideas today? What evidence convinced you to do so?

**Indicators of meeting Criterion V.3:
**1. The material engages (or provides specific suggestions for teachers
to engage) students in monitoring their progress toward the benchmark.

2. The material asks students to think about how their ideas have developed or changed.

3. The material gives students an opportunity to revise their initial ideas about the benchmark based on what they have learned.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets 2 of the 3 indicators.

Low: The material meets 1 of the 3 indicators.

None: The material does not meet any of the indicators.

**Category VI: Assessing Student Progress in Mathematics**

Assessment provides information to students about what is important and to teachers about what has been learned. Just as important, assessment provides information to both the students and the teacher about adjustments that should be made in learning and instruction. Because assessment is so important to the teaching-learning process, it must match the mathematics learning goal of the curriculum materials. Further, assessments must address the range of skills, applications, and contexts that reflect what students are expected to learn. All of this is possible only if assessment takes place throughout instruction, not only at the end of a chapter or unit.

VI.1 Aligning Assessment. Are assessment items or tasks
included that match the ideas, concepts, or skills of the benchmark? |

**Clarification:
**This criterion highlights the necessity of including assessment items
for each benchmark that is important in the material. To judge whether the
items provided match the standard, the same procedure should be followed as
in examining the content match between an activity and a benchmark. That is,
examine whether the assessment item addresses the substance rather than only
the topic of the benchmark, the level of sophistication of the benchmark,
and what part of the benchmark is assessed.

**Indicators of meeting Criterion VI.1:**

1. The material provides at least one assessment task that addresses the specific
ideas of the benchmark (assessment item should not be answerable by reading
comprehension, general intelligence, or test-wiseness alone).

2. The assessment items that do address the benchmark require no other, more
sophisticated, ideas.

3. If the material provides a test that is given to the students, an appropriate
number of assessment items are content-matched to the benchmark. If the material
provides a bank of assessment items that teachers select from, an appropriate
proportion of assessment items are content-matched to the benchmark.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets indicators 1 and 2.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.

VI.2 Assessment through Applications. Does the material
include assessment tasks that require application of benchmark
ideas, concepts, or skills and avoid allowing students a trivial
way out, like using a formula or repeating a memorized term or
rule without understanding? |

**Clarification:
**Rather than checking whether students have memorized certain bits of information,
assessment needs to test students’ mathematical understanding, reasoning,
and the application of knowledge. In addition, it needs to include tasks that
engage students in activities similar to those they will engage in in their
lives outside the classroom. Literate persons use mathematical knowledge to
describe, explain, and predict real world phenomena, solve a practical problem,
or discuss issues. Accordingly, assessment tasks need to engage students in
descriptions, explanations, predictions, design, and discussion of issues.
This, however, does not necessarily dictate the format that the assessment
should include. For example, assessments of students’ use of knowledge
to explain a concept could include a "multiple choice" or "constructed
response" format.

**Indicators of meeting Criterion VI.2:
**1. The material provides assessment tasks that focus on application of
benchmark ideas.

2. The material includes assessment tasks that are familiar as well as tasks that are novel or non-routine.

3. If the material provides a test that is given to the students, an appropriate number of assessment items focus on application. If the material provides a bank of assessment items that teachers select from, an appropriate proportion of items or tasks focus on application.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets 2 of the 3 indicators.

Low: The material meets 1 of the 3 indicators.

None: The material does not meet any of the indicators.

VI.3 Using Embedded Assessment. Are some assessments
embedded in the curriculum along the way, with advice to teachers
as to how they might use the results to choose or modify activities? |

**Clarification:
**This criterion highlights the need for assessment to be in the service
of instruction to guide teaching and learning. The criterion requires that
materials include assessments that can be used as diagnostic or formative
instruments, which help determine learners’ needs, rather than largely
as instruments for grading students at the end of a unit or chapter.

Responding to this question involves examining whether the material (a) provides assessment tasks only at the end of a unit of study to help grade student achievement, or also along the way to help monitor student progress, (b) encourages and provides guidance to teachers about how to probe beyond students’ first response to clarify and further understand student answers, and (c) encourages teachers to use the information from the assessments to make instructional decisions about what ideas need to be addressed by further activities with the whole group or smaller groups of students.

**Indicators of meeting Criterion VI.3:
**1. The material uses embedded assessment as a part of the instructional
strategy or design.

2. The material includes assessments that provide opportunities, encouragement, or guidance for students on how to further understand benchmark ideas.

3. The material includes suggestions or guidance for teachers on how to probe students’ understanding of benchmark ideas.

4. The material provides specific suggestions to teachers about how to use the information from the embedded assessments to make instructional decisions about what ideas need to be addressed by further activities.

**Scoring Scheme:
**High: The material meets all 4 indicators.

Medium: The material meets indicator 1, along with 2 of the other 3 indicators.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.

**Category VII. Enhancing the Mathematics Learning Environment**

Several other important considerations are involved in the selection of curriculum materials—for example, the help they provide teachers in encouraging student curiosity and creating a classroom community where all can succeed, or attractiveness. These can influence student learning or even whether the materials are used appropriately by the teacher and students. The criteria listed in this category provide reviewers with the opportunity to comment on these and other important features.

VII.1 Providing Teacher Content Support. Does the material
help teachers improve their understanding of mathematics and its
applications? |

**Clarification:
**The issue here is whether the material includes a "content background"
section or other features that help teachers develop their understanding of
the mathematical knowledge addressed in the material. Responding to this question
involves commenting on the quality of the support, not merely on whether such
support is included. Just providing teachers with a list of resources that
may enhance their understanding of mathematics does not adequately address
this question. A minimum requirement is that such lists are annotated to describe
the resources and specify what can be learned from them.

**Indicators of meeting Criterion VII.1:
**1. The material provides content information or recommends resources for
improving specific skills or understanding of particular ideas.

2. The material provides content information that is in the form that is useful and appropriate for teachers, no matter what their background knowledge.

3. The material indicates how the ideas or skills are relevant and important to teaching the material to students.

**Scoring Scheme:
**High: The material meets all 3 indicators.

Medium: The material meets 2 of the 3 indicators.

Low: The material meets 1 of the 3 indicators.

None: The material does not meet any of the indicators.

VII.2 Establishing a Challenging Classroom. Does the
material help teachers to create a classroom environment that
welcomes student curiosity, rewards creativity, encourages a spirit
of healthy questioning, and avoids rigidity? |

**Clarification:
**Responding to this criterion involves examining whether teachers are given
guidance to (a) encourage students to raise questions about the material being
studied and suggest productive ways for finding answers, (b) use activities
in which students’ creativity and imagination will pay off, (c) respect
and value students’ ideas, and (d) avoid conveying the impression that
they themselves or the textbooks are absolute authorities whose conclusions
are always correct. In addition, the criterion involves examining whether
the materials give a vision of what the curriculum might look like in action
(i.e., teacher hints and suggestions, dialogue boxes, vignettes, or video
clips that show desirable student teacher interactions).

**Indicators of meeting Criterion VII.2:
**1. The material provides opportunities for students to express their curiosity
or creativity.

2. The material provides occasions for students to take risks and ask questions.

3. The material suggests how to encourage students to weigh and challenge their own and others’ ideas.

4. The material avoids sending a message that mathematics consists only of rules and single correct answers.

**Scoring Scheme:
**High: The material meets all 4 indicators.

Medium: The material meets 2 of the 4 indicators.

Low: The material meets 1 of the 4 indicators.

None: The material does not meet any of the indicators.

VII.3 Supporting All Students. Does the material help
teachers to create a classroom that encourages high expectations
for all students, enables all students to experience success,
and provides all students a feeling of belonging in the mathematics
classroom? |

**Clarification:
**Several pedagogical criteria presented in previous categories highlight
the need for materials to incorporate principles of teaching and learning
that are likely to promote mathematics understanding for all students. This
question highlights the importance of reviewing curriculum materials for features
that might distract or impede the progress of females, minorities, students
whose first language is not English, students with disabilities, or others
from the intended work. Further, the criterion requires that materials provide
specific suggestions and resources for encouraging all students to be able
to learn mathematics and express their competence and performances during
instruction and in assessment tasks.

**Indicators of meeting Criterion VII.3:
**1. The material avoids stereotypes or language that might be offensive
to a particular group.

2. The material illustrates the contribution or participation of women, minorities, and persons with disabilities to mathematics-related fields.

3. The material suggests alternative formats for students to develop or express their mathematics knowledge during instruction and assessment.

4. The material includes specific suggestions on how teachers can modify activities for students with special needs, interests, or abilities.

**Scoring Scheme:
**High: The material meets all 4 indicators.

Medium: The material meets indicator 1, along with 2 of the other 3 indicators.

Low: The material meets indicator 1.

None: The material does not meet any of the indicators.