Project 2061 LogoAAAS Project 2061
AAAS :: Project 2061 :: Textbook Evaluations

Middle Grades Science Textbooks: A Benchmarks-Based Evaluation

Criteria Used in Evaluating the Programs’ Quality of Instructional Support

In evaluating the instructional effectiveness of the individual middle grades science programs covered in this document, the review, or evaluation, teams have applied a set of research-based criteria developed by Project 2061 and presented below. The criteria are organized in seven categories, each of which focuses on a specific aspect of instructional support. The criteria, in turn, are annotated briefly to provide users with additional guidance.

Each criterion is then followed by a list of indicators that the reviewers used to judge how well the curriculum material meets the criterion. Lastly, accompanying each criterion’s list of indicators is a rating scheme that describes which indicators must be met for the range of ratings. Typically, only “excellent,” “satisfactory,” and “poor” ratings were specified. Intermediate ratings of “very good” and “fair” were given only when indicator judgments fell between specified ratings.
  Categories
I.
 Providing a Sense of Purpose
II.
 Taking Account of Student Ideas
III.
 Engaging Students with Relevant Phenomena
IV.
 Developing and Using Scientific Ideas
V.
 Promoting Students’ Thinking about Phenomena, Experiences, and Knowledge
VI.
 Assessing Progress
VII.
 Enhancing the Science Learning Environment


I. Providing a Sense of Purpose

This category consists of criteria for determining whether the curriculum material attempts to make its purposes explicit and meaningful to students, either in the student text itself or through suggestions made to the teacher. The sequence of lessons or activities is also important in accomplishing the stated purpose, since ideas often build on each other.

Conveying unit purpose. Does the material convey an overall sense of purpose
and direction that is understandable and motivating to students?
Indicators of meeting the criterion
  1. A problem, question, representation (or otherwise identified purpose) is presented to students.1, 2
  2. The problem, question, representation (or otherwise identified purpose) is likely to be comprehensible to students.
  3. The problem, question, representation (or otherwise identified purpose) is likely to be interesting and/or motivating to students.
  4. Students are given an opportunity to think about and discuss the problem, question, representation (or otherwise identified purpose).
  5. Most lessons are consistent with the stated purpose and those that are not are explicitly labeled as digressions.
  6. The material returns to the stated purpose at the end of the unit.

Rating Scheme
Excellent: The material meets indicators 1–6.
Satisfactory: The material meets indicators 1–3 and 5.
Poor: The material meets indicator 1 at best.

Conveying lesson/activity purpose. Does the material convey the purpose of each lesson or activity and its relationship to others?

Indicators of meeting the criterion
  1. The material conveys or prompts teachers to convey the purpose of the activity to students.
  2. The purpose is expressed in a way that is likely to be comprehensible to students.
  3. The material encourages each student to think about the purpose of the activity.
  4. The material conveys or prompts teachers to convey to students how the activity relates to the unit purpose.
  5. The material engages students in thinking about what they have learned so far and what they need to learn/do next at appropriate points.

Rating Scheme
Excellent: The material meets all indicators.
Satisfactory: The material meets any three out of the five indicators.
Poor: The material meets no more than one out of the five indicators.

Justifying lesson/activity sequence. Does the material involve students in a logical or strategic sequence of lessons or activities (versus being just a collection of lessons or activities)?

Indicators of meeting the criterion

  1. The material includes a logical or strategic sequence of activities.
  2. The material conveys the rationale for this sequence.

Rating Scheme
Excellent: The material meets both indicators.
Satisfactory: The material meets the first indicator.
Poor: The reviewer can infer a logical rationale for the sequence of only a few activities.

II. Taking Account of Student Ideas

Fostering understanding in students requires taking time to attend to the ideas they already have, both ideas that are incorrect and ideas that can serve as a foundation for subsequent learning. This category consists of criteria for determining whether the curriculum material contains specific suggestions for identifying and addressing students’ ideas.

Attending to prerequisite knowledge and skills. Does the material specify prerequisite knowledge/skills that are necessary to the learning of the key ideas?

Indicators of meeting the criterion

  1. The material alerts the teacher to specific prerequisite ideas or skills (versus stating only prerequisite topics or terms).
  2. The material alerts teachers to the specific ideas for which the prerequisites are needed.
  3. The material alerts students to prerequisite ideas or experiences that are being assumed.
  4. The material adequately addresses (provides instructional support for) prerequisites in the same unit or in earlier units (in the same or other grades). (The material should not be held accountable for addressing prerequisites from an earlier grade range. However, if a material does address such prerequisites they should count as evidence for this indicator.)
  5. The material makes adequate connections (provides instructional support for connections) between ideas treated in a particular unit and their prerequisites (even if the prerequisites are addressed elsewhere).

Rating Scheme
Excellent: The material meets indicators 1, 2, 3 or 4, and 5 for all or most prerequisites.
Satisfactory: The material meets indicators 1, 2, 5, and either 3 or 4 for some prerequisites.
Fair: The material meets indicators 5 and either 3 or 4 for some prerequisites.
Poor: The material meets no more than one indicator.

Alerting teachers to commonly held student ideas. Does the material alert teachers to commonly held student ideas (both troublesome and helpful), such as those described in Benchmarks for Science Literacy, Chapter 15: The Research Base (American Association for the Advancement of Science, 1993)? Read the Research on Commonly Held Student Ideas provided to analysts for the middle grades science textbooks evaluation.

Indicators of meeting the criterion
  1. The material accurately presents specific commonly held ideas that are relevant to the key ideas and have appeared in scholarly publications (rather than just stating that students have difficulties with particular ideas or topics).
  2. The material clarifies/explains commonly held ideas (rather than just listing them).

Rating Scheme
Excellent: The material meets indicators 1 and 2 for a considerable proportion of commonly held ideas that have appeared in scholarly publications.3
Satisfactory: The material meets indicators 1 and 2 for some commonly held ideas that have appeared in scholarly publications.
Poor: The material meets indicator 1 at best.

Assisting teachers in identifying their students’ ideas. Does the material include suggestions for teachers to find out what their students think about familiar phenomena related to the key ideas before the scientific ideas are introduced?

Indicators of meeting the criterion

  1. The material includes specific questions or tasks that could be used by teachers to identify students’ ideas.
  2. The questions/tasks are likely to be comprehensible to students who have not studied the topic and are not familiar with the scientific vocabulary.
  3. The questions/tasks are identified as serving the purpose of identifying students’ ideas.
  4. The material includes questions/tasks that ask students to make predictions and/or give explanations of phenomena (rather than focus primarily on identifying students’ meanings for terms).
  5. The material suggests how teachers can probe beneath students’ initial responses to questions or interpret student responses (e.g., by providing annotated samples of student work).

Rating Scheme
Excellent: The material provides a sufficient number and variety4 of questions/tasks that meet indicators 1 and 2 and meet indicators 3–5.
Satisfactory: The material provides some questions/tasks that meet indicators 1–4.
Poor: The material provides some questions/tasks that meet indicator 1 or indicators 1 and 2.

Addressing commonly held ideas. Does the material attempt to address commonly held student ideas?

Indicators of meeting the criterion
  1. The material explicitly addresses commonly held ideas.
  2. The material includes questions, tasks, or activities that are likely to help students progress from their initial ideas, for example, by5
    1. explicitly challenging students’ ideas, for example, by comparing their predictions about a phenomenon to what actually happens
    2. prompting students to contrast commonly held ideas with the scientifically correct ideas, and resolve differences between them
    3. extending correct commonly held ideas that have limited
      scope.
  3. The material includes suggestions to teachers about how to take into account their own students’ ideas.

Rating Scheme
If there is research on commonly held student ideas:
Excellent: The material meets indicators 1 and 2 for a considerable proportion of commonly held ideas that are documented in the literature.
Satisfactory: The material meets indicators 1 and 2 for some commonly held ideas that are documented in the literature.
Poor: The material meets the first indicator at best.

III. Engaging Students with Relevant Phenomena

Much of the point of science is to explain phenomena in terms of a small number of principles or ideas. For students to appreciate this explanatory power, they need to have a sense of the range of phenomena that science can explain. The criteria in this category examine whether the curriculum material relates important scientific ideas to a range of relevant phenomena and provides either firsthand experiences with the phenomena or a vicarious sense of phenomena that are not presented firsthand.

Providing variety of phenomena. Does the material provide multiple and varied phenomena to support the key ideas?

Indicators of meeting the criterion

  1. Phenomena could be used to support the key ideas.
  2. Phenomena are explicitly linked to the relevant key ideas.6, 7

Rating Scheme
Excellent: The material provides a sufficient number and variety of phenomena that meet indicators 1 and 2.
Satisfactory: The material provides some phenomena that meet indicators 1 and 2.
Poor: The material provides, at best, one phenomenon that meets indicators 1 and 2.

If the “key idea” for which the material is analyzed includes several ideas, the reviewers should proceed as follows:

  1. Identify the ideas for which there is a content match.
  2. Score the treatment of each idea that results from step a. The overall score for this criterion will be the average of the scores for each idea.

Providing vivid experiences. Does the material include activities that provide firsthand experiences with phenomena when practical or provide students with a vicarious sense of the phenomena when not practical?

Indicators of meeting the criterion
  1. Each firsthand experience is efficient (when compared to other firsthand experiences) and, if several firsthand experiences target the same idea, the set of firsthand experiences is efficient. (The efficiency of an experience equals the cost of the experience [in time and money] in relation to its value.)
  2. The experiences that are not firsthand (e.g., text, pictures, video) provide students with a vicarious sense of the phenomena. (Please note that if the material provides only firsthand experiences, this indicator is not applicable.)
  3. The set of firsthand and vicarious experiences is sufficient.

Rating Scheme
Excellent: The material meets all indicators or just indicators 1 and 3 or indicators 2 and 3, if firsthand experiences are not possible.
Satisfactory: The material includes some efficient firsthand experiences and, if several firsthand experiences target the same idea, the set of firsthand experiences is sufficient. When firsthand experiences are not practical, the material provides students with a vicarious sense of the phenomena for some of the experiences that are not firsthand.
Poor: The material includes at best only one efficient firsthand experience or provides students with a vicarious sense of one phenomenon that is not firsthand.

Reviewers should proceed as follows:

  1. Identify the key ideas for which there is a content match.
  2. Score the treatment of each idea that results from step a. The overall score for this criterion will be the average of the scores for each idea.



IV. Developing and Using Scientific Ideas

Science literacy requires that students understand the link between scientific ideas and the phenomena that they can explain. Furthermore, they should see the ideas as useful and become skillful at applying them. This category consists of criteria for determining whether the curriculum material expresses and develops the key ideas in ways that are accessible and intelligible to students, and that demonstrate the usefulness of the key ideas and provide practice in varied contexts.

Introducing terms meaningfully. Does the material introduce technical terms only in conjunction with experience with the idea or process and only as needed to facilitate thinking and promote effective communication?

Indicators of meeting the criterion
  1. The material links technical terms to relevant experiences that develop the idea as the term is used (rather than just having students learn definitions of terms).
  2. The material restricts the use of technical terms to those needed to communicate intelligibly about key ideas.

Rating scheme
Excellent: The material meets both indicators.
Satisfactory: The material fully meets one indicator and partially meets the other.
Poor: The material marginally meets both indicators at best.

Representing ideas effectively. Does the material include accurate and comprehensible representations of the key ideas?

Indicators of meeting the criterion
  1. Representation is accurate (or, if not accurate, then students are asked to critique the representation).
  2. Representation is likely to be comprehensible to students.
  3. Representation is explicitly linked to the real thing.

Rating Scheme
Excellent: The material includes a sufficient number and variety of representations that meet indicators 1–3 and none of the representations included in the material are inaccurate. (In order to judge whether there is a sufficient number and variety of representations, reviewers should first consider which key ideas require representations and then decide whether these are adequately represented. However, reviewers are not expected to evaluate and rate each idea separately and average the scores.)
Satisfactory: The material includes some representations that meet indicators 1–3 and few (if any) of the representations included in the material are inaccurate. (In some cases, including one accurate and comprehensible representation for a specific idea may be sufficient for a material to receive a “satisfactory” rating. However, most of the key ideas must be adequately represented.)
Poor: Even though the material includes a few representations that meet indicators 1–3, few or none of the key ideas are adequately represented.

Demonstrating use of knowledge. Does the material demonstrate/model or include suggestions for teachers on how to demonstrate/model skills or the use of knowledge?

Indicators of meeting the criterion
  1. The material consistently carries out (or instructs teachers to carry out) the expected performance (e.g., the student text explains a particular phenomenon using the kinetic molecular theory). (Teacher’s guides often include responses to questions posed in the student text. If the material does not instruct the teacher to use the answers to model the use of knowledge, such responses do not count as instances of modeling.)
  2. The performance is step-by-step.
  3. The performance is explicitly identified as a demonstration of the use of knowledge or skill.
  4. The material provides running commentary that points to particular aspects of the demonstration and/or criteria for judging the quality of a performance.

Rating Scheme
Excellent: The material meets all 4 indicators.
Satisfactory: The material meets indicators 1 and 2.
Poor: The material meets indicator 1 at best.

Providing practice. Does the material provide tasks/questions for students to practice skills or to use knowledge in a variety of situations?

Indicators of meeting the criterion
  1. The material provides a sufficient number of tasks in a variety of contexts, including everyday contexts. (In order to determine whether the task/question addresses the actual substance of the key idea, reviewers will need to consider both the question and the expected response in the teacher’s guide.)
  2. The material includes novel tasks.
  3. The material provides a sequence of questions or tasks in which the complexity is progressively increased.
  4. The material provides students first with opportunities for guided practice with feedback and then with practice in which the amount of support is gradually decreased.

Rating Scheme
Excellent: The material meets indicators 1, 2, and either 3 or 4.
Satisfactory: The material provides some tasks/questions, including novel tasks.
Poor: The material provides at best some tasks/questions, but no novel tasks.

Reviewers should proceed as follows:

  1. Identify the key ideas for which there is a content match.
  2. Score the treatment of each idea that results from step a. The overall score for this criterion will be the average of the scores for each idea.



V. Promoting Students’ Thinking about Phenomena, Experiences, and Knowledge

Engaging students in experiences with phenomena (category III) and presenting them with scientific ideas (category IV) will not lead to effective learning unless students are given time, opportunities, and guidance to make sense of the experiences and ideas. This category consists of criteria for determining whether the curriculum material provides students with opportunities to express, think about, and reshape their ideas, as well as guidance on developing an understanding of what they experience.

Encouraging students to explain their ideas. Does the material routinely include suggestions for having each student express, clarify, justify, and represent his or her ideas? Are suggestions made for when and how students will get feedback from peers and the teacher?

Indicators of meeting the criterion
  1. Material routinely encourages students to express their ideas.
  2. Material encourages students not only to express but also to clarify, justify, and represent their ideas (a material is not expected to encourage students to clarify, justify, and represent ideas each time they are asked to express their ideas; however, in the course of teaching a particular key idea the material should provide students with opportunities to clarify, justify, and represent ideas).
  3. Material provides opportunities for each student (rather than just some students) to express ideas.
  4. Material includes specific suggestions on how to help the teacher provide explicit feedback to students or includes text that directly provides students with feedback.
  5. Material includes suggestions on how to diagnose student errors, explanations about how these errors may be corrected, and recommendations for how students’ ideas may be further developed.

Rating Scheme
Excellent: Material meets all indicators.
Satisfactory: Material meets 3 out of 5 indicators.
Poor: Material meets no more than 1 out of 5 indicators.

Guiding student interpretation and reasoning. Does the material include tasks and/or question sequences to guide student interpretation and reasoning about experiences with phenomena and readings?

Indicators of meeting the criterion
  1. The material includes specific and relevant tasks and/or questions for the experience or reading.
  2. The questions or tasks have helpful characteristics such as
    1. framing important issues
    2. helping students to relate their experiences with phenomena or representations to presented scientific ideas
    3. helping students to make connections between their own ideas and the phenomena or representations observed
    4. helping students to make connections between their own ideas and the presented scientific ideas
    5. anticipating common student misconceptions
    6. focusing on contrasts between student misconceptions and scientific alternatives.
    Please note that while a single high quality task or question sequence might have only one of these characteristics, the set of sightings should exhibit several of them.
  3. There are scaffolded sequences of questions or tasks (as opposed to separate questions or tasks).

Rating Scheme
Excellent: Material consistently meets all three indicators.
Satisfactory: Material consistently meets indicators 1 and 2.
Poor: Material meets indicator 1 at best.

Encouraging students to think about what they have learned. Does the material suggest ways to have students check and reflect on their own progress?

Indicators of meeting the criterion

  1. The material gives students an opportunity to revise their initial ideas based on what they have learned (without asking them explicitly to think about how their ideas have changed).
  2. The material engages (or provides specific suggestions for teachers to engage) students in monitoring how their ideas have changed, but does so infrequently in the unit.
  3. The material engages (or provides specific suggestions for teachers to engage) students in monitoring how their ideas have changed and does so periodically in the unit.

Rating Scheme
Excellent: The material meets indicator 3.
Satisfactory: The material meets indicator 2.
Poor: The material meets indicator 1 at best.



VI. Assessing Progress

This category consists of criteria for evaluating whether the curriculum material includes a variety of aligned assessments that apply the key ideas taught in the material.

Aligning assessment to goals. Assuming a content match between the curriculum material and a key idea, are assessment items included that match the same key idea?

Indicators of meeting the criterion
  1. The specific ideas in the key ideas are necessary in order to respond to the assessment items.
  2. The specific ideas in the key ideas are sufficient to respond to the assessment items (or, if other ideas are needed, they are not more sophisticated than key ideas and have been taught earlier).

Rating Scheme8
Excellent: The material provides a sufficient number9 of assessment items that meet indicators 1 and 2.
Satisfactory: The material provides some assessment items that meet indicators 1 and 2.
Poor: The material provides no more than one/a few assessment items that meet indicators 1 and 2.

Testing for understanding. Does the material include assessment tasks that require application of ideas and avoid allowing students a trivial way out, like using a formula or repeating a memorized term without understanding it?

Indicators of meeting the criterion
  1. Assessment items focus on understanding of key ideas.
  2. Assessment items include both familiar and novel tasks.

Rating Scheme
Excellent: The material provides a sufficient number of assessment items that meet indicators 1 and 2.
Satisfactory: The material provides some assessment items that meet indicators 1 and 2 or sufficient assessment items that meet indicator 1.
Poor: The material provides no more than one/a few assessment items that meet indicator 1.

To judge whether there is a sufficient number of assessment items, the reviewers need to consider whether all ideas in a given key idea are adequately assessed. However, reviewers are not expected to evaluate and rate each idea separately and average the scores.

Using assessment to inform instruction. Are some assessments embedded in the curriculum along the way, with advice to teachers as to how they might use the results to choose or modify activities?

Indicators of meeting the criterion
  1. The material uses embedded assessment as a routine strategy (rather than just including occasional questions).
  2. The material assists teachers in interpreting student responses to diagnose what learning difficulties remain.
  3. The material provides specific suggestions to teachers about how to use the information from the embedded assessments to make instructional decisions about what ideas need to be addressed by further activities.

Rating Scheme
Excellent: The material meets all indicators.
Satisfactory: The material meets indicators 1 and 2 or indicators 1 and 3.
Poor: The material meets indicator 1 at best.

Some materials suggest that teachers will use all practice tasks included in the unit as embedded assessment (typically, these suggestions are made in the introduction to the teacher’s guide). In these cases, reviewers need to examine these tasks (even if they were examined before in the “Providing practice” criterion of category IV).



VII. Enhancing the Science Learning Environment

The criteria in this category provide analysts with the opportunity to comment on features that enhance the use and implementation of the curriculum material by all students. For this category, the reviewers used criterion-specific ratings in lieu of the general ratings used for categories I through VI.

Providing teacher content support. Does the material help teachers improve their understanding of science, mathematics, and technology as is necessary for teaching the material?

Indicators of meeting the criterion
  1. Alerts teachers to how ideas have been simplified for students to comprehend and what the more sophisticated versions are (even though students are not required to understand the more sophisticated versions).
  2. Provides sufficiently detailed answers to questions in the student book for teachers to understand and interpret various student responses.
  3. Recommends resources for improving the teacher’s understanding of key ideas.

Encouraging curiosity and questioning. Does the material help teachers to create a classroom environment that welcomes student curiosity, rewards creativity, encourages a spirit of healthy questioning, and avoids dogmatism?

Indicators of meeting the criterion
  1. Includes suggestions for how to encourage students’ questions and guide their search for answers, respect and value students’ ideas, raise questions of evidence, and avoid dogmatism.
  2. Provides examples of classroom interactions—e.g., dialogue boxes, vignettes, or video clips—that illustrate appropriate ways to respond to student questions or ideas, etc.

Supporting all students. Does the material help teachers to create a classroom community that encourages high expectations for all students, that enables all students to
experience success, and that provides all different kinds of students with a feeling of belonging in the science classroom?

Indicators of meeting the criterion
  1. The material avoids stereotypes or language that might be offensive to a
    particular group.
  2. The material illustrates the contribution of women and minorities to science
    and brings in role models.
  3. The material suggests alternative formats for students to express their ideas
    during instruction and assessment.
  4. The material includes specific suggestions about how teachers can modify activities for students with special needs.
  5. The material provides strategies to validate students’ relevant personal and social experiences with scientific ideas.

Advancing science, Serving society

Copyright 2002 by the American Association for the Advancement of Science. All rights reserved.