AAAS » Project 2061

Reprinted here with the permission of Science Books & Films. No further republication or redistribution is permitted without the written permission of the editor. Source: Science Books & Films, November/December 1999 - Volume 35 - Number 6

HEAVY BOOKS

LIGHT ON LEARNING

AAAS Project 2061 Evaluates
Middle Grades Science Textbooks

George D. Nelson, Director of Project 2061, and Jo Ellen Roseman, Project 2061 Curriculum Director, report on the results of the Project 2061 middle grades science textbooks evaluation at a press conference in Washington, DC. (photo © John Metelsky, Imagecatcher News)

Project 2061 director Dr. George Nelson recently announced the results of an in-depth evaluation of middle grades science textbooks. Like the project's earlier evaluation of middle grades mathematics textbooks (reported in Science Books & Films, Vol. 35, No. 4), the focus of this effort was to see which textbooks had potential for helping students learn key ideas. But unlike the math textbook study, not one of the middle grades science texts evaluated by Project 2061 rated satisfactory.

The evaluation examined how well textbooks for the middle grades help students learn key ideas in earth science, life science, and physical science, drawn from AAAS's Benchmarks for Science Literacy and the National Research Council's National Science Education Standards. The study probed beyond the usual superficial alignment by topic heading and examined each text's quality of instruction aimed specifically at the key ideas, using criteria drawn from the best available research about how students learn. (See the sidebars on page 244 for more details about the ideas and criteria used in the evaluation.) The evaluation procedure was developed and tested over a period of three years in collaboration with more than 100 scientists, mathematicians, educators, and curriculum developers, with funding from the National Science Foundation.

View the Chart: Rating of Instructional Quality

The analysts--middle school teachers, curriculum specialists, and professors of science education--examined nine textbooks, including widely used books and those that are relatively new to the market. They found that the textbooks covered too many topics and didn't develop any of them well. In addition, the texts included many classroom activities that either were irrelevant to learning key science ideas or didn't help students relate what they were doing to the underlying ideas.

The study also looked at three stand-alone units that are not part of any textbooks--Matter and Molecules; Food, Energy, and Growth; and Chemistry That Applies. Developed at Michigan State University and the Michigan Department of Education, these units are based explicitly on research about how students learn. Across the board, the units rated much higher than the textbooks. These encouraging results show that good science materials can indeed be developed.

Focus on Student Learning

But what does a satisfactory textbook look like? The following examples illustrate some of the instructional characteristics that analysts looked for in the books they evaluated. These characteristics are described in the criteria used to rate the instructional quality of the books. The examples below focus on physical science ideas, but there are similar examples for ideas in life and earth science as well. Among a number of other important features, a satisfactory textbook should have instructional support that is designed to:

Alert teachers to commonly held student ideas.

Of the materials examined, only the stand-alone unit Matter and Molecules was rated excellent on this criterion in physical science. Matter and Molecules lists all relevant misconceptions that are reported in student learning research, explains each one and why students believe it, and describes what has to change for them to find credible the scientific idea that molecules are perpetually in motion.

In contrast, while a few textbooks have notes in the teacher's guide labeled "Misconceptions" or "Prior Knowledge," the statements included are not very helpful. For example, one textbook merely states that "students may confuse the terms atoms and elements." Other well-documented student misconceptions (such as the idea that matter is continuous or the idea that molecules are in substances) are not even mentioned. Teachers need insights like these to plan effective instruction.

Provide students with a variety of phenomena.

A phenomenon is an event that can be scientifically described. Much of the point of science is explaining phenomena in terms of a small number of principles or ideas. For students to appreciate this explanatory power, they need to have a sense of the range of phenomena that science can explain. The Matter and Molecules unit and three of the textbooks provide a satisfactory variety of physical science phenomena related to the kinetic molecular theory.

Unfortunately, none of the materials provides phenomena to help make plausible the (less believable) idea that the particles of a solid are in continuous motion. For example, while it takes considerably longer to observe, solids like gold and lead can mix a bit.

But experiences with phenomena are not enough! Students need help to understand and appreciate how the phenomena relate to the scientific ideas.

Guide student interpretation and reasoning.

Of the materials examined, only Matter and Molecules provides excellent guidance to help students interpret and reason about phenomena. For example, after students see that air can be compressed in a plastic syringe and read about the large amount of air that can be compressed in a small scuba tank, the unit provides questions to guide student thinking about the phenomena. To probe the common student misconception that molecules are not perpetually in motion, students consider the following:

Students are also asked a question that anticipates a common misconception (that molecules are not perpetually in motion but only move if the substance appears to move):

None of the other materials have questions with these characteristics and hence were not rated very good or excellent on this criterion.

Once an idea begins to take hold, students need many and varied opportunities to apply it.

Provide practice in using scientific ideas.

The Matter and Molecules unit scores higher than all textbooks examined in providing a sufficient number and variety of practice tasks for most of the physical science ideas examined. These include novel tasks that ask students to develop descriptions and explanations of phenomena they see all around them. Since explaining real world phenomena often requires using more than a single idea, the phenomena students are asked to explain increase in complexity. For example, the following questions (taken from several places in the unit) have students move from using mainly the idea that "molecules are in perpetual motion" to using this and other related ideas:

• Draw pictures to show how water molecules are moving.
• Can water molecules in ice slow down and stop?
• If you want something to dissolve fast, should you mix it with hot water or cold water? Why?
• Explain how you can smell an open bottle of vinegar even though you are across the room. What is actually reaching your nose? How did the vinegar molecules get into the air? How did the vinegar molecules reach your nose?
• When food covered with plastic wrap in the refrigerator (or when soup is warming on the stove, but not boiling, with a lid on the pot), water evaporates and then condenses. Where does the water evaporate from? Where does the water condense? How do the water molecules get from the place where water evaporates to the place where water condenses?

In contrast, other textbooks include few, if any, relevant practice tasks. Those that are included are usually like this one:

Since a similar statement appeared in the text a few pages earlier, filling in the blank requires little more than copying the correct answer. The only textbook to include several decent practice tasks did not do so consistently across the set of ideas that were used as the basis for the physical science analysis.

But what good is practice if students don't understand what they are to practice? (Imagine being asked to practice the breast stroke in swimming if you've never been shown what it looks like done well!) So, rather than expecting students to figure out what a good explanation should be like, the Project 2061 procedure includes a criterion to probe whether a material demonstrates how to use scientific ideas.

Demonstrate the use of knowledge.

Only Matter and Molecules receives an excellent rating on this criterion. The unit first shows students how to explain phenomena and then coaches them through a few explanations before turning them loose to explain how the world works.

Earlier in the unit, students read about the two characteristics of a good explanation, and the teacher posts them as criteria to use when judging their own explanations and those of others. When students first begin to practice explaining phenomena, they are reminded of the criteria and asked whether their explanations meet them. As students proceed through the unit, they are reminded less often until they are explaining on their own. No other material was rated even satisfactory on this criterion.

Improving Textbooks

Although Project 2061 does not write textbooks, its goal is to provide guidance for those who do. Project 2061 hopes the detailed information in the reviews not only will guide textbook development in the future but also will be valuable for middle school teachers today. Understandably, schools that are using these texts are bound to be disturbed by these negative evaluations. But teachers should be able to use the explanations in the full reports to start looking for ways to compensate for the texts' shortcomings. Until better texts are developed, for example, schools might consider keeping their current texts and spending their money on professional development to help teachers to supplement the books.

This is the second in a series of Project 2061 textbooks evaluations funded by the Carnegie Corporation of New York. The findings from Project 2061's analysis of middle school mathematics textbooks (see www.project2061.org), released in January 1999, are influencing textbook adoptions across the nation. For more information, visit www.project2061.org. Full reports on each textbook will be available early next year.

---

Key Ideas Used for the Project 2061 Evaluation of Middle Grades Science Textbooks

Physical Science Topic: Kinetic Molecular Theory

Ideas that served as the basis for the analysis were drawn from Chapter 4, Section D of Benchmarks for Science Literacy and from Physical Science Content Standard B of National Science Education Standards:

• All matter is made up of particles called atoms and molecules (as opposed to being continuous or just including particles).
• These particles are extremely small--far too small to see directly through a microscope.
• Atoms and molecules are perpetually in motion.
• Increased temperature means greater molecular motion, so most materials expand when heated.
• Differences in arrangement and motion of atoms/molecules in solids, liquids, and gases:
• In solids, particles (i) are closely packed, (ii) are [often] regularly arranged, (iii) vibrate in all directions, (iv) attract and "stick to" one another.
• In liquids, particles (i) are closely packed, (ii) are not arranged regularly, (iii) can slide past one another, (iv) attract and are loosely connected to one another.
• In gases, particles (i) are far apart, (ii) are randomly arranged, (iii) spread evenly through the spaces they occupy, (iv) move in all directions, (v) are free of one another, except during collisions.
• Explanation of changes of state--melting, freezing, evaporation, condensation, and perhaps dissolving--in terms of changes in arrangement, interaction, and motion of atoms/molecules.
• Connection between increased temperature and increased energy: Increased temperature means greater average energy of motion, so most substances expand when heated.

Life Science Topic: Transformation and Transfer of Matter and Energy

Ideas that served as the basis for the analysis were drawn fromChapter 5, Section E of Benchmarks for Science Literacy and from Life Science Content Standard C of National Science Education Standards:

• Food (for example, sugars) serves as (molecules that provide) fuel and building material for all organisms.
• Plants make their own food, whereas animals obtain food by eating other organisms,
• Matter is transformed in living systems:
• Plants make sugars from carbon dioxide in the air and water.
• Plants break down some of the sugars they have synthesized back into simpler substances--carbon dioxide and water--and assemble some of the sugars into the plants' body structures (including some energy stores).
• Other organisms break down the stored sugars or the body structures of the plants they eat (or in the animals they eat) into simpler substances; reassemble them into their own body structures (including some energy stores).
• Decomposers transform dead organisms into simpler substances, which other organisms can reuse.
• Energy is transformed in living systems:
• Plants use the energy from light to make "energy rich" sugars.
• Plants get energy by breaking down the sugars, releasing some of the energy as heat.
• Other organisms get energy to grow and function by breaking down the consumed body structures to sugars and then breaking down the sugars, releasing some of the energy into the environment as heat.
• Matter and energy are transferred from one organism to another repeatedly and between organisms and their physical environment.

Earth Science Topic: Processes that Shape the Earth

Ideas that served as the basis for the analysis were drawn from Chapter 4, Section C of Benchmarks for Science Literacy and from Earth and Space Science Content Standard D of National Science Education Standards:

• The (seemingly solid) earth is continually changing (not only has it changed in the past but it is still changing).
• Several processes contribute to building up and wearing down the earth's surface.
• The processes that shape the earth today are similar to the processes that shaped the earth in the past (not comparing rates).
• Some of the processes are abrupt, such as earthquakes and volcanoes, while some are slow, such as continental drift and erosion.
• Slow but continuous processes can, over very long times, cause significant changes on Earth's surface (e.g., wearing down of mountains and building up of sediment by the motion of water).
• Matching coastlines and similarities in rocks and fossils suggest that today's continents are separated parts of what was long ago a single vast continent.
• The solid crust of the earth consists of separate plates that move very slowly, pressing against one another in some places, pulling apart in other places.
• Major geological events, such as earthquakes, volcanic eruptions, and mountain building, result from these plate motions.

---

Criteria for Evaluating the Quality of Instructional Support

Category I. Providing a Sense of Purpose

Conveying unit purpose. Does the material convey an overall sense of purpose and direction that is understandable and motivating to students?

Conveying lesson purpose. Does the material convey the purpose of each lesson and its relationship to others?

Justifying activity sequence. Does the material involve students in a logical or strategic sequence of activities (versus just a collection of activities)?

Category II. Taking Account of Student Ideas

Attending to prerequisite knowledge and skills. Does the material specify prerequisite knowledge/skills that are necessary to the learning of the benchmark(s)?

Alerting teacher to commonly held student ideas. Does the material alert teachers to commonly held student ideas (both troublesome and helpful) such as those described in Benchmarks Chapter 15: The Research Base?

Assisting teacher in identifying students' ideas. Does the material include suggestions for teachers to find out what their students think about familiar phenomena related to a benchmark before the scientific ideas are introduced?

Addressing commonly held ideas. Does the material attempt to address commonly held student ideas?

Category III. Engaging Students with Relevant Phenomena

Providing variety of phenomena. Does the material provide multiple and varied phenomena to support the benchmark idea(s)?

Providing vivid experiences. Does the material include activities that provide firsthand experiences with phenomena when practical or provide students with a vicarious sense of the phenomena when not practical?

Category IV. Developing and Using Scientific Ideas

Introducing terms meaningfully. Does the material introduce technical terms only in conjunction with experience with the idea or process and only as needed to facilitate thinking and promote effective communication?

Representing ideas effectively. Does the material include accurate and comprehensible representations of scientific ideas?

Demonstrating use of knowledge. Does the material demonstrate/model or include suggestions for teachers on how to demonstrate/model skills or the use of knowledge?

Providing practice. Does the material provide tasks/questions for students to practice skills or using knowledge in a variety of situations?

Category V. Promoting Student Thinking about Phenomena, Experiences, and Knowledge

Encouraging students to explain their ideas. Does the material routinely include suggestions for having each student express, clarify, justify, and represent his/her ideas? Are suggestions made for when and how students will get feedback from peers and the teacher?

Guiding student interpretation and reasoning. Does the material include tasks and/or question sequences to guide student interpretation and reasoning about experiences with phenomena and readings?

Encouraging students to think about what they've learned. Does the material suggest ways to have students check their own progress?

Category VI. Assessing Progress

Aligning assessment to goals. Assuming a content match between the curriculum material and this benchmark, are assessment items included that match the same benchmark?

Testing for understanding. Does the material include assessment tasks that require application of ideas and avoid allowing students a trivial way out, like using a formula or repeating a memorized term without understanding?

Using assessment to inform instruction. Are some assessments embedded in the curriculum along the way, with advice to teachers as to how they might use the results to choose or modify activities?

Category VII. Enhancing the Science Learning Environment

Providing teacher content support. Would the material help teachers improve their understanding of science, mathematics, and technology necessary for teaching the material?

Encouraging curiosity and questioning. Does the material help teachers to create a classroom environment that welcomes student curiosity, rewards creativity, encourages a spirit of healthy questioning, and avoids dogmatism?

Supporting all students. Does the material help teachers to create a classroom community that encourages high expectations for all students, that enables all students to experience success, and that provides all students a feeling of belonging in the science classroom?

###

About the authors: Jo Ellen Roseman, Project 2061's curriculum director, led the middle grades science textbook evaluation with assistance from senior research associate Sofia Kesidou and research associates Luli Stern and Ann Caldwell.

###

---

Roseman, J. E., Kesidou, S., Stern, L., Caldwell, A. 1999. Heavy Books Light on Learning: AAAS Project 2061 Evaluates Middle Grades Science Textbooks. Science Books & Films, 35 (6).