| |
 |
Policy & Student Learning: What Textbooks, Assessment, and Professional
Development Can Contribute
A Conference Report
Drafted by Andrew Porter
University of Wisconsin–Madison
September 12, 2002
[Return to the conference
documents]
Introduction
In its continuing effort to significantly improve student learning in science,
mathematics, and technology, Project 2061 of the American Association
for the Advancement of Science hosted a conference on state and
district policies
that influence student learning May 15-17, 2002, in Washington,
D.C. The conference,
the third in a series
of Project 2061 conferences dedicated to improving science and mathematics
textbooks and curriculum, examined policies that affect the quality
of instructional materials, student assessment, and teacher professional
development.
Jo Ellen Roseman, acting director of Project 2061, opened the conference. She
noted that, although the past 2 decades of research have produced some curriculum
materials and examples of teaching focused on helping students understand
important science and mathematics ideas, the scaling up of these effective
materials and practices has been difficult. With regard to curriculum
materials, recent evaluations have shown that in science—unlike mathematics,
where several effective research-based textbooks series have been developed
and are being implemented on a national scale—very little has been done
to incorporate the best that research and experience have to offer.
Furthermore, student assessments, which are becoming increasingly important
in science, may not align well with what states and districts agree are important
ideas for students to learn.
The conference focused on five case studies that shed light on state and district
policies that contribute to students’ learning important ideas in math
and science. The case studies addressed the selection and implementation
of curriculum materials, the role played by state assessment instruments,
and the professional development available to teachers. Discussion attempted
to identify patterns across the case studies that can inform state and district
efforts to improve student learning.
Dr. David K. Cohen, Professor of Education and Public Policy at the University
of Michigan, gave the keynote address, presenting findings from his recent
book, Learning Policy: When State Education Reform Works (coauthored
with Heather Hill). Gary Sykes, Professor of Education at Michigan State
University, chaired the conference and provided the closing analysis and summary.
[TOP]
Conference Attendees
The conference was attended by 81 people representing a rich mixture of groups
knowledgeable about and having a stake in math and science textbooks, assessments,
and professional development. Approximately 20% of the attendees came
from school district staff, and another 20% from higher education. There
were about 10% from each of the following groups: textbook publishers,
education professional organizations, state departments of public instruction,
the U.S. government, technical assistance
centers, and research houses. There were a couple of teachers, and a
principal as well.
[TOP]
Intervening in Instruction to Enhance Teaching and Learning
In his keynote address, Cohen stressed the importance of research that shows
connections between policy and student achievement and gives careful attention
to the nature and quality of instruction as an intervening variable.
He emphasized that teachers’ responses to curriculum policies (as evidenced
in their instructional practices) are filtered through their knowledge and
norms. Surprisingly, relatively little research on the effects of policies
has included good measures of teachers’ content knowledge and pedagogical
content knowledge. Yet these two sets of variables may offer strong
explanations for how policies lead to instructional practice. More work
needs to be done to operationalize both teacher content knowledge and teacher
pedagogical content knowledge and to develop adequate measures of these.
Cohen listed curriculum materials, assessments, professional opportunities
to learn, and incentives as instruments of practice. He characterized
these instruments as working in concert to produce desired change, concluding
that curriculum materials alone are unlikely to succeed in creating the types
of instruction sought by today’s reforms.
Cohen concluded that teachers’ knowledge of science and math is modest,
school managers know even less, teachers and managers both receive conflicting
guidance about what should be taught and how, and generally professional development
is weak. He concluded that tests, academic standards, and accountability
by themselves rarely influence practice in the directions and to the extent
hoped.
Cohen sketched a more hopeful picture of California's math reform from 1985
to 1995 (the reform investigated in his book with Heather Hill). He
described the California reform as consisting of a math framework, a new state
test matched to the framework, new curriculum materials aligned with the framework,
and professional development aligned with both the framework and the test.
The curriculum materials were replacement units, each covering a single topic
and lasting 4–6 weeks. Cohen described the replacement units as
grounded in practice, supported with aligned professional development, and
fitting together so that as a set they aligned with the state framework.
Cohen and Hill surveyed 600 elementary school teachers to investigate teachers’
opportunity to learn, their instructional practices, their school conditions,
and their students’ achievements. Cohen and Hill found positive effects
on student achievement from replacement units, teachers’ scoring of
performance items on the state test (the California Learning Assessment System,
or CLAS), and professional development on replacement units. The major
distinction between the successful California approach and the general approach
described earlier by Cohen was California's provision of aligned and curriculum-embedded
professional development.
Cohen ended his talk where he began, calling for more research and evaluation
on the effects of reforms on student achievement mediated by instructional
practices. He urged researchers to undertake research on the alignment
among standards, assessments, curriculum materials, and professional development.
Cohen further advocated that professional associations should play a larger
role in reform so that education becomes less dependent on the government.
[TOP]
Alignment
Four interactive sessions at the conference provided a look at the AAAS Project
2061 procedures for judging alignment between (a) national and state science
and mathematics standards (Kathleen Morris and Lori Kurth), (b) curriculum
materials and standards (Linda Hackett and Ann Caldwell), and (c) assessments
and standards (Leah Bricker). All served to highlight the importance
of examining alignment to specific learning goals in standards (rather than
to just their general topic). Since both benchmarks and standards specify
a carefully and deliberately chosen subset of knowledge and skills on any
particular topic, judgments about alignment should only be made with this
subset in mind. At the same time, it is important when assessing alignment
to recognize any content in the assessments or materials that is not in the
standards as well as content in the standards but not in the assessments or
materials.
These four interactive sessions were useful in clarifying Project 2061 procedures
for judging alignment. Exercises were conducted in a small-group format,
allowing all participants to have firsthand experience with attempting to
judge alignment among standards, assessments, and materials. Group discussion
identified differences in judgment and helped all to better understand how
difficult alignment can be to achieve.
Not only was alignment discussed during the four interactive sessions, but
alignment was a major focus of the entire conference. Conference presentations
and discussions went a long way toward clarifying what is meant by alignment,
how alignment should be judged, who should do the judging, and how the results
should be used. In addition to the Project 2061 procedures for judging alignment,
recognition was given to methodologies developed by Achieve, Inc.; and the
Council of Chief State School Officers, working with the National Institute
for Science Education.
Conference discussions made clear that alignment remains for many a somewhat
elusive concept, notwithstanding its essential role in standards-based reform.
Uniformity does not exist in the criteria used for judging alignment, the
objectivity and replicability of the procedures followed, policies on use
of third parties to make judgments, or the uses to which alignment is put.
There remain many questions for which consensus answers are net yet available.
For example, is it important to have a quantitative index of alignment? Since
alignment exists in degrees, how much is enough? When can a state say its
tests are sufficiently aligned to our content standards? Is it necessary
to look at alignment between assessments and standards not only from the perspective
of the assessment (i.e., is everything on the test in the content standards?),
but also from the perspective of the content standards (i.e., is everything
in the content standards assessed?)? When judging alignment, what “grain
size” is most useful? If alignment must be an exact match between,
say, a test item and a standard so that each is virtually identical, then
no assessment will be judged to be aligned (very small grain size). On the
other hand, if one only asks if a test item is about mathematics and the standard
is about mathematics, then every test would be found to be aligned (very large
grain size). Generally, participants agreed that questions of alignment of
assessments to standards should not be left to test publishers because of
their vested interest in marketing their products.
[TOP]
Case Studies
El Centro, California
(Presenter: Olga Amaral, San Diego State University)
The El Centro reform initiative addresses science reform in kindergarten through
sixth grade and involves a local and regional K-16 collaborative partnership
among districts, business, volunteer scientists, and the Imperial Valley Campus
of San Diego State University. The El Centro reform is comprehensive,
encompassing (a) curriculum, (b) professional development, (c) materials support,
(d) administrative and community support, and (e) assessment and evaluation.
Located in the extreme southwest corner of California, Imperial County is the
poorest of all 58 counties in the state, with 30% unemployment and a mean
income of approximately $16,000. Of the total K-16 student population,
73% receive free or reduced-price lunch, 51% are designated English language
learners, and 81% are Hispanic.
The science curriculum focuses on life, earth/space, and physical science,
with three to four science units covered each year at each grade level.
The curriculum units are designed to be (a) developmentally appropriate, (b)
research-based, and (c) focused on “big ideas.” The units give
students the opportunity to explore, investigate, inquire, question, test
hypotheses, and collect and analyze data.
A sustained professional development program involves over 100 hours across
5 years, along with both preservice and master’s degree components.
Inservice professional development is embedded in the content of the science
units and includes a lesson lab component.
The reform project’s administrative and community support component offers
administrator training on content, pedagogy, classroom supervision, teacher
evaluation, and assessment. A volunteer scientist initiative trains
volunteers to serve as content consultants, validate the curriculum, model
questioning, model inquiry, and become community advocates. The project
includes a parent education initiative, as well.
Evidence of the reform’s effects can be seen in student achievement.
The project has compared the SAT-9 science scores of participating and nonparticipating
classes and of individual students based on their cumulative years in the
program. In each case, the program has been found to yield strong positive
effects. Admittedly, student stability is confounded with years in the
program, and there is no randomization of participation versus nonparticipation.
Still, the results in science are encouraging, and there are similar positive
results for reading, math, writing proficiency, and science notebooks.
The writing and reading improvements are likely a function of the reform’s
science literacy components. The project’s effects on mathematics
scores are less easily understood.
The El Centro reform is highly focused and ambitious, with an impressive, comprehensive
set of program components.
Baltimore
(Presenters: Patricia Campbell, University of Maryland; Melva Greene
and Andrea Bowden, Baltimore City Public Schools)
The Baltimore reform focuses on kindergarten through fifth-grade mathematics,
drawing in part on National Science Foundation urban systemic and local systemic
initiative funding and in part on special state funding. The collaboration
among the Baltimore City Public Schools, the University of Maryland–College
Park, Morgan State University, and the Maryland State Department of Education
has undertaken an ambitious and comprehensive approach to reforming elementary
school mathematics.
The Baltimore City Public School System serves approximately 50,000 students.
In 122 elementary schools, 74% of the students receive free or reduced-price
lunch, and 87% are African American. Scores on student achievement tests
have been low historically relative to state and national norms.
When the reform began in 1995, the mathematics program was diagnosed as exhibiting
“curriculum overload,” too many objectives, lack of uniformity
in instructional materials, too little and poor-quality professional development,
and conflicting curriculum policies. Six years of reform have resulted
in (a) a standards-based K-5 mathematics curriculum; (b) one districtwide
textbook aligned to the curriculum; (c) an instructional model that shifts
instruction from a show-and-tell demonstrated practice approach to an approach
based on asking questions and questioning answers (called Mathematics: Applications
and Reasoning Skills, or MARS); (d) an aligned program of professional development,
with a requirement of at least 100 hours for each teacher; (e) instructional
leaders in mathematics in two thirds of the elementary schools; and (f) systemwide,
grade-level assessments, consisting of 15 assessments aligned with the curriculum
and administered and scored by teachers.
Since 1998, Baltimore district student achievement medians on national percentiles
are up at every grade level, both for concepts and applications and for computation.
Increases are dramatic on the Comprehensive Test of Basic Skills (CTBS), but
improvement is also seen on the Maryland School Performance Assessment Program
(MSPAP) test.
Clearly, the Baltimore reform made excellent use of additional funding from
NSF and the state. The reform initiative also profited from excellent
leadership both at the university level and in the district.
Pittsburgh
(Presenter: Diane Briars, Pittsburgh Public Schools)
The Pittsburgh reform focuses on mathematics in Grades K-12, though the case
presented was limited to Grades K-8. Like other reform initiatives described
at the conference, the Pittsburgh reform is multifaceted; like the Baltimore
effort, the Pittsburgh initiative received some additional funding from NSF.
Beginning in 1990 with kindergarten, the initiative has phased in new instructional
materials for an additional grade each year. In elementary school, the
curriculum materials are Everyday Mathematics; at middle school,
Connected Mathematics. Assessments are the Iowa Test of
Basic Skills (ITBS) and the world-class New Standards. Student performance
is reported by performance level. There is a curriculum-embedded and
sustained professional development program for both teachers and administrators,
delivered in part by school-based resource teachers. A parent component
attempts to elicit parental support and involve parents in the education of
their children. Accountability is at the school and teacher level.
The Pittsburgh Public Schools serve approximately 40,000 students, with 59
elementary schools, 19 middle schools, and 11 high schools. Of the Pittsburgh
student population, 56% are African American, and 62% qualify for free or
reduced-price lunch.
Leaders of the Pittsburgh reform conclude that the key barrier to successful
implementation is lack of teacher content knowledge. Their vigorous
professional development program includes in-class support using resource
teachers and professional development embedded in standards-based instructional
materials. The Pittsburgh reform puts an unusually heavy emphasis on
measuring the quality of reform implementation in the classroom.
Effects on student achievement are seen through pre-reform/post-reform comparisons,
as well as comparisons between classrooms with a strong reform implementation
and those with a weaker implementation. For all grade levels and on
both New Standards and ITBS, the comparisons have shown improved achievement
in favor of the reform effort.
Once again, the importance of external funding can be seen. Also, the
reform was led by highly qualified, charismatic leaders and put an emphasis
on instructional leadership at both the district and school principal levels.
Indian River School District, Delaware
(Presenters: Linda Bledsoe, Cedar Lane Elementary School, Middleton,
Delaware,and Jon Manon, University of
Delaware)
In 1995, the Indian River School District decided that, although its elementary
school mathematics achievement compared favorably to the state average, achievement
in the district was not good enough. In particular, the district wished
to address the achievement gap between African American and White students.
Reform planning began in 1995; by 1998, the district had adopted the NSF-funded
curriculum Math Trailblazers. With the new curriculum materials
as its lead policy instrument, Indian River built a system of reform with
a heavy emphasis on professional development. All elementary mathematics
teachers participated at least monthly in a professional development program
known as the Math Club. The initiative created a cadre of lead teachers
to provide school-based leadership, engaged a district mathematics specialist
to provide district-level leadership, and sought additional funding through
grants. Other components of the reform include curriculum-embedded
assessment, monitoring of individual student performance to ensure that no
child falls behind, increased instructional time in mathematics, and parents’
involvement in support of the reform and their own children’s learning.
Strategically, Indian River seeks to create increased teacher responsibility
for student learning. More generally, the district wishes to shift its
focus from teaching to student learning. The curriculum is ambitious,
aligns with state standards, provides vertical articulation of “big
ideas,” focuses on problem solving and communication, and achieves a
balance between conceptual understanding and skills.
Performance on the state mathematics test has been impressive, with steady
increases from 1998 through 2001 at both third- and fifth-grade levels.
Student achievement, disaggregated by race, shows steady reductions in the
achievement gap from 1998 to 2001.
The program is now being replicated in eight other Delaware school districts.
The success of these replications is not yet known.
Battle Creek Area Mathematics and Science Center, Michigan
(Presenters: Theron Blakeslee, Connie Duncan, Chris Lapekas, and Barry
Linscott)
The state of Michigan has regional mathematics and science centers, one of
which serves the Battle Creek area. The state has also taken on a leadership
role in reforming K-12 science education. The case study presented at
the conference described the collaborative relationships between the Battle
Creek center and the state, on the one hand, and districts and schools drawing
on the center’s expertise, on the other.
Since 1989, with Project 2061’s publication of Science for All
Americans, the Michigan Department of Education has been pursuing a
new framework for science education. This ambitious framework was revised
in 2000. State tests in Grades 5, 8, and 11 are aligned to the framework.
These tests, required of all students, are demanding: just under 50% of 5th
graders pass, 20% of 8th graders pass, and 60% of 11th
graders pass. Student incentives are tied to performance on the state
test, with a $2,500 college scholarship available to students who pass the
11th-grade tests in all subjects and a $500 award available to
students who pass the 8th-grade tests.
Recognizing that assessment alone cannot bring about deep and meaningful reform
at the classroom level, Michigan developed a number of curriculum units in
science, including Chemistry That Applies and Food, Energy,
and Growth. These units are made available to anyone in the state
who wishes to use them.In addition, a number of the regional math
and science centers have developed complete unit-based curricula for the elementary
grades.Over 100 districts across the state use the New Directions
Teaching Units distributed by the BattleCreek area center. The motivation
for local development of science curriculum units is the inability to find
off-the-shelf materials that are sufficiently well aligned to the state framework
and test.
Science reform in Michigan is also supported through professional development.
Again, the regional math and science centers play an important role, with
state funding augmented by Eisenhower and NSF funds. The emphasis is
on curriculum-embedded professional development.
The Battle Creek area math and science center reforms are working best in schools
with strong principal leadership. Another key to successful reform appears
to be careful record keeping relating to teacher coverage of state science
framework material. Finally, the use of outside observers to assess
implementation has proved useful.
The effects of the Battle Creek reform on achievement are not well documented.
Nevertheless, when a cluster of Michigan schools—the Michigan Invitational
Group—participated in the Third International Mathematics and Science
Study–Repeat (TIMSS-R), students scored extremely well.
[TOP]
Assessment
The last conference session brought together a panel of three to talk about
science assessment and particularly issues of alignment of science assessments
to science content standards: Virginia Malone, from Harcourt Publishing;
Edward Smith, from Michigan State University; and David Potter, from the South
Carolina Education Oversight Committee.
The challenges of building assessments aligned to content standards are sharply
different for a publisher and a state. The Michigan educational assessment
of progress in science, developed in partnership with Michigan State University,
provides one example of how a state can build aligned assessments. At
least two aspects of the Michigan assessment are innovative and merit consideration
by others:
- Use of both framework-wide and area-specific assessment.
Michigan's framework-wide assessment covers a representative sample of
objectives from the state framework. Area-specific assessment provides
a more in-depth assessment of a small set of closely related objectives.
- Assessment of student skills in investigation. In the month
prior to state testing, all science classrooms in Michigan conduct the
same investigation. The state then includes questions about that
investigation in its assessment. This type of curriculum-embedded
state assessment has great potential—not only for upgrading the
enacted curriculum statewide, but also for allowing an on-demand paper-and-pencil
test to assess students’ understanding of how to carry out and interpret
a science experiment.
Michigan hopes to convey to teachers that intensive work on a particular topic
pays off in student achievement and that students need to be writing in science,
doing investigations, and engaging in critical reasoning based on science
source materials. By taking innovative approaches to assessment, Michigan
has ensured that some of the more difficult-to-assess content in its framework
is being tested.
South Carolina offers another example of a successful approach to a state building
science assessments aligned to their content standards. South Carolina
is particularly aggressive in analyzing the alignment of its assessments to
its standards. Alignment is a key consideration at every step of the
test preparation process, from test construction to field testing to reporting
and interpretation of student performance. South Carolina believes that
its work on alignment has improved the quality of its assessment and accountability
system.
But many states lack the expertise and funds to develop their own assessments.
Such states rely on test publishers to supply an assessment product that meets
their needs. At the same time, a test publisher must have a product
that is both affordable and useful. Typically this has meant developing
one set of assessments per academic subject area for a nationwide market.
Virginia Malone represented the publishing industry in explaining how her
company copes with the challenges between the need for science assessments
in each state that are aligned to the state standards and the need to market
nationally assessments that are of high psychometric quality yet affordable.
While the challenges of marketing an assessment nationwide yet aligned to
each state’s content standards are challenging in any academic subject,
they are particularly challenging in science, where a very broad range of
content might be taught and where prerequisites are often not well established.
Malone stated that the test publishing industry is finding new ways to partner
with states to provide assessments aligned to state standards. With
the No Child Left Behind legislation’s call for science assessment aligned
to state standards, the challenges will surely become more pressing over the
course of the next few years.
[TOP]
Gary Sykes’ Summary
At the conclusion of the conference, facilitator Gary Sykes summarized what
he had learned from the presentations and discussions. For him, there
were two “stories” running through the conference—one elaborating
on the nature, the complexities, and the measurement of alignment; and the
other exploring the role of instructional materials and policies in determining
what happens in schools.
A commitment to standards-based reform served as the context for Sykes’
remarks. Standards-based reform—originally thought to be technically
easy but politically difficult—has turned out, according to Sykes, to
be both technically and politically difficult. A core problem appears
to be the desire to cram too much content into the curriculum, resulting in
standards that are too broad and textbooks that are too fat. Still,
Sykes concluded from the case studies presented at the conference that standards-based
reform can work. He urged conference attendees to continue to pursue
such reform, not only because success stories exist, but also because the
alternatives of school choice and privatization seem even more problematic.
Moreover, as standards-based reform is pursued, the focus should be kept squarely
on the education of children from low-income families.
According to Sykes, the success of standards-based reform depends on doing
a number of things well. In particular, the failure to make any of the
following a strong part of a standards-based reform effort represents a fatal
flaw:
- Alignment. According to Sykes and as seen in the case studies,
alignment is complicated and difficult, requires considerable interpretation,
yet must be understood by both teachers and principals. Alignment
is needed (a) among the instruments of policy (e.g., standards, assessments,
instructional materials, professional development) and (b) between state
instruments of policy and instruction as delivered in the classroom.
- Professional development. Professional development must
be embedded in aligned curriculum materials. The professional development
should focus on content and the manner in which students learn content,
teachers’ use of curriculum materials in classrooms, and the involvement
of teachers in helping other teachers. Duration of professional
development must be sufficient (both in contact hours and across months).
Sykes contended that there is much too much low-quality professional development
in K-12 math and science, and he called for a moratorium on all weak professional
development.
- Time. Sykes expressed concern that we are trying to incorporate
too much content into too little time, especially at the middle school
and high school levels, and that at the elementary school level, we are
devoting too little time to science and perhaps even mathematics.
- Leadership. The Baltimore and Pittsburgh case studies brilliantly
captured the value of leadership at both district and school levels.
- External assistance. From the case studies, Sykes concluded
that help from a local university was a common ingredient to success.
- Funding. In all of the cases presented, the reforms were
supported in part by “extra dollars,” money beyond the school’s
and district’s regular budgets. In particular, NSF leadership
and support were present in virtually every case. Thus, it appears
that outside funding is a key ingredient to success. On the other
hand, Sykes concluded that money is available, and so perhaps may not
really be a problem.
- Strategy. Each successful case laid out a strategy in advance,
typically in the form of a 5- to 10-year plan. Strategies involved
careful selection of policy instruments and a thoughtful approach to phasing
them in over time. However, Sykes noted one issue without a definitive
answer—namely, how scripted strategies should be for teachers.
He also noted that, although research and development were a part of each
strategy, there is a need for higher quality evaluations against student
achievement data, a concern expressed at the beginning of the conference
by David Cohen and recognized as a shortcoming in virtually every case.
- Politics. In Sykes’ words, standards-based reform
should not be “too avant-garde.” He worried about a
possible backlash to “fuzzy math.” He also worried about
the difficulties of a revolving door in district superintendent and school
board positions. Sykes favored standards-based reform that is focused
on content, not too controversial, and supported by stable leadership.
Stable funding may be an issue as well.
As a final thought, Sykes observed that high school reform was missing from
the conference agenda.
[TOP]
In Closing
The case studies presented at the conference make clear that standards-based
reform, to be successful, must go well beyond ambitious content standards
and high-quality, aligned assessments. High-quality, aligned curriculum
materials must also be readily available. Professional development must
be embedded in the curriculum materials, require a significant and sustained
investment of time by teachers, and use teachers to teach teachers.
Leadership was seen as essential in the conference case studies, at both district
and school levels. External agents played an important role, as well;
the National Science Foundation may have been the impetus for many, if not
all, of the reforms, and certainly provided some of the funding. Finally,
universities typically played an important collaborative role, providing expertise
and support.
There were a number of things missing from the cases, as well. First,
although one should be careful when generalizing from so few cases, mathematics
was more visible than science. Similarly, elementary school reform was
more visible than high school reform. A parent component was mentioned
in many of the cases but seemed poorly planned and weak. Not one mention
was made of comprehensive, research-based, whole-school reform. Surprisingly
little was said about accountability: There was some evidence of accountability
for students sometimes and for schools and teachers at other times, but accountability
was not built into the reforms in the way called for in the No Child Left
Behind legislation. One cannot help wondering how these reforms will
adjust to the new Title I requirements. Finally, although every case
had a research and evaluation component, none was as strong as one would wish.
Comparison groups were often lacking, and baselines were not well established
for across-cohort trend data. Longitudinal student data was nonexistent.
Analyses were descriptive; the reform initiatives did not include sufficient
controls to provide convincing arguments of cause and effect.
Although much remains to be done, participants left the conference clearer
about the future agenda, excited about what can be accomplished, and inspired
by the importance of the work.
[TOP]
|
