I. Exploration

Before evaluating an assessment item, reviewers should first become familiar with the learning goal to which the item is targeted.  They should carefully read the content standard or benchmark and any clarification materials that are provided.  They should also become familiar with the assessment task and identify the specific knowledge and skill that students need to have in order to respond successfully.  These initial observations will be used later in the analysis of the item.

In addition to the learning goal and the assessment task, materials provided may include:

• Clarification statements that describe in more detail what students are and are not expected to know.
• Assessment maps that show the relationship between the targeted learning goal and other learning goals, especially earlier and later learning goals.
• Summaries of research on student learning that describe difficulties students may have as well as misconceptions and other ideas they may hold. 

IA. Exploring the Learning Goal and Related Materials. More Details

1. Is there anything you disagree with or do not understand in the learning goal, the clarification statement, the map, or the description of misconceptions from the research on student learning? Suggest changes here.
   

IB. Exploring the Assessment Task. More Details

1. Describe the multiple strategies (if any) that can be used to complete the task. More Details
   One example of multiple strategies would be using the information in the stem OR the information in a provided diagram.  “Eliminating distractors” (vs. identifying the correct choice), or “intuition” do not qualify as different strategies under this field.  Furthermore, this field should be reserved for cases where there are multiple strategies within the same learning goal(s).  If an additional strategy requires different goals, then don’t enter anything here; make the comment under Necessity/Sufficiency.
   
   
2. State the knowledge needed to select or eliminate each answer choice. As much as possible, state the knowledge that is needed in terms of a general principle, not in terms of the instance being referred to in the item.
   
   
3. What common misconceptions might affect how a student answers?  For each answer choice indicate the misconceptions that might affect how a student answers.
   
4. What general cognitive abilities beyond recall of scientific knowledge are needed to successfully complete this task? More Details

   Examples include predicting, explaining, analyzing, comparing, evaluating, inferring, interpreting a representation.

   
   
5. What mathematical knowledge and skills are needed to successfully complete this task? More Details
   Examples include understanding concepts such as probability, interpreting graphs and tables, or using various computational skills.
   
   
   
   
   

In this section, the alignment of the task content to the targeted learning goal content is evaluated.  Two criteria, Necessity and Sufficiency, are used.  The necessity criterion addresses if the learning goal is needed to complete the task, while the sufficiency criterion addresses if the learning goal is adequate by itself to complete the task.

II. A. Necessity More Details

Using your responses in the Explorations section as a guide, check all applicable boxes and provide explanations as appropriate.

1. Evaluation of every answer choice requires knowledge of the ideas specified in the learning goal and supporting materials. State the part of the learning goal that is needed for each answer choice.
   
2. Evaluation of some of the answer choices requires knowledge of the ideas specified in the learning goal and supporting materials. State the part of the learning goal that is needed for each of those answer choices.
   
3. Evaluation of none of the answer choices require knowledge of the ideas specified in the learning goal and supporting materials.
   
4. Evaluation of the answer choices may come from having learned a specific instance of the general principle (SIGP) being tested. Explain
   
   When testing a student’s understanding of a general principle, it is sometime difficult to distinguish knowledge of the general principle for knowledge of the particular instance used in the assessment task. For example, suppose the learning goal contains the general principle, “all bodies attract all others gravitationally”, and the assessment task asks about the attraction between two particular objects, the earth, and the moon. In this case we cannot be sure that the student understands the general principle. He/she may have learned the specific fact that gravity is the force of attraction between the earth and the moon without knowing that all bodies attract all others gravitationally. When is likely that a student could know the specific instance without knowing the generalization, the corresponding box should be checked and an explanation about why you think it is likely that knowledge of the particular instance that appears in the assessment task is all that is needed and not knowledge of the general principle that is being assessed should be provided. (SIGP (Necessity) scoring instructions are given below.)

Score select value 1 2 3 4 5 NA Guidelines

Scoring Necessity of Multiple-Choice (4-Choice) Items: Score 1 point for each distractor that can be evaluated using the targeted Idea, and 2 points if the correct answer can be evaluated using the Idea. Note that each answer choice should be evaluated individually. By evaluated, we mean that the learning goal can be used to determine the truthfulness and content relevance of that answer choice.

SIGP Penalty for Necessity: First score the item as described in the table below. Then apply an additional penalty of 2, 1, or 0 points depending on the extent to which knowledge of the SIGP enables students to complete the item without knowing the targeted learning goal.

If the learning goal is needed to evaluate...	score
All four answer choices	5
Two distractors and the correct answer	4
Three distractors	3
One distractor AND the correct answer	3
Two distractors	2
The correct answer	2
One distractor	1
Zero distractors	1

Scoring notes:

Using your responses in the Explorations section as a guide, check all applicable boxes and provide explanations as appropriate.

1. Evaluation of each of the answer choices does NOT require any more knowledge of skill than is specified in the learning goal or that can be assumed as general knowledge and ability of students this age.
   
2. Successful evaluation of some or all of the answer choices requires:
   

1. more knowledge or skill than is specified in the learning goal or that can be assumed as general knowledge and ability of students this age.  Specify the additional knowledge or skill that is needed to evaluate each answer choice. 
   
   
   The assessment task may require more advanced knowledge than is stated in the learning goal.  For example, the task might require a molecular explanation of a phenomenon (e.g., evaporation, chemical reactions) whereas the learning goal itself specifies a substance explanation rather than a molecular explanation.  Students may very well know the exact ideas or skills in the learning goal (the substance explanation) but not the more advanced knowledge or skills required by the assessment task.  If more advanced ideas and skills are required, a brief explanation should be given in the box below.
   
   Sometimes successfully completing a task requires the application of mathematical knowledge and skill even though it is not stated in the learning goal.  This may involve computation, graphing, solving equations, or reading tables and charts that contain numerical information.  It is not necessarily true that because a mathematics idea is taught in an earlier grade, students will have learned it well enough to use it in new contexts.  It is often difficult for students to apply skills that they have learned in a mathematics class to problems in science.  If there is any question about whether students should generally be expected to have the mathematics skills needed to successfully complete the task, it should be noted below.
2. familiarity with science or mathematics terminology that may be unfamiliar to students.  Specify the specific terms.
   
   In some cases, successful completion of a task requires understanding the meaning of technical science or mathematics vocabulary. If the terminology used in the assessment task goes beyond that which is used in the learning goal itself, students may very well understand the ideas or skills specified in the targeted learning goal but respond unsatisfactorily because they do not understand the technical vocabulary used in the task.
3. knowledge of specific instances of the general principle (SIGP) that all students are not expected to have.  Explain.
   
   The student may have some familiarity with a particular piece of information, but may not know that it is a specific instance of the general principle (SIGP) relevant to the item. For example, an assessment task might require the student to choose an animal that licks salt. The student may have some familiarity with deer, and may understand licking salt, but may not know that deer lick salt. There are no additional scoring instructions for the case of SIGP (Sufficiency): follow the table below.
4. familiarity with specific representations or models of ideas in the learning goal that all students are not expected to have.  Explain.
   
   Does the task require the student to be familiar with specific representations or models of ideas in the learning goal that all students are not expected to have. For example, the strength of a force on an object can be represented by the length or the width of the arrow.
5. Other.  Provide a brief explanation.
   

Score select value 1 2 3 4 5 NA Guidelines

Scoring Sufficiency for Multiple-Choice (4-Choice) Items: Score 1 point for each distractor that can be evaluated using the targeted Idea, and 2 points if the correct answer can be evaluated using the Idea.

If the learning goal is sufficient to evaluate...	score
All four answer choices	5
Two distractors and the correct answer	4
Three distractors	3
One distractor AND the correct answer	3
Two distractors	2
The correct answer	2
One distractor	1
Zero distractors	1

Scoring notes:

This section focuses on features of the assessment task that could reduce the likelihood that students’ responses will provide information about what they know and do not know.  Students should not get a question wrong if they understand the targeted idea and they should not get it right if they do not.  Three criteria are provided: comprehensibility, appropriateness of task context, and resistance to test-wiseness.  Additionally, a section on cost effectiveness, in which the reviewer reflects on the amount of time needed to complete the task, is present.

III. A. Comprehensibility More Details

Check all applicable boxes and provide explanations as appropriate.

1. It is not clear what question is being asked.  Explain.
   
2. The task uses unfamiliar vocabulary that is not clearly defined, or words or phrases that have unclear, confusing, or ambiguous meanings.  Explain.
   Some words, although correct, may lead students astray.  For example, in the phrase “scientific finding,” the word “finding” has a special meaning in science that may be unfamiliar to students.  Note all places where words, both general and scientific, do not have clear and straightforward meanings.
   
   
3. The task uses unnecessarily complex sentence structure when plain language could have been used, or ambiguous punctuation that make the task difficult to comprehend.  Explain.
   
4. There is inaccurate information in any part of the task (including diagrams and data tables) that may be confusing to students who have a correct understanding of the science. Explain.
   
5. There are comprehensibility issues with the diagrams, graphs, and data tables.  Explain.
   
6. There is more than one correct answer. Explain
   
7. Other.  Provide a brief explanation.
   

Score select value 1 2 3 4 5 NA Guidelines

Scoring Comprehensibility for Multiple-Choice (4-Choice) Items
Problems in the stem should be weighted more than problems in the answer choices. The reviewer is expected to use his/her professional judgment regarding the overall effect on comprehensibility.  For cases where a part of the item is found to be incomprehensible to students then the following scoring scheme is suggested:

If box 6 is checked (i.e., there is more than one correct answer), subtract one point for each answer choice that can be considered correct.

Students will NOT be able to evaluate:	Score
Zero answer choices	5
One answer choice	4
Two answer choices	3
Three answer choices	2
The stem	1
All four answer choices	1

Scoring notes:

Check all applicable boxes and provide explanations as appropriate.

• This task has no context.  Check here and skip to the next section.
• The task makes use of a context.

1. The context may be unfamiliar to most students. Explain.
   
2. The context may advantage or disadvantage students because of their interest or familiarity with the context. Explain.
   
3. The context is complicated so that students might have to spend a lot of time trying to figure out what the context means. Explain.
   
4. The information and quantities that are used are not reasonable or believable. Explain.
   
5. The context does not accurately represent scientific or mathematical realities or make clear when idealizations are involved. Explain.
   
6. Other. Provide a brief explanation.
   

score select value 1 2 3 4 5 NA Guidelines

Scoring the Appropriateness of Task Context for Multiple-Choice Items: Deduct 1 point (from a total of 5 points) for each statement that is checked, if the student's response would be affected by it.

Scoring notes:

III. C. Resistance to Test-Wiseness More Details

Check all applicable boxes and provide explanations as appropriate.

1. Some of the distractors are not plausible. Explain.
   
   
2. One of the answer choices differs significantly in length or contains a different amount of detail from the other response options. Explain.
   
3. One of the answer choices is qualified differently from the other response options, and/or uses giveaway words such as “usually” or “never.” Explain.
   
   
4. The use of logical opposites leads the student to eliminate the other answer choices.  Explain.
   
5. One of the answer choices contains vocabulary at a different level of difficulty from the other response options. Explain.
   
6. The language in one of the answer choices mirrors or is obviously closely related to the language in the stem. Explain.
   
7. Other.  Provide a brief explanation.
   

score select value 1 2 3 4 5 NA Guidelines

Scoring Resistance to Test-Wiseness for Multiple-Choice (4-Choice) Items: Deduct 0-1 points (from a total of 5 points) for each distractor, and 1-4 points for the correct answer choice, if they can be evaluated (successfully or unsuccessfully) through test-wiseness. The table below gives scoring ranges for some possible situations. The ranges are intended to allow the reviewer to use his/her professional judgment regarding the overall effect on test-wiseness. In particular, the large range associated with the correct answer choice allows the reviewer to assess a severe penalty, if the person can select the correct answer without any content knowledge. Note: Successfully evaluating the correct answer choice means identifying it as the correct answer. Successfully evaluating a distractor means identifying it as a wrong answer. Unsuccessfully evaluating the correct answer choice means thinking that it is a wrong answer, and unsuccessfully evaluating a distractor means thinking that it is the correct answer.

	Score
No test-wiseness issues	5
One distractor can be eliminated using test-wiseness methods	4-5
Two distractors can be eliminated using test-wiseness methods	3-5
Three distractors can be eliminated using test-wiseness methods	1-4
The correct answer can be determined using test-wiseness methods	1-4

Scoring notes:

III. D. Cost Effectiveness More Details

Check the applicable box and provide explanations as appropriate.

• The item does not require an inordinate amount of effort or time. If you have any comments please enter them here.
  
• The item requires an inordinate amount of effort or time.  Provide a brief explanation of why the task is not cost effective and how the same information might be elicited more efficiently. 
  

Does it take a long time for students to:
Read the question
Decipher the answer choices
Interpret the graph, read the data table
Make calculations
Other

Score select value 1 2 3 4 5 NA

Scoring notes:

In this section, numerical ratings are assigned to the judgments made thus far. Using the Assessment Task Evaluation Form indicate on a scale of 1-5 the extent to which you think the assessment criteria are met for the task you are evaluating. A rating of 1 means a failure to meet the criterion and a rating of 5 means the criterion is completely met. Scores in the middle mean the criterion is partially met. Choose N/A if the criterion does not apply or if you are unable to answer.

After assigning scores, please re-read your analysis, and modify it if necessary, to ensure that a future reader will be able to quickly identify and understand your justifications for the scores that you have assigned.

Finally, recall that the purpose of this procedure is to judge (1) the alignment of the item content to the targeted learning goals, and (2) the likelihood that the item will provide information about what the student knows and does not know. If you feel that the scores resulting from the procedure are not a reasonable representation of these two factors, please explain below and state the scores that you would prefer.

Necessity

select value 1 2 3 4 5 NA

Sufficiency

select value 1 2 3 4 5 NA

Comprehensibility

select value 1 2 3 4 5 NA

Task Context

select value 1 2 3 4 5 NA

Test-Wiseness

select value 1 2 3 4 5 NA

Cost Effectiveness

select value 1 2 3 4 5 NA

Comments on Scoring:

Revisions for improving the task or reasons for eliminating the task from further consideration should be listed below. If the item is satisfactory as written, the corresponding box should be checked.

Check the appropriate box and provide suggestions and explanations as appropriate.

• The task is effective and does not need to be revised.
• The task should be revised.  Please make specific suggestions for revision.
  
• The task should be eliminated from further consideration. Please explain.