Readings

Read Chapter 7, Developing Assessment Instruments, from Dick, Carey & Carey.

Introduction

Now that you have clear objectives of what your learners need to do, you can write the assessment items that will measure their achievement of the performance stated in the objectives. The emphasis on assessment is an important one as creating an appropriate and well-thought out assessment helps us determine which objectives have or have not been learned by the learners. t Mager again provides us with some wisdom on assessment in his book "Making Instruction Work". He tells us that "if it's worth teaching, it's worth finding out whether the instruction was successful. If it wasn't entirely successful, it's worth finding out how to improve it" (pg. 83). If you think of objectives as describing where one needs to go, the assessment items are the means by which you determine whether she got there.

You may wonder why test items are created now when you haven't even developed your instruction. The belief is that your assessment items should come directly from your objectives. The performance asked for in the assessment item should match the performance described in the objective. It should not be based on what you think are good or fun test questions or on your instructional activities. If you've written worthwhile objectives, you already know what performance needs to be tested.

Designing Test Items

Designing test items is not as easy as we may think. There are several issues to consider when designing assessment items. Each is discussed in this section.

Centering the Test Items

Test items should be goal-centered
As we have discussed, test items should align with the performance stated in the objective. The wording of the objective should guide the process of writing the assessment items. A well- written objective will prescribe the form of test item that is most appropriate for assessing achievement of the objective. Appropriate assessment items should answer "yes" to the following questions:

  1. Does the assessment item require the same performance of the student as specified in the instructional objective?
  2. Does the assessment item provide the same conditions (or "givens") as those specified in the instructional objective?
For example, if the performance of an objective states that learners will be able to state or define a term, the assessment item should ask them to state or define the term, not to choose the definition from a list of answers.

You should not use a test item that asks for a different performance than the one called for by your objectives. For example, if you have an objective that says students need to be able to make change, it would be unfair to the learner to have test items such as the following:

  1. Define money
  2. Name the president on the fifty-dollar bill.
  3. Describe the risks of not being able to count.
None of these items asks the student to do what the objective asks, which is to make change. As a result you will not know if your students can perform as required.

Test items should be learner-centered

Test items should take into consideration the characteristics and needs of the learners. This includes issues such as learners' vocabulary and language levels, motivational and interest levels, experiences and backgrounds, and special needs. To start with, test items should be written using language and grammar that is familiar to the learners. Another important aspect of learner- centered assessment is that the level of familiarity of experiences and contexts needs to be taken into consideration. Learners should not be asked to demonstrate a desired performance in an unfamiliar context or setting. The examples, question types, and response formats should also be familiar to learners, and your items should be free of any gender, racial, or cultural bias.

Test items should be context-centered

Remember the context analysis you wrote? Well, when writing test items you should consider both the performance context and the learning context your wrote about. It is important to make your test items as realistic and close to the performance setting as possible. This will help ensure the transfer of skills from the learning environment to the eventual performance environment. According to Dick, Carey & Carey, "the more realistic the testing environment, the more valid the learners' responses will be" (pg. 153). It is also important to make sure the learning environment contains all the necessary tools to adequately simulate the performance environment.

If your test items require special equipment and facilities - as specified in the "conditions" component of your objective - you will need to make sure that those things will be available to them. If not, you will need to create a realistic alternative to the ideal test item. Keep in mind that the farther removed the behavior in the assessment is from the behavior specified in the objective, the less likely you will be able to predict if learners can or cannot perform the objective.

Test items should be assessment-centered

Test items should be well written and free of spelling, grammar, and punctuation errors. Directions should be clearly written to avoid any confusion on the part of the learner. It's also important to avoid writing "tricky" questions that feature double negatives, deliberately confusing directions, or compound questions.

Assessment Performance and Learning Domain

The next issue we want to look at is that of analyzing the assessment performance and determining the learning domain. Remember that the performance expected in the assessment is determined by the objective. Classifying each objective according to learning domain can also aid you in selecting the most appropriate type of assessment item.

Assessment Performance Assessment Item Types
Verbal – involves reciting something from memory Simple objective-style test items. This includes short-answer, matching, and multiple-choice, but requires that the learner responds orally.
Intellectual Skills – involves the use of discrimination, concrete concepts, defined concepts, rules, and/or higher order rules Objective-style test items, the creation of some product, or a performance of some sort. The product or performance would need to be judged by a checklist of criteria.
Psychomotor Skills – requires a physical activity, coordination of all or certain body parts, and involves special techniques in the performance The learner must perform a set of tasks that lead to the achievement of the goal. It also requires a checklist or rating scale so that the instructor can determine if each step is performed properly.
Attitudes – requires an internal state influencing the choice of personal action toward a class of people, things, or events. These assessments are more problematic since there is usually no way to directly measure a person's attitude. Assessment items generally involve observing learners in action and inferring their attitudes, or having learners state their preferences on a questionnaire.

Types of Assessment Items

The wording of your objective and the learning domain of the performance expected in the assessment guides the selection of item type. You should select the type of item that gives learners the best opportunity to demonstrate the performance specified in the objective. For example, if we expect our learners to state the capital of Virginia, they must respond orally (to "state" is a verbal skill) from memory. This could be assessed by presenting the learner with a fill-in-the-blank assessment item, but not a multiple-choice item.

There are several types of assessment items. The most frequently used ones include essay, fill- in-the-blank, completion, multiple-choice, matching, product checklist, and a live performance checklist.

How many test items are needed?

The question often arises regarding the number of assessment items necessary to achieve mastery of an objective. For some skills, only one item is necessary. For others, it may require more than one item. It is essential to keep in mind that no matter how many items are created for an objective, the conclusion aimed for should not be how many they got correct but rather if the number of items is necessary to indicate mastery of the objective.

While two items may be better than one, it may also yield a 50-50 result, with a student getting one right and one wrong. Would this indicate mastery? Gagne suggests having three items in this case instead of two, as two out of three provides a better means of making a reliable decision about mastery.

Examples of Test Items

Here are some examples of good and bad assessment items:

Example 1
Objective: The student will state the time shown on an analog clock to the nearest 5 minutes.

Good assessment: Students are given pictures of analog clock faces [conditions] and are asked to state the time indicated on each clock [performance].
Bad assessment performance: Students are given a time [conditions] and are asked to draw the corresponding minute and hour hands on a blank clock diagram [performance].

Example 2
Objective: The student will set up an attractive merchandise display in the student store with appropriate signs.

Good assessment performance: Students are scheduled a week to set up an attractive merchandise display [performance] in the student bookstore.
Bad assessment performance: Students are asked to write a paragraph [performance] describing the six elements of an attractive merchandise display.

Example 3
Objective: Students will write a descriptive essay of at least 300 words.

Good assessment: Have students choose a topic [performance] and write an essay [performance] describing it.
Bad assessment: Have students read [performance] several examples of good essays [conditions].
Bad assessment: Write a descriptive essay [performance] in class by having each student contribute a sentence [conditions].
Bad assessment: Have each student orally describe [performance] an unknown object [conditions] until the other students can guess what the object is.

Developing a Checklist or Rating Scale

Should your assessment performance learning domain indicate that a checklist is needed, follow these steps suggested by Dick, Carey & Carey:

1. Identify the elements to be evaluated - These elements should be taken directly from the behaviors and criteria included in your objectives. Make sure that the elements you select can be observed during the performance.

2. Paraphrase each element - Elements should be paraphrased to cut down on the length of the instrument. Also, make sure that a "Yes" response on the instrument always corresponds with a positive performance, and a "No" response with a negative performance.

3. Select the type of judgment to be made by the evaluator - When evaluating a performance, product, or attitude, judgments can be made using checklists, rating scales, or frequency counts. Checklists provide a simple "yes" or "no" as to whether or not a learner meets a criterion or element. Rating scales take this a step further by allowing for in-between ratings instead of strictly "yes" or "no". Frequency counts are used for indicating the number of times a learner meets or displays a criterion or element. This is good if the element can be or must be observed more than once.

4. Determine how the instrument will be scored - With checklists you can simply add up the "yes" answers to obtain a score for each objective and for the entire process or product. With rating scales you can add up the numbers assigned for each element. Frequency counts are a little more complicated as you have to determine how to create a score. You can add up the frequencies for an element, but you would still have to determine what constitutes a good score and whether a lack of an occurrence would be detrimental.

Evaluating Alignment

One of the most crucial aspects of the assessment phase of the design process is to be able to evaluate the alignment of the action stated in the step, sub-steps, and subordinate skills with the objective and the assessment item. This is part of the systematic approach to instructional design. One way to clearly represent this relationship is to create a three-column table that lists each of the skills from your instructional analysis, the accompanying objective, and the resulting assessment item. At the bottom of the table you would finish up with your main instructional goal, the terminal objective, and the test item for the terminal objective.

Activity: Assessment Items

Now that you have drafted a list of objectives from the previous assignment describing what you want your learners to be able to do, it is time to create test items that will determine whether they have achieved the objectives. When finished, your assignment for this activity should be added to the document you began in the last lesson (objectives.doc).

Step 1: Analyze each objective including the terminal objective.

1. According to your objective, what is it that the learner will need to do when you assess the performance?
2. What conditions will need to be provided for in an assessment for this performance to occur? For conditions you cannot provide, describe approximations that will help the learner complete the performance as close as possible to the action noted in the instructional analysis flowchart.
3. What type of learning domain is covered by this performance: verbal information, intellectual skill, psychomotor skill, or attitude?
4. What type of test item will you need for this objective? Will it be an objective-style test item? Will it be a checklist or rating scale to evaluate a product, performance, or attitude? If it is an objective-style test item, which type of item will align with the prescribed behavior and conditions (e.g., multiple-choice, matching, essay, etc.)?

If you are assessing a product, performance, or attitude, a checklist or rating scale may be a better fit. In these items, describe the performance you will expect or the product they will create. In assessing a performance, the assessment items may need to indicate the steps required. In assessing a product, the assessment items should include some characteristics of the product. Finally, indicate the response to the assessment item. This could be simply a check-off of "Yes" or "No" or a rating scale.

Step 2. Create an assessment item for each objective.

When you have answered these questions, create an assessment item for the objective. The items should provide the conditions stated in the objective or approach the conditions as closely as possible. They must accurately assess the performance stated in the objective and meet the standard indicated in the criterion. They do not need to be paper-and-pencil tests.

If you feel you must have more than one item to test an objective, it should be because (a) the range of possible conditions is so great that one performance won't tell you that the student can perform under the entire range of conditions, or (b) the performance could be correct by chance. Be sure that each item calls for the performance stated in the objective, under the conditions called for. However, at this point you are only required to create one item per objective.

Step 3. Format your analysis and assessment items.

Here's an example of what your document should look like for each objective. This example shows an intellectual skill with an objective-type test item.

Objective 4.0 - Given a set of rules for determining the credibility of information on a web site, learners will correctly identify web sites that are credible.
1. What will they need to do? The learners should be able to identify which web sites are credible when viewing the web site.
2. What conditions will need to be provided? The learners will have the process for determining the credibility of a web site.
3. Domain - Intellectual Skills: Rules. Students have to apply a set of criteria in order to make a decision.
4. This objective will require an objective-style fill-in-the-blank test item.
Test Item 1

In a web browser of your choice, use the AltaVista search engine to find websites related to your topic. On the assignment sheet, list your topic, the first five web sites, and if the sites contained credible information according to the process described in the handout.

Step 4. Create the Design Evaluation Chart

The last step is to create a design evaluation chart that indicates the alignment among your skills, objectives, and assessment items as discussed earlier in this lesson. In the left column list all of your goal steps, substeps, and subordinate skills in numerical order. In the middle column list the accompanying objective for each skill. In the last column list your assessment item(s) for the objective. Make sure that everything is aligned properly, so that it is obvious which skill goes with which objective, and which objective goes with which test item.

Submitting your Assignment

This assignment should be produced using Microsoft Word. Your assignment for this lesson must be appended to your assignment for the previous lesson. The title of this assignment is "Objectives and Assessment". Beneath that enter your name, email address, and the date. Save your assignment using the filename "objectives". After you have saved your file, go to the student interface and submit your assignment for grading. Click here if you need additional information regarding submission of your assignment.

Assignment: Assessment Items

Points: 20

Grading Criteria

- For each objective, includes a description of what the learner will need to do in the assessment. (1)
- For each objective, includes a description of the conditions that will need to be provided in the assessment. (1)
- For each objective, describes the learning domain covered by the objective. (1)
- For each objective, identifies the type of test item that is appropriate for assessing that objective. (1)
- Assessment item created for each objective that will accurately assess the behavior or performance called for by the objective. Realistically provides for the conditions stated in the objective. Product, performance, or attitude assessment items should describe the product or performance and list the criteria that would be included on a checklist or rating scale for that item. (10)
- Three-column design evaluation chart created that indicates the alignment among skills, objectives, and assessment items. (6)