Assignments
Top

Lesson 10 - Formative Evaluation

Lesson 10 Readings

  • Read Chapter 10, Formative Evaluation, from Dick and Carey

Background Information

Since you are not developing your instructional materials in this course, you will not be able to conduct a formative evaluation of them. However, you will learn more about evaluation in an upcoming module. With this in mind, we will once again limit our discussion here.

Martin Tessmer, in his book Planning and Conducting Formative Evaluations, defines formative evaluation as "a judgment of the strengths and weaknesses of instruction in its developing stages, for purposes of revising the instruction to improve its effectiveness and appeal" (pg. 11). Note that in a formative evaluation, evidence of an instructional programís worth is gathered for use in making decisions about how to revise the program while it is being developed. This is why it is called "formative" evaluation, because the instruction is in its developmental stages and is not yet "grown up". The idea is to find out if your newly developed course works at teaching the objectives you need to teach to the learners who need to learn them, before you present it to your target audience.

In any given formative evaluation, you can find out how to make your instruction more:

  • Effective
  • Efficient
  • Interesting/Motivating
  • Usable
  • Acceptable

You do this by carrying out procedures that will provide you with evidence as to the effectiveness of your instruction. The emphasis is on collecting data and revising the instruction. Notice in the graphic of the Dick and Carey model that there is a "Revise Instruction" box that hovers above the rest of the steps, with dotted lines connecting everything. This revision process stems from the results of your formative evaluation activities and can affect any of your previous steps.

Many believe that formative evaluation is the key to instructional success. Conducting a formative evaluation allows you to try out your materials to find out how they work and how you might make them better before you present them to your target audience. In many cases, however, evaluation isnít done because of cost and timing issues. Itís hard enough to get the design and development work done, and often by that time most people involved are ready to be finished with it. Keep in mind, however, that if we were talking about anything other than instruction, evaluation would be a given. Robert Mager, in his book Making Instruction Work, provides a couple of nice to-the-point examples:

"Has this plane been tested?"
"What?"
"Has this plane gone through tryouts to make sure it works?"
"We donít have time for that. We consider the maiden flights the tryouts."
"Oh."

Or,

"Has this medicine been FDA approved?"
"What?"
"Has this medicine been tested and approved safe to use?"
"No need for that. We find out on the first few patients who die from it."
"Oh."

So you see, no matter how good you might be at instructional design, thereís still no way to find out how well it works until you try it out. Since nothing is perfect, everything can be improved.

Dick and Carey focus on three phases of formative evaluation. However, Tessmer identifies four distinct phases:

  1. Expert Review Ė Experts review a rough version of the instruction with or without the evaluator present to determine its strengths and weaknesses. They can be content experts, technical experts, designers, or instructors.
  2. One-to-One Ė One learner at a time reviews the instruction with the evaluator. The evaluator observes the learner using the instruction, notes the learnerís comments, and questions the learner during and after the instruction.
  3. Small Group Ė The evaluator tries out the unfinished instruction with a group of learners and records their performance and comments.
  4. Field Test Ė The evaluator observes the instruction being tried out in a realistic environment with a group of learners.

Generally, these four phases are carried out in order, although Expert Review and One-to-One are often carried out at the same time. Based on the data you collect in each stage of the evaluation, decisions should be made regarding how you will revise the instruction.

Instructional Design Project Part Six (cont.)

The activities in this lesson should be added to the document you began in the last lesson (development.doc). If you recall, in the last lesson you began Part Six of your ID Project by giving some thought to how you might acquire or develop instructional materials based on your instructional strategy. You also considered your role in the development and delivery of the instruction. Now itís time to give some thought to the formative evaluation process.

Depending on your individual situation, it may not be realistic to think that you will be able to perform each type of formative evaluation with your instructional materials. Even so, we would like you to think about how you might realistically perform each type of evaluation. There may be ways for you to fit in an expert review or one-to-one evaluation. And perhaps you can find a small group to try out your instruction with? No matter how you end up going about it, the goal is to find ways in which to gather as much information as possible about the instruction so that it can be appropriately revised. Anything is preferable to "going public" with your first draft.

After reading Chapter 10 in the book, briefly answer the following questions regarding the potential evaluation of your instructional materials. Please answer each numbered question separately:

  1. Describe how you might conduct an Expert Review of your instructional materials. What subject matter expert(s) would you consult with, and what specific types of information would you hope to collect?

  2. Describe how you might conduct a One-to-One evaluation of your instruction. How would you go about selecting a few learners to participate who are representative of your target population? What factors would you take into account in the selection process? Also, in conducting the one-to-one evaluation, what specific information would you want to gather about your instruction, and what tools and methods might you use to gather that information (e.g., checklists, questionnaires, interviews, etc.)?

  3. Describe how you might conduct a Small Group evaluation of your instruction. How would you go about selecting a small group of learners to participate who are representative of your target population? In conducting the small group evaluation, what specific information would you want to gather about your instruction, and what tools and methods might you use to gather that information?

  4. Describe how you might conduct a Field Test of your instruction. What instructional setting would you use for the field test? How would you go about selecting a group of learners to participate who are representative of your target population? In conducting the field test, what information would you want to gather about your instruction that was not already gathered during the small group evaluation, and what tools and methods might you use to gather that information? If your situation makes it impossible to perform a true field test, then you may want to consider your first formal presentation of the instruction as a sort of field test. However, if you do this leave yourself some time to revise the materials after this initial instructional session. Donít set everything in concrete.

When you have addressed each of these questions, write a summary describing your proposed formative evaluation procedures.

Submitting Part Six of Your ID Project

At the end of this lesson you will submit your completed Part Six. To recap, Part Six of your ID Project should be typed up in Microsoft Word. At the top of the paper type "ID Project Part Six: Development and Formative Evaluation". Underneath that include your name, email address, and the date. Also, make sure the file is named "development.doc". When you have completed Part Six, upload the Word document to the "instrdes" folder in your Filebox. When you have finished uploading your file, proceed to the online student interface to officially submit your activities for grading.

Assignment: ID Project Part Six (cont.)
Points:
5
Grading Criteria:

  • Describes proposed Expert Review evaluation procedures, including types of subject matter experts they would consult with, and the information they would hope to collect. (.5)
  • Describes proposed One-to-One evaluation procedures, including how they would select learners to participate, the specific information they would be looking for, and the tools and methods they would use to collect it. (.5)
  • Describes proposed Small Group evaluation procedures, including how they would select learners to participate, the specific information they would be looking for, and the tools and methods they would use to collect it. (.5)
  • Describes proposed Field Test procedures, including the setting they would use, how they would select learners to participate, the information they would be looking for beyond what they Small Group evaluation provided, and the tools and methods they would use to collect it. (.5)
  • Summary of proposed formative evaluation procedures. (3)