It is so fitting that we end our exploration of distance education with a lesson about evaluation! You probably know from any of the design models you chose for your final project there is an element of evaluation that should be worked into your design. As your chapter and the additional article point out, the term “evaluation” is sometimes (incorrectly) used interchangeably with “assessment” and “research.” It is best to keep the terms separate as their purposes are different. Using the term “assessment” works well when discussing learner achievement and may be used in an evaluation. Evaluation, in the context of learning, performance, and change initiatives, can be defined as a form of inquiry that seeks to address critical questions concerning how well a program, process, product, system or organization is working. The results are designed to benefit a variety of stakeholders for them to use for decision-making purposes and, if shared throughout an organization, may lead to organizational learning. Three types of evaluation that are performed to provide a variety of results are formative, summative and developmental. (We will focus on Formative and Summative):
Formative evaluations are usually performed to improve a product or a program and are often performed by members of the staff. These internal formative evaluations are designed, throughout the process to improve or refine what is being evaluated. The results of a formative evaluation can be used to help an organization come closer to a goal of a product or program. Instructor feedback and target audience member feedback are helpful in this process.
Summative evaluations are different from formative evaluations as they are usually performed to decide if a product or program is worth keeping, if it needs changes or an overhaul or even if it has merit. Its results have more of a final judgment of a program that is mature or has already been completed. Unlike the formative evaluation the results are not necessarily designed to improve what is being evaluated; instead the results may help stakeholders decide if the product or program should stay in existence, or if it “passed” or “failed.” There are various types of summative assessments such as monitoring and auditing where some fine-tuning can occur, outcome which can be an ongoing evaluation schedule for an organization program participants, and impact which focuses on the participants of a particular program.
Developmental evaluation is a form of long-term evaluation where the evaluator is more closely linked to the stakeholders during the development of a program and is actually a member of the design team. Feedback is provided in real-time as the project progresses. There is a wonderful metaphor from one of the readings “When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative.” An organization can implement all three types of evaluations depending on their need and where they are in the program or product’s design or implementation. Looking more closely at Kirkpatrick’s evaluation approach (discussed in Chapter 12) he provides four levels of evaluation outcomes: In the chart below each level is named, the purpose is described and specific challenges or limitations of each level area listed. As a designer, you will want to consider what outcome you are looking for? What do you want know about your program/training/lesson? Do you just want to know if the learners “liked it?” Do you want to evaluate if any learning has occurred? How will you do that? Who will the results serve? Being an Evaluator can be its own profession, but to be able to evaluate your own lesson, training, program will be a valuable tool for you, your learners and whatever organization you are working with.
| Kirkpatrick’s Level | Purpose | 1 Challenge/Limitation |
|---|---|---|
| Reactions | The purpose here is to gather a participant’s reactions to ascertain if the participants enjoyed the training, if the environment was suitable, if the trainers were skilled …basically using the participants reactions to the training. | Positive responses at this level does not necessarily mean that any learning has occurred. There are mixed results on whether a positive response in training is related to increased learning or greater use on the job |
| Learning | This determines what the participants learned as a result of the training or to determine if they know more now than they did before the training. This helps answer the question “have the participants improved or increased their knowledge or skills?” | There are times when a learner’s feeling that they learned something is not an accurate assessment of what they actually learned. Also, testing right after training may show positive results that may not translate into retention. |
| Behavior | This level determines if the behavior of the participants is improved in the workplace. It helps to determine if any new knowledge from the training has transferred as new behaviors in the workplace. | There may be little or no relationship between the post training behaviors to the pre-training behavior. Also, if there are better results, that may be due to the trainer and not the program. There is not always a good way to make such determinations. |
| Results | This focuses on what the results of the training are for the organization. Has the business benefitted from the training or has it helped the organization move closer to its goals or achieve its goals? | It is difficult to evaluate the training as a monetary investment that will reap some type of monetary return. Also the time lag between training and results creates a problem for an accurate evaluation. |
There are many data sources available when performing an evaluation. The most commonly used four are (a) observation, (b) survey, (c) interview, and (d) archival data. One or all can be used when evaluating. Below is a small glance at some advantages, disadvantages, real-world examples and factors influencing the choice to use it.
| Data Source | Advantages | Disadvantages | Real-World Example | Factors influencing this choice |
|---|---|---|---|---|
| Observation | First hand documentation is valuable data which, when added to the participant’s data, could prove to be quality data sources. Observations can allow evaluation data to be collected during real events occurring in real-time. They can be videotaped or photographed also. This can be referred to more than once if necessary. | The observer must be qualified and understand the product and/or process completely in order to be able to accurately observe. The observer’s presence could negatively affect the data collections. Also if there are not enough observations, the data may not be helpful and if there are too many, this could take an evaluation over time/budget. | When a process or a procedure is being evaluated, such as how a participant interacts with a computer program or application or an employee using a new tool in the environment of the workplace or the flow of a procedure or practice, then observation might be a good data source. | When a process needs to be evaluated. To evaluate in real-time versus after the fact. The need to observe in context. |
| Survey | Surveys allow an evaluator the opportunity to receive data from a large number of participants and possibly over a large geographical area. Also, if the participants might be influenced by the presence of others, this allows them a certain privacy of time and place. |
Often the answers received do not provide the information desired. The skill of the evaluator or the one who designs the questions is necessary in order to obtain relevant data. Also, people may be burned out on surveys and then participation numbers will lag. | If an evaluation needs to be done quickly, for a company that has offices all over the country a survey could be a best choice option. A computer program designed to organize, calculate, evaluate the data could be used to help in the process and meet deadlines to gain as much relevant data as possible. | The number of participants Their locations Time-table restrictions Privacy or flexibility for the participant |
| Interview | This allows for a deeper response. Allows the participant to expand, if necessary on their responses which is sometimes preferred to a survey. This also allows the interviewer to narrow in on the major topics of interest. Interviewers can meet with more than one person as in a focus group. This can save time. | There may be a bias that affects the data based on the questions asked, how they were asked. Sometimes the participant might be responding to the interviewer and not the question so the data is not as accurate as possible. This is another data source that requires a highly skilled interviewer who can stay on track, cover all the bases, redirect the participant and know when to probe deeper. It is also time consuming which might push the time-table and budget. | A real-world example might be when a rich description is needed such as customer satisfaction or a user-experience that would require details and explanations in order to fully understand their reaction to the program or process being evaluated. | Deeper responses needed. Clarification needed. |
| Archival | This data already exists in some form and just needs to be requested, retrieved and reported. This data originated in a natural, real-world context which can provide rich cultural data and can be reviewed more than once. |
To gain access to this information often requires assistance and therefore it may not be a timely response. Certain biases may show through such as not including all the data or tweaking data before the evaluator receives it. |
This may be a great way to begin an evaluation or if much data is needed and an evaluator doesn’t have a great deal of time. Ex. Being able to review exit surveys from previous years as a launch pad to new data collection. | Previous years’ data is required. Data is available and accessible. |
In the design phase of a distance learning course/lesson/training, you can perform formative assessments throughout to gain helpful feedback that helps to get closer to your goal. The Summative evaluation is post-finish line; your course or program has already been completed by your learners and you perform an evaluation (that is part of your design process,) designed to let you know if it has achieved all you have designed it to achieve. There are always places to tweak for a variety of reasons. Did the technology you chose perform as you had hoped for? Did the lesson you design bring about the results you wanted? Did your audience walk away with the learning you had hoped? These are not the end. This is just the beginning of the iterative process of instructional design. Sometimes, however, in the hectic environment of business or education, evaluation of the program or course or lesson can fall to the wayside.
This final lesson is included to offer a suggestion that evaluation is a very important element that will help lead us to a quality product, more meaningful learning for our learners and evidence (to you or the powers that be) that things are going in the right direction.
READ: "Measuring Success: Evaluation Strategies for Distance Education"
READ: Chapter 12 - Teaching and learning at a distance, "Evaluating Teaching and Learning at a Distance"
Consider the distance learning project you created. Think of your learners, the purpose of the lesson, and its place in the larger system it is a part of. What do you consider “success” with regards to your lesson? How will you evaluate its success? In a 1-2 page word document, outline a summative form of evaluation, using your readings and the table above. Select one or more methods of data collection (observation, survey, interview archival) that you will use to evaluate the “success” of the instruction you created after its intended learners have gone through it. Describe what approach you will take and what you hope to discover and your plan to make any necessary changes should your evaluation lead you to that.
(This response should be in MS Word (or similar) and should be submitted together with Assignments 11.2 & 11.3.)
You will design a small survey that you can ask your learners to fill out upon completion of your instructional project. Please create the document in MS Word first, limit it to 6-8 questions and then create a working survey using a survey tool of your choice. Virginia Tech students may use QuestionPro (you may need to sign in with your PID). There are many tutorials available to you. If you click the button below, you can explore some of the frequently used tutorials.
Assignment 11.3 is as follows:
Final: Small Reflective Feedback The overall goal for this course was to help create opportunities for you to learn about the subject of distance education from several perspectives and also be able to apply this knowledge in your own life. When you have completed this course, it is designed (and hoped) that you will have accomplished the following:
This final assignment evaluates whether or not the overall goal for the course was met. We want to know if learning actually occurred and if, after completing this course, you have a greater understanding of Distance Education and a greater sense of confidence when it comes to planning, designing and implementing various forms of distance learning.
Assignment
This assignment should provide a short narrative of your progression from the beginning of this course to now. The purpose is to reflect upon your journey throughout the course. Moreover, this is an opportunity for you to share your observations regarding the design and delivery of this course.
As instructors (educators, trainers, etc.) we strive towards excellence in teaching and learning. The desire here is that learning does occur. ITMA will utilize your constructive feedback to improve on the design and development of this course in the future. This assignment will be graded upon the following criteria only:
18 Points Possible
6 – Paragraph 1 – Provide an introductory statement as to how you rated your knowledge of Distance Education when you first began this class.
6 – Paragraph 2 – Explain your understanding of the overall course goals. Include what your perceptions were when you began this class and your understanding of Distance Education now.
6 – Paragraph 3 – Explain the transformation that might or might not have occurred over the past several weeks as a result of being in this course (be sure to cite specific examples of when you might have had an "ah ha" moment).
This paper should be typed and double-spaced. Please be sure to answer each section thoughtfully and completely, citing specific examples from your experiences.
(This response should be in MS Word (or similar) and should be submitted at the same time as Assignments 11.1 & 11.2.)
When you have completed all three assignments (11.1, 11.2 &11.3) please submit them all at the same time in a single document. At the top of your document you should have the lesson name, and underneath that should be your name, email address, and the date. Your paper should be double spaced using a standard font such as Arial, Helvetica, or Times New Roman. Most formatting questions can be resolved by using the APA Publication Manual guidelines. Make sure you have saved your file with the name “Assignment11”. After you have saved you file, go to the student interface and submit your assignment for grading. Click here if you need additional information regarding submission of your assignment.
Your final assignment guidelines were provided in Lesson 7. Your Final Assignment will be assessed based on the following criteria based on the guidelines you were provided.
| Final Project Evaluation Criteria (300 Points) | |
|---|---|
| 1. Instructional Product (100 points) | |
| 20 | Consistent with design goals described in Section 2 of your paper. |
| 20 | Consistent with instruction where learner and instructor are separated by space and possibly time. |
| 20 | A complete chunk of instruction (complete enough that someone could teach or learn without the developer present) |
| 10 | Lists objectives |
| 10 | Contains Directions for leaners |
| 20 | Includes an assessment element that aligns with goals/objectives |
| 2. Descriptive Paper Section 1 (100 points) | |
| 10 | Program Description |
| 10 | Program Objectives |
| 10 | Level of the Program |
| 10 | Target Audience |
| 5 | Project Team |
| 5 | Planning Stages |
| 10 | Budget Plan |
| 10 | Program Delivery |
| 10 | Faculty and Student Support |
| 10 | Evaluation and Assessment |
| 10 | Disability Accommodations |
| 3. Descriptive Paper Section 2 (100 points) | |
| 10 | Overview of the topic |
| 5 | Justification (why is instruction needed for this topic?) |
| 10 | Instructional Goal Statement. |
| 5 | Your instruction design choice and instructional theory used |
| 10 | A Learner Analysis. (Include any relevant general characteristics of your target audience (age, gender, education level, etc.) as well as characteristics such as, prior knowledge, and attitudes). |
| 20 | A list of objectives based on the main learning goal for your instruction. |
| 20 | A description of choices you made about:
|
| 20 | A description of how this lesson not only responds to the needs of your learners, but also the organization that is providing the instruction. |
Submitting Your Final Assignment
When you have completed your Final Assignment please submit the reports in a single document. At the top of your document you should have “Final Assignment” and underneath that should be your name, email address, and the date. Your paper should be double spaced using a standard font such as Arial, Helvetica, or Times New Roman. Most formatting questions can be resolved by using the APA Publication Manual guidelines. Make sure you have saved your file with the name “FinalAssignment”. Some of you will be turning your Report in with a PPT and might be wondering how to send in the two files together. For those who are wondering how to do this, you can create a zip file and pull the Word doc and the PPT into the one file. If you have not done this before, HERE is how. After you have saved your file, go to the student interface and submit your assignment for grading. Click here if you need additional information regarding submission of your assignment.