Executive Summary

When designing instructions for a virtual learning environment, it is essential to pay special attention to the provision of feedback. On-line students can feel isolated, unsupported, frustrated and overwhelmed if they can not get feedback or if the feedback is too slow. These learners are usually working independently so it is essential that there is some mechanism for making sure their learning is progressing as expected by the teacher. Even when tutors have attempted to give clear instructions the student can be unsure about what is required. It is easy to resolve ambiguities in face-to-face lessons but it is more difficult when only written communication is available. The omission of formative assessment [1] can be dangerous because the sole use of summative assessment [2] can mean that problems are not identified until it is too late.

The number one criterion throughout a unit of work on this aspect of Educational Design Methods is to maximize the number of opportunities to assess student work, give feedback and allow them the chance to revise. This feedback is often comments on work in progress that are aimed at assisting students improve. By maximizing opportunities for formative assessment, this course is teaching by example; students are learning about feedback first hand.

Formative assessment consists of two parts: the assessment of performance (formally or informally) and the provision of constructive feedback. Effective formative assessment helps the student to see the gap between their current understanding of course components and the goals of the course so that they can take the appropriate action to achieve these goals.

see formative assessment as the cycle of performing a task, assessing the outcome, giving constructive feedback that makes the student reflect then take action to improve.

William and Black (1996) describe this cycle in terms of eliciting evidence, interpreting it and acting upon it by giving feedback. “What makes good feedback? … Good feedback causes thinking.” (Black and Wiliam, 2003: 631).

Sadler (1989) views it in terms of “student responses, judgments about the quality of the student responses and feedback to improve the student’s competence”.

The occurrence of Feedback in This Unit of Work

The course culminates in summative assessment after receiving formative assessment four times from three different sources (technology, peers and tutor) as well as having the opportunity to give peer feedback. The students experience four cycles of assessment and revision before they do the test. Each time they have a chance to revise and improve.

(The assignment receives both formative and summative assessment by the tutor: feedback in the form of comments with the chance to resubmit the work if it is not satisfactory and a mark.)

What forms of cognitive feedback are available within a virtual learning environment that will help students achieve competency?

We have entered “A New Era of Assessment” (Dochy, Segers & Sluijsmans, 1999: 1). We can not continue to solely assess students at the end of a unit of work. Assessment and instruction complement each other when the cycle of formative assessment is used throughout a course. The student must take an active role in the assessment of their performance and use all forms of feedback to improve their understanding and competency. This issue is particularly significant when we move away from the face-to-face classroom into a virtual learning environment.

A critique of the course design

This course can be found here

I have used three ways of approaching this critique: there were some obvious errors or weaknesses, I asked if the course met its objectives and if it adhered to the seven principles of good feedback suggested by Juwah et al. (2004). By carrying out this analysis I have found strengths and weaknesses in the design.

Strengths

Through reading the text or first hand experience this course succeeded in meeting the following objectives:

• Understanding the role feedback has in the learning process
• Identifying the essential nature and characteristics of effective formative feedback
• Applying this knowledge to the design of learning tasks
• Giving formative feedback to peers
• Encouraging teacher and peer dialogue around learning

The course adhered to the following principles of good feedback:

• Facilitating the development of self assessment (reflection) in learning
• Provides opportunities to close the gap between current and desired performance
• Delivers high quality information to students about their learning. (Juwah et al., 2004: 4).

This all depends on the quality of the feedback; how well it supports and enhances learning. It should ‘have embedded within it some degree of prescription about what must be done’ (Wiliam & Black, 1996: 543). Assessment and feedback need to focus on understanding not just memorization.

Weaknesses

Two errors immediately present themselves:

• A minor error was the title “Feedback and Formative Evaluation in a Virtual Learning Environment should have been “Feedback and Formative Assessment in a Virtual Learning Environment”. Assessment involves evaluation and feedback.
• An error was made when referring the students to two texts. What was listed as a second reference was actually a section of the first text cited. This would have been misleading and confusing for the students.
  Other texts need to be provided for the students to read; one that specifically focuses on giving formative feedback in a virtual environment and another about self-assessment. This would address the objective to identify and use different forms of cognitive feedback that are available within a VLE. An article by Peat and Franklin on ‘Supporting student learning: the use of computer-based formative assessment modules’ looks at a variety of computer-based formative and summative assessment tools. An article by Sadler (1989) on ‘Formative assessment and the design of instructional systems’ looks closely at the value of developing self-monitoring skills instead of being dependent on teachers and peers for feedback.
• At the beginning of a course, students need to go over the goals, objectives and expectations of the course and have the chance to ask questions. It can be useful to show several exemplars of the desired outcome. This needed to be in the course design.
  “knowing the goal of learning, self-assessing in respect to the goal, revising to achieve the goal…”.(Baron et al., 1998: 273)
• There should also be some attempt to find out what level of prior learning each student has so that the course can be tailored to the students needs. Determining the entry point will help to identify the gap between current level and desired goals.
• A learner-centered approach to this unit would have given students the chance to make choices about the content and the evaluation methods. This was only one unit of an entire course so a needs analysis could have been done at the beginning of the course.
• There was no activity in the design that actually addressed the objective to place formative assessment and feedback within the context of various instructional design models. The course assumes that the students have already studied instructional design models but they have not been asked to place it into the models. One can not say they can do something if they have not had the chance to do so.
• There is no evidence that this course design encourages positive motivational beliefs and self-esteem. To some extent the way the tutor gives feedback can address motivation and self-esteem.
  “[Feedback can] results in drawing student attention away from the task and may have a negative effect on learning. [The] development of sophisticated feedback skills by teachers is critical.” (Moreland and Jones, 2000: 5)
• “…formative assessment involves the use of assessments… as sources of feedback to improve teaching and learning” (Bransford, Brown and Cocking, 2000: 128). The teacher can use this information to help shape the teaching. After the teacher has given feedback on the critique, the teacher could use that information to decide if students needed further guidance before they did the assignment. This does give an opportunity for the teacher to shape the teaching but it really needs to be explicitly built into the design to ensure that it happens.
• This course could have had a practice test to provide another opportunity for feedback. Familiarity with the technical procedure and the type of questions would have reduced stress, given a chance to identify areas of weakness for appropriate revision or even suggested alternative learning strategies or approach.
• One last criticism is about the workload placed on the teacher. Teachers too can feel overwhelmed by the volume of emails and workload. It would have been more productive to focus only on automated formative assessment from the outset. However, it was only through my research that I discovered this. Bransford, Brown and Cocking, (2000) set out a research agenda with committee recommendations that include:
  “new assessment research to focus on improving and implementing formative assessments.” (Bransford, Brown and Cocking, 2000: xx)
  I would add to this that new assessment research should focus on improving and implementing automated formative assessments.
  “Because many new technologies are interactive, it is now easier to create environments in which students can learn by doing, receive feedback, and continually refine their understanding and build new knowledge.” (Bransford, Brown and Cocking, 2000: xix)

The emphasis on formative feedback in this course, although a good thing, was not enough to create a balanced design. This process of checking if objectives of the course are achieved would have been better done during the design phase of the course- this is what has been termed as ‘proactive evaluation’ (Sims et al., 2003) to ensure that the design assists learning.

The Reflection on this aspect of educational design

Chosen Approach

In the executive summary I have outlined reasons why I chose to focus on formative assessment and the provision of feedback for on-line learning. Two of the instructional design methods we have studied this year include the use of formative assessment. Gagnes 6th, 7th and 8th events of instruction [3] are; elicit performance, provide feedback and assess performance. The 4C/ID-model [4], Component 2: Supportive Information – includes feedback that is provided on the quality of performance.

In their recommendations for teaching for understanding, Baron et al. (1998: 273) suggest four principles of design:

1. Learning-appropriate goals
2. Scaffolds that support both student and teacher learning,
3. Frequent opportunities for formative self-assessment and revision, and
4. Social organizations that promote participation and result in a sense of agency

There is much support in favor of providing formative assessment through out courses but the error that has been made in this course design is in not giving enough attention to all of the other steps in the design method.

This investigation has highlighted two particular ways of providing formative assessment in a virtual learning environment which I would like to now discuss; Multiple Choice Tests and Peer/Self assessment.

Multiple Choice

When the reading list for a course is extensive, carried out over a mater of weeks, a student can have difficulty deciphering the important issues. There is general agreement that the use of “computer managed learning system as a formative assessment tool improves student performance on summative assessments” (Sly & Rennie, 1999: 1). Bocij & Greasley (1999) found that students “repeating formative computer-based assessments showed an average increase of approximately 12%” [on a summative assessment]. An important feature of effective formative assessment is the provision of immediate feedback. A multiple choice test is a powerful tool in this sense.

Buchanan (2000) points out that it is important not to give students answers. It is more productive to give the text reference that deals with the question. Buchanans’ study found that this timely feedback is likely to benefit students. “To help students take charge of their learning, the feedback suggests resources that students can consult to help with the concepts.” (Baron et al., 1998: 294)

There is software available such as ISV’s Question Tools to assist teachers, who have basic computing skills, to create and deliver assessments, tests, quizzes and surveys with minimum effort.

Peer/Self Assessment

Peer assessment is of benefit to the teacher and the students. In a large class it can be difficult for the teacher to give individualised feedback to everyone. When the student is required to do peer assessment, they are forced to think about their product in relation to products generated by their peers and they are required to think about the criteria for making judgments. These metacognitive processes will assist students in the revision and improvement of their own product.

Sadler points out that it is important to explicitly build these peer/self assessments into a course in order to train students “to develop self-assessment skills and gap-closing strategies simultaneously and therefore to move towards self-monitoring.” (Sadler, 1989: 140) Relying on teacher assessment will not develop these qualities. Baron et al. discusses the need to change classroom culture. Students need to be willing to accept constructive criticism from their peers as a positive experience, assisting in the improvement of their work. “Revision was not seen as a chore but rather as a natural component of learning and growing.” (Baron et al., 1998: 284)

An interesting electronically assisted peer assessment tool has been developed by Bhalera & Ward (2001). This tool is called OASYS. It automatically marks multiple choice questions then controls the anonymous distribution of products amongst learners for peer assessment of free response answers. This overcomes the current limitations of computers not being able to automatically mark this form of answer.

Conclusion

The benefits of formative assessment are two fold. Firstly, to assists in the learning process of acquiring skills, knowledge and competence. Secondly, to help develop students from being a passive receptacle of knowledge to an active, reflective, mindful, independent learner: a learner who is no longer dependent on the teacher, a learner who is capable of self-regulation. Developing these qualities is part of the journey to becoming a life-long learner.

Footnotes

[1] “Formative assessment involves the use of assessments as sources of feedback to improve teaching and learning” (Bransford, Brown and Cocking, 2000: 128)

[2] “Summative assessment measures what students have learned at the end of some set of learning activities”, e.g. test at end of unit. (Bransford, Brown and Cocking, 2000: 128)

A very simple way of explaining the difference between formative and summative evaluation is to say one is evaluation for learning and the other is evaluation of learning.

[3] Robert Gagne (1965b). “Nine Events of Instruction: Gain attention, Inform learner of objectives, Stimulate recall of prior learning, Present stimulus material, Provide learner guidance, Elicit performance, Provide feedback, Assess performance, and Enhance retention transfer. This concept leads to the important notion that instruction should be designed so as to ensure that learners acquire subordinate skills before they attempt to acquire super ordinate ones.”

[4] The 4C/ID-model. Four interrelated components are essential in blueprints for complex learning: learning tasks, supportive information, just-in-time (JIT) information, and part-task practice. Cognitive feedback is a final part of supportive information which relates to feedback that is provided on the quality of performance.

2726 words including quotes, tables and referencing.

Reference

• Barron, B. J. S., Schwartz, D. L., Vye, N. J., Moore, A., Petrosino, A., Zech, L., Bransford, J. D. ‘Doing with Understanding: Lessons from Research on Problem: and Project-Based Learning’, The Cognition and Technology Group at Vanderbilt. The Journal of the Learning Sciences, Vol. 7, No. 3/4, Learning through Problem Solving (1998), pp. 271-311
• Bhalerao.A. and Ward. A. ‘Towards Electronically Assisted Peer Assessment. A Case study’ Association for Learning Technology Journal, 2001
• Black. P. & Wiliam. D. ‘In praise of Educational Research: Formative assessment’, 2003. British Educational Research Journal, Vol 29, No 3, 2003.
• Bocij, P. & Greasley, A. 1999. ‘Can computer-based testing achieve quality and efficiency in assessment?’ Journal of Educational Technology.
• Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). ‘How people learn: Brain, mind, experience, and school’. Washington: National Academy Press
• Buchanan, T. ‘The efficacy of a World-Wide Web mediated formative assessment’. Journal of Computer Assisted Learning. Volume 16, Issue 3 , Page 193 – September 2000
• Dochy, F., Segers, M. & Sluijsmans. D. 1999 ‘The use of self- peer and co-assessment in higher education: A review’, Studies in Higher Education. Abingdon: Oct 1999.Vol.24, Iss. 3; pg. 331, 20 pgs
• Hanna, D.E., Glowacki-Dudka. M., and Conceicao-Runlee. S. 2000. ‘147 practical tips for teaching online groups’ Essentials of Web-Based Education. Atwood publishing.
• Hofmann, D.W. 2002, ‘Internet-based distance learning in higher education’. Tech Directions. Aug 2002.Vol.62, Iss. 1; pg. 28, 5 pgs
• Juwah, C., Macfarlane-Dick, D., Matthew, B., Nicol, D., Ross, D. & Smith, B. ‘Enhancing student learning through effective formative feedback’. June 2004 – The Higher Education Academy Generic Centre.
• Moreland, J. and Jones, A. ‘Emerging Assessment Practices in an Emergent Curriculum: Implications for Technology’. International Journal of Technology and Design Education. Volume 10, Number 3. January, 2000. Pages: 283 – 305
• Peat. M. and Franklin. S. 2002. ‘Supporting Student Learning: computer based formative assessment models’. British Journal of Educational Technology, Volume 33, No 5, 2002.
• Sadler, D.R., ‘Formative assessment and the design of instructional systems’. Instructional Science 18:119-144 (1989) 119
• Sims. R., Dobbs.G. & Hand. T. ‘Enhancing quality in on-line learning: Scaffolding planning and design through proactive evaluation’. Distance Education. Vol 23, 2003
• Sly. L. & Rennie. L.J. (1999) ‘Computer Managed Learning: Its use in formative as well as summative assessment’. Proceedings of the 3rd Annual CAA Conference, June, 1999 – caaconference.com
• Wiliam, D. & Black. P. ‘Meanings and Consequences: a basis for distinguishing formative and summative functions of assessment’. British Educational Research Journal, Vol 22, No 5, 1996.