Evaluate
Evaluation Plan

e3Learning project: Evaluation of a content-rich website

Version:

1

Date of creation:      

4 February 2003

Course:

A course on medical communication skills

Period of site use

Jan 2003 - May 2003

Project description:

Students viewed video clips on medical communication skills improvements and completed interactive exercises in class and in self-study.

 

A. Evaluation questions:

1. Regarding the online notes

Questions

Data

a. Students

 

Do students like the way the online notes have been put up on the site?

Course-end student survey,

Focus-group interview

Do students feel that the online notes are useful to them in learning the subject?

Course-end student survey,

Focus-group interview

What are the ways to further improve the online notes?

Course-end student survey,

Focus-group interview

Do students regularly access the online notes?

Site log data,

Course-end student survey,

Focus-group interview

2. Regarding the role-play video clips

Questions

Data

a. Students

 

Do students like the way the role-play clips have been put up on the site?

Course-end student survey,

Focus-group interview

Do students feel that providing the role-play clips online is useful to them in learning the subject?

Course-end student survey,

Focus-group interview

What are the ways to further improve the online role-play clips?

Course-end student survey,

Focus-group interview

Do students view the online video clips?

Site log data,

Course-end student survey,

Focus-group interview

3. Regarding the overall site

Questions

Data

a. Teacher

 

What is the workload of web-based teaching compared with the traditional one?

Course-end student survey,

Focus-group interview

Does web-based teaching make teachers able to enjoy their teaching more?

Course-end student survey,

Focus-group interview

What insights in teaching (if any) has the teacher gained after implementing web-based teaching?

Course-end student survey,

Focus-group interview

b. Students' satisfaction

 

Do students appreciate how the teacher has taken effort in using technology to assist teaching?

Course-end student survey,

Focus-group interview

What other e-learning materials do students think can also be added to the site?

Course-end student survey,

Focus-group interview

c. Students' learning

 

Does web-based learning generate higher learning outcomes?

Course-end teacher survey,
Course-end student survey,

Focus-group interview

Does web-based learning generate higher student participation?

Course-end teacher survey,
Course-end student survey,

Focus-group interview

 

B. Evaluation instruments and timetable

Item

Timing

Description of item

Strategies

Data on how the teacher feels

Course-end teacher survey

End-of-course

Teacher reflects on what she feels about the workload and enjoyment of employment web technology in assisting the teaching of the course. Teacher also reflects on how she sees the e-experience in connection with students' learning.

1. Evaluation officer (EO) prepares survey draft.

2. Teacher and EO finalize survey questions.

3. Teacher fills in survey.

Data on how students feel

Course-end student survey

End-of-course

Elicit data concerning students' feelings towards the online notes, the role-play video clips, and the overall site.

1. EO prepares survey drafts.

2. Teacher and EO finalize survey questions.

3. Teacher administers the surveys.

4. EO processes and analyzes the data.

Focus group interview

End-of-course

A meeting with 5 to 6 voluntary students in a 45-minute session with the Evaluation Officer to elicit more data concerning students' feelings.

1. EO prepares topics to be covered in the interview.

2. Teacher and EO finalize the topics.

3. EO meets with the students.

4. EO processes and analyzes the data.
Data on how students do
Site log data Weekly data feed-in Elicit data concerning how the various resources (the site, notes, and video clips) have been accessed.

1. EO suggests levels of keeping site log.

2. Teacher and EO finalize the site log levels.

3. EO helps to set up site counters to monitor students' access of the resources.

 

C. Evaluation data use policy

Evaluation data is returned to the website owner at the end of the evaluation but a copy of data will be kept by the e3Learning Project for reporting purposes. The general principles regarding the use of the evaluation data are as follows:

  1. The e3Learning Team reserves the rights to use the data in our project reports in ways in which details of the individual sites and/or courses are not mentioned.
  2. In reports where details of the individual sites and/or courses are mentioned, the e3Learning Team will send the written section to the site owner for revision and comments before publication.
  3. In cases where substantial evaluation has been done there may be a possibility of academic publishing about the design, development and evaluation of particular sites and/or courses. The e3Learning Team would welcome collaborative publishing with joint-authorship.
  4. When a site owner refers to evaluation data in reports and articles, a reference is expected to be made to the e3Learning Project in the text. Also, the Project will be most grateful if a copy of the reports or articles could be sent to us for the purpose of keeping records.

-end-