Questionnaires are a very useful way to receive feedback. They may be created on paper or using an online package. They can be made up of both open and closed questions. One problem is that return rates can be very low. Here we will consider:

• Some hints
• Open questions
• Closed questions
• Combination of Open and Closed Questions

Some hints <top>

When preparing a questionnaire it is important to:

• Be clear about the nature of the audience.
• Be clear about the purpose of the questionnaire.
• Consider the issues of anonymity.
• Make the purpose of the questionnaire and how the data will be used clear to the respondents.
• Give clear instructions on how to complete it.
• Keep questions clear and unambiguous.
• Group questions on a particular theme together.
• Keep a consistent style.
• Avoid double-barreled questions e.g. Which activity did you like best, Activity A or Activity B?
• Too many questions may lead to shallow responses.
• If possible, pilot the questionnaire.
• Put sensitive issues towards the end.
• Include a thank you statement at the end.

Open questions <top>

These are better for exploratory evaluations. With open questions, respondents write a textual response to a question.

• This allows respondents to detail their own experiences.
• A richer range of experience is often obtained.
• Also, the process can capture unexpected outcomes.

For example:

1. Outline some ways in which the use of the technology is affecting your approach to designing the answers to your assignments.
2. Explain how the use of online technology has affected your learning in this subject.

Open questions are harder to analyse than closed questions. Analysis involves coding responses to identify common themes.

Closed Questions <top>

These provide reliable data and are easy to analyse. Closed questions are easy to complete but good questions take longer to prepare. Closed questions:

• have a fixed response for the respondent to choose from;
• are better to use when the possible responses are known and you want to compare the results; but
• do not allow for unexpected results to emerge

There are several types of closed questions. Some are Likert questions, true/false or yes/no questions, multiple choice questions, and multiple response questions.

Likert questions provide a graduated series of responses from which the respondent chooses one. If you have a small sample size responses may be analysed better using the mode (most frequently occurring response) rather than the mean.

Variation: Use a scale on which respondents can place a mark.
For example:
I found the online materials easy to access.

Strongly Agree	Strongly Disagree

True/False          Yes/No questions

For example:
1. I was able to work at my own pace through the materials
True False

Multiple choice questions provide a range of responses from which the respondent chooses one option. For example:
1. How would you describe the ease of moving around within the subject website?
consistently easy
consistently confusing
easy in section A but confusing in section B
easy in section B but confusing in section A

Multiple Response questions provide a range of responses from which the respondent can select one or more.
For example:
Select the locations from which you usually accessed the online learning material:
university computer lab university Library at home at work community centre other

Combination of Open and Closed Questions <top>

For example:
1. Select any one or more of the following, which describe how online learning has changed your learning experiences.
I had more interaction with my peers.
I had more self-paced activities.
I received more feedback during the semester.
Other

2. If you answered 'other', please explain your response.

A guided or structured interview is essentially a conversation between the interviewer and the interviewee. It is important to identify the key areas that are to be evaluated and then generate five or six key questions. The more open the questions, the more the individual experience is able to be captured. It may be that supplementary or clarification questions are asked during the interview; it is best to be flexible.

Variation:

• Use in conjunction with other more quantitative methods (e.g. questionnaire, checklist).

These are moderated meetings of a sample of a class. They are useful for both formative and summative evaluation. They help to identify or clarify the key issues. Some points:

Web logs <top>

Information about which parts of your website students visit, and when, can be very helpful.

These are useful for collecting a lot of information quickly. They can be useful for checking if online materials meet specific criteria. They may identify issues to be followed up further in a focus group or interviews.

For example:
Check all the words which describe this course website:
clear interesting easy to navigate boring frustrating relevant confusing informative

Variation
Use in conjunction with observations of the innovation in action, such as a test or a trial.

These are used to judge how confident students are with a specific activity. They are applied in order to:

Hints:
Identify the purpose of the log.

In this strategy, there is an analysis of the interchanges between students in real time (synchronous) and asynchronous discussions, examining the nature of the interaction process and the quality of what is said. This can be time-consuming, but very informative.

This is in-situ observation of a process or innovation. The aim is to observe the process or innovation within its intended environment with minimal effect on the environment. The ethnographic study acknowledges the environment in which it is happening. The data being collected are interpreted in context. This relies on describing the 'feel' as well as the ‘facts’. The evaluator becomes an observer. Anonymity in reporting is an issue.

The outcome of an expert review is a report provided by an appropriate expert or group of experts. The type of expert depends on the aspect being investigated. It might be a subject matter expert, an instructional design expert, a graphical interface expert, or a combination of these.

The purpose of this strategy is to obtain students' interpretations of the process of understanding and learning. Students are asked to explain in writing how the technology may have assisted them to develop their understanding and learning of key ideas, with emphasis upon the understanding and learning processes. This requires careful structuring and exemplification if the journal is to move beyond a fairly low-level description of events and experiences. A useful framework is that of the five 'R' levels (Bain et al., 1999):

The purpose of testing is to verify that the product matches the requirements as outlined in the specification document. Testing should occur during the development stage to ensure the project is functioning successfully. It is a cyclical process and its time span depends on the size and scope of the project. The product needs to be specified clearly at the outset of the project so that comparisons may be made to see if the development is on track. As prototypes are developed, they undergo testing in terms of functionality, robustness of any programming code, compatibility across platforms, content, interactivity, look and feel. Problems are best addressed here.

The 'alpha' version is a near-complete build of the product which is advanced enough to undergo user trials. These are to provide usability information so that the design of the product can be modified as required.