General information about criteria catalogues

What is meant by a criteria catalogue or a checklist is a systematic compilation of various questions and rating scales on individual product variables. This allows the coverage of various characteristics of learning software, such as:

• Screen design;
• Readability;
• Crash security;
• Comprehensibility of instructions;
• Appropriateness of learning contents; and many others

Based on the criteria catalogue, an expert checks for the presence of a series of individual characteristics which good learning software should have. On the one hand, use of criteria catalogues helps evaluation experts reach quick conclusions about individual categories of the software being tested. On the other hand, the evaluator can also draw a relatively complete picture of the overall quality of the software as well.

Why are criteria catalogues used?
When developing a learning system, criteria catalogues can be used for quality control to accompany the development process and for the final evaluation. They are also suitable for comparing and selecting learning systems if the project plans to access an existing product. Here, it is recommended that knock-out criteria are defined (absolutely necessary requirements which the product must meet).

However, there will be hardly any catalogue which is suitable in all areas for the respective evaluation task or the learning program to be evaluated. It is usually necessary to adapt the catalogue by, say, weighting the criteria to meet your own needs, leaving certain characteristics areas out, or defining your own additional criteria.

Required resources
Criteria catalogues are time and cost-efficient. However, the actual time taken can be quite different depending on the criteria catalogue in use and there is of course a trade-off between effort and assessment quality. A comprehensive assessment might not be possible using a possibly time-saving but less detailed catalogue. Short checklists are certainly the method of choice if the aim is an initial, rather superficial quality assessment, whereas comprehensive (and thus more complex) criteria catalogues make sense for in-depth analyses. In any case, complementary use of further evaluation methods should be considered.

Advantages
Criteria catalogues

• can be used with just minor changes for a wide variety of questions and evaluation areas;
• are easy to work with;
• help prioritise the many characteristics which must be considered when evaluating of learning software (Schenkel, Tergan & Lottmann, 2000).

Disadvantages
The reasons for not using criteria catalogs are that ...

• different evaluators can arrive at different assessments, i.e. the reliability of the evaluation is not always guaranteed.
• it is not always correct to assume that the aspects of the learning software being evaluated are related in some meaningful way to the learning results.
• catalogues usually do not consider the context in which the learning system is used. However, this context is at least just as important for the learning results as the characteristics of a learning system.
• they frequently reflect an over-emphasise technical aspects;
• they do not consider individual conditions such as learning prerequisites and habits (Schenkel et al., 2000).