Evalguide - Design

The participants are assigned to an intervention group (IG) or to a control group (CG) randomly (and IG and CG are therefore equivalent).
A specific ’treatment’ (X) is introduced, i.e. the intervention group receives the intervention to be analysed (e.g. training using an e-learning tool) whereas the control group does not (or receives an appropriate control treatment).
The target value (variable in question, e.g. learning performance) is measured in both groups, if necessary, before the intervention, but always after the intervention (measurement = O).

The purpose is to exclude all factors which could possibly have an influence on the target value (except the treatment). The differences which arise between the intervention and control groups or with respect to the learning performance can be ascribed only to the treatment. An experimental design is necessary, for example, if the added value of a learning technology is to be demonstrated relative toa “more conventional” learning system.

One-off measurement with a control group:

IG: X --> O

CG: (X_C) --> O

Example: A group (IG) of students is asked to work with a learning program on topic z and afterwards a variable of interest, e.g. knowledge on topic z, is measured. The same measurement is also conducted in the control group (CG) that has not used the learning program (or that has received a ‘control treatment’ (XC), e.g. learning using a different teaching aid).

The control treatment should be chosen in such a way that both treatments only differ in the factors to be evaluated. For example, in a comparison of technology-based learning with non-technology-based learning, all other learning-relevant factors should be kept constant. This means, for example, that both groups should be taught by the same teacher, that the content should be the same, that the didactic design of the course should be identical, etc. (Tergan’s presentation "What makes learning successful?" explains a large number of learning-related components and can be downloaded in German from http://www.evaluationsnetz.de/index.php?cat=0&id=10005000&SID / Ergebnisse des Fachgesprächs / Vortrag Lernerfolg).

Before and after measurement with a control group:

IG: O --> X --> O

CG: O --> (X_C) --> O

Example: A group (IG) of students is asked to work with a learning program on topic z and the variable in question, e.g. knowledge on topic z, is measured before and after the event. Before and after measurements are also conducted on a control group (CG) that has not learned using a learning program (or that has received a ‚control treatment’ (X_C), e.g. learning using a different teaching aid). By contrast to one-off measurement, this design has the advantage that the learning performance can be measured reliably by constructing a difference between before and after measurement. Furthermore, it can be verified whether there is already a difference between the two groups in terms of prior knowledge.

Needless to say, if a knowledge test is used, the same questions should not be asked iin both the before and after measurement as the test experience for the first measurement can itself bring about learning. Firstly, a set of test questions should be generated from the learning material and these should then be assigned to the before or after measurement randomly.