Evalguide - Untersuchungsdesign

Quasi-experiments differ from experiments in that the participants are not assigned randomly to the intervention and control group, i.e. naturally existing, non-randomised groups are investigated (e.g. two different school classes). The non-random assignment means that it cannot be ruled out that both groups differ with respect to variables that have an effect on the target value in question. Thus, in the interpretation of possible differences between the groups one cannot simply assume that the difference is ascribed to the ‘treatment’ (the intervention to be evaluated).

Quasi-experimental designs with (non-randomised) control group

One-off measurement with a non-randomised control group:

IG: X --> O
CG: (X_C) --> O

Example: A group (IG) of students is asked to work with a learning program on the topic z and, afterwards, a variable in question, e.g. knowledge on the topic z, is measured. The same measurement is conducted also in the control group (CG) that has not used the learning program (or that has received a ‚control treatment’ (X_C), e.g. having learned with another teaching aid). If the measurement in the IG results in a ‚better’ score than that of the CG, it can be argued with more certainty than in a one-off measurement without a control group (see below) that the learning program has had an effect. However, the groups could have been differed even before the treatment with respect to a variable that relates in some way to the variable in question (knowledge)!

Before and after measurement with a non-randomised control group:

IG: O --> X --> O
CG: O --> (X_C) --> O

Example: A group (IG) of students is asked to work with a learning program on the topic z and the variable in question, e.g. knowledge on the topic z, is measured before and after the event. Before and after measurements are also conducted with a control group (CG) that has not learned with a learning program (or that has received a ‚control treatment’ (X_C), e.g. having learned with another teaching aid). By contrast to a one-off measurement, this design has the advantage that the learning performance can be measured reliably by constructing a difference between before and after measurement. Nevertheless, the interpretation of possible differences between the groups should take into account that the groups differ possibly not only with respect to the treatment!

Quasi-experimental designs without a control group

One-off measurement without a control group:

X (Treatment) --> O (Measurement)

Example: A group of students is asked to work with a learning program on the topic z and, afterwards, a variable in question, e.g. knowledge on the topic z, is measured. In many cases, this survey design is problematic since it does not allow any conclusion as to whether the variable (knowledge) can be really attributed to the treatment. Nevertheless, the effect of the treatment can sometimes occur, if, for example, it can be ruled out based on the syllabus that the students had had any knowledge about the topic z before the treatment. It still remains problematic that there is no value existing for comparison. Even if one is able to prove that something has been learned, the question still remains whether ‘a lot’ or ‘a little’ has been learned and whether the quality of learning is ‘good’ or ‘poor’.

Before and after measurement without a control group:

O --> X --> O

Example: A group (IG) of students is asked to work with a learning program on the topic z and the variable in question, e.g. knowledge on the topic z, is measured before and after the event. By contrast to one-off measurement, this design has the advantage that the learning performance can be measured reliably by constructing a difference between before and after measurement. The improvement of the values between the first and second measurement can only be interpreted as the effect of the treatment (X) when there is no other plausible explanation!