Evalguide - Design

Quasi-experiments differ from experiments in that the participants are not assigned randomly to the intervention and control group, i.e. naturally existing, non-randomised groups are investigated (e.g. two different school classes). The non-random assignment means it cannot be ruled out that both groups differ in variables that have an effect on the target value in question. Thus, when it comes to the interpretation of possible differences between the groups, one cannot simply assume that the difference can be ascribed to the ‘treatment’ (the intervention to be evaluated).

Quasi-experimental designs with (non-randomised) control group

One-off measurement with a non-randomised control group:

IG: X --> O
CG: (X_C) --> O

Example: A group (IG) of students is asked to work with a learning program on topic z and, afterwards, a variable of interest, e.g. knowledge on topic z, is measured. The same measurement is also conducted in the control group (CG) that has not used the learning program (or that has received a ‘control treatment’ (XC), e.g. using a different teaching aid). If the measurement in the IG results in a ‘better’ score than that of the CG, it can be argued with more certainty than in a one-off measurement without a control group (see below) that the learning program has had an effect. However, there could already have been a difference in the groups with respect to a variable that relates in some way to the variable in question (knowledge) prior to the treatment!

Before and after measurement with a non-randomised control group:

IG: O --> X --> O
CG: O --> (X_C) --> O

Example: A group (IG) of students is asked to work with a learning program on topic z and the variable of interest, e.g. knowledge on topic z, is measured before and after the event. Before and after measurements are also conducted on a control group (CG) that has not learned with a learning program (or that has received a ‘control treatment’ (XC), e.g. using a different teaching aid). By contrast to one-off measurement, this design has the advantage that the learning performance can be measured reliably by constructing a difference between the before and after measurements. Nevertheless, the interpretation of possible differences between the groups should take into account that the groups may not only differ with respect to the treatment!

Quasi-experimental designs without a control group

One-off measurement without a control group:

X (Treatment) --> O (Measurement)

Example: A group (IG) of students is asked to work with a learning program on topic z and, afterwards, a variable of interest, e.g. knowledge on topic z, is measured. This survey design is often problematic since it does not allow any conclusion as to whether the variable (knowledge) can really be attributed to the treatment. Nevertheless, the effect of the treatment can sometimes occur, if, for example, it can be ruled out based on the syllabus that the students had had any knowledge about topic z before the treatment. It is still problematic that there is no value existing for comparison. Even if one is able to prove that something has been learned, the question still remains whether ‘a lot’ or ‘a little’ has been learned and whether the quality of learning is ’good’ or ‘poor’.

Before and after measurement without a control group:

O --> X --> O

Example: A group (IG) of students is asked to work with a learning program on topic z and the variable of interest, e.g. knowledge on topic z, is measured before and after the event. By contrast to one-off measurement, this design has the advantage that the learning performance can be measured reliably by constructing a difference between the before and after measurements. The improvement of the values between the first and second measurements can be interpreted as the effect of the treatment (X) only when there is no other plausible explanation!