Dept. of Educational Psychology & Learning Systems
Florida State University
True Experiments
Experiments can be classified as True experiments and Quasi-Experimentsand their differences will become obvious later. Let's begin first with true experiments which are very specialised tools ideally suited for the task of causal analysis. No other method of scientific inquiry permits the researcher to say with confidence that "X caused Y to happen". eg. praising young learners motivates them to do the task repeatedly. However, experiments are difficult to conduct. Many experiments in educations are concerned with testing the effect of certain interventions, the effectiveness of new eduational materials or certain educational practices on learning, attitudes, preceptions, etc.among students.
The experimental method is the only method that attempts to show "effectiveness" (keberkesanan) of a particular treatment (eg. teaching method, curriculum innovation). That is, it is a method that can claim to show "cause" and "effect". In other words, to show whether the inquiry method "causes" students to perform better on critical thinking ("effect"), the experimental method will have to be used. Hence, it is important that you use the word "effectiveness" carefully, as it only applies if you are using the experimental method.
The key problem in conducting experiments is establishing suitable CONTROL,so that any change in the posttest can be attributed only to the treatment that was manipulated by the researcher. Control means ruling out other possible causes for the changes on the posttest. There are many extraneous variables that need to be controlled so that they do not contaminate or interfere with the findings on the posttest. The problem of control is more serious in educational experiments where it is difficult HOLDING all variables that might affect the posttest.
The "true" experiment is one in which subjects are randomly assigned to the experimental and control groups. Random assignment means that each sampling unit (eg. student, teacher, class, etc) has an equal chance of being selected in the experiment. In designing an experiement, you should ensure random assignment as it is the best technique available in establishing that the two or more groups are equivalent. Equivalent means that the subjects in the two or more groups have more or less similar characteristics, such as similar ability levels, similar attitudes, similar number of males and females, similar experiences, similar socio-economic backgrounds, etc.
A popular technique used to ensure random assignement is to use the Table of Random Numbers. Say you have 100 subjects to be assigned to an Experimental Group and a Control Group. Assign number 1 to 100 to the subjects. Then, refer to the Table of Random Number and select a starting point, let's say the top of the fourth column has the numbers 25, 38, 145, 78, 98, 56 and so forth. You will select subject no. 25 assigned to the Experimental Group followed by subject no. 38 assigned to the Control Group. Of course you will ignore number 145 because it is outside the 100 subjects. You will continue this procedure until all 100 subjects have been assigned to the two groups.
Introduction
The experimental method is the most scientific of methods used in the behavioural sciences. Its origins are traced to the field of agriculture where experiments were conducted to test the effectiveness of various kinds of treatments (eg. fertilisers) on plant growth and yield. The method is also widely used in the medical sciences especially in testing the effectiveness of various kinds of treatments (eg. new drugs, methods, procedures) on patients.
Internal Validity of an Experiment
In conducting experiments, you should ensure that your design has the highest internal validity possible. The internal validity of an experiement is the extent to which extraneous variables have been controlled or ruled out by the researcher. Internal validity is an indication that the results you obtain are caused by the treatment you administered and not some other variable or factor. For example, in your experiment you treated the experimental group of 4 year olds with the 'whole word method of reading' and discovered that their reading ability increased by 50% compared to the control group who were treated with the 'phonics method of reading'. How do you know that the increase in reading ability scores of the experimental group is DUE to the 'whole word method' and not some other factors or variables? In other words, an experiement with high internal validity means that the probability that the treatment caused the change is higher.
However, there are many threats to internal validity such as the following:
History: Events that have changed in the subjects environment. Between the pre-test and the posttest, certain events occured that might affect the scores.
Maturation: Subjects may change between the pre-test and posttest resulting in subject becoming more mature.
Instrumentation: The change in instruments used in the pretest and the posttest can lead to changes in measurement. For example, an easier test used in the posttest will result in better performance in the posttest because of the instrument and not the treatment.
Testing: Subjects remember the questions in the pretest and if the same test is given as a posttest, the chances are they may score higher in the posttest. i.e. they have become "test-wise". The time period between the pretest and the posttest should not be too short such that subjects can recall.
Mortality: Sometimes referred to as "attrition" is when subjects drop out from the experiment which can affect the experiment. This is especially serious when subjects of a particular characteristic (eg. hogh ability) systematically drop out.
Selection Bias: This happens when the subjects selected for the experimental group and the control group were not equivalent before the treatement leading to misleading conclusion. For example, if the experimental group had 50% high ability subjects while the control group had only 25% high ability subjects, higher performance on the posttest may not be attributed to the treatment but due to non-equivalent subjects in terms of ability.
Regression to the Mean: When subjects with extreme scores on a test are selected, there is a likelihood that when they are retested later on a measure that is correlated with the first test, their scores will move towards the mean. For example, students who performed poorly are selected for training; their average posttest scores will be ghiger than their pretest scores because of statistical regression even if no training were given.
What is an Experiment?
An experiment is a researh method in which the researcher wants to know the effect of using a particular treatment on a group of subjects. To show that a particular treatment has an effect, the researcher has control all other factors that might influence the subjects. For example, the effect of the inquiry method on the acquisition of science concepts among Form 1 students. The TREATMENT is the Inquiry Method which is also the INDEPENDENT VARIABLE; the DEPENDENT VVARIABLE is Acquisition of Science Concepts. The PRETEST (testing the acquisition of science concepts) is given before the treatment while the POSTTEST (preferably the same test as the pretest) is given at the end of the treatment. The EXPERIMENTAL GROUP consists of subjects who are randomly assigned to get the Treatment while the CONTROL GROUP consists of subjects who are randomly assigned to get NO Treatment.
External Validity of an Experiment
When one conducts an experiement, one is interested in establishing the extent to which the findings of the study can be applied to another setting and this is called EXTERNAL VALIDIY. However, it is possible for the findings of your experiment to be to be externally valid for setting X, less externally valid for setting Y and not all externally valid for setting Z. Glen Bracht and Gene Glass in their article, The External Validity of Experiments, American Educational Research Journal, 5, 1968. 437-474, identified the following factors that might affect the external validity of an experiement (i.e. generalisable to other settings).
Clear Description of the Treatment: To enable someone else to replicate your study, the researcher needs to know in detail what is your treatment and how you administered it. Otherwise, the researcher will not be sure the experiment is done the same way you did it and so the your finidngs cannot be generalised to the intended study.
Hawthorne Effect: If a researcher is not careful, there is a tendency for some of them to pay special attention to subjects in the experimental group which may influence their behaviour. External validity is compromised when in another study reseachers are aware of this phenomenon and take precautions.
Novelty Effect: Subjects in the experimental group succumb to the novelty effect because the treatment given is different from what they are used. Subjects tend to be enthusiastic in the beginning but as the treatement continues the novelty effect wears off and the results of the experiment have low generalisability.
Experimenter Effect: A particular treatment is effective or ineffective because the teacher or instructor influences the way the treatement is administered (introducing certain biases). This jeopardises the findings to another setting when the teacher is a different person with different characteristics.
Pretest Sensitisation: If the pretest is part of the treatment, it will obviously affect the results or findings.
History and Treatment Interaction: It is possible that because of certain events, subjects in the experimental group expressed strong feelings in the posttest. At another point of time where such events are not present, the subjects did not react accordingly; thus, challenging the generalisability of results of the the earlier experiment.
Measurement of the Dependent Variable: In your experiement, you had used a multiple-choice test to measure the effect of the treatment. To generalise your results, the next time the experiment is conducted multiple-choice test should be used. If an essay test is used, you cannot generalise earlier findings.
Other techniques to ensure that the subjects in the control group and experimental group are IDENTICAL
One of the difficult tasks for a researcher using the experimental method is getting two or more identical groups. Imagine the difficulty of finding two people who are similar on every characteristic such as IQ, attitude, aptitude, mathematical abilty, etc. As mentioned earlier, random assignment is a powerful way to ensure that subjects assigned to the various groups will have more or less similar characteristics. Are there other techniques?
1) Matching
Determine a particular factor, for example, academic performance which is measurable and categorised as High and Low. From the sample, select two High Ability subjects and randomly assign them to the control group and the experimental group. Next, select two Low Ability subjects and assign them randomly to the control group and the experimental. Continue doing this until all subjects have been assigned and your two groups are matched in terms of academic performance.
Another technique of matching is to give the pretest and based on the scores obtained assign subjects to the control group and the experimental group. However, you should ensure that the average score or mean score of the pretest should be the same for the two groups. eg. two subjects with mean of 23; two subjects with mean 30; two subjects with mean 34 and so forth.
2) Holding One or More Variables Constant
Another technique is to hold a particular variable constant. For example, in an experiment you have difficulty ensuring that the control group and the experimental have an equal number of high SES and low SES subjects. You could only take only low SES subjects and assign them randomly to the control group and experiemental group if you are not interested in comparing high and low SES subjects. What you have done is to eliminate the SES factor or variable by including only low SES subjects; i.e. controlling by holding SES constant across the two groups you are comparing.
3) Including an Extraneous Variable in the Research Design
You could control a variable by including it in the design by making it another independent variable. For example, you design an experiment to test the effectiveness of getting students to define concepts using their own words on performance in economics. However, you find it difficult to control for prior knowledge in economics among your subjects. You could include only those who have low prior knowledge based on a test on economics you administered or you could categorise prior knowledge as High, Medium and Low based on test scores and treat prior knowledge levels as an independent variable. However, you should use this technique only if you are interested in the influence of prior knowledge on performance. What you have done is to control the influence of prior knowledge on other independent variables by including it in the research design.
4) Analysis of Covariance (ANCOVA)
It is a statistical method used to ensure that the subjects in the control group and the experimental group are equivalent on various factors. ANCOVA adjusts the scores on the dependent measure for the differences found on the pretest and statistically equates the subjects in the control and experimental group. For example, you are conducting a study on the effectiveness of metacognitive training on the critical thinking skills of Form 4 students. However, you find that some subjects in your experiment are high achievers while the others are low achievers which may influence performance on the critical thinking test (i.e. the dependent variable). To ensure that all subjects in the control group and the experimental group are equal in academic achievement, the ANCOVA is used to adjust scores on the critical thinking test for the difference in academic achievement.
Experiments can be classified as True experiments and Quasi-Experimentsand their differences will become obvious later. Let's begin first with true experiments which are very specialised tools ideally suited for the task of causal analysis. No other method of scientific inquiry permits the researcher to say with confidence that "X caused Y to happen". eg. praising young learners motivates them to do the task repeatedly. However, experiments are difficult to conduct. Many experiments in educations are concerned with testing the effect of certain interventions, the effectiveness of new eduational materials or certain educational practices on learning, attitudes, preceptions, etc.among students. 
