CRITIQUE

Hicks, R. B. & Laue, H. (1989) A Computer-Assisted Approach to Learning. Physics Concepts,

57(9) , 807-811.

My summary of the article

Introduction

First-year physics is a difficult subject for many students. Large classes make for restricted
access to tutorial help from professors and graduate assistants. To address this problem, the
department of physics with assistance from faculty of science, has developed a computer assisted
learning resource called CALIPH ( Computer Assisted Learning In Physics ). The resource
consists of more than 50 tutorial lessons in first year physics, including mechanics, fluid
mechanics, kinetic theory, and thermodynamics.

Much of the available instructional software provides drill in numerical problem solving or
acquaints students with physical situations by means of simulations. However, there is a lack of a
comprehensive set of tutorial lessons to attack the most common cognitive problems alluded to
previously in a systematic pedagogical way. It is extremely important that students be alerted to
all relevant details and assumptions when presenting new concepts and ideas. In the authors and
their colleagues' experience, errors resulting from incorrect preconceptions and a reluctance or
inability to apply appropriate definitions and physical laws are made by many students even
though lectures cover the relevant material repeatedly

Therefore individualized tutorial instruction is required for (a) the effective teaching of details
such as those which tend to get lost in a large lecture theatre; (b) students need to act upon the
information given to them at their own pace; and(c) students need individual feedback to their
thinking.

Assumption

Students in first year physics usually do not perform well. Poor performance may be due to a
variety of common misconceptions and difficulties. Apparent familiarity with the objects under
discussion may actually be one of the factors contributing to the malaise because one is tempted
to think about familiar objects in familiar way. Often students do not seem to realize that they are
being asked to make a significant change in their way of looking at the material world around
them. Therefore, the implementation of new instructional strategies focusing on restructuring of
the preexisting knowledge to produce a conceptual change is an important problem in physics
instruction.

Subjects
In total, 96 student volunteers from the current first year physics courses in the fall and winter of
1987-88 participated in the study. No computer experience was stipulated. Unfortunately, many
students failed to follow through with the second tutorial session due to workload or other
pressures and were replaced by new volunteers where possible. As a result, 46 students attended
the first session only, 17 attended the second only and 33 attended both sessions.

Instrument

In this research only 5 of the 50 tutorial modules were used. (Two in kinematics and three in
dynamics ). The first module dealt with calculation of average speed for a simple one-
dimensional motion. The purpose of this tutorial is to make students aware that care must be
taken to distinguish between a simple arithmetic average and a time-weighted average, as
required by the definition of average speed.

The second kinematics module attacked the "speed comparison

task" in which a uniformly accelerating object (a bus) starting from rest is first passed by, and
then passes an object (a bicycle) moving with constant speed. Students are shown the screen and
are asked whether the two objects had the same speed (a) only once (b) twice or (c) never.

The final three tutorials dealt with the frictionless block- and-pulley systems. These three
modules examine the relations among the weights and tensions in each system, demonstrate how
to draw correct "free-body" force diagram for each block in the system and help the student to
learn a systematic method of derivation to find expressions for the tensions and accelerations.

One specification of the tutorial modules is that multiple choice questions are encountered
frequently, requiring the student to consider material in an active way before proceeding; where
possible, questions address typical student misconceptions. Qualitative questions are presented
first before proceeding to quantitative questions. Also some practice with numerical problems is
provided, but is secondary to the presentation of concepts.

Students are occasionally asked to perform simple derivations or moderately lengthy calculations
using pencil and paper; the computer provides the correct answer and method as feedback.

Concepts and definitions are emphasized, graphics and animation are used liberally. Tutorials are
entirely mouse driven; no keyboard entry is required and considerable help is available in the
form of menu items and optional explanation sequences.

The other instrument used in this research questionnaire to measure the attitude of students
towards the tutorials which is not well explained in the article.

Hypothesis

Although the research hypothesis was not clearly stated but it could be inferred from the
introduction that the goal was to show that "it is possible to construct relatively simple,
attractive, and interactive computer program that facilitate the learning of fundamental concepts
and definitions in the absence of a human tutor while DISPELLING erroneous preconceptions
and other types of misunderstanding". It was also mentioned that considerable emphasis would
be placed on qualitative understanding and derivations.

Procedure

Each student agreed to attend two 2-hours sessions on Saturday mornings. The first session
involved the two kinematics modules only, while the second (3-4 weeks later) dealt with the
block-and- pulley tutorials. In all cases students sat at individual microcomputers with help from
the one or two instructors present being kept to a bare minimum.

To assess tutorial effectiveness, short written tests were provided at the beginning and at the end
of each session. Most test questions were aimed directly at the content of the tutorials for the
day. Test questions at the beginning and at the end of each session were very similar, although
modified in some case by changing input data. The initial test of the second session also included
a question on average speed in order to test retention of material learned in the first session.

A brief questionnaire filled out at the end of each session was used to get the feedback from the
participants. Student anonymity on written work was maintained throughout.

Analysis

By comparing test scores on similar written questions immediately before and after the tutorial
sessions, it was possible to determine whether an individual student had improved his conceptual
understanding of material by exposure to the computer tutorial. Most question asked were simple
enough so that a binary scoring system ( 0= unsatisfactory , 1=satisfactory ) was sufficient to
characterize student responses. Other tasks, such as derivations, required in addition an
intermediate score ( 1/2 =substantially correct but incomplete ). A table summarizes results for
selected tasks tested in the study. Only frequencies and percentages are computed and compared.

Results

It is apparent from the first column of percentages in the Table 1 that a sizable fraction of the
students in this study brought common misconceptions with them to the study even though the
relevant material had been covered in lectures. It is also evident that the tutorials achieved
significant success in attacking student deficiencies in a number of tasks but were less successful
with others. Specifically most deficient students (75% or more) clearly learned the definition of
average speed and its application and became familiar with aspects of block-and-pulley systems,
including derivations of acceleration and tension in the two simplest systems discussed.

The fallacy that the tension in a string supporting an accelerating weight is equal to the weight
was clearly dispelled, and students appeared able to generalize this concept to Atwood's
machine. However they were less successful in performing the derivation for Atwood's machine
even though they had learned derivation for similar systems.
It was noted that the success rate for the speed comparison task was relatively low (37%) even
though an entire tutorial module was devoted to aspects of this task.

To test the persistence of knowledge acquired in the first session, students returning 3-4 weeks
later for the second session were asked to define average speed. Of 20 students who had been
able to give a satisfactory definition at the outset of the study without any exposure to the
computer tutorials, 3 failed to do so at the second session. Among the returning students were 8
who had learned the correct definition in the first session; 4 of these could not give a satisfactory
definition at the second session.

The results of the questionnaire showed that students experienced little difficulty in operating the
tutorials even where experience with computers was totally lacking. Responses of the majority of
students were supportive. Students liked the use of graphics and animations. They also
commented that the explanations found in tutorials were clearer and more direct than those in the
textbook.

Most students indicated that they enjoyed being able to work at their own pace. Negative
comment was rare. Several students asked for treatment of more complex situations, stating that
the material covered by tutorials was "too easy".

Conclusions

The study shows that tutorial software constructed along fairly simple lines can be effective in
improving student performance in the virtual absence of the human tutors. Although many the
results of the study could have been predicted by experienced teachers, pilot studies of this type
are essential to test strategies prior to massive development of materials. Although no effort was
made to control the selection of students entering the study, favorable student responses and the
success rate achieved more than 80% for a number of important fundamental tasks have
convinced the authors that the development of further tutorials is desirable as perhaps the only
way in which one can actively engage many students in the thought process required to master
basic physics concepts.

Critiques
Importance of the problem investigated in the article.

An important problem in physics instruction is "how to help students achieve a deep conceptual
understanding of the subjects and how to help them develop powerful problem solving skills".
The importance of the problem can be easily realized by comparing the number of studies in
physics instruction with other sciences such as chemistry, biology or social sciences. So I think
the goal of this research is clearly stated and its importance is well justified.
Although the topic is not a new ground but it is important. It should be mentioned here that most
of the research studies which are reported in different physics instruction journals, are not really
experimental. However this one is at least a quasi-experimental one.

The scope of the problem is reasonable but the conclusions are over generalized. Actually, the
theoretical bases for the effectiveness of CAI in physics is not conclusive enough to give the
researcher a good theoretical work frame. Therefore, this study and most of studies in this field
are more application oriented rather than theoretical based.

Review Of Literature

Given the fact that the CALIPH is made by the authors of this article and that this study is the
first study on its effectiveness, one can not expect to find a similar or even a related study on
CALIPH. Furthermore, generally the effectiveness of CAI in physics is a new field in science
education.

However, there are enough related literature which could have been mentioned in the present
study. For example, there are several classical and meta-analytical studies (Kulik,1980, Mc Neil,
1991) which are worth mentioning to justify the importance of the problem and to give the reader
some information about the controversies in the field. Only two not very related 11 studies
( Trombridge & Reife, 1982 ) were mentioned, which is not sufficient and also they were not
well explained in this article.

Hypothesis.

The hypothesis "the CALIPH can dispel common student misconceptions in the first year
physics" is not clearly stated. One must search through the whole article to find the indirectly
stated hypothesis. The dependent and independent variables are also stated implicitly but not
appropriately. No operational definition and even a simple definition of variables were stated.
For example, several words (e.g. significant, effective, sizable, majority, conceptual
understanding, several, most ... ) are not defined. Therefore, one can not conclude, for example,
what proportion of students are called majority or what the conceptual understanding means.

The underlying assumption was that "physics students come into class with some preexisting
misconceptions". This assumption had been made on the authors' and their colleagues'
experience. It is not based on research or theoretical models. It should be mentioned here that the
content of tutorials is based on this assumption and so it was expected that the authors would
have tried their assumption and to clarify what these misconceptions were.

No pilot study was done before the construction of CALIPH and before starting the present
study. However at the end of the study the authors realized that it had been necessary and that
they should had done it.

Sample
The subjects of this study were 96 volunteer students from first-year physics courses. It is not
stated that the subjects were just physics majors or other students who take physics courses were
also included ( for example, engineering or other science students). Also demographic
characteristics of the subjects (for example, gender, age, SES) was not reported in the article.

Furthermore, a volunteer group is not a representative sample because they may be higher
motivated or higher achiever students. This problem specially pertains to this study because
previous researches have shown that motivation has a crucial role in effectiveness of computer
assisted instruction. The result of this study also supported such a possibility because it is
reported that students enjoyed the tutorials and stated that they are "TOO EASY." This shows
that mainly top students participated in this research.

Another major problem with the sample was that many students failed to follow with the second
tutorial session and they were replaced by new volunteers. It is obvious that when we add a new
group to an experiment, any conclusion should be considered with caution. Although it is
claimed that 96 students participated in the study, only 33 attended both sessions. It would have
been better to analyze the mixed and the original subjects separately.

Procedure & Instruments

Three instruments were used in this study. One was the short written test provided at the
beginning and at the end of each session. These short tests are briefly described but their
reliability and validity is not mentioned at all.

The second instrument was the questionnaire to measure the attitude of the students towards the
tutorials and the research. This questionnaire and its content and its validity and reliability was
not explained. Only the results of some of its questions were mentioned

The third instrument which actually was the independent variable (the tutorial modules ) was
well explained in the article. It is not clear which kind of test was used (pretest and posttest) to
measure the effectiveness of the tutorials, but it seems that they were not standard or even
validated tests.

A major problem with this report is that it is not clear exactly how many subjects were studied in
each session and how the new group was entered into the experiment. Actually the numbers
given in the page 809 and the numbers in the given table are different. Generally the goal, the
procedure and the results of the second session is not well explained and well justified.

Another problem is that the content of the tutorials was a repetition of the materials that had been
covered in the lectures. In fact any gain after working on the tutorials can not be merely the
effect of the independent variable but also can be the result of repeating the lectures in a new
form. Furthermore, this repetition may have caused a considerable reduction in the students'
motivation and may have had a grate impact on the results of the study. This methodological
problem could have been removed if two different groups (one with no experience with the
subject matter and one with such an experience) were studied.
Analysis

The major part of the results of the research is presented in a table, a copy of which is enclosed
with this paper. As it could be seen, the effectiveness of different tutorials is measured by
computing the percentage of students that were classified as unsatisfactory before working on
tutorials with the percentage of those that were improved by working on tutorials.

For example, in the first row ( definition of average speed ) 48% of students did not know the
definition before working on tutorials. But 81% of them were improved by using tutorials. No
statistical test has been used to measure the extent of the effectiveness of the tutorials. Therefore
the conclusion that the tutorials achieved significant success in attacking student deficiencies is
statistically meaningless.

The important analytical problem with this research is that the researchers used a binary scoring
system to measure students achievement before and after the treatment. It was concluded that
about 80% of students who were classified unsatisfactory in pretest were classified as
satisfactory in posttest. Why was a binary or a nominal scale used instead of an interval scale?
We know that the statistical analyses such as t-test or ANOVA are much more powerful than
non-parametric analyses. It is possible that if they had used a parametric analysis probably the
difference between the pretest and posttest means would not have been not significant.

The table used to present the results also had some problems. First it is not clear from the table
which result belongs to the first session and which is belong to the second session. Secondly as
mentioned earlier the number of students in the first session is reported (75) in the table, while it
is reported differently (79= 46+33) in section three of the article. It is not clear whether the
different

numbers is the result of missing values or other factors.

Furthermore, given that the research had been done in two sessions it would have been better to
use another table to show the rate of retention of learning during the time interval between two
sessions. If an interval scale was used in both pretest and posttest the rate of retention could have
been measured by a t-test.

Results and Conclusions

The authors concluded that the tutorials had been effective in improving students' performance in
the virtual absence of human tutors. So they believe that their hypothesis is supported by the
reported results.

It was hypothesized that the tutorials can dispel some preexisting misconceptions of the physics
students. These misconceptions are not clearly explained and systematically measured. After
reading the results of the article one can not find whether these misconceptions were observed or
measured. Furthermore, there is no evidence to show that after working with tutorials these
misconceptions were dispelled. It should be noticed that the improvement of students from
pretest to posttest does not necessarily imply the existence and the dispelling of such
misconceptions. A wrong answer to a question may have several reasons, not necessarily one,
which the authors predicted.

Furthermore, as mentioned earlier, several methodological and analytical problems with this
research imply that such conclusions are not well justified. The absence of a control group, the
invalid measurement of achievement (pretest and posttest), the inappropriate scale of
measurement and statistical method, the non-representativeness of the sample, and lack of
motivation of subjects all in all does not permit one to conclude that the hypothesis is supported.

Therefore this research not only did not show the existence and dispelling of specific
misconception, but also it did not show any statistically significant and meaningful improvement
in the students' achievement.

An important limitation of the results of this research is due to lack of attention to the
demographical characteristics of the subjects. For example if the sex, age, major, SES, and other
characteristics of the subjects were asked in the questionnaire it would have been possible to get
much more valuable information about the differential effectiveness of the tutorials. Some
previous research studies on other software programs have reported different effects for different
groups.

One of the advantages of this article is that whenever there was a shortcoming or a limitation in
the procedure or results it was frankly stated. For example, when the results showed that the
confusion of average and instantaneous quantities was evident in many responses, the authors
stated that the tutorials did not specifically address the concept of instantaneous speed. Or when
a negative effect of the tutorial was noticed for 3 of the subjects it was honestly mentioned in the
results. Also in the conclusion part it is stated that "Although many of the results of the study
could have been predicted by experienced teachers, pilot studies of these type are essential to test
strategies prior massive development of material "development of materials"

Some good suggestions are offered as a conclusion of the research. For example, when a low
success rate with the concept of equal tension on both sides of a massless, frictionless pulley was
noticed, it was suggested that this notion must be placed on an equal footing with other topics in
the tutorial rather than being relegated to a "HELP" item only.

Generally it seems that the authors have paid a particular attention to show the effectiveness of
the "CALIPH" and have not noticed the limitations and weaknesses of the research method. Even
the simplest and the most fundamental outline of research method was not noticed carefully in
this research. For example, the general format of the research report (introduction, procedure,
results...) was not in accordance with the standard format.

Furthermore, something, which should be mentioned in conclusion, was in fact mentioned in

introduction and something, which should be mentioned in the introduction, was mentioned in
the conclusion.

Some personal impressions and opinions can be seen in different parts of the article, which are
not data based or research based. For example, in the conclusion part it is stated that "the use of
attractive graphics and animation sequences engages students' interest and makes dynamic
physical situations more intuitively accessible."

All in all this study has several methodological and report problems, however, in comparison
with other research articles in the field (applications of computer in physics instruction ) is better
designed and more conclusive. It should be noticed that in this research, only 5 of more than 50
tutorials were considered and these five tutorials were not randomly selected. Actually they were
the simplest ones.

Although the is paper is criticizing the Hicks and Laue's (1989) article it should not be concluded
that CALIPH is not effective. All I have to say is that their article has methodological problems
and therefore, does not reflect the effectiveness of CALIPH.

I have seen and worked on many tutorials of the CALIPH. I think that it is one of the best
tutorials in first-year physics. I have talked with its authors about the popularity of CALIPH in
Canada. I know how well they have done on this 5-year project. But I think this report is not a
good report on a very good project. A much more comprehensive and well-designed research is
required to evaluate its effectiveness.