CHAPTER 3 METHODS

Participants The participants in this study were 220 students at Kyoto University, one of the leading institutions in Japan, whose ages ranged from 18 to 25. All the participants were Japanese nationals who grew up in Japan speaking Japanese. They were enrolled in four English courses I taught in the academic years 2009 and 2010. All of them had passed a highly competitive university entrance examination that included an English section. The English section of the entrance examination required the participants to translate passages both from English into Japanese and from Japanese into English. They had had at least 6 years of formal pre-college English education that was focused mostly on reading and translation, (i.e., the grammar-translation method). Their level of communicative proficiency, including listening comprehension, was relatively low. The average TOEFL iBT section scores for those who took the test and submitted their scores voluntarily were 23, 19, 15, and 22 respectively for Reading, Listening, Speaking, and Writing. The mean total score was 78. The participants were informed that their proficiency profiles were being measured and they agreed to the use of scores

for statistical analyses.

Who are they (e.g., first year high school students)? How many? How many are male/female? What is their age range (and the mean)? What is their ethnicity? How/why were they selected? How were they assigned to groups? What is their general English proficiency? How many years of formal English education do they have? How many classes (and what classes) are they currently taking? What do readers need to know about the school (e.g., highly competitive? public/private? all girls/boys/coeducational?). How were the participants informed of the study? If they signed a written consent form, put an example of that form in an Appendix. Do not refer to the people who took part in your study as subjects. Instead, use words such as participants, individuals, college students, or respondents. Use active voice when describing what the participants did. For instance, The participants completed the questionnaire is preferable to The questionnaire was administered to the participants.

Instrumentation Short Listening Test The Short Listening Test (Appendix A) is a listening comprehension test used in the TOEFL PBT. Listeners hear a conversation between a male and a female speaker of about 10 seconds in duration. The conversation is followed by a question. Students are to select the correct answer out of four printed choices. Many questions require some inferential thinking rather than a straightforward understanding of factual information. There are 30 questions, and the test takes approximately fifteen minutes to complete. The test is designed to measure students understanding of the content of a short conversation. While they have to comprehend implied meanings correctly as well as factual information, they are not required to synthesize many pieces of information to construct a holistic understanding of the passage. An example is given below. 1. (woman) Thanks a lot! This scarf will be perfect with my blue jacket. (man) Made a good choice, did I? (narrator) What does the man mean? Answer Choices (A) He wants to know which scarf the woman chose.

(B) He wants to know what color the jacket is. (C) He thinks he selected a nice scarf. (D) He thinks any color would go well with the jacket.

Long Listening Test The Long Listening Test (Appendix B) is a TOEFL iBT-type listening comprehension test made up of lectures and conversations lasting 3 to 5 minutes. The conversations are typically 3 minutes and the lectures 5 minutes. While listening, students are allowed to take notes. There are 5 to 6 questions accompanying each conversation or lecture. Examinees are asked to choose the correct answer(s) typically from four printed options. Approximately one fifth of the questions require listening to a part of the conversation or lecture again. In this case, the question is not printed, but otherwise, the questions are printed on the answer sheet along with the answer choices. This is a test of whether listeners have gained a holistic understanding of a longer text, be it a conversation or lecture. Students are asked to answer questions including bridging inference questions and implication questions. This type of listening comprehension is the dependent variable in this study.

Vocabulary Size Test The Vocabulary Size Test (Appendix C) is a newly developed 140-item vocabulary size test designed to measure written receptive knowledge of the first 14,000 words of English (Beglar, 2010; Nation & Beglar, 2007). There are 10 items for each of the 14 levels ranging from 1,000- to 14,000-word level. In this study, the participants completed the 3,000 through the 8,000 word levels. Students were given 5 minutes for each level to complete a 10-item test. SO WERE THERE 50 ITEMS ON THE TEST? IS 5 MINUTES CORRECTIT SEEMS VERY SHORT. Given below is an item at the 5000 word level. 1. deficit: The company had a large deficit. a. b. c. d. spent a lot more money than it earned went down a lot in value had a plan for its spending that used a lot of money had a lot of money stored in the bank

Written Expression: Grammatical Error DetectionTest The Written Expression: Grammatical Error Detection Test (Appendix D) is a Written Expression test that is based on a section on the TOEFL PBT. There are 25 sentences and four segments, words and expressions are underlined in each sentence. Test takers

are asked to identify the segment that contains a grammatical error. This measures the students receptive grammatical knowledge. Here is the first item on the test. 1. Margaret Mead (A) studied many (B) different cultures, and she was one (C) of the first anthropologists to photograph (D) hers subjects.

Structure: Gap Filling Test On the Structure: Gap Filling Test (Appendix E), students are asked to fill in the blank in a sentence with one of the four given choices, much like in TOEFL-PBTs Structure Section. There are 30 items on this test. The test is designed to measure the students syntactic awareness and vocabulary knowledge. A typical item is shown below. 4. The discovery of the halftone process in photography in 1881 made it ------photographs in books and newspapers. (A) the possible reproduction (B) possible to reproduce (C) the possibility of reproducing (D) possibly reproduced

Listening Cloze Test The Listening Cloze Test (Appendix F) is not a strictly listening cloze test. Such a test

has been prepared and successfully administered before (Buck, 2001, p. 69; Templeton, 1977). However, technical difficulties prevent me from making this test strictly listening-based. Students are given a script with approximately every 10th word missing and they fill in the blanks with the words they hear. There are two passages with 25 blanks each for a total of 50 blanks. Spelling mistakes are permitted, but not grammatical ones such as the incorrect tense and singular-plural errors. As a result, this is to some extent a test of syntactic and grammatical knowledge in addition to word recognition in a flow of speech. As for the effect of written text in terms of general listening comprehension, known as the aural-written verification effect, Chang (2009) has shown that the gain in comprehension was only 10%. In addition, the speed of speech makes it almost impossible to engage in extensive backtracking and repetitive reading.

Reading Cloze Test The Reading Cloze Test (Appendix G) is made up of a text with approximately every 10th word removed. Students are asked to supply the words to fill in the blanks as they read the passage. The first one or two letters are provided where such was deemed appropriate and/or necessary. There are two passages with 25 blanks each for a total of

50 points. Spelling mistakes are allowed, but not grammatical ones such as a wrong tense and singular-plural distinction. As a result, this is to some extent a test of syntactic and grammatical awareness in addition to vocabulary knowledge. This test is one of the instruments to shed light on the differences between listening and reading.

Difficult Words 50: Word Recognition Test In the Difficult Words 50: Word Recognition Test (Appendix H) test, students listen to 50 words that are presumed difficult to recognize aurally for native speakers of Japanese. After hearing each word, they have 5 seconds to write it down. This test is designed to measure word recognition skill when the words are presented in isolation and out of context. Here are some examples of the words included in the test. annihilate, conscience, heir, plumber, thorough

Reading Comprehension of Listening Scripts Test The Reading Comprehension of Listening Scripts Test (Appendix I) requires test takers to read a listening script in the same amount of time that it takes to listen to it and answer the same questions that appear on the corresponding listening test. The time limit for the entire test is also equal to the listening test, which is typically 10 minutes. Because all the tests are to be given within one semester, the students read the scripts of

comparable listening tests, but did not read the scripts of the listening comprehension tests they took. The use of the listening scripts, both of conversations and lectures, is one unique feature of this test. This is being done in an effort to control for the effect of the text type, written vs. spoken.

Productive Vocabulary Levels Test Nations Productive Vocabulary Levels Test Version A (2,000-word level and 3,000word level test) (Appendix J) was adopted in this study. The test is made up of 18 sentences with one omitted word for each level. The first few letters of the omitted words are already provided, and the students complete each word in 15 seconds. While grammatical mistakes are not acceptable, examinees are not penalized for spelling mistakes. Admittedly, this is a very restrictive method to measure productive vocabulary compared with, for example, analyzing free speech or writing. It is probably best to regard this test only as one parameter that is related to productive vocabulary. One example each from the 2000 and 3000 level test is exhibited below. THE 2,000 WORD LEVEL A 9. They managed to steal and hi some knives.

THE 3,000 WORD LEVEL A 12. They get a big thr out of car-racing.

Dictation Test The Dictation Test (Appendix K) is made up of tree short sentences and three longer sentences. One short sentence has 8 words, and the other two are 11-word sentences. The longer sentences were 14 words, 16 words, and 17 words, respectively. The examinees have 30 seconds to write each of the three short sentences. The three longer sentences are read twice. After each reading, there is 30 seconds of silence during which the examinees write the sentence. Each correct word is awarded 1 point, and there is no penalty for misspelled words. Grammatical mistakes are not awarded any credit. Furthermore, if there is an extraneous word not in the original sentence, one point is taken away. This test is designed to be a measure of aural word recognition in context as well as grammatical and syntactic knowledge. One short sentence example and one long sentence example are given below. 2. You did an excellent job on that presentation. 6. I really want to see the play at the outdoor theater tonight. Will you come with me?

Phoneme Distinction Test: Consonant Pairs Phoneme awareness and distinction are known to be greatly assisted by literacy. As one of Kyoto University students notable strengths is reading comprehension, it is possible

that their aural consonant distinction shows significant correlation with reading, let alone listening. This instrument (Appendix L) measures the ability to tell the difference between two consonants, embedded in a word, which are confusing to native speakers of Japanese. The consonant pairs were [b, v], [f, h], [l, r], and [s, th] as shown below. Five pairs of words were given for each of these consonant pairs. 1 6 11 16 21 lice clue vase mouse food rice crew base mouth hood

Metacognitive Awareness Listening Questionnaire (MALQ) Metacognition is knowledge of cognitive processes and the ability to monitor, control, and integrate these processes in an orchestrated manner; i.e. self-appraisal and selfmanagement of cognition also described as self-reflection and self-direction (Flavell, 1976, 1979). Successful learners share the ability to apply metacognitive knowledge. Vandergrift et al developed a questionnaire called Metacognitive Awareness Listening Questionnaire (MALQ) to study this aspect of listening ability development and have shown, using multiple regression analysis, that up to 13% of variance in listening comprehension can be explained by metacognition (Vandergrift, Goh, Mareschal, & Tafaghodtari, 2006). MALQ measures person knowledge which is awareness of ones

own listening and learning processes including abilities and difficulties, task knowledge which means such knowledge about the task as the purpose, demands, and nature, and strategy knowledge which guides one in applying and adapting a range of strategies at their disposal to meet the demands of different listening situations. It is a 21-item questionnaire with a 6-point Likert scale ranging from strongly agree to strongly disagree. The first question is shown below. In the study conducted by Vandergrift, participants answered the questionnaire in English. However, we translated the questionnaire into Japanese to minimize the interference/noise due to reading comprehension in L2 (Appendix M). Wherever applicable, the same question was asked about listening in their mother tongue of Japanese as well. Typical items look like these. 2 English

1 1

2 2

3 3

4 4

5 5

6 6

17 English

Research Design If you are doing experimental research, it is a good idea to diagram the design. Diagrams for mixed-method studies can be found in Cresswell (2003) pages 213-214. Explain the strengths and weaknesses of the design and your reasons for choosing it.

Procedures Describe the procedures clearly enough that readers can replicate your study. Use chronological order to describe what happened first, second, third, etc. The following is a step by step description of our data analysis procedures.

Outlier Detection and Elimination The data will be purged of univariate outliers first. As the sample size is larger than 80 cases, cases with a z-value beyond the interval [-3, +3] will be regarded as an outlier (Schwab). Second, Mahalanobis distance will be computed to detect multivariate outliers. If the probability associated with the distance is .001 or less, the case will be regarded as an outlier (Schwab).

Rasch Analysis

Many of the tests had a large number of items and were conducive to Rasch analysis, as discrete data, mostly dichotomous with one polytomous data, were obtained. Rasch analysis rescales raw data to an interval scale on which item difficulty and person ability estimates are placed. The Rasch analysis also allows for the identification of misfitting items and can be used to detect the presence of secondary dimensions in the data.

Correlation Studies Pearson correlations will be calculated for all possible combinations of the HOW MANY? 9 (David, this may change downward a bit depending on how I decide to combine the test results.) variables to get an overview of how the variables are interacting. Because there will be 36 (This may change accordingly.) correlations, some might be products of random fluctuations at p < .05. Additionally, partial correlation coefficients will be examined to control for potential linear effects of other variables. In this study of multiple aspects of linguistic fluency with an emphasis on listening comprehension, correlations among multiple variables are unavoidable by the very design of the study.

Factor Analysis and Principal Components Analysis I use factor analysis as an umbrella term covering both factor analysis and principal

components analysis in this section. According to Wilkinson, Blank, and Gruber (1996), Principal component and common factor solutions for real data rarely differ enough to matter. Factor analysis might provide one solution to dealing with the network of correlations described in the previous section. However, there is no guarantee that the underlying constructs are easily interpretable and conducive to ready applications to teaching and learning. I consider factor analysis as yet another approach to investigating the relationships among predictors of listening and reading and defer the decision as to whether I will use variables or the factors for further investigation until the factors have been closely examined.

Multiple Linear Regression Linear regression analyses will be conducted with the Long Listening Test result as the dependent variable and other variables as predictors. This is one way to estimate the variance accounted for by each independent variable. When Rasch analysis is conducted, person ability measures instead of raw scores will be used. *Removed* SIMPLE LINEAR REGRESSION WILL JUST GIVE YOU THE SAME ANSWER AS THE PEARSON CORRELATION, SO THERES NO NEED TO DO IT. There will necessarily be correlations among all the measurements, if not exact

collinearity. Automated procedures including forward variable selection, backward elimination, and stepwise variable selection will be tried first followed by adjustments by hand as recommended by Noruis (2008, p. 260). Admittedly, nothing in nature is truly linear, and this is an approximation and estimation. However, this technique has the advantage of easy interpretation. Because so many researchers employ this method to assess the amount of variance each independent variable is responsible for, this allows me to compare my results with the existing literature.

Time Schedule 1. More than 90% of the tests have been given and the corresponding data have been collected. Some tests have been given twice to two sets of students in 2009 and 2010. The remaining tests will be administered by the end of July. 2. Data analysis will proceed through the summer of 2010 and will be completed by the end of September. 3. The dissertation proposal will be ready for defense in the Spring Semester of 2011.

I CHANGED #3 TO DISERTATION PROPOSAL. WHEN DO YOU HOPE TO DEFEND THE PROPOSAL? TO BE DECIDED IN CONSULTATION WITH YOU LATER.

ADD A TIME SCHEDULE FOR CONDUCTING THE STUDY AT THE END OF THE METHODS CHAPTER. THIS SCHEDULE CAN BE DELETED ONCE THE FINAL DRAFT OF THE PROPOSAL HAS BEEN ACCEPTED AND FILED.

REMINDER: USE CORRECT HEADINGS Level One Heading: Centered, First Letter of Each Word Capitalized

Level Two Heading: Centered, Italicized, First Letter of Each Word Capitalized

Level Three Heading: Left Justified, Italicized, First Letter of Each Word Capitalized

Note 1: If part of your dissertation involves the validation of an instrument, it may be best to devote CHAPTER 4 to the validation effort and move the chapter titles back accordingly (i.e., CHAPTER 5: RESULTS; CHAPTER 6: DISCUSSION, etc.). In this case, you could name CHAPTER 4 THE VALIDATION OF AAA.

Note 2: If you are conducting a large number of preliminary analyses (e.g., factor analyses of questionnaires) and those preliminary analyses do not answer any research

questions, it may be best to devote CHAPTER 4 to that purpose and start CHAPTER 5 (RESULTS) by immediately answering research questions. In this case, you could name CHAPTER 4 PRELIMINARY ANALYSES.

Note 3: If you are conducting multiple experiments, it may be best to devote one chapter to each experiment. In that case, you may name chapters as follows: CHAPTER 4: EXPERIMENT 1, CHAPTER 5: EXPERIMENT 2, etc. Each of these chapters would be self-contained in that you would describe the methodology and results for the experiment in that chapter. Your RESUTS chapter would then become a summary of the results of the individual experiments and your DISCUSSION chapter would be a discussion of those results.