What does TOEFL test sometimes? Vyara Istratkova & Ellie Boyadzhieva South-Western University of Blagoevgrad General Introduction The choice of this topic has been provoked by some observations upon the preparation work and the successful accomplishment of some tasks by would-be Bulgarian takers of the TOEFL. Traditionally, Bulgarian takers of the TOEFL have very good results. It is mainly teenagers in their last year at high schools (most notably English Language schools) that take the test, the most common purpose being to continue their education at American universities. Many of them go in for courses offered for preparation for the TOEFL examination. The Bulgarian candidates‟ approach to the test is based mainly on the presumption that their high level of language proficiency is the necessary prerequisite for success. It does hold true to some extent as it is necessary indeed. Yet, language proficiency is not a sufficient condition for success, and that is what we will try to prove below. Aims of the TOEFL The TOEFL in used as a standard measure of the „English proficiency‟ of the candidates taking it. The TOEFL score is a compulsory requirement of the admission offices of different educational institutions and organizations so that foreign candidates are considered for admission. (In the 1996 edition on p.6 we read: “Many Universities use TOEFL scores to fulfill the foreign language requirement for doctoral candidates whose first language is not English” (TOEFL 1996:6). Actually, this applies to nearly all (American) universities and it refers to undergraduates as well. An interesting point is that the TOELF is to be taken even by British students. The material in focus and the authors’ goal As it is well known the TOEFL consists of three sections: Listening Comprehension, Structure and Written expression and Reading Comprehension & Vocabulary section and it tests the candidates‟ five skills - listening, structure, written expression, vocabulary and reading (TOEFL 1989:8). The material in focus presents samples from the 1st, 2nd and the 3rd part of the Listening Section of two different editions of TOEFL which (also) differ in their format – the Listening Section of the Sixth edition of Barron‟s TOEFL (1989) consists of a statement-restatement part, conversations and mini-talks, whereas in the Eight edition of Barron‟s TOEFL (1996) the Listening Section is comprised of short conversations, longer conversations and talks. The examples included (in the corpus) are illustrative, the authors‟ goal being to show that sometimes something more than language proficiency and proficiency in how to approach the examination is needed for the successful accomplishment of the listening tasks of the test. Theoretical background Tests are measurement instruments and (according to Carrol, 68) a “procedure designed to elicit certain behavior from which one can make inferences about certain characteristics of an individual” (Carrol 1968:46). In fact, tests are (or, ideally, should be) simulated real-life situations. On the basis of the performance of the candidate one should be able to make predictions about his/her language behavior in real life. The test itself should be based on the real needs and serve as basis for predictions about the ability of the students to cope with people and matters in reality. In the theory of testing and evaluation, it is a well-known fact that it is “neither theoretically nor practically possible to define either an absolutely perfect level of actual language performance or an individual with perfect language ability” (Bachman 1990:11). This can be regarded as a direct consequence of the fundamental dilemma in foreign language testing, namely that “the tools we use to observe language ability are themselves manifestation of language ability” (Bachman 1990: 9). Analysis Our personal observations show that in the TOEFL examination the most troublesome area appears to be the Listening Section of the test. Why should reading for example be easier than listening? After all, both are receptive skills. Yet, the phenomenon observed is not a surprising one - listening comprehension is found to be among the most difficult tasks for the learners due to several reasons. First, spoken language is ephemeral, and the examinees cannot refer to it whenever they want to as is the case with Reading comprehension texts. Consequently, they have to rely mainly on their memory abilities, which differ throughout individually. Second, examinees are pressed for time as the time is limited within the length of every particular text. Third, they have to develop different strategies for the different types of tasks depending on the types of exercises. Accordingly, candidates sometimes fail to successfully accomplish the listening tasks. This may be owing to the fact that they do not know the meaning of a word or an expression. However, even if they did, it still might not make sense in the context, as the ideas of the text (the discourse) present a different point of view on the reality. When so, situations occur in which the students make wrong decisions even in cases when they know every single word in the string because of the fact that what they have heard merely does not meet their expectations, and sounds weird to them. Thus, candidates often fail to see the point because the language in use is strongly culturally biased. Discussion We shall now consider several types of most problematic discourses for Bulgarian takers of the TOEFL excerpted from the Listening sections of the two TOEFL textbooks already mentioned and classified as academic and political discourse, education, different kinds of services, everyday practices, and social behavior. I. Academic environment The examples connected with academic environment outnumber the rest. In general, the main factors for making mistakes in items reflecting the specific American academic environment are a consequence of the crucial differences between the American educational system and the one we have in Bulgaria. Basically, these differences concern the syllabus structure, the types of courses offered, the existence/lack of credit system, the academic positions. Not rarely do students face the problem of the so called false friends, as it is the case with the word “professor” for instance. e.g. 1. (Barron‟s TOEFL, 1989) A: I‟d like to take Dr. Sullivan‟s section of Physics 100, but my advisor is teaching it too, and I don‟t want her to be offended. B: Who cares? The question is: What should the woman do? The correct answer is A: The woman should not consider her advisor in the decision. e.g. 2 (Barron‟s TOEFL 1996) A: So the course is closed. This is terrible. I have to have it to graduate. B: You are o.k. Just Dr. Collin‟s section is closed. There is another section that is still open, but nobody knows who‟s teaching it. It‟s marked staff. The question is: What would the student probably do? The correct answer is C: Enroll in the section marked „staff‟ (cf. B. Graduate at a later date; D. Find out who‟s teaching the other section - confusing) In both examples, in order to work out the right answer Bulgarian test-takers should know what the duties of the advisor are and what his/her functions are (advisors are members of the academic staff at American Universities). Moreover, at American universities one and the same subject is taught by two (sometimes even more) people and the students are free to choose whose course to attend which presents different practice from what we have in Bulgaria. One is also to be aware of the existence of credit system at US universities. e.g. 3 (Barron‟s TOEFL 1996) A longer conversation between a student and a professor from which it becomes clear that the student couldn‟t register for the class of the instructor because the course was closed by the time he got to the front of the line. The two problematic questions are: 1. What‟s Mike‟s problem (The correct answer: He must have the permission of the instructor); 2. What does the professor decide to do(T the correct answer: Allow Mike to take the class this term). . Here again, different regulations at US universities are reflected which do not have equivalents within the Bulgarian educational system - Bulgarian students are to be aware of the fact that there is a competition among the students to register for a course on first-come first-serve basis as well as of the possibility for exceptions allowed by the professors themselves. e.g. 4 (Barron‟s TOEFL, 1996) A talk concerning assessment and evaluation: “You have a midterm examination the last week of October, and a final examination the second week of December. The midterm is worth 25 points and the final is worth 50 points. That leaves 25 points for the project that …… and you have several choices to fulfill that requirement. You either write a paper or make a half hour presentation”…… The question is: What are the course requirements? The correct answer is D: A midterm, a project and a final exam (cf. A. A midterm and a final exam; B. A midterm and either a final exam or a project; C. A midterm and a paper or a presentation) Obviously, the language in use is again culturally biased - knowledge is required about the content of the notions midterm and project (written paper/oral presentation); projects do not constitute part of the continuous assessment, they are part of the final grade. Students themselves own up that the whole seems to be a complete mess and they take pot luck when answering the question. II. Education e.g. 5 (Barron‟s TOFEL, 1996) The text presents a talk – in fact, this is a public service announcement telling about possibilities offered by a college for distance learning provided by the help of video telecourses. The question is : What is the announcement mainly about? (The correct answer: Video telecourses). In the explanatory notes the other two options (The Sun up semester programme & The community college campus) are said to be „secondary‟ as they have been „used to develop the main topic‟. Bulgarian takers of the TOEFL exam are generally not aware of the existence of distance education as these are usually young people at school age.This type of education seems to be quite extended in the US whereas in Bulgaria such a possibility has been provided for only a couple of years now by few institutions and does not have as long traditions as in the US. III. Political discourse e.g. 6 (Barron‟s TOEFL 1996) This is a lecture on history with the following key expressions: Each political party… nominates a slate of electors pledged to support the party‟s nominees… each state has the same number of electors in the college as it has members of Congress …. registered voters go to the polls to choose the electors. The ballots list only the names of the candidates… This vote by the people for electors is called the popular vote and the candidates who receive the most popular votes win all the electoral votes in a state. The text includes a lot of specific terms and the test-takers need specific subject knowledge about the political electoral procedure. At short notice and after having heard the talk once only they have to identify all the names and to adjust them to the roles played by the different groups of people at the elections. The very word college is rather confusing - “An organized body of persons with shared functions and privileges” (Oxford Reference Dictionary). Thus, it is beyond doubt that language proficiency is far from sufficient if a Bulgarian student is to answer the questions: How are the people nominated for the electoral college? What is the popular vote? IV. Everyday practices – food and drinks e.g. 7 (Barron‟s TOEFL 1996) This example presents a longer conversation between two speakers in front of a coffee machine which has just stolen the man‟s money. Finally they decide to go to the library to try the vending machine there. The questions to be answered are: 1. Why did they decide to go to the library? (The correct answer:To have coffee); 2. What prompted the conversation? (The answer: The speaker wanted coffee); 3. What do the speakers mainly discuss? (The correct answer: The coffee). Vending machines are typical for all public places in the US and other countries, yet hardly ever seen in our country. So, you can go to any public place (library including) and be sure to find a machine there whereas to a Bulgarian a library is a place to only borrow or read books. An additional problem is the fact that the word “vending” has no equivalent in Bulgarian – no word for non-existing artifact. e.g. 8 (Barron‟s TOEFL 1996) A: What do you want on that? B: Everything and extra catsup, too, please. The question is: What does the man mean? The correct answer is: lettuce, pickles, onions, mustard, mayonnaise, and catsup The TOEFL textbook explains: “Everything is an idiomatic (?) expression that means all the condiments included”. Obviously, this item does not test language proficiency at all. You have to have lived in the US for some time to know the expression. As Bulgarian takers of the TOEFL examination are usually young people at school age, most would have no prediction as to what the phrase means. V. Health services e.g. 9 (Barron‟s TOEFL 1989) This is a conversation between a pharmacist and a patient of Dr. Williams‟s, the doctor being supposed to give a prescription. Key phrases: Aspirin is the strongest medication I can give without a prescription; I can call him…. Dr. Williams will give me a pain prescription over the phone. The question to be answered: 1.Where does the conversation take place? (The correct answer: In the drug store); 2. Why did the man call Dr. Williams? (The correct answer: To get a prescription for the woman on the phone) Something one should take into consideration here is the existence of different words in the two varieties - drugstore in American and chemist‟s in British English. It is true though it does not usually cause problems for Bulgarians, but it might well be a problem for some native speakers of British English. The main problem for a Bulgarian here might be the behavior of the chemist which seems rather strange since it is not possible to get a prescription on the phone in our country. What is more, in Bulgaria, no prescription would be needed a for a pain killer. VI. Services e.g. 9 (Barron‟s TOEFL 1996) A: You turn on the TV by pulling on this button. The heat control is on the wall. Will there be anything else, Ma‟am? B: No, thank you. The question is: What can be inferred about the man? A. He‟s a TV repairman B. He‟s a bell boy (the correct answer) C. He‟s a tailor. D. He‟s a security guard. In order to decide on which the correct answer is, one is to first identify the settings - deduction should be made about a place with both a TV and a heating device for which someone is responsible. This is more or less a matter of common sense, however only possible for experienced travelers since very few hotels provide bell boys services in Bulgaria and these are usually very expensive ones. VII. Social behavior e.g. 10 (Barron‟s TOEFL 1989) This is a restatement example: My invitation has RSVP printed at the bottom. A. I should dress formally. B. I should tell the hostess whether I‟ll go (the correct answer) C. I should take liquor. D. I should buy a ticket. It is of crucial importance here that the abbreviation be interpreted correctly. The abbreviation itself is a borrowing from French and means repondez s‟il vous plait. This phrase has almost no circulation in Bulgaria except on very formal occasions . We should not except the average Bulgarian to know it. Some conclusions 1. The TOEFL textbooks do never provide any information about cultural specifics and concentrate on grammar and lexis only. Consequently, many prospective test takers go in for courses which prepare them for the successful sitting of the examination. 2. The courses but most often provide instruction only in grammatical and vocabulary issues and focus mainly on the techniques and building strategies to complete the test tasks within the time required. 3. The material discussed above and the situation observed lead inevitably to some important consequences concerning the type of instruction necessary for the preparation for the TOEFL not only in BG but elsewhere. The basic presumption is that every candidate targets higher scores, hence a correct answer to every test item seems to be crucial for him/her. 4. This puts additional load onto the instructors as they have to provide at least the minimum information on several cultural specifics, and especially in the academic discourse as this topic area presents abundance of different cultural practices included in the test content. 5. From everything said so far one general question follows: who will make a better instructor for the TOEFL examination – a native or a non-native teacher? In the authors‟ opinion a non-native teacher seems to be better suited to perform this task as cultural issues involve implicit or explicit comparisons with another culture and only a non-native teacher can possess the knowledge and intuition what the similarities and the differences in the target culture are so that to elicit the most problematic topics and issues. 6. In the Bulgarian environment the above problem might sometimes have an alternative solution, namely – a British native-speaker teaching in TOEFL courses. Having in mind that sharing one mother tongue does not necessarily imply sharing one and the same culture, we could conclude that a British teacher is often a better choice than an American one because the former will also feel the specifics of the social practices adopted in the US. 7. The analysis of the material shows that the TOEFL ( as well as many other standardized proficiency tests of English – cf. FCE) tests language proficiency as well as cultural awareness although it is often pointed out that culture should be subject to a different testing procedure. Still, in our opinion it is often absolutely impossible in cases of proficiency tests to divide language from cultural competence as it is intrinsically embedded into the use of the particular language. 8. One additional point to be made here is that the preparatory TOEFL courses can successfully provide information about the social, educational, behavioral and political specifics in the US . 9. Finally, we can conclude that the TOEFL aims at testing not only the language competence of non-native speakers of American English, but also their communicative competence which encompasses both linguistic competence and cultural awareness without which language cannot be used appropriately in different social contexts. So, the TOEFL reflects the general tendency in ELT which is to move further on from pure accuracy (or language competence) to pragmatic competence.