Best watched in Full Screen.

Although this NASILP video is a little dated, many of the topics covered in this video are still relevant today in NASILP programs. See Wikipedia for more information about Dr. Walton. This video was directed by Dan Gross. (Note the NASILP secretariat information at the end of this video has changed).

In the video, Dr. A. Ronald Walton discusses the concept of “prochievement,” a blend of proficiency and achievement in language learning. He emphasizes the importance of assessing both a learner’s ability to use the language effectively (proficiency) and their accomplishments in specific language tasks (achievement).

Dr. Walton outlines various assessment methods to evaluate these aspects, including performance-based evaluations and standardized testing. He also highlights the role of self-instructional language programs in fostering autonomous learning and the need for tailored assessment strategies to measure learners’ progress accurately.

Ronald Walton “Testing for Prochievement” Video Transcription (by AI)

(made by restream.io 10/19/24 and lightly edited)

Hi. Before coming to the main topic of this video, let’s take a little time to review the characteristics of a successful self-instructional language program. And we’ll begin with the coordinator because the coordinator is the backbone of all self-instructional programs. This person is often a faculty member, sometimes a separate administrative appointment, and this person is often a language teacher or a linguist. And their duties are considerable in making for the smooth running of a self-instructional program.

Amongst these duties would be screening students. This is a very important task because the demands of a NASILP program, are such that students need to be chosen with some care. In addition, the coordinator needs to locate and train the tutors in the program. They must provide a program orientation to the students, at the beginning of each semester. They’ll be contacting and consulting outside examiners who will be coming in to do the testing.

In addition to this, they need to be able to explain and promote their program on the campus so that everybody understands exactly what the program does and does not do. And in addition, they’ll be asked to handle another, a range of administrative duties. And all of this is best done in constant consultation with the headquarters of NASILP. Now let’s say a few words about the students in a self-instructional program. These students need to be highly self-disciplined and determined to succeed.

If they only have a casual interest in language learning or a casual interest in this particular language, the chances are they’re not going to do well in this particular format. These students must work well independently. They must be highly motivated and very self-directed. They’ll need to set up and stick to a rigorous daily schedule of tape practice. This taped practice is the core of learning in a self-instructional program.

But more than anything else, students in a self-instructional program must take much more responsibility for their own learning than in the normal classroom-based program. The tutor is yet another critical component of a NASILP program. The tutor is an educated native speaker of the principal dialect of the language to be studied. Typically, the tutor meets with students 2 to 3 times per week to enhance the mastery of material that’s already been studied on the tapes. A typical tutorial session should include 4 to 5 students, and it must be remembered that this is not an opportunity for introducing new material, but rather an opportunity to practice material already studied in a communicative context with a native speaker.

The tutor is trained by the coordinator using various resources, including videos produced by NASILP. The tutor is expected to be monitored by the coordinator. The tutor is a valuable component of the NASILP program, but it must be remembered that this person is not a trained teacher, and, therefore, cannot be expected to choose materials, set pacing, explain grammar, examine, or give grades. Another crucial ingredient of NASILP programs are the instructional materials. Ideally, this is a textbook that has been specifically prepared for self-study, and it must include a high quality, complete, well programmed set of accompanying tapes.

While audio tapes are the norm today, it is also possible that in the future more and more of these tapes will be on video. Now let’s turn to the role of the examiner in an NASILP program. The examiner is the key to the success of the academic integrity and the academic accountability of any NASILP program. The examiner is normally a trained teacher of the target language, at an accredited institution and is familiar with the NASILP programs and with the procedures followed by NASILP. Now this examiner administers what is termed a pro achievement test, which is actually an oral interview test based on the material that the student has covered up to the time of the exam.

This pro achievement test is an achievement test to be sure, but a test that’s given in an oral proficiency format. This oral interview is given to each student individually at exam time and is normally conducted at the end of the semester, although it is highly recommended that a midterm exam be given whenever possible. While the test is primarily an oral achievement test, it is quite common to test the students control of the written language at the same time. It is important to remember that the examiner alone gives the course grade in a NASILP program based on the exam. Now, without an effective examiner who gives an appropriate exam, it is absolutely impossible to ensure the academic rigor and the integrity of a NASILP program.

A few additional points, normally, NASILP itself selects the materials to be used and recommends the pacing. However, the examiner may suggest the use of these materials. And finally, the examiner should make clear to the program the goals of the test and of the testing procedures prior to the exam, and after the exam should provide feedback to the program on how student performance can be improved. The purpose of this video is to inform both the experience and the would be examiner in the techniques of achievement testing that have been developed by NASILP over the years. Now what precisely is a pro achievement test?

Well, as I said earlier, this is an oral interview given in the target language, and it’s an achievement test given in a proficiency format. But exactly how is the student evaluated in terms of performance? We are testing for grammatical accuracy, vocabulary use, and pronunciation, but most importantly for communicative competence, which includes fluency, comprehension, and cultural appropriateness. Now let’s take a look at a typical pro achievement exam stopping along the way to discuss and explain. You’ll note that this particular exam is given in English in order to serve as a generic example.

And please note that there will be slight differences in examining procedures and techniques depending on the target language. I’m the coordinator of, self-instruction language program at my institution and we’re offering a new language this year and I have a new examiner. How do I go about telling the new examiner how to prepare for a Pro Achievement exam? First, the new examiner is going to have to understand that there’s a considerable amount of work to be done. The very first thing that you should have a new examiner do is coming online is take a look at the video tape that was prepared especially for this purpose called What Did the Student Learn?

This was prepared by NASILP some years ago. You should ask this, potential examiner, this new examiner to study this tape very closely. Write down any questions they’ve got, call you up and talk to you about it. But I would say the first, very first step is going through this tape. It’s very clear, gives you a very clear idea of what’s expected.

In addition, there was a document produced by NASILP called, Testing the Student in Self-instructional  Language Programs. It’s a written document. That is also worth studying. And again, discussing, any questions that the examiner has over that document. But at the very beginning, I would say the best orientation is to do the videotape of the document and have a discussion with the examiner, make sure the examiner understands all the points that were raised.

It’s a little different than what they’re used to. Besides, what’s next then? I think the next thing is that they definitely should become familiar with the, the text materials and the tapes that are that are used, in your program. Now ideally, if they’re using the very same materials, and the very same audio tapes, we’re in the best shape because then they’re actually have been involved in teaching these particular materials. That’s not always true.

So if not, they’re going to have to have a copy of the materials. They’re going to need to become familiar with the materials. They’re going to pretend they’re a student, go through the lessons, and particularly go through the audio tape component, making sure that it agrees with the, text component. And they need to understand what the student has been working on and been exposed to, in preparing for that test because that test should really simply be ideally, that test is just a almost like another step in what the student’s been doing on a weekly basis.

I can think of it that way. What’s the test format like? Well, the test format certainly can vary depending on the level that the student is at. It might have a little bit to do with reading and writing at certain stages. But generally, we prefer a kind of a 3 part, 3 section test which begins with what we call drills.

And these are very carefully structured, almost like doing a regular drill that you would do a drill in the classroom. The second part are questions, series of questions, realistic questions that the student is expected to answer. And these first two parts are really preparation for the most important part, which is, which is the, the actual communication situation. We use situations and place the student in a situation and we try to make those situations as realistic and as possible. Now let me make it really clear about this test because a lot of people do get confused.

This is a pro achievement test, not a proficiency test. The achievement part of it, it definitely deals with the drills. In the drills, they serve as a warmup, but we’re also trying to test discrete points. It might be discrete grammatical points. We’re listening for grammatical accuracy.

We’re listening for word usage. We’re trying to check on pronunciation, and there is no pretense that this is communication, realistic communication. This is previous to realistic communication, but it is definitely an achievement test. We are trying to find out, do you control the material studied up to this point, but control it in a in a use format, not just knowing about it, but actually knowing how to use it. Then the questions move much closer to, to the proficiency end of a for achievement test.

It’s still based on the materials, but now it’s more realistic. We’re asking the kinds of on the materials, but now it’s more realistic. We’re asking the kinds of questions that would really come up in real life. Then the final part of the situations, that’s the most realistic. We try to create a situation and have the student play in that role play in that situation.

We want those situations to be very realistic. This is why a good textbook makes a big difference. The more that the textbook is in line with the kind of testing that we’re doing, the better off we are. So that the examiner, stimulates the student to speak by mainly asking questions? Well, it’s a little more complicated than that.

The examiner will always tell the learner, the student, at this time of year is what we’re going to do next. So we’re going to do drills and here’s the way that’s going to work and now we’re going to do quests. So they always know where they are, but no, as we move through the sequence, we will want to try to use as many visual aids of pieces of realia as we can to to make whatever we’re doing realistic. What kind of visual aids would be used? It depends a lot on the situation, but for example, in the early stages, most courses are teaching numbers through prices.

And, so what we might have is a situation, for example, or even some questions dealing with prices in which we’d have some items on the table and each of those items would have a price tag on it much as a store would. And the student, the, the dialogue between the examiner and the student might be confirming prices, asking prices on items you can’t see, let’s say a book until you open it up and then you see the price. But we’re trying to get the student to to use the numbers in a realistic context. Another example might be directions in which we would like to be using some sort of visual map, visual aid, maybe a map or something else that one can sort of follow, the directions. Now no test is ever realistic because it’s a test, but we would like these visual aids to be used to the degree that you can to give that added reality all the way through.

Practical question. How many students do I schedule in say an hour’s time to be tested? That’s another good question. It depends a lot on the level of the students. It depends on how many lessons this particular program may be covering in this particular language.

Obviously the more you’ve covered, you’re going to need a little more time for the testing. For example, it turns out that students who are sort of on the on the less able end of the scale, let’s say, often take longer because you want to be fair. You want to really find out what they know and what they don’t know and that may take quite a bit of of repeating and examining and so forth. That can take a long time. It’s not true that students at the elementary level have a shorter test period than those who are at the advanced have more.

It really depends a lot on the students’ performance ability. I would say a minimum most of the time of 15 minutes up to a maximum of probably 30 minutes. A 30 minute oral exam is a very trying experience and you’re getting into the upper end here. That might happen with somebody who has covered a lot of material, maybe in a more advanced stage and you’re really trying to do the widest coverage you can. Then you ask how many per hour, so you have to remember that the examiner needs some time between exams to go back over that student’s performance, make notations, jot down things that happened in the test and so forth, and to clear their head a little bit to get ready for the next exam.

Yeah, 2 an hour might be your max, might be, the minimum that you could do maybe in depending on the kinds of students, you might be able to do 3 in an hour, maybe 4, but there’s no rule here. And the more experienced the examiner becomes, the clearer it is to them, even in the early parts of the test, how long this test may take. Hi. Hi. Come on in.

Have a chair. Thanks. Make yourself comfortable. This is nothing more than greeting the student and putting him or her at ease. At this point, the examiner normally turns to the first person of the exam, the drills.

The whole introduction may certainly be done in the target language if the student is at the appropriate level. Okay, Susie. To get started, we’re gonna do just a a section called drills. This is very simple and I’m not trying to be communicative here. I’m just checking your control of some specific things.

Okay. I’m gonna just start off with the verb to go. Mhmm. I’m gonna say something and you say what I say and finish it up. Okay?

Okay. With this verb. So I Okay. I go. Mhmm.

She. She goes. They. They go. We.

We go. You. You go. John. John goes.

John and Mary. They go. Okay, that’s fine. Now the next thing I’m gonna do is I’m gonna say something in the positive. I want you to just say it in the negative.

It’s very simple. Okay. He went to the movies. He didn’t go to the movies. I ate out in a restaurant last night.

I didn’t eat out in a restaurant last night. Now let’s try a few something a little different. Mhmm. I’m going to ask you a question, and you answer in the negative. Okay?

Have you seen the latest Spielberg film? No. I haven’t seen it. Have you ever visited Hollywood? No.

I’ve never been there. Never been there. Okay. That’s good. Remember that these are only examples for the purpose of demonstration.

Actual drills will vary depending on the appropriate particulars of the target language studied by the student. Note that the drills serve to warm up the student to the target language prior to the more extensive linguistic demands. They are also useful in testing key discrete items of a particular language. Where do you come from? From Guadalupe.

Guadalupe. Do you have any brothers or sisters? Yeah. I have a big family. We are 8.

8? Yeah. 8. How many are brothers and how many are sisters? Four sisters and three brothers.

And how old is the oldest of these brothers or sisters? I think she’s 36. Are you the youngest? No. I’m the 6th.

You’re the 6th? Yeah. Big family. Yeah. Have you studied any other language besides English?

Yes, Spanish. Spanish. How long have you studied Spanish? 5 years, I think. 5 years?

Yeah. How long have you been at this at this college? Only 2 months. 2 months? Yeah.

Do you like it so far? Yeah. I like it so far. Okay. And what are you gonna do after you leave?

I don’t know. Maybe I want to be a teacher, but I’m not sure. Okay. Susie, on this part, we’ve got some weather scenes here. I’m just gonna indicate the weather scene Mhmm.

And then you just give me a very short description. Very, very simple. Okay? Okay. How about this one?

Number 5. I think it’s snowing because, the boy wears winter clothes and we can see the snow falling down. That’s fine. How about number 2? It’s raining.

He has a hand and umbrella. Mhmm. And he he’s trying to make his way further away. That’s fine. How about number 1?

It’s sunny. It’s a very beautiful day. Everything is fine. Everything’s fine. What about number number 6?

6. It’s windy and she can’t stand. I mean That’s fine. And how about number how about number 3? No.

It’s very hot and he can’t breathe. Okay. That’s fine. Okay, Susie. In in this case, we’re gonna do a little situation now.

First, I’m giving you some money. Mhmm. There’s some dollars. You can count that. And here’s some change.

Thanks. Alright. Now here’s the situation. Mhmm. I’d like to get a Coke, but I don’t have any change.

I only have a $5 bill. Could you give me some change for this so that I can Yeah. Sure. So I can use the Coke machine? Okay.

Okay. Yeah. Great. Thanks. Now this thing doesn’t have one of those dollar changers.

Would you have some smaller change? You gotta break the $1 down? I think so. Yeah. Thanks.

Let’s see. Yeah. I think. Better count that out for me. I have 2 closes Uh-huh.

5 4 dimes Okay. And 2 needles. Well, that’s great. That’s perfect. Alright.

Thank you very much. You’re welcome. You said you were from Guadalupe. I I’ve never been there. Could you tell me a little bit about it?

Describe it a little bit? Sure. It’s a French island. It’s, in the Atlantic Ocean. Do you know where Cuba is?

Yeah. It’s not very far from Cuba. And we speak French but also Creole. Mhmm. Yeah.

And it’s very different from France because first, it’s a tropical climate and I mean, we have a different history and different customs. Mhmm. Yeah. And, you know, I’m French but I’m also different, you know. It’s French and also different.

Yeah. It’s difficult for me to explain it. To explain, but it’s, you know Well, that’s fine. That was good. Well, you say you, you speak French in Guadalupe.

Now what about here on campus? Do you have an opportunity to use your your French? Yeah. I live in the French house actually and we at times we speak French. We we see French movies and every Wednesday we do something in French.

Tell me a little bit more about this French house. Who lives there? There are 10 or 11. 11 people live in a French house and, they learn how to speak French. That’s why we live here.

And, it’s a very big house. It’s quite nice. We have something like, let’s see, 6 or 7 rooms and a big kitchen and living room, everything. I think it’s better, when it does. Sounds a lot better than it does.

Yeah. I think so. Okay. Thanks. Again, remember that your conversational interview will depend on the specific material being tested.

The interview you just saw is meant to be only a guide. Unlike the drills, which warm students up and test short term learned patterns and discrete items, the conversational interview tests the student’s ability to communicate, albeit in a limited way. The examiner should check the student’s ability to appropriately use the vocabulary and the grammatical structures and their ability with regard to comprehension and accurate pronunciation. Finally, the conversational interview permits the examiner to see if the student can communicate effectively with respect to cultural appropriateness and fluency without hesitation. Okay, Susie.

So far, we did some drills and then we did some conversation where I asked you some real questions about real things and and you, tried to answer and so forth. Now for this third part, we call this interpretation or role playing. And I’m gonna set up some situations. Mhmm. And, you have to kind of play along with the situation.

We normally try to do this with 3 people and you interpret back and forth, but this is another way of doing it. Okay? The first task we’re going to do is based on a restaurant bill. Alright. Here’s the situation.

Bill. Mhmm. Okay? And we’re trying we’re going to split this, but you’re going over the bill to figure out the cost and so forth. Okay?

Okay. So here’s the bill. Thanks. Okay. Alright.

Now so here’s the situation. You ready? Yeah. I’m ready. Okay.

Well, how much does the bill come to? 14.51. 14.51? Yeah. It seems awfully expensive.

What do they have me down for? A cheeseburger. Yeah. And well, how much was a cheeseburger? 395.

395? Yeah. Okay. And then what did you have? I had a soup and a salad bowl.

It was 450. Soup and salad bowl is 450. Yeah. Okay. And then now did what do we have to drink?

A Coke. Oh, did you add a Coke too? Yeah. 2 Cokes. Two Cokes.

So how much was that? 1.50. Okay. So that’s 75 each. And then, I had that hot fudge sundae.

What was that? $3? 2.25. Oh, 2.25. And what did you have?

I had, the apple pie. Yeah. It’s 1.1 20 25. 1.25. Okay.

And there’s a tax, right? No. No. Sorry. 175.

175. For the apple pie? Yeah. And what are the tax? 56.

56. Okay. Yeah. Yeah. Okay.

Well, that does come in 14. Now we’ll have to add the tip Mhmm. And then we’ll and then we’ll split the difference. Okay? Okay.

Alright. Okay, Susie. In this situation, I’m a visitor to your campus. I’ve come for the day and I’m I wanna see a basketball game that’s going on here. Mhmm.

And it’s in the gym. But I’m totally lost. Now I have this map. Mhmm. Okay.

Except don’t know how to use it because I don’t know where I am. I’m coming up to you and I’m trying to get your help, okay? Yeah, okay. Okay, okay. Okay, okay.

Excuse me, I’m trying to find the gymnasium. I want to see a bogie but I’m lost here. I don’t know where I am. Can you help Sean make it there? Yeah, Sure.

Yeah. Here. Fine Arts Center. Oh, this is the Fine Arts Center. Okay.

And you have to go down, Park Street. Uh-huh. Right. And you have to pass, the bookstore. Yeah.

I see that. The Then you turn right, you take the 8th Avenue. Okay, 8th Ave. Alright. And Darby Jim is just Yeah.

Okay, right there. Where’s the entrance? Is it on this side or? No, you have to turn and take the entrance is here. Uh-huh.

Go by the through the parking lot? Yeah. Okay. And then come in? Yeah.

It’s very simple. I’m sure you will. Alright. Thank you very much. You’re welcome.

Okay. Susie, in this situation, the campus is you’re in a college campus. They’re having what’s called a book fair and the students are trying to to sell some of their old books. Okay. And you’re standing at a little table trying to get rid of some of your books.

And I’ve come along and I’m interested in in buying some. Okay? Thank you. Right? Okay.

Well, I, you know, I’m kind of interested in these 3 books, but I I couldn’t seem to find the prices. How much is that one? Let me see. Oh, it’s 4.75 5, I think. 475?

Yeah. Gee, that’s a lot of money. Is that is that new? Yeah. Is that new?

Okay. And this one, I really like that one too, I think, about French cultural history. But, Let me think. I don’t remember what this point again. Oh, 17/95.

17/95? Yeah. All right. Okay. That’s a lot of money.

This one says it’s 19.95. Actually, I like this one the best. Would you be willing to lower the price on this just a little bit? You know, I bought it 30. Really?

Yeah. I’ll do that. Alright. I’ll tell you what. What about for these 2 books?

Mhmm. Alright? Yeah. Make it 25 for both. 20 let me see.

Okay. 25? 25 for both. 25? This was 19.

So it would be 27 or 28 and I’ll just give you 25 for those 2. Maybe 30. 30? 25 is 2. Okay.

Alright. That’s a deal. Okay. Alright. Thanks.

Yeah. Again, remember that these interpretations should serve only as examples and that specific questions will vary depending on the cultural context of the target language and importantly, on the material covered during the semester. This part of the exam is designed to elicit from the student the total communicative oral aural act. That is, to be able to interact with control of grammatical structures learned, extensive application of learned vocabulary, accurate pronunciation, and communicative competency, including the amount of time required to formulate utterances as well as fluency, cultural appropriateness, and the like. If the pro achievement exam in NASIL is primarily speaking and understanding, do you ever test reading?

Well, again, this is a good question, but no simple answer. It depends on a number of things, the level that the student has reached in the language, the nature of the difficulty of the language, the difficulty of the writing system in the language, what kind of role the writing system actually plays in the textbook of the language that’s being studied. And all of these things vary. For example, if you’re dealing with a Roman alphabetic language, like say Portuguese, no one anticipates there’s gonna be huge problems, in reading. So it might be that the course has its usual emphasis on the oral, oral abilities, but somewhere along the line there might begin to be some checking out of reading comprehension ability, not really the real focus.

More likely, however, that you’re gonna get a kind of focus if you want to think of it on the written language and on reading. Probably has more to do with languages like Arabic, Greek, Hebrew, Russian, where the writing system itself is a challenge to match sound to symbol. And there because again, we just this is pro achievement, course pro achievement testing. We want to know if the student can indeed handle this orthography. For some textbooks, this is the only way into the to the spoken language through the writing system.

That is, they’re not providing any other access other than through the native orthography. For instance, Cyrillic alphabet in Russian. So you’re going to have to know that. And it seems fair to test students on that just as we would test for the pieces of the sound system, we test for pieces of the orthography to see if you can control this. So reading, yes, but maybe in that case it might be something like, asking the student during the test situation, do some reading aloud just to be checking the correspondence between the sound and the symbol.

In a language like, Russian or, Russian or, excuse me, a language like Chinese or Japanese, however, here the sound symbol relationship can get to be very distant. Therefore, it may make sense to very much delay, getting into the native orthography. And most textbooks don’t require that you have control of the native orthography in order to do conversation. For instance, there are romanization systems that are used in Chinese and Japanese that allow you to to prepare and work with, oral material without having to recognize that that native orthography. So a huge range in reading and why you would want to be reading and reading aloud maybe at that level is and let me add to that that, if you wanted to check genuine reading ability, which is more on the proficiency end of pro achievement, probably would be slightly different there.

You would have students read silently and then you would ask them some questions in English to see if they’re comprehending. What about writing? Again, writing sort of depends on the purpose again. It’s not the point in in these courses to teach anything like composition and so forth, but writing as an enabling skill for reading or as an enabling skill for learning the orthography can be important. So there might be a little testing of the writing in the form of a dictation, something like that.

So you’d be testing writing mainly to find out if they’re making the sound symbol correspondence, which is an indirect test of reading, which itself is really not real reading sometimes, but more testing the sound symbol correspondence. So, yes, it could fit in. It will never be a big component. When would be a good time to test reading? At what point?

Depends again on the language, and like I’d say in in languages with very difficult orthographies, and if they have other transcription systems that can be used to access spoken language, it would probably become later in the program. For, orthographies that are easier to comprehend, you might start a little bit earlier. I don’t think there’s any hard and fast rule here. You mentioned Chinese and Japanese, and that the students don’t learn the writing system from the beginning. Why is this?

Well, these writing systems, for instance, in Chinese and Japanese, and I suppose a lot of people would make the argument in Arabic as well, there’s a sort of a difficult match between the writing system and the sound system. In the beginning, when you learn a language, we’re emphasizing the spoken language, that is to speak it and to listen to it, much as as as a child learns their first language by by by speaking and listening and so forth. And they study the orthography later, that is you learn to read and write later. In a sense, we’re trying to make sure that the students have some grasp of the language, pronunciation, understanding the structures, able to use the structures, the vocabulary, and so forth. Before burdening them in these languages with a very ponderous writing system, something that’s gonna take a long time, something that’s complicated.

If you try to introduce the native orthographies early in in these courses, it will take so much time away from the time that you would have to practice the spoken language that you pull the whole learning process down into a slow grind. Therefore, we’re suggesting that you start off with emphasis on the oral, oral skills and that as those begin to fall into control, you phase in this more difficult writing system step by step by step by step. But again, it might be, it might be quite might be deep into a program before you would actually be reading for comprehension. Good. Thank you.

Susie, in this next part, we’re gonna do just a little check of, it’s not actually reading so much as it is sort of matching the written language to the sound system. Mhmm. And so I’m gonna ask you to read a little bit out loud. And the document that we’re gonna use here is a letter. Okay.

This letter is from one friend to another. Okay? So all I’d like you to do now is just start at the very top and just read out loud for me in a normal way just a little bit of this letter. Okay? Yeah.

Okay. Dear Mary, well, I’ve been here at the university for almost 1 year now. I’m always busy and I just can’t seem to find time to sleep, much less to write to my friends. Please forgive me for taking so long to get in touch with you. How are you and your family?

What are your plans for next year? Do you think that you may be coming here to go to school or do you still want to go to k four? Okay. That’s fine. Alright.

Now let’s try something a little different. This time, I really do want you to sort of read this letter from top to bottom for understanding and comprehension. Okay. And when you’re done, I’m gonna ask you to give me the letter back and I’m gonna ask you a few questions in English. Okay.

Okay? Alright. Take your time. Okay? Yeah.

Alright. Let me just ask you a few questions. This this letter is to someone called Mary. Yeah. Do we know from the letter what Mary’s gonna be doing next year?

No. We don’t know. Because she she asked her questions. She Mhmm. She asked her if she’s going to California.

California. Yeah. Right. Okay. The letter is from what someone named Jane.

Yeah. And Jane talks a little bit about going to school. Mhmm. Does she like the professors at the university or the college that she’s at? Yes.

She does. Because, she thinks, they are interested in the students. Mhmm. And they were always ready to help them. To help them.

Okay. When Jane is writing this letter, she says a little bit about, Mary’s mother. Do you what does she say about Mary’s mother? I think she says that she misses her because, she’s a good cook and she doesn’t like, what she eats in the in school. In school.

Okay. That was great. Thank you. The of this test will depend on the difficulty in the particular writing system of the language in question as previously stated. The purpose here is to test the student’s ability to vocalize the writing system in an acceptable way as well as to comprehend the message of the text at an appropriate level of difficulty.

Okay. Susie, on this part, we’re gonna do a little check out again just to to see if you can put sound and symbol together and do a little writing, and I’m just going to say a couple of short sentences and I just want you to write them down just like I say. Okay. Okay? The first one is the bank closes at 12 o’clock on Saturdays.

Okay. Okay. And the next one is, please send me a copy of the letter. Alright. Susie, this is the very last thing.

In in this case, I really wanna check a little bit more about just your ability to write something everyday, common, useful, like a message. So I’m gonna describe to you a message I’d like you to write. I want you to do it in your own words. Don’t write anything down while I’m talking. Just listen and then just do this message your way.

Here’s the situation. You’d like to meet a friend at the library in the morning at 9 o’clock. Alright? If your friend can’t make it, she should phone you tonight. Alright?

If you’re not in, she should leave a message with your roommate. Okay? So can you write that message for her? Okay. Great.

Testing for writing production may not be necessary in languages with a Roman alphabet writing system, but for languages with difficult orthographies or writing systems, such as Chinese or Japanese, such checkouts can be quite important. Well, look, that’s all there is to it and, overall, I think you’re coming along very, very nicely. Just in general, you probably could do a little more on the tape work, pay a little more attention to pronunciation. Yeah. I know.

Some of the vocabulary in the materials, you still aren’t real sure of how to use, but Mhmm. You know, I think you’re coming along. I’ve made a lot of detailed notes, and I’ll be going over this with your with your coordinator, and you’ll find out how everything went. Okay. Okay.

So I I thank you very much. Okay. I thank you too. Thanks a lot. You bet.

This is simply the examiner closing the interview and saying goodbye to the student. At this point, the examiner may include a bit of feedback concerning the student’s performance with comments regarding strengths, weaknesses, and suggestions for further improvement. This will be quite short and will be done in English. There is no discussion at this point of the grade, nor is this a opportunity for detailed evaluation and feedback. Now that we’ve just seen an example of a pro achievement exam, tell me how does the examiner evaluate the student?

How does an examiner give the student a grade? Okay. We the main thing we’re after here is some sort of systematic process. And first, it’s important to remember yet again that the student is this is a curriculum based test. Pro achievement test is based on the curriculum.

So it’s what the student has covered in terms of a set amount of material. So that part is fairly easy. We know we know the tester knows what they’re supposed to be looking for because it’s based on the finite amount of material. The tester, again, would want to look over the what did the student learn videotape to get some ideas about how you assign ratings because that tape devotes quite a bit of time to assigning some sort of evaluation to the student’s performance. Second thing is the examiner wants to take a very long look at the document on the testing of students in self-instructional  language courses.

That document actually goes into quite a bit of detail on on the categories that are rated and on some suggested scales. For instance, there’s ratings in language competence portion, there are low ratings and high ratings and some details there. There is a system. There are lots of suggested ways of going about formulating this evaluation in a coherent manner. What kind of format does the examiner use to evaluate the student?

Okay. It’s true that as I said, the document that we have outlines the categories and the type rating, but you have to make up, you have to actually create your own kind of rating sheet. Almost all examiners do this. They create some sort of form that they can use for each student. And one way to think about the pro achievement test oral interview is that there are gonna be a certain number of exchanges that are going to be going on.

And in a way, you would like to get a rating for each of these exchanges. Sometimes you’re rating comprehension, sometimes you’re rating production. So we normally suggest that you make out some sort of rating sheet that is going to allow for all the different utterances that you’re going through. And that you’ll need to make up some sort of scale and you’ll need to have categories down there. It would be typical to have a rating for pronunciation, a rating for vocabulary use, a rating for grammatical accuracy, a rating for language use or communicative competence.

That might include a rating for fluency and how long it takes a student to respond and so forth. There’s as much detail as you want to put in and you can keep up with. So that you are actually getting and a lot of people like to use some sort of numerical scale. 1 through 5 or some people are more comfortable with 1 through 10. So that as you’re going through the exam, you’re sort of able to assign some sort of numerical value fairly quickly.

Are there any special considerations in making up this evaluation sheet? Absolutely. The, it’s got to be in a way sort of light and transparent because you don’t want to be disrupting the student or making the student feel uncomfortable making all sorts of detailed marks. And you yourself won’t have time. These exams move fairly quickly.

And so you can’t be pausing and stopping and figuring out some complicated chart. You need something that’s fairly light, transparent, fast that you can go through and make marks and checks on, write fast comments or whatever. Again, this is up to the individual examiner and experience helps. I mean there are we can give you some ideas on how to get started, but most examiners are going to have to go work on this a little bit on their own to find out what kind of rating sheet or evaluation sheet is best for them. Now what does the examiner do with a student who’s clearly nervous during such an exam?

Well, our training materials will try to address this issue and we try to make sure the examiner understands that, we know from years of of working with students in exam situations in in NASIL that students are gonna well, all students are nervous. Some are more nervous than others. What we’re always trying to figure out is what is the source of this nervousness? Now the first thing we try to do is a warm up, and that’s to put the student at ease. It’s to get the student moving, to get them to forget a little bit at least that this is an exam and start thinking about actually getting into the language.

The warm up phase is there for that to begin to get that started. I think most examiners would agree that the keenest source of nervousness is when students are doing something, when what’s being done in this exam somehow does not match the kind of preparation and work that they’ve been doing in the course. Because ideally, all that this exam should be doing is very much the same sort of things that have been going on in the classroom situation and even with the tapes. So it it should be a day, just another day in some senses. There should be nothing unusual.

When we find a really nervous student, most of the time the reasons are they weren’t preparing properly, they haven’t been taking the course properly, they haven’t been using the tapes, oral language, They’ve been doing a lot of eye work and suddenly they’re confronted with oral work and they can’t handle it. And I think it’s rare. Given that these exams go for 20 minutes and can go past 30 minutes. Given that there’s that much time, there’s lots of settle down time. If that nervousness persists, if it persists all the way through the exam, chances are that it’s not just nervous as the student really is not prepared.

If that nervousness is heavy at the beginning and then diminishes, this this can be typical. You have to remember that this exam is the only exam and the examiner is giving the grade, so there’s reasons to be nervous. But if you’re prepared, there’s nothing in this exam that should be frightening. It’s only material you cover. There’s not going to be any tricks.

The student has seen this tape about what did the student learn. They know what’s coming. So everything’s been thought of and planned. And I’ll just repeat that if the nervousness persists, it’s usually because the student really cannot handle the material. And in that case, they’re not going to make a good grade anyway.

Are there any considerations for the student who performs poorly and unexpectedly poorly? Well, let’s take this example. The student performs unexpectedly poorly. The examiner is bound to give a low grade. That’s it.

And people can make all the noise they want about, well, this student has been doing great and so forth. We only know. The examiner only knows what the examiner is seeing in a in a pro achievement test. But certainly after the exam, it’s not atypical to try to sit down and figure out with the coordinator what went wrong here. Was is there some problem with the tutor?

Is there some problem with the course design? Is there some problem the student didn’t understand things? If there’s a huge discrepancy between how the student performed on the exam and how they seem to be performing into the course, this is worth looking into. Not directly with the student necessarily and not directly with the tutor, but with the coordinator. This is worth probing.

Not so much as the idea of changing the grade, but of preventing this from happening in the future if the student goes on. So I think certainly there’s that element that we want to know why someone who seemed to be doing very well in a course would suddenly, have problems with the exam. And I’ve we’ve covered the nervousness angle, maybe there’s something else. Maybe the impression the student gave was not really accurate altering the course. So it’s worth probing into that, but the grade gets assigned based on the exam.

We hope that this video has answered many questions that you may have had about testing in NASILP programs and that it has provided useful guidance for constructing pro achievement exams in the self-instructional  language program format. If you have any questions that were not covered in this video, please feel free to contact the NASILP secretariat whose address and phone number will appear in just a second.