Adventures in Foreign Language
Volume 2, Issue 1
How Do You Measure Success?
The University of Utah Assesses Student Oral Proficiency -at Home and Abroad
The Great Sphinx, Giza, Egypt The University of Utah offers a five-week program in Egypt as well as seven other countries.
Table of Contents
Feature Story ..................1 Learn how the University of Utah measures the success of its language programs with OWL!
New OWL Hosted ..........3 Welcome to Our New Employees ......................4 Conference Schedule .....4
r. Fernando Rubio and his fellow educators at the University of Utah decided to plan ahead. Like many universities, the University of Utah requires four semesters of foreign language for students who wish to receive a bachelor of arts degree. “With the push for accountability across the country, we knew that sooner or later, the university was going to ask us to measure the communicative abilities of all foreign language students by the end of the second year of language study,” says Dr. Rubio, Chair of the university’s foreign language department. “We wanted to be ready with the answers.” In addition to the imminent demands by the university administration, some of the foreign language faculty were questioning the school’s policy regarding students who participated in the summer study-abroad programs. These students, who spent five weeks in the summer in one of eight foreign countries, were able to Could a mere five weeks of study eliminate two semesters of study in the target adequately replace an entire year’s language. The faculty were worth of course work on campus? concerned: could a mere five weeks of immersion adequately replace an entire year’s worth of course work on campus?
D
Special Promotions!
OWL Celebrates Foreign Language Week with special offers throughout March!
Call (412) 436-0559 or email to get more information!
The University of Utah offers summer semesters in Egypt, China, Japan, Italy, Spain, France, Germany and Chile. Approximately 500 students participate, and each one takes a proficiency exam before and after the trip.
Under Dr. Rubio’s leadership, the foreign language department developed assessments to measure the communicative abilities of its students. At first, they relied on audio cassette tapes to conduct the SOPI-type tests, but even when all of the students and raters were on campus, this tended to be an administrative nightmare. The assessment tapes of the study-abroad students were even more troublesome, and the university needed a new solution. In an effort to use technology to their advantage, Dr. Rubio tried a widely-used online course management software, but it didn’t meet the department’s needs. Ideally, students would be able to take the assessment wherever they were in the world, and likewise, the raters could rate the speech samples from any computer, whether in Utah, Germany or Japan. The software would also allow them to test languages that do not use Roman alphabets, since the university offers Arabic, Japanese and Chinese as well as Spanish, French, Italian and German.
Testing
Dr. Rubio found the solution in OWL Testing Software, which is specifically designed to simplify oral proficiency assessment. Unlike the previous applications they had tried, OWL gave the language department the features and control it needed to manage both the on-campus and distance assessment program. Today, all foreign language students take a test at the end of the second semester on campus. Those students who participate in the study-abroad program also take the test at the end of the five weeks abroad, while they are still on location. No special software needs to be installed on the computers; each student uses a web browser to access the test from virtually anywhere in the world, no matter which language the student is learning.
Rating
The Pantheon, Rome, Italy
Even though all of the raters are based in Utah, many of them travel extensively, including Dr. Rubio himself. In the past, one rater might be available in Utah, but the second rater would be abroad, and the whole rating process would be held up. Now, instead of having to wait until a program director hauled cassettes back to campus and both raters were in town, raters can access the tests via their web browser. “Arabic raters in Utah can access the results the day after the students take the test in Alexandria. Now, we can have results much more quickly than we could before,” says Dr. Rubio.
Results
Every year, the department receives special funding from the university to conduct the assessments, giving the university a true measurement of program success. Another advantage of assessing oral proficiency, says Dr. Rubio, is that the aggregated data provides good quality control of the study-abroad programs. “If we see that out of the eight programs, six are getting good results, but two are not, we recognize that there may be problems with the two programs and can make adjustments,” he notes.
2
Adventures in Foreign Language • Vol. 2, No. 1 • OWL Testing Software
And as to the question about whether five weeks abroad can truly take the place of an entire year’s course work on campus? “After these four years, we’ve learned that students who go abroad learn as much or more than students who take language on campus. They experience a significant gain in proficiency between pre-test and post-test. They speak and write more and better,” says Dr. Rubio. He notes that the results are complicated to generalize. Because the students who travel abroad might be considered a select sample, they may not be entirely representative; students who go abroad tend to be more motivated and must achieve a certain G.P.A. But the students who travel abroad tend to show greater progress even as compared to the best students who stay on campus.
Research
In addition to testing student proficiency levels, part of Dr. Rubio’s research involves extensive analysis of student pronunciation. Using the students’ tests prior to their trip abroad, he might isolate certain sounds that challenge the student. For instance, he might measure vowel sounds to determine if they are pure vowel sounds or dipthongs. The digital audio recordings not only allow him to more easily listen to the speech samples, but he can also see visual representations of the sounds using spectrograms. Until this year, Dr. Rubio’s research has been used internally at the University of Utah, though he has presented some of his findings at conferences. By the end of summer 2008, however, Dr. Rubio expects to have enough data for publication and looks forward to sharing the success of the university’s proficiency assessment program with other institutions and researchers. He can be reached at fernando.rubio@utah.edu. For more information about powering your assessment program with OWL Testing Software, contact Greg Russak at grussak@owlts.com, or by calling (412) 436-0559.
Dr. Fernando Rubio, a native of Spain, received his PhD in Applied Linguistics from State University of New York in 2000 and has been with the University of Utah since 2002. He is currently working on an electronic 3rd year Spanish textbook.
NEW!
OWL Offers Hosted Testing Software!
President & CEO Chris Dalessandri is pleased to announce the company’s latest product offering targeted to smaller organizations wishing to create, administer and rate oral proficiency testing for second language learning. “Many school districts and colleges want a proven testing solution but don’t have the IT or financial resources to manage the software on their own servers. OWL Hosted provides them with an affordable testing suite that meets their needs.” OWL will host your school’s testing website, eliminating the extra burden on your IT staff. For larger organizations or ones that prefer to host OWL inside their firewall, the server-installed version provides more flexibility as well as a few additional features unavailable to hosted customers. Learn more.
Adventures in Foreign Language • Vol. 2, No. 1 • OWL Testing Software
3
Welcome
Welcome New Employees!
Len Brush
Veteran technology expert Len Brush joins the company as our Senior Advisor, Higher Education. Len brings more than 40 years experience serving the information technology, networking and systems integration needs of institutions of higher education such as Carnegie Mellon University, Penn State University and Loyola University in Chicago. “Len’s expertise ranges from planning, designing, developing and implementing enterprise-wide information systems to process reengineering, data security and diasaster recovery, and his particular understanding of the needs of colleges and universities is perfectly aligned with our mission,” says President & CEO Chris Dalessandri. Richard “Dick” Smith takes over as Chief Financial Officer & Lead Recruiter for the organization. Dick brings more than 35 years experience leading successful organizations with skills in all core operational and financial functions. He founded, grew and sold a prosperous financial services firm, then joined Omega Systems in various high level positions including Board of Directors and CFO; eventually, he helped execute the sale of the company to Keane Inc. Most recently, Dick has provided executive coaching and strategic planning services for businesses, and is excited to put his skills to work for OWL Testing Software. Our new Vice President of Sales, Greg Russak is a 26-year veteran of technology sales and marketing. His career has included numerous leadership roles in the telecommunications, enterprise software, and technology consulting industries. He is a graduate of Penn State and Robert Morris Universities and holds degrees in engineering and organizational studies.
Dick Smith
Greg Russak
OWL in person
Be sure to visit Chris Dalessandri and Greg Russak at Digital Stream and CALICO with IALLT! Look for the OWL Logo!
DIGITAL STREAM - March 17-19, 2008 Hosted by California State University Monterey Bay School for World Languages and Cultures in Monterey, CA Presentation: OWL Software: Making It Easy to Test and Rate Speech Samples March 18, 2008 @ 11:30 AM - 12:15 PM in Building 46 Room 103 CALICO with IALLT - March 18-22, 2008 Hosted by the University of San Francisco Held at the Holiday Inn Golden Gateway in San Francisco, CA Presentation: Pittsburgh Students Are Doing “LAPS” for Proficiency March 21, 2008 @ 4:30 PM
OWL Testing Software • 1484 Washington Road • Pittsburgh, PA 15228 • (412) 436-0559 • www.owlts.com OWL publishes this newseltter quarterly and welcomes comments and suggestions. Send your feedback to mbelan@owlts.com
4
Adventures in Foreign Language • Vol. 2, No. 1 • OWL Testing Software