Departmental Faculty

Document Sample
Departmental Faculty Powered By Docstoc
					                                                                               Graduate Degrees 1

                Graduate Degrees in Educational Measurement and Evaluation

                                       Jeffrey Kromrey
                                         Melinda Hess
                                       Constance Hines
                                        Dorian Vizcain
                                         Bethany Bell
                                   Gianna Rendina-Gobioff
                                         Lisa Adkins
                                  University of South Florida

                                       Jeanine Romano
                                     University of Tampa

Paper presented at the Annual Meeting of the Association for Institutional Research, May 14-18,
2006, Chicago
                                                                                   Graduate Degrees 2

       The Department of Educational Measurement and Research at the University of South
Florida (USF) provides graduate preparation for students interested in pursuing advanced studies
in the areas of educational assessment, educational research, program evaluation, and educational
statistics. Many students interested in research are not familiar with the professions associated
with measurement and research, and the opportunities which this field presents. The need for
skilled researchers in education and other social and behavioral sciences is increasing at a rapid
pace. As the call for accountability at virtually every level and within every professional field of
education increases, so does the need for specialists in statistics, measurement, and evaluation.
Development of defensible, valid and reliable measures as well as the ability to evaluate the
effectiveness of educational interventions, programs and curricula is a complex process requiring
well trained professionals. Recent national surveys have reconfirmed the conclusions of the
National Council of Measurement in Education that there is a shortfall in the number of
measurement professionals graduating from educational programs relative to the number of
available employment opportunities and that this shortfall is especially severe among minority
professionals. A review of job boards sponsored by the American Educational Research
Association, the Chronicle of Higher Education, the American Institutes of Research, the
Educational Testing Service, and ACT Inc. further underscores the job opportunities available to
individuals with earned graduate degrees in this field.
       This paper describes the graduate programs in Educational Measurement and Evaluation
provided at the University of South Florida. The program philosophy, professional preparation
components, and student experiences are detailed, and the major domains of post-graduate
professional practice are described. Finally, information about program application procedures
and points of contact for additional information are provided.

                      Program Context: The University of South Florida

       USF was established in 1956 as a public university and is a comprehensive multi-campus
research university serving more than 42,000 students. With four campuses, USF is home to
medical clinics and hospitals, a major mental health research institute, and two public
broadcasting stations. USF generates over $290 million annually in sponsored research. In
FY2003/2004 there were 1,610 active sponsored projects. Its endowment exceeds $244 million
                                                                                    Graduate Degrees 3

and includes 55 endowed chairs. USF is classified as a Doctoral/Research University-Extensive,
which is the highest classification by the Carnegie Foundation. The largest metropolitan research
university and the second largest in total enrollment in the State University System of Florida,
USF offers a wide variety of degree programs with both basic and applied orientations, including
86 baccalaureate, 86 master’s, and 32 doctoral degrees, including the M.D.

               Graduate Programs in Educational Measurement and Research

       The Ph.D. program in Educational Measurement and Evaluation was initiated in 1971
and was the first Ph.D. program in the College of Education. Since awarding its initial Ph.D.
degree in 1975, the Department of Educational Measurement and Research has produced over 70
professionals with Ph.D. degrees. In addition to the Ph.D. degree, the Department also offers the
M.Ed. and Ed.S. degrees in Educational Measurement and Evaluation. The faculty and students
in our programs are diverse in terms of professional perspectives and philosophies, gender, race
and culture. The Department includes international students and faculty who are engaged in
research, teaching, and service locally, regionally, nationally, and internationally. It is notable
that students of color accounted for 27% of the admissions over the past five years.

                                       Program Philosophy
       Our current graduate programs prepare students through coursework and other applied
experiences in three broad areas: educational measurement, methods of inquiry, and evaluation.
The goal is to prepare scholars and future leaders in the fields of educational measurement,
evaluation, and methods of inquiry who will not only be producers of new knowledge in their
field, but will also be skilled and knowledgeable in the content and advances in their field, and
who will engage in critical and reflective thought in the applications of methods to solve
problems and improve educational practice. Accordingly, emphasis is placed on the development
of systematic inquiry skills essential to the study and evaluation of educational processes and
outcomes in education and the social sciences.
       In keeping with the interdisciplinary nature of the field of education and the program
faculty’s valuing of diversity and diverse perspectives, students in the program are exposed to
multiple research traditions that are employed in education and the social sciences.
                                                                                    Graduate Degrees 4

Methodological skills are developed within a programmatic context that encourages development
of knowledge about education, considers important principles of and ethics in research,
assessment, and evaluation, and provides a clinical setting in which these elements can be fused
into professional applications.
       Program faculty value and seek to create and nurture a dynamic environment in which a
shared commitment to learning, engagement in scholarly inquiry, and the development of
intellectual curiosity and other habits of mind, is part of the graduate school experience for full-
time students. This is pursued through collaborative research groups of students and faculty as
well as students working with individual faculty members within the department, the College of
Education or in other Colleges across the university. Students are encouraged to participate
actively in research and to present their research in refereed conferences and journals. Part of
faculty load is used to support this requirement. In addition, each student engages in a
supervised practicum that provides opportunities for the application of methods of systematic
inquiry in various settings.
Teaching, Research and Service: Contributions to the Goals of the Graduate Programs
       The three traditional activities of teaching, research, and public service form the
foundation of the University’s mission. The graduate programs in the Department of
Educational Measurement and Research embrace each of these activities in promoting an
environment in which students can achieve the goals of the graduate program.
       Teaching. The program faculty is committed to excellence in teaching and uses a multi-
modal approach to excellence in teaching practices. This approach includes: 1) traditional
methods of didactic classroom instruction, 2) case-study methods within a cooperative learning
paradigm, 3) problem-based learning and assessment strategies, 4) extensive use of technology in
teaching, research, and assessment, and 5) distance learning opportunities for some required
courses. In addition, faculty members work collaboratively with students to identify, conduct,
and disseminate research on practical educational problems. Department faculty members
believe the teaching process is best when it is informed by research, is on-going in every
interaction between students and faculty, and uses a wide range of resources and settings.
       In addition to the emphasis on excellence in teaching by its faculty, all doctoral students
are encouraged to acquire skills themselves in teaching at the university level. These skills are
developed in many ways, including completion of a Graduate School sponsored teaching
                                                                                     Graduate Degrees 5

seminar, participation in a semester-long teaching internship under the supervision of a full
professor and experienced teaching assistant, and teaching independently under the supervision
of the faculty member who supports them through the process of teaching at the undergraduate
level. After their undergraduate teaching experience and completion of appropriate doctoral
coursework, they may serve as a supervised teaching assistant in various graduate level courses.
Students focused on becoming faculty members themselves typically volunteer to work in
several courses.
       Research. The Department of Educational Measurement and Research has a strong
commitment to the scientist-practitioner model that emphasizes the relationship between research
and practice. All faculty members in the Department are experienced researchers with strong
records of scholarly productivity. Graduate students demonstrate their research skills through
ongoing collaborative research with faculty, directed research courses, assignments within the
Department’s Center for Research, Evaluation, Assessment, and Measurement (CREAM), and
through their thesis or dissertation. It is important to note that the research efforts of faculty and
students are focused on seeking solutions to real education problems in universities, school
districts, government agencies, commercial test publishing, and program evaluation enterprises
ensuring a breadth of skill and experience for graduates. This breadth is critical for alumni today
with accountability models being applied routinely across all these arenas.
       Service. Consistent with the University’s mission, program faculty and students provide
extensive service to school districts, charter schools, hospital-school programs, community
initiatives, university community service, and their professional organizations. The faculty
provides this service, in part, to model service commitment to students in the program; therefore,
faculty and students work together in providing community and professional service.
       In summary, the goals of the program are achieved through the integration of excellence
in teaching, research that informs and serves, and programs that provide benefit to educational
enterprises in schools, government, and business. It is the successful integration of these three
activities (the foundation of the university’s mission) that enables the Department of Educational
Measurement and Research to achieve programmatic goals for each student and to receive the
national and international recognition that the program has enjoyed.
                                                                                     Graduate Degrees 6

                                      Departmental Faculty
       The faculty members in the Department of Educational Measurement and Research
include six tenured/tenure-earning members (including the Department Chair) and three Visiting
Assistant Professors. Four members of the Department hold the rank of Professor, one is an
Associate Professor, and one is an Assistant Professor. The expertise of the faculty members
spans the areas of research design, measurement and psychometrics, statistics and data analysis,
and program evaluation. The typical teaching assignment for faculty members is two graduate
courses each semester. In addition to course assignments, each faculty member serves on
dissertation and thesis committees for students within the Department as well as committees
within other graduate departments in the College and across the university.
       Each member of the faculty in the Department of Educational Measurement and Research
has a prolific publication record and publishes a number of articles or book chapters each year.
In addition to such publications, faculty regularly present research papers at scholarly
conferences including national and international research conferences, regional research
conferences, and state conferences. The majority of these conferences are focused on educational
research, but presentations at general statistics conferences (e.g., the American Statistical
Association) and research software conferences (SAS Users Group International) are also routine
outlets for disseminating the results of scholarly activities. Further, a large proportion of these
publications represent collaborative work between the faculty members and graduate students in
the Department. Finally, three of the faculty members in the Department currently serve as
journal editors or members of editorial boards for journals in the discipline.
       The Department of Educational Measurement and Research has been very successful in
obtaining external funding to support research, graduate assistantships, and faculty support
(computers, travel, equipment). The Department has secured more than 800,000 dollars in
external funding over the past five years, including funding through federal and state grants, as
well as local school districts. In addition to this independent funding, faculty members
collaborate with other faculty across the university in joint research proposals.

       The Department of Measurement and Research has formed numerous partnerships within
the college, across campus, and in the community. These partnerships include ongoing research
                                                                                    Graduate Degrees 7

projects as well as supportive activities for faculty and students. Although there are numerous
partnerships that exist with diverse organizations and in various guises, the department’s students
and faculty often work with local centers and institutes to further their missions, gain new
insights and experiences, and provide support for the university and community.
       Within the department, there are two centers with diverse, yet complementary missions.
The Center for Research, Evaluation, Assessment, and Measurement (CREAM, established in
2003) provides research and evaluation services to the University, businesses, government, and
Pk-12 school communities. It brings faculty, students, and community partners together to
assess current programs and to aid planning for the future. The Center is staffed primarily by a
full-time director and graduate students in the College of Education. Members of the teaching
faculty also work on many of the projects, thus building relationships with students, each other,
and outside organizations. Research and evaluation activities undertaken by CREAM provide
students with real-world research and evaluation experiences and foster partnerships with school
districts, colleges, departments, and organizations on and off campus. In addition to CREAM,
the Department also supports the Consulting Office for Research in Education (CORE).
Established in 2005, CORE supports faculty and students within the College of Education by
offering consulting services for their research and evaluation projects and activities. CORE is
staffed by advanced Measurement and Research doctoral students who advise educational
researchers on methods, techniques, and analyses appropriate for their research. This service is
invaluable to members of the college who are not primarily methodologists and enhances the
research mission of the college and university. In addition, it provides the students working in
CORE with diverse situations to hone their skills and expand their personal and professional
knowledge base in research, measurement, and evaluation.
       Although there are many centers and institutes outside of the department that may work
with department faculty and students, there are four specific agencies with whom the department
has formed strong bonds: (1) The Institute for Instructional Research and Practice (IIRP) was
created in 1984 and focuses on a wide range of testing and measurement issues at the state,
national and international levels. IIRP is involved in certification/licensure, student testing,
program evaluation, charter schools, teacher preparation, and community development. (2) The
Center for Research on Children’s Development and Learning, established in 1980, focuses on
improving environments in which children of all ages develop and learn, including, in particular,
                                                                                    Graduate Degrees 8

family, community, school, and classroom contexts. As part of its mission the Center is involved
in developing, validating, and disseminating effective interventions for childhood developmental
and learning problems. (3) The Research and Training Center for Children's Mental Health at
USF's Louis de la Parte Florida Mental Health Institute was initiated in 1984 to address the need
for improved services and outcomes for children with serious emotional/behavioral disabilities
and their families. Since that time, the Center has conducted research, synthesized and shared
existing knowledge, provided training and consultation, and served as a resource for other
researchers, policy makers, and administrators in the public system, and organizations
representing parents, consumers, advocates, professional societies, and practitioners. (4) The
David C. Anchin Center strives to assure excellence in the educational future of Florida and the
nation. Locally, the Center is concerned with providing teachers the help that they need to create
superior schools and students by directing efforts toward professional development of teachers
and the infusion of technology into teaching in ways that encourage student-centered learning.
Nationally, the Center is involved in large-scale evaluation research projects on national reform
efforts. Through this research, the Center is able to make recommendations at the federal level
that may impact the course of educational reform.
       Faculty and graduate students from the Department of Measurement and Research have
been actively involved in collaborative research activities with all of these centers and institutes
and continue to partner with them on various projects. As students and faculty join the
department, other partnerships are anticipated to be formed based on the interests and skills they
bring with them to the college and university.

                 The Graduate Program: Courses and Research Experiences

       Graduate degrees offered through USF’s Department of Educational Measurement and
Research include the Masters of Education (M.Ed.), Education Specialist (Ed.S.) and Doctor of
Philosophy (Ph.D.). Each graduate program in the department offers a balance of coursework
and applied measurement, evaluation and research experiences. More specifically, typical
statistical courses taken by Measurement and Research graduate students include topics such as
ANOVA, regression, multivariate statistics, interrupted time series analysis, structural equation
modeling, and multilevel modeling. Measurement courses provide training in criterion-
                                                                                  Graduate Degrees 9

referenced and norm-referenced testing, classical test theory, item response theory and computer-
based testing. Finally, evaluation and research methods courses provide training in qualitative,
quantitative, and mixed methods focused on both decision making and knowledge generation.
       In addition to the general courses described above, Measurement and Research doctoral
students pursue additional training through a selected cognate area. Cognates can be
methodological or applied in nature and can consist of courses offered within or outside the
College of Education. Previous methodological cognates pursued by Measurement and Research
doctoral students include Research Methods and Issues as well as Measurement Methods and
Issues. Previous applied cognates include Developmental Psychology, Instructional Technology
and Community and Family Health. Examples of courses taken with the Research Methods and
Issue cognate include Biostatistical Case Studies and Collaboration, Survey Sampling Methods
in Health, Research Methods in Anthropology and Meta Analysis. Examples of courses taken
under the Developmental Psychology cognate include Language and Literacy Development,
Resilience in Human Development, Developmental Issues in Instruction and Developmental
Research Methods.
       To ensure our graduate students obtain practical, hands on experiences outside the
academic classroom, all degree programs require students to complete a practicum. In addition to
the practicum requirement, students pursuing the Ph.D. fulfill directed research requirements.
Besides increasing students’ knowledge and skills in research design, measurement, statistics,
and program evaluation, the practicum requirement allows students the opportunity to work in an
applied setting, whereas the directed research requirement allows doctoral students to actively
engage in a research project with one or more faculty members.
       The nature of the practicum and directed research are tailored to the student’s interests
and program. For example, common practicum experiences for M.Ed. students include
conducting needs assessments and program evaluations under the supervision of Department
faculty members and local education administrators. For Ed.S. and Ph.D. students, the practicum
experience for students interested in measurement issues may include developing an instrument
or gathering validity evidence for an existing instrument being applied to a new population or
new context. In contrast, the practicum experiences for students interested in applied research
methods may include designing and conducting field studies to test educational interventions.
The same flexibility exists for fulfillment of the directed research requirement. Previous projects
                                                                                  Graduate Degrees 10

undertaken by doctoral students to fulfill their directed research requirement include simulation
studies on the behavior of statistics, a case study on collaborative learning in undergraduate
education, and the evaluation of technological tools for the management of statistics anxiety.
       In addition to gaining research experience through the practicum and directed research
requirements, many Measurement and Research graduate students are employed as graduate
research assistants, both within and outside the College of Education. As graduate research
assistants, students are able to further apply the knowledge and skills obtained in the classroom
to a variety of research projects. For example, a Measurement and Research graduate research
assistant who works in the Department’s Center for Research, Evaluation, Assessment, and
Measurement will have the opportunity to work on a breadth of projects including grants funded
through the National Science Foundation and American Cancer Society, initiatives advanced by
the University Provost, and projects contracted by local public education districts.
       Measurement and Research doctoral students also have the opportunity to teach the
undergraduate measurement course for teachers at USF’s College of Education and assist the
Department’s faculty in the delivery of graduate courses. The opportunity to teach undergraduate
measurement students is a great advantage for graduates of the USF program because many
graduate students from other universities with similar programs do not have the opportunity to
get independent teaching experience. In addition, the opportunity to develop and deliver
graduate level courses with Department faculty provides students with diverse teaching
       Measurement and Research graduate students are also involved in presenting research at
professional conferences and publishing papers in scholarly, peer reviewed journals both on their
own and in collaboration with department faculty and fellow graduate students. For example,
Measurement and Research graduate students have presented at the annual meetings of the
American Educational Research Association, the Florida Educational Research Association, the
American Public Health Association, and the American Statistical Association. Recent examples
of student presentations from the 2006 AERA annual meeting include:
       o      What Do Graduate Students Value in Their Ideal Mentor – Bethany A. Bell-Ellison
              and Robert F. Dedrick (Doctoral Student and Faculty Member)
                                                                                 Graduate Degrees 11

       o    Examining the Internal Structure of the Trends in International Mathematics and
            Science Study (TIMSS) 2003 Assessment: A Confirmatory Factor Analysis Approach
            - Ha Phan (Doctoral Student)
       o    Hispanic/Latino Male Student Dropout: Logistic Regression Analysis Versus
            Proportional Hazards Analysis – Dorian C. Vizcain (Doctoral Student)

Measurement and Research graduate students have also published in a number of scientific peer
reviewed journals including Educational and Psychological Measurement, Journal of College
Teaching and Learning, and Communications in Statistics. Examples of student publications
       o    An empirical comparison of regression analysis strategies with discrete ordinal
            variables. – Jeffrey D. Kromrey and Gianna Rendina-Gobioff (2002). Published in
            Multiple Linear Regression Viewpoints 28(2), 30-43. (Measurement and Research
            Faculty Member and Measurement and Research Doctoral Student)
       o    The quality of factor solutions in exploratory factor analysis: The influence of
            sample size, communality and overdetermination. – Kristine Y. Hogarty, Constance
            V. Hines, Jeffrey D. Kromrey, John M. Ferron, and Karen R. Mumford (2005).
            Published in Educational and Psychological Measurement, 65, 202-226.
            (Measurement and Research Doctoral Students, Measurement and Research Faculty
            Members, and the Director of Assessment for the College of Education)
       o    A multilevel study of the associations between economic and social context, stage of
            adolescence, and physical activity and BMI. – Caroline Mae McKay, Bethany A.
            Bell-Ellison, Kirsten Wallace, and John Ferron (2006). Accepted for an upcoming
            special issue of Pediatrics. (Measurement and Research Doctoral Students along
            with Public Health Doctoral Student, and Measurement and Research Faculty

                The Graduate Program and Beyond: Students’ Perspectives

       As one begins the academic goal of an advanced degree in Measurement and Research,
many of the opportunities are not yet visible. The excitement of the educational environment and
                                                                                  Graduate Degrees 12

the overwhelming feeling of full time coursework leave little time to contemplate future
employment possibilities. Prior to discussing areas of employment in the field of Measurement
and Research, a few thoughts on the student experience.
       The rigorous coursework as one begins in the graduate program is enhanced with
opportunities to collaborate with faculty and other students on current research projects many of
which result in proposals and submissions for conference presentations. Along the way, the
readings on the major thinkers in their respective specialties is where much of the learning is
taking place: learning the various research approaches, details of the myriad of methodological
choices, and most effective presentations of research results. The learning among graduate
student colleagues is integral to the educational experience as ideas are shared and discussed.
       As one develops their educational research skills, students are exposed to and participate
in more complex advanced methods, with larger samples and longitudinal research designs.
Having faculty close and available for feedback and suggestions on “best” procedures is
invaluable as the student approaches the end of their coursework and preparation for the
Qualifying Exam leading to Doctoral Candidacy.
       The graduate student now embarks on research of interest and defends a proposal plan of
conducting their dissertation research. Once successfully presented to committee, the researcher
collects, analyzes, and reports the findings. The final dissertation defense is the culmination of
years of preparation and the author’s original contribution to empirical knowledge.
       The types of employment that graduates of our program achieve are as diverse as our
student population. There are four settings that graduates typically find employment: higher
education, Pk-12 education systems, testing/research organizations, and independent consulting.
Within the higher education setting graduates have continued their research and teaching as
faculty and directors. Another organization that many of our graduates choose to apply their
evaluation, research, and measurement skills is within public school districts or state
departments. Often these graduates are administrators (directors, primary researchers,
measurement specialists) in the research/evaluation/measurement departments. Testing and
research organizations are another opportunity for graduates of our program. Examples of
testing organizations in which our graduates have continued work as psychometricians are ACT,
Harcourt, and Institute for Instructional Research and Practice. Finally, some graduates choose
                                                                               Graduate Degrees 13

to seek clients and contracts independently by becoming measurement, evaluation, and research

                      Program Application and Additional Information

       Application for admission to the graduate programs in the Department of Educational
Measurement and Research begins with an application to the University of South Florida’s
Graduate School ( In addition to transcripts of
undergraduate and previous graduate work, applicants to our programs provide GRE scores,
letters of recommendation, a writing sample, and a statement of professional goals. Following
our review of these materials, applicants are interviewed by three members of the Department
faculty. Additional information about our programs is available on the Department’s website
( Questions about the programs and
program admissions should be directed to the Department Chair (Jeffrey D. Kromrey,, or 813 974-5739).

Shared By: