Helping students learn through the assessment of student outcomes: A faculty responsibility
Edwin C Jones, Jr University Professor Emeritus, Iowa State University Adjunct Professor, University of St Thomas
Quote from Derek Bok
I would have liked more discussion among the faculty as a whole in really discussing methods of teaching. But … they don't engage in what well-run organizations in every other part of society do, that is, a continual process of self-improvement-identifying problems and experimenting with ways to solve those problems. The professors,…, spend almost all their time figuring out what courses should be required and in what sequence. They spend almost no time discussing how the courses should be taught, even though most people … say the lasting influence of colleges is focused much more on how [classes] are taught.
U S News & World Report, 3/6/06.
25 September 2006
2
Early work on instructional development in engineering
1970s—we considered Mager We wrote instructional objectives at all 6 levels We developed personalized self instruction We worked on computer-based education We considered the role of the laboratory Some publishers worked with authors to incorporate the ideas But, was it enough?
3
25 September 2006
Is Engineering Different?
Engineering Education has been well studied
Wickenden Report Grinter Report Goals Report
National Academy of Engineering Dean’s Council of ASEE National Science Foundation
1990s—Three Studies
25 September 2006
4
Observations and Activities
The early studies were primarily content 1990s studies considered process In addition, many, especially larger, institutions were questioning ABET ABET responded to the institutional concerns and to the studies Industry/Government concerns also considered
5
25 September 2006
What was concluded?
Major reform of engineering education needed The field is changing so rapidly that emphasis on content was inappropriate Engineering educators needed to develop BS graduates who can be productive for a career Graduate study provides this for some, not all ABET had a major role to play
25 September 2006
6
ABET’s Response
EC2000 Criteria Major Components
Students Program Educational Objectives Program Outcomes and Assessment Professional Component Faculty Facilities Institutional Support and Resources
7
25 September 2006
Comments
ABET’s use of terms not used in all literature—goals, objectives, mission, outcomes, etc. Major Criteria Change is addition of #3, Program Outcomes and Assessment Educational Objectives had long been there, but had generally been downplayed Early on, ABET did not define objectives and outcomes. Has led to much confusion. Definitions added about 2003.
8
25 September 2006
Definitions
Program Educational Objectives Although institutions may use different terminology, for purposes of Criterion 2, program educational objectives are broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve.
9
25 September 2006
Definitions (cont’d)
Program Outcomes (and Assessment) Although institutions may use different terminology, for purposes of Criterion 3, program outcomes are statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that student acquire in their matriculation through the program.
10
25 September 2006
Comments on Objectives
They are marketing tools
They define the program
Potential students Employers
Assessment is straight forward
What it is What it is not--implicitly
Find out what the graduates are doing Find out what suggestions they have for program improvement
11
25 September 2006
More Comments on Objectives
Criterion 2-a. Must be published Criterion 2-b. Developed with constituents Criterion 2-c. Map curriculum to objectives Criterion 2-d. Show achievement of objectives—faculty responsibility. Show also improvements.
12
25 September 2006
Program Outcomes (ABET/EAC)
Each program must formulate program outcomes that foster attainment of the program objectives articulated in satisfaction of Criterion 2 of these criteria. There must be processes to produce these outcomes and an assessment process, with documented results, that demonstrates that these program outcomes are being measured and indicates the degree to which the outcomes are achieved. There must be evidence that the results of this assessment process are applied to the further development of the program.
13
25 September 2006
Comment
Attaining outcomes does not guarantee achievement of objectives Not all graduates need to have achieved all objectives Significant changes are possible and have been observed
25 September 2006
14
What are the desired outcomes?
a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability
15
25 September 2006
Outcomes--continued
(d) an ability to function on multidisciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively
16
25 September 2006
Outcomes--continued
(h) the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.
17
25 September 2006
Additional Proviso
In addition, an engineering program must demonstrate that its students attain any additional outcomes articulated by the program to foster achievement of its education objectives.
25 September 2006
18
Where did these “a-k” come from?
Development Other professions developing similar outcomes (possibly with other names) Not limited to professional education
25 September 2006
19
Principal types of assessment
Direct
Student work evaluated by professionals considering the degree to which the work shows the designated ability Homework, tests, laboratory work, design projects, papers, portfolios Assessment is of designated ability. Goes beyond grades—deeper, focused.
20
25 September 2006
Principal types of assessment
Indirect
Surveys Other self assessment tools
A common opinion of assessment consultants—3 measurements needed
One may be indirect Direct measurements should be independent There are new companies doing assessment Remains a faculty responsibility
21
25 September 2006
How can we assume our responsibility?
Design courses with abilities in mind
e.g., EE 333 addresses 3-b, experiment design and data interpretation It uses content to do this—content relevant to the program Made clear to students and faculty Not too much responsibility in each course—suggest a H/M/L classification
22
25 September 2006
How can we assume our responsibility? Course Level
Instructional activity reviewed by, say, 3 qualified persons—including instructor When appropriate, usually soon after the end of the course, the 3 persons review student work Evaluate work on a Likert scale Compile and interpret data Suggest improvements Repeat—did improvements work?
23
25 September 2006
How can we assume our responsibility? Course Level
Not all changes we make are effective. This is OK Some might even make it worse. Again, this is OK Some changes do work, happily Continue assessment on a regular basis
25 September 2006
24
How can we assume our responsibility? Program Level
Design Project—common to all engineering programs. Other programs usually have some sort of integrating project.
Be sure project requirements exercise all of the abilities Instead of, or in addition to grades, ask project evaluators to evaluate attainment of the outcomes in the project—Likert scale
25
25 September 2006
How can we assume our responsibility? Program Level
Portfolios
Common in some disciplines New venture in engineering Start as first year students Organize by content and abilities Ask evaluators to consider portfolios with respect to “a-k”—again, Likert scale Can be done on the web
26
25 September 2006
How can we assume our responsibility? What do we do with the data?
Faculty responsible for setting targets and for evaluating the data Some results will be outstanding Some results will be OK There will be room for improvement— there always is Caution—do not overdo it!
27
25 September 2006
Interesting Comparison with the Deming Cycle for Continuous Improvement
PLAN
ACT
DO
CHECK
25 September 2006
28
Implementation Matrix
Program Educational Objectives have been established and maintained Constituents are involved in helping set program objectives and in evaluating the level to which they are being achieved The required Processes are operational Outcomes Assessment is being practiced Results of outcomes and the various processes are being used to improve programs and assure objectives are being achieved An overall System is in place to meet the accreditation requirement
29
25 September 2006
Implementation Matrix
Horizontal—Objectives and Outcomes Vertical—Level of attainment, 1-5 A good accreditation suggest 3 or higher
25 September 2006
30
Part of the Matrix
Level 1 3 5 Educational Objectives
Not well defined Comprehensive; defined, documented and measurable; clearly tied to mission and
constituent needs
Comprehensive; defined documented, measurable and flexible; clearly tied to mission; readily adaptable to meet constituent needs; systematically reviewed and updated
25 September 2006
31
Part of the Matrix
Level 1 3 5 Outcomes Assessment--Results
Anecdotal
Good outcomes; positive trends in several major areas; some evidence that results caused by systematic approach
World-class outcomes; sustained results; results clearly caused by systematic approach
25 September 2006
32
Educational Objectives
Constituents
Informal contact Somewhat involved in defining objectives and desired outcomes, and assessment Clearly involved in defining objectives and desired outcomes, and assessment; evidence of some sustained strategic partnerships High degree of involvement in defining objectives and desired outcomes; evidence of many sustained strategic partnerships in all constituent groups
Processes
Few, if any processes defined and documented Some major processes defined and documented; clearly tied to mission and program objectives Processes for all major elements of criteria defined, documented, and controlled; clearly tied to mission, program objectives, and constituent needs
Outcomes Assessment
Limited to ad hoc efforts Some outcomes defined and improved in systematic manner; problems recognized and corrected
Results
Anecdotal Satisfactory outcomes; some evidence of positive trends in areas deployed Good outcomes; positive trends in several major areas; some evidence that results caused by systematic approach
System
None evident Early stages; partial deployment within the program and college
1
Not well defined Broadly defined and documented; clearly tied to mission; evidence of constituent input
2
3
Comprehensive; defined, documented and measurable; clearly tied to mission and constituent needs
All major outcomes defined; systematic evaluation and process improvement in place; problems anticipated and prevented
All outcomes defined; systematic evaluation and process improvement in place; many support areas involved; sources of problems understood and eliminated
In place; deployed throughout the program and college; driven by mission and objectives
4
Comprehensive; defined, documented and measurable; clearly tied to mission; responsive to constituent needs; systematically reviewed and updated
Processes for all elements of criteria are quantitatively understood and controlled; clearly tied to mission, program objectives, and constituent needs
Excellent outcomes; positive trends in most areas; evidence that results caused by systematic approach
Integrated; deployed throughout the program, college, and support areas; driven by mission and objectives
5
Comprehensive; defined documented, measurable and flexible; clearly tied to mission; readily adaptable to meet constituent needs; systematically reviewed and updated
High degree of involvement in defining objectives and desired outcomes, assessment; and improvement cycles; sustained evidence of strategic partnership with all key constituents
Processes for all elements of criteria are quantitatively understood and controlled; clearly tied to mission, program objectives, and constituent needs; seen as benchmarks by other institutions
All outcomes defined; systematic evaluation and process improvement in place; all support areas involved; common sources of problems understood and eliminated
World-class outcomes; sustained results; results clearly caused by systematic approach
Sound, highly integrated system; deployed throughout the program, college, and institution; driven by mission and objectives
Summary
Assessment is a developing faculty responsibility Good techniques exist—will get better Considerable startup required Eventually, readily accomplished while the process improves Results are worth the effort
34
25 September 2006