Helping students learn

Helping students learn through the assessment of student outcomes: A faculty responsibility Edwin C Jones, Jr University Professor Emeritus, Iowa State University Adjunct Professor, University of St Thomas Quote from Derek Bok  I would have liked more discussion among the faculty as a whole in really discussing methods of teaching. But … they don't engage in what well-run organizations in every other part of society do, that is, a continual process of self-improvement-identifying problems and experimenting with ways to solve those problems. The professors,…, spend almost all their time figuring out what courses should be required and in what sequence. They spend almost no time discussing how the courses should be taught, even though most people … say the lasting influence of colleges is focused much more on how [classes] are taught.  U S News & World Report, 3/6/06. 25 September 2006 2 Early work on instructional development in engineering        1970s—we considered Mager We wrote instructional objectives at all 6 levels We developed personalized self instruction We worked on computer-based education We considered the role of the laboratory Some publishers worked with authors to incorporate the ideas But, was it enough? 3 25 September 2006 Is Engineering Different?  Engineering Education has been well studied    Wickenden Report Grinter Report Goals Report National Academy of Engineering Dean’s Council of ASEE National Science Foundation  1990s—Three Studies    25 September 2006 4 Observations and Activities      The early studies were primarily content 1990s studies considered process In addition, many, especially larger, institutions were questioning ABET ABET responded to the institutional concerns and to the studies Industry/Government concerns also considered 5 25 September 2006 What was concluded?      Major reform of engineering education needed The field is changing so rapidly that emphasis on content was inappropriate Engineering educators needed to develop BS graduates who can be productive for a career Graduate study provides this for some, not all ABET had a major role to play 25 September 2006 6 ABET’s Response   EC2000 Criteria Major Components        Students Program Educational Objectives Program Outcomes and Assessment Professional Component Faculty Facilities Institutional Support and Resources 7 25 September 2006 Comments     ABET’s use of terms not used in all literature—goals, objectives, mission, outcomes, etc. Major Criteria Change is addition of #3, Program Outcomes and Assessment Educational Objectives had long been there, but had generally been downplayed Early on, ABET did not define objectives and outcomes. Has led to much confusion. Definitions added about 2003. 8 25 September 2006 Definitions  Program Educational Objectives  Although institutions may use different terminology, for purposes of Criterion 2, program educational objectives are broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve. 9 25 September 2006 Definitions (cont’d)  Program Outcomes (and Assessment)  Although institutions may use different terminology, for purposes of Criterion 3, program outcomes are statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that student acquire in their matriculation through the program. 10 25 September 2006 Comments on Objectives  They are marketing tools    They define the program   Potential students Employers  Assessment is straight forward   What it is What it is not--implicitly Find out what the graduates are doing Find out what suggestions they have for program improvement 11 25 September 2006 More Comments on Objectives     Criterion 2-a. Must be published Criterion 2-b. Developed with constituents Criterion 2-c. Map curriculum to objectives Criterion 2-d. Show achievement of objectives—faculty responsibility. Show also improvements. 12 25 September 2006 Program Outcomes (ABET/EAC)  Each program must formulate program outcomes that foster attainment of the program objectives articulated in satisfaction of Criterion 2 of these criteria. There must be processes to produce these outcomes and an assessment process, with documented results, that demonstrates that these program outcomes are being measured and indicates the degree to which the outcomes are achieved. There must be evidence that the results of this assessment process are applied to the further development of the program. 13 25 September 2006 Comment    Attaining outcomes does not guarantee achievement of objectives Not all graduates need to have achieved all objectives Significant changes are possible and have been observed 25 September 2006 14 What are the desired outcomes?    a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability 15 25 September 2006 Outcomes--continued     (d) an ability to function on multidisciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively 16 25 September 2006 Outcomes--continued     (h) the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. 17 25 September 2006 Additional Proviso  In addition, an engineering program must demonstrate that its students attain any additional outcomes articulated by the program to foster achievement of its education objectives. 25 September 2006 18 Where did these “a-k” come from?    Development Other professions developing similar outcomes (possibly with other names) Not limited to professional education 25 September 2006 19 Principal types of assessment  Direct    Student work evaluated by professionals considering the degree to which the work shows the designated ability Homework, tests, laboratory work, design projects, papers, portfolios Assessment is of designated ability. Goes beyond grades—deeper, focused. 20 25 September 2006 Principal types of assessment  Indirect   Surveys Other self assessment tools  A common opinion of assessment consultants—3 measurements needed     One may be indirect Direct measurements should be independent There are new companies doing assessment Remains a faculty responsibility 21 25 September 2006 How can we assume our responsibility?  Design courses with abilities in mind     e.g., EE 333 addresses 3-b, experiment design and data interpretation It uses content to do this—content relevant to the program Made clear to students and faculty Not too much responsibility in each course—suggest a H/M/L classification 22 25 September 2006 How can we assume our responsibility? Course Level       Instructional activity reviewed by, say, 3 qualified persons—including instructor When appropriate, usually soon after the end of the course, the 3 persons review student work Evaluate work on a Likert scale Compile and interpret data Suggest improvements Repeat—did improvements work? 23 25 September 2006 How can we assume our responsibility? Course Level     Not all changes we make are effective. This is OK Some might even make it worse. Again, this is OK Some changes do work, happily Continue assessment on a regular basis 25 September 2006 24 How can we assume our responsibility? Program Level Design Project—common to all engineering programs. Other programs usually have some sort of integrating project.   Be sure project requirements exercise all of the abilities Instead of, or in addition to grades, ask project evaluators to evaluate attainment of the outcomes in the project—Likert scale 25 25 September 2006 How can we assume our responsibility? Program Level  Portfolios       Common in some disciplines New venture in engineering Start as first year students Organize by content and abilities Ask evaluators to consider portfolios with respect to “a-k”—again, Likert scale Can be done on the web 26 25 September 2006 How can we assume our responsibility? What do we do with the data?      Faculty responsible for setting targets and for evaluating the data Some results will be outstanding Some results will be OK There will be room for improvement— there always is Caution—do not overdo it! 27 25 September 2006 Interesting Comparison with the Deming Cycle for Continuous Improvement PLAN ACT DO CHECK 25 September 2006 28 Implementation Matrix       Program Educational Objectives have been established and maintained Constituents are involved in helping set program objectives and in evaluating the level to which they are being achieved The required Processes are operational Outcomes Assessment is being practiced Results of outcomes and the various processes are being used to improve programs and assure objectives are being achieved An overall System is in place to meet the accreditation requirement 29 25 September 2006 Implementation Matrix    Horizontal—Objectives and Outcomes Vertical—Level of attainment, 1-5 A good accreditation suggest 3 or higher 25 September 2006 30 Part of the Matrix Level 1 3 5 Educational Objectives Not well defined Comprehensive; defined, documented and measurable; clearly tied to mission and constituent needs Comprehensive; defined documented, measurable and flexible; clearly tied to mission; readily adaptable to meet constituent needs; systematically reviewed and updated 25 September 2006 31 Part of the Matrix Level 1 3 5 Outcomes Assessment--Results Anecdotal Good outcomes; positive trends in several major areas; some evidence that results caused by systematic approach World-class outcomes; sustained results; results clearly caused by systematic approach 25 September 2006 32 Educational Objectives Constituents Informal contact Somewhat involved in defining objectives and desired outcomes, and assessment Clearly involved in defining objectives and desired outcomes, and assessment; evidence of some sustained strategic partnerships High degree of involvement in defining objectives and desired outcomes; evidence of many sustained strategic partnerships in all constituent groups Processes Few, if any processes defined and documented Some major processes defined and documented; clearly tied to mission and program objectives Processes for all major elements of criteria defined, documented, and controlled; clearly tied to mission, program objectives, and constituent needs Outcomes Assessment Limited to ad hoc efforts Some outcomes defined and improved in systematic manner; problems recognized and corrected Results Anecdotal Satisfactory outcomes; some evidence of positive trends in areas deployed Good outcomes; positive trends in several major areas; some evidence that results caused by systematic approach System None evident Early stages; partial deployment within the program and college 1 Not well defined Broadly defined and documented; clearly tied to mission; evidence of constituent input 2 3 Comprehensive; defined, documented and measurable; clearly tied to mission and constituent needs All major outcomes defined; systematic evaluation and process improvement in place; problems anticipated and prevented All outcomes defined; systematic evaluation and process improvement in place; many support areas involved; sources of problems understood and eliminated In place; deployed throughout the program and college; driven by mission and objectives 4 Comprehensive; defined, documented and measurable; clearly tied to mission; responsive to constituent needs; systematically reviewed and updated Processes for all elements of criteria are quantitatively understood and controlled; clearly tied to mission, program objectives, and constituent needs Excellent outcomes; positive trends in most areas; evidence that results caused by systematic approach Integrated; deployed throughout the program, college, and support areas; driven by mission and objectives 5 Comprehensive; defined documented, measurable and flexible; clearly tied to mission; readily adaptable to meet constituent needs; systematically reviewed and updated High degree of involvement in defining objectives and desired outcomes, assessment; and improvement cycles; sustained evidence of strategic partnership with all key constituents Processes for all elements of criteria are quantitatively understood and controlled; clearly tied to mission, program objectives, and constituent needs; seen as benchmarks by other institutions All outcomes defined; systematic evaluation and process improvement in place; all support areas involved; common sources of problems understood and eliminated World-class outcomes; sustained results; results clearly caused by systematic approach Sound, highly integrated system; deployed throughout the program, college, and institution; driven by mission and objectives Summary      Assessment is a developing faculty responsibility Good techniques exist—will get better Considerable startup required Eventually, readily accomplished while the process improves Results are worth the effort 34 25 September 2006