Helping students learn

Helping students learn through the assessment of student outcomes:Afaculty responsibilityEdwin C Jones, JrUniversity Professor Emeritus, Iowa State UniversityAdjunct Professor, University of St Thomas25 September 20062Quote from Derek BokI would have liked more discussion among the faculty as a whole in really discussing methods of teaching. But … they don't engage in what well-run organizations in every other part of society do, that is, a continual process of self-improvement--identifying problems and experimenting with ways to solve those problems. The professors,…, spend almost all their time figuring out what courses should be required and in what sequence. They spend almost no time discussing how the courses should be taught, even though most people … say the lasting influence of colleges is focused much more on how [classes] are taught. U S News & World Report, 3/6/06.25 September 20063Early work on instructional development in engineering1970s—we considered MagerWe wrote instructional objectives at all 6 levelsWe developed personalized self instructionWe worked on computer-based educationWe considered the role of the laboratorySome publishers worked with authors to incorporate the ideasBut, was it enough?25 September 20064Is Engineering Different?Engineering Education has been well studiedWickenden ReportGrinter ReportGoals Report1990s—Three StudiesNational Academy of EngineeringDean’s Council of ASEENational Science Foundation25 September 20065Observations and ActivitiesThe early studies were primarily content1990s studies considered processIn addition, many, especially larger, institutions were questioning ABETABET responded to the institutional concerns and to the studiesIndustry/Government concerns also considered25 September 20066What was concluded?Major reform of engineering education neededThe field is changing so rapidly that emphasis on content was inappropriateEngineering educators needed to develop BS graduates who can be productive for a careerGraduate study provides this for some, not allABET had a major role to play25 September 20067ABET’s ResponseEC2000 CriteriaMajor ComponentsStudentsProgram Educational ObjectivesProgram Outcomes and AssessmentProfessional ComponentFacultyFacilitiesInstitutional Support and Resources25 September 20068CommentsABET’s use of terms not used in all literature—goals, objectives, mission, outcomes, etc.Major Criteria Change is addition of #3, Program Outcomes and AssessmentEducational Objectives had long been there, but had generally been downplayedEarly on, ABET did not define objectives and outcomes. Has led to much confusion. Definitions added about 2003. 25 September 20069DefinitionsProgram Educational ObjectivesAlthoughinstitutionsmayusedifferentterminology,forpurposesofCriterion2,programeducationalobjectivesarebroadstatementsthatdescribethecareerandprofessionalaccomplishmentsthattheprogramispreparinggradutestoachieve.25 September 200610Definitions (cont’d)Program Outcomes (and Assessment)Althoughinstitutionsmayusedifferentterminology,forpurposesofCriterion3,programoutcomesarestatementsthatdescribewhatstudentsareexpectedtoknowandbeabletodobythetimeofgraduation.Theserelatetotheskills,knowledge,andbehaviorsthatstudentacquireintheirmatriculationthroughtheprogram.25 September 200611Comments on ObjectivesThey are marketing toolsPotential studentsEmployersThey define the programWhat it isWhat it is not--implicitlyAssessment is straight forwardFind out what the graduates are doingFind out what suggestions they have for program improvement25 September 200612More Comments on Objectives Criterion 2-a. Must be publishedCriterion 2-b. Developed with constituentsCriterion 2-c. Map curriculum to objectivesCriterion 2-d. Show achievement of objectives—faculty responsibility. Show also improvements.25 September 200613Program Outcomes (ABET/EAC)EachprogrammustformulateprogramoutcomesthatfosterattainmentoftheprogramobjectivesarticulatedinsatisfactionofCriterion2ofthesecrteria.Theremustbeprocessestoproducetheseoutcomesandanassessmentprocess,withdocumentedresults,thatdemonstratesthattheseprogramoutcomesarebeingmeasuredandindicatesthedegreetowhichtheoutcomesareachieved.Theremustbeevidencethattheresultsofthisassessmentprocessareappliedtothefurtherdevelopmentoftheprogram.25 September 200614CommentAttaining outcomes does not guarantee achievement of objectivesNot all graduates need to have achieved all objectivesSignificant changes are possible and have been observed25 September 200615What are the desired outcomes?a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability 25 September 200616Outcomes--continued(d) an ability to function on multi-disciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively 25 September 200617Outcomes--continued(h) the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. 25 September 200618Additional ProvisoInaddition,anengineeringprogrammustdemonstratethatitsstudentsattainanyadditionaloutcomesarticulatedbytheprogramtofosterachievementofitseduationobjectives.25 September 200619Where did these “a-k” come from?DevelopmentOther professions developing similar outcomes (possibly with other names)Not limited to professional education25 September 200620Principal types of assessmentDirectStudent work evaluated by professionals considering the degree to which the work shows the designated abilityHomework, tests, laboratory work, design projects, papers, portfoliosAssessment is of designated ability. Goes beyond grades—deeper, focused.25 September 200621Principal types of assessmentIndirectSurveys Other self assessment toolsA common opinion of assessment consultants—3 measurements neededOne may be indirectDirect measurements should be independentThere are new companies doing assessmentRemains a faculty responsibility25 September 200622How can we assume our responsibility?Design courses with abilities in minde.g., EE 333 addresses 3-b, experiment design and data interpretationIt uses content to do this—content relevant to the programMade clear to students and facultyNot too much responsibility in each course—suggest a H/M/L classification25 September 200623How can we assume our responsibility? Course LevelInstructional activity reviewed by, say, 3 qualified persons—including instructorWhen appropriate, usually soon after the end of the course, the 3 persons review student workEvaluate work on a Likert scaleCompile and interpret dataSuggest improvementsRepeat—did improvements work?25 September 200624How can we assume our responsibility? Course LevelNot all changes we make are effective. This is OKSome might even make it worse. Again, this is OKSome changes do work, happilyContinue assessment on a regular basis25 September 200625How can we assume our responsibility? Program LevelDesign Project—common to all engineering programs. Other programs usually have some sort of integrating project. Be sure project requirements exercise all of the abilitiesInstead of, or in addition to grades, ask project evaluators to evaluate attainment of the outcomes in the project—Likert scale25 September 200626How can we assume our responsibility? Program LevelPortfoliosCommon in some disciplinesNew venture in engineeringStart as first year studentsOrganize by content and abilitiesAsk evaluators to consider portfolios with respect to “a-k”—again, Likert scaleCan be done on the web25 September 200627How can we assume our responsibility? What do we do with the data?Faculty responsible for setting targets and for evaluating the dataSome results will be outstandingSome results will be OKThere will be room for improvement—there always isCaution—do not overdo it!25 September 200628Interesting Comparison with the Deming Cycle for Continuous ImprovementDOPLANCHECKACT25 September 200629Implementation MatrixProgram Educational Objectiveshave been established and maintainedConstituentsare involved in helping set program objectives and in evaluating the level to which they are being achievedThe required Processesare operationalOutcomes Assessmentis being practicedResultsof outcomes and the various processes are being used to improve programs and assure objectives are being achievedAn overall Systemis in place to meet the accreditation requirement25 September 200630Implementation MatrixHorizontal—Objectives and OutcomesVertical—Level of attainment, 1-5A good accreditation suggest 3 or higher25 September 200631Part of the MatrixLevelEducational Objectives1Not well defined3Comprehensive; defined, documented and measurable; clearly tied to mission and constituent needs5Comprehensive; defined documented, measurable and flexible; clearly tied to mission; readily adaptable to meet constituent needs; systematically reviewed and updated25 September 200632Part of the MatrixLevelOutcomes Assessment--Results1Anecdotal 3Good outcomes; positive trends in several major areas; some evidence that results caused by systematic approach 5World-class outcomes; sustained results; results clearly caused by systematic approach Educational ObjectivesConstituentsProcessesOutcomes AssessmentResultsSystem1Not well definedInformal contactFew, if any processes defined and documentedLimited to ad hoc effortsAnecdotalNone evident2Broadly defined and documented; clearly tied to mission; evidence of constituent inputSomewhat involved in defining objectives and desired outcomes, and assessmentSome major processes defined and documented; clearly tied to mission and program objectivesSome outcomes defined and improved in systematic manner; problems recognized and correctedSatisfactory outcomes; some evidence of positive trends in areas deployedEarly stages; partial deployment within the program and college 3Comprehensive; defined, documented and measurable; clearly tied to mission and constituent needsClearly involved in defining objectives and desired outcomes, and assessment; evidence of some sustained strategic partnershipsProcesses for all major elements of criteria defined, documented, and controlled; clearly tied to mission, program objectives, and constituent needsAll major outcomes defined; systematic evaluation and process improvement in place; problems anticipated and preventedGood outcomes; positive trends in several major areas; some evidence that results caused by systematic approachIn place; deployed throughout the program and college; driven by mission and objectives4Comprehensive; defined, documented and measurable; clearly tied to mission; responsive to constituent needs; systematically reviewed and updatedHigh degree of involvement in defining objectives and desired outcomes; evidence of many sustained strategic partnerships in all constituent groupsProcesses for all elements of criteria are quantitatively understood and controlled; clearly tied to mission, program objectives, and constituent needsAll outcomes defined; systematic evaluation and process improvement in place; many support areas involved; sources of problems understood and eliminatedExcellent outcomes; positive trends in most areas; evidence that results caused by systematic approachIntegrated; deployed throughout the program, college, and support areas; driven by mission and objectives5Comprehensive; defined documented, measurable and flexible; clearly tied to mission; readily adaptable to meet constituent needs; systematically reviewed and updatedHigh degree of involvement in defining objectives and desired outcomes, assessment; and improvement cycles; sustained evidence of strategic partnership with all key constituentsProcesses for all elements of criteria are quantitatively understood and controlled; clearly tied to mission, program objectives, and constituent needs; seen as benchmarks by other institutionsAll outcomes defined; systematic evaluation and process improvement in place; all support areas involved; common sources of problems understood and eliminatedWorld-class outcomes; sustained results; results clearly caused by systematic approachSound, highly integrated system; deployed throughout the program, college, and institution; driven by mission and objectives25 September 200634SummaryAssessment is a developing faculty responsibilityGood techniques exist—will get betterConsiderable startup requiredEventually, readily accomplished while the process improvesResults are worth the effort