092110caption.doc - NCRTM

Document Sample
092110caption.doc - NCRTM Powered By Docstoc
					Date: 09/21/2010
Time: 9:00am -10:30am (PST)
Title: Making a Difference through Program Evaluation
Speaker: Dr. Michael Quinn Patton

>> DR. PATTON HAS WORKED WITH MANY ORGANIZATIONS
AND PROGRAMS AT THE INTERNATIONAL STATE AND LOCAL
LEVELS WITH PEOPLE FROM DIFFERENT CULTURES AND
PERSPECTIVES.
HE HAS EXPRESSED AN INTEREST IN OUR DEVELOPING
COMMUNITY PRACTICE AND PROGRAM EVALUATION.
PLEASE WELCOME DR. MICHEAL QUINN PATTON.
[APPLAUSE]
>> THANK YOU VERY MUCH.
I AM VERY HONORED TO BE HERE AND EXPRESS MY THANKS
TO THE ORGANIZING COMMITTEE AND ALL THAT INVOLVED IN
MAKING A CONVERSATION LIKE THIS POSSIBLE.
I HAVE SERVED AS A PROGRAM CHAIR SO I KNOW HOW MUCH
WORK BEHIND THE SCENES GOES ON TO BRING ABOUT
SOMETHING LIKE THIS.
I APPRECIATE ALL THOSE INVOLVED AND HONORED TO GET A
CHANCE TO ADDRESS YOU.
I RESIGNATED TO THE FUN FACT THAT OREGON HAS LOSTS
OF GHOST TOWNS.
IT MADE ME THINK OF THE EXPERIENCE I HAD JUST LAST
WEEK AT A CONFERENCE I WAS KEYNOTING, A WOMAN RAN
UP TO ME, DR. PATTON, I DIDN'T REALIZE YOU WERE STILL
ALIVE?
[LAUGHTER]

I HAVE BEEN DOING THIS FOR A LONG TIME.
[LAUGHTER] AND SOME OF YOU WHO USED THE FIRST BOOK I
WROTE IN THE 70S FIGURE CAN THAT ANYBODY THAT WROTE
A BOOK THAT LONG AGO MUST BE DEAD.
I AM HANGING ON AND I ENJOY CHANCES TO COME OUT OF
THE GHOST TOWN AND MAKE AN APPEARANCE.
I SEE THERE ARE OTHER GRAY HEADS IN THE AUDIENCE.
FOR YOU FOLKS, THERE WILL BE A BIT OF A WALK DOWN
MEMORY LANE.
FOR THE YOUNGSTERS IT IS A BIT OF REMINDING YOU OF
THE HISTORY OF THIS WORK AND THE SHOULDERS ON
WHICH WE ALL STAND BOTH AS A LARGER SOCIETY AND
SPECIFICALLY IN TERMS OF EVALUATION.

I AM GOING TO BEGIN BY GOING BACK TO THE 60S.
I KNOW THOSE OF YOU THAT DIFFERENT LIVE THROUGH THE
60S GET TIRED OF HEARING ABOUT US FROM THOSE THAT
DID.
IT WAS AN IMPORTANT TIME.
IT WAS THE TIME OF THE GREAT SOCIETY VISION AND
LYNDON JOHNSON, LBJ AFTER THE ASSASSINATION OF JFK
WAS AN IMPORTANT PART OF THAT VISION.
WHERE ARE THE TEXANS IN THE AUDIENCE?
THEY CAN TELL YOU MORE ABOUT LBJ.
I AM GOING TO TELL YOU ONE STORY NOT ALTOGETHER KIND
SO YOU FOLKS CAN SET THE RECORD STRAIGHT.
LBJ WAS A VERY, VERY POLITICALLY SAVVY SENATE LEADER.
LBJ WAS FAIRLY AUTHORITARIAN IN MANOR.
HE MISSED THE WORKSHOPS ON MANAGEMENT.
PRETTY TOUGH GUY.
HE LIKED TO INTIMIDATE PEOPLE AND WENT THROUGH STAFF
VERY QUICKLY.
INCLUDING SPEECH WRITERS.
CHARLES OZ GOOD TELLS A STORY OF LBJ HAD GONE
THROUGH A SPEECH WRITER.
HE HADN'T TOLD LBJ HE WAS QUITTING.
HE HAD ONE POLICY SPEECH.
THIS WAS BEFORE TELL PROMPTERS, THEY WERE IN PRINT
ON A PAGE.
LBJ ARRIVED LATE AT THIS BIG CONFERENCE WHERE HE WAS
GOING TO DO A BIG POLICY SPEECH.
HANDED THE SPEECH AND AFTER HE DID HIS WARMUP, HE
BEGAN TAKING THE SPEECH AND SAYING UM, TO CREATE A
GREAT SOCIETY, WE MUST TAKE CARE OF THE LESS
ADVANTAGED AND THOSE IN POVERTY, TURN THE PAGE.
THEY ALWAYS PUT THAT AT THE BOTTOM SO THEY
WOULDN'T FORGET.
NO DETAIL LEFT UNINTENDED.
IN THE FACE OF THE GLOBAL COMMUNIST THREAT, WE MUST
HAVE A -- AND MUST STIMULATE TAXES, TURN THE PAGE,
HOW CAN WE FUND A WAR ON POVERTY, STRENGTHEN THE
DEFENSE AND CUT TAXES?
TURN THE PAGE.
LET ME TELL YOU RIGHT NOW, TURN THE PAGE, YOU ARE ON
YOUR OWN, YOU BASTARD.
[LAUGHTER]

ONE OF THE WONDERFUL THINGS ABOUT A CONFERENCE
LIKE THIS IS THAT YOU ARE NOT ON YOUR OWN.
YOU ARE PART OF A LARGER COMMUNITY, NOT ONLY WITHIN
THE DISABILITY COMMUNITY AND EVALUATION COMMUNITY,
BUT LARGER EVALUATION COMMUNITY.
THAT IS PART OF WHAT I AM GOING TO TALK ABOUT WITH
YOU.
I WANT TO PAY ONE MORE TRIBUTE.
THAT IS TO LBJ'S VICE PRESIDENT WHO HAILED FROM
MINNESOTA, HUBERT HORITIO HUMP FREE.
OUR MINNESOTAIANS CAN TELL YOU ABOUT THE GOOD
HAPPY WARRIOR.
IT WAS HUMPHREY'S COMMITMENT TO CIVIL RIGHTS AND
INDEED DISABILITY RIGHTS THAT WAS THE HALLMARK OF HIS
DISTINGUISHED CAREER IN THE SENATE AND AS VICE
PRESIDENT.
HUMPHREY SAID THE MORAL TEST OF GOVERNMENT IS HOW
THAT GOVERNMENT TREATS THOSE WHO ARE IN THE DARN
OF LIFE, THE CHILDREN.
THOSE IN THE TWILIGHT OF LIFE, THE ELDERLY AND THOSE
IN THE SHADOWS OF LIFE, THE SICK, NEEDY, AND
HANDICAPPED.
THIS WAS THE PERSPECTIVE THAT HUMPHREY WAS KNOWN
FOR, THAT HE CHAMPIONED, AND I HAVE ALWAYS FELT THIS
QUOTE AS MUCH AS ANY OTHER AND I HAIL FROM
MINNESOTA AND HAVE FOR MY ADULT LIFE, THAT THIS
QUOTE IS THE CHALLENGE TO US WHO DO EVALUATION.
THIS GIVES US THE OVERALL VISION OF HOW WE EVALUATE
OUR OVERALL SOCIETY AND GOVERNMENT.
HUMPHREY FELT THAT YOU EVALUATE BOTH A SOCIETY AND
A GOVERNMENT BY HOW IT CARES FOR THOSE WHO ARE
LESS ADVANTAGED.


AND OUR WORK IS A PART OF REALIZING THE EXTEND TO
WHICH THAT VISION BECOMES REALITY.

EVALUATION IN THE LAST DECADE HAS BECOME A GLOBAL
ENTERPRISE.
THE FIRST EVALUATION SOCIETIES WERE FORMED IN THE
1970S, THE OSCILLATION SOCIETY, CANADIAN SOCIETY AND
PREDECESSOR SOCIETY AND NETWORK WHICH MERGED IN
1984.
EUROPEAN SOCIETY CREATED IN THE 90S, AN UMBRELLA OF
SEVERAL NATIONAL EUROPEAN SOCIETIES.
AND IN THE LAST DECADE OVER 80 NATIONAL ASSOCIATONS
HAVE BEEN FORMED AROUND THE WORLD.
INDEED YOU COULD SPEND YOUR ENTIRE YEAR GOING FROM
EVALUATION CONFERENCE TO EVALUATION CONFERENCE
AROUND THE WORLD.
YOU ARE PART OF A GLOBAL COMMUNITY OF PEOPLE
AROUND THE WORLD WORKING ON MAKING EFFECTIVE
PROGRAMS, IMPROVING PROGRAMS, INFORMING DECISIONS
ABOUT PROGRAMS OF EVERY KIND IN ALL PARTS OF THE
WORLD.


ONE OF THOSE PLACES IS AFRICA.
UM, I WAS A PEACE CORP. VOLUNTEER IN AFRICA.
THIS MAP SHOWS THE LOCATION OF THE COUNTRY I WAS A
PEACE CORP. VOLUNTEER NOW CALLED BURKINA FASO IN
THE CENTER OF WEST AFRICA, SOUTH OF THE SA HAR RA
DESERT ONE OF THE POOREST COUNTRIES IN THE WORLD.
HOW MANY PEACE CORP. VOLUNTEERS DO WE HAVE?
A COUPLE.
THE PEACE CORP. WAS A VERY FORMATIVE EXPERIENCE AND
THE IDEAS I BROUGHT INTO EVALUATION CAME FROM THAT
EXPERIENCE.
I WORKED AMONG THE GORMA PEOPLE ON THE BORDER A
GROUP OF A HALF A MILLION SUBSIST TANS FARMERS.
IT WAS WHAT THE FRENCH COLONISTS.
THEY CALLED ME BOMPIANO WHICH TRANSLATS AS "WHITE
THING." I WAS IN THOSE VERY REMOTE VILLAGES DOING
BASIC AGRICULTURAL WORK, DIGGING WELLS, BUILDING
SCHOOLS.
SO WHEN THE AFRICAN EVALUATION SOCIETY FORMED IN
NAIROBI IN 1999, I WAS HONORED TO LAUNCH THE AFRICAN
SOCIETY.
I WAS HOPING THERE WOULD BE REP TIFS FROM BURKINA.
THE GORMA ARE NOT THE MAJORITY GROUP.
BUT THERE WAS A MAN FROM GORMA, THE LAND OF THE
GORMA.
I SOUGHT HIM OUT.
WE HAD LUNCH AT THE UNITED NATIONS COMPOUND THAT
FIRST DAY AND CAUGHT UP ON THE NEWS, THE ROAD
BETWEEN THE CAPITOL OF BURKINA.
NONE OF THE ORGANIZATIONS HAD BEEN THERE YET WHEN I
WAS THERE.
THE COUNTRY WAS JUST BEGINNING TO OPEN UP.
THERE HAD BEEN SEVERAL CHANGES IN THE GOVERNMENT.
AS WE HAD THE CONVERSATION, I SPECULATED ON HIM
WHETHER A CERTAIN, SHALL WE SAY POLITICAL
CORRECTNESS MIGHT HAVE EVEN ENTERED INTO THE DEEP
BUSH OF BURKINA FASO.
I SAID IF I CAME BACK TOGETHER WOULD I BE CALLED
BUMPIAN O.
THEY SAID WOMPIANO.
OLD WHITE THING.
[LAUGHTER]
WHICH IS HOW I COME TO YOU TODAY.
[LAUGHTER] TO TAKE A WALK DOWN MEMORY LANE ABOUT
WHAT HAS HAPPENED IN THIS FIELD OF EVALUATION THAT
YOU ARE INVOLVED IN.
SYSTEM OF YOU ARE FULL TIME EVALUATORS, SOME ARE
RESEARCHERS, SOME ARE PART TIME, SOME ARE SUBJECT
MATTER SPECIALISTS, A DIVERSE GROUP OF PEOPLE.
I WANT TO SHARE WITH YOU, MY PERSPECTIVE ABOUT HOW
THE FIELD HAS EMERGED AND CHALLENGES AND
OPPORTUNITIES THAT BRINGS FORTH.
AS EVALUATION HAS BECOME A TRANSDISCIPLINE.
A TRANSDISCIPLINE IS A DISCIPLINE THAT IS USED BY OTHER
DISCIPLINES IN ORDER TO DO THEIR WORK.

THERE ARE ONLY A FEW TRANSDISCIPLINES, STATISTICS IS A
TRANSDISCIPLINE, ALL FIELDS USE STATISTICS, PHILOSOPHY
IS A TRANSDISCIPLINE.
EVERY FIELD HAS TO DEAL WITH EVIDENCE, QUALITY OF
EVIDENCE, PEER REVIEW IS AN EVALUATION PROCESS.
EVERY DISCIPLINE HAS TO DECIDE WHAT CONSTITUTES
KNOWLEDGE.
YOU ARE PART OF NOT ONLY A PROFESSION, WE'LL TALK
ABOUT WHAT THAT MEANS, BUT A TRANSDISCIPLINE WITH ITS
OWN QUALITY ASSURANCE STANDARDS AND PROFESSIONAL
DEVELOPMENT OPPORTUNITIES OF WHICH THIS
CONFERENCE IS ONE.

ONE OF THE EXPERIENCES I HAD WHEN THE GREAT SOCIETY
PROGRAM REALLY BEGAN THROUGH THAT LEGISLATION
CREATING A DEMAND FOR EVALUATION, THE FIRST TIME
EVALUATION REQUIREMENTS WERE WRITTEN IN THE
FEDERAL LEGISLATION WAS IN THE WAR ON POVERTY
LEGISLATION IN THE LATE 60S AND EARLY 70S.
ALREADY AT THAT TIME, THERE WAS SOME PUSH BACK
ABOUT SPENDING FUNDS ON EVALUATION.
I REMEMBER CONVERSATIONS WITH NON FOR PROFIT
GROUPS AND GOVERNMENT GROUPS WHEN WE WERE
BEGINNING EVALUATIONS.
THEY WOULD SAY, SO WHO IS EVALUATING THE
EVALUATORS?
AT THAT POINT WE DIDN'T HAVE ANY STANDARDS.
A BLUE RIBBON COMMISSION WAS CREATED IN THE LATE
1970S REPRESENTING EVALUATION SCHOLARS, AMERICAN
PSYCH ASSOCIATION, POLITICAL SCIENCE ASSOCIATON THAT
CAME TOGETHER AND OVERA 2-YEAR PERIOD HAMMERED
OUT STANDARDS FOR THE PROFESSION PUBLISHED IN 1980
AND REVISED AGAIN.
IT IS A REQUIREMENT OF PROFESSIONS THAT HAVE
STANDARDS BY THE INTERNATIONAL STANDARDS
ASSOCIATON THAT STANDARDS BE REGULARLY REVIEWED
AND UPDATED AND READOPTED BY PEOPLE IN THE
PROFESSION.
THERE ARE 30 DETAILED STANDARDS, THEY ARE GROUPED
IN 4 CATEGORIES.
THESE ARE THE 4 CATEGORIES SHOWING ON THE SCREEN.
THEIR SET OF STANDARDS AROUND UTILITY TO ENSURE
REVALUATIONS ARE USEFUL.
SET OF STANDARDS AROUND FEASIBILITY, PROPRIETY AND
ETHICS AND ACCURACY.

EVERY NEW CLIENT I WORK WITH, I AM A FULL TIME
INDEPENDENT EVALUATION CONSULTANT, I HAND OUT THE
STANDARDS AND TELL MY CLIENTS THIS IS WHAT I AM
ACCOUNTABLE FOR.
I AM A PART OF A GLOBAL PROFESSION THAT HAS ADOPTED
THESE AS THE STANDARDS.
THE UTILITY STANDARDS CALL FOR EVALUATIONS TO BE
JUDGED ON WHETHER OR NOT THEY ACTUALLY GET USED.

AND WHAT THIS MEANS IS, THAT NO MATTER HOW METHOD
LODGELY SOPHISTICATED IT IS, IF IT IS NOT USED TO
IMPROVE PROGRAMS OR INFORMED DECISION MAKING, IT IS
A BAD EVALUATION.

WE ARE ACCOUNTABLE FOR USE.
PART OF THE STRUGGLES IS AS EVALUATION HAS GROWN
BECOME MANDATED, MORE AND MORE, IT BECOMES A
COMPLIANCE FUNCTION WHERE PEOPLE SIMPLY GO
THROUGH THE PAPERWORK MOTIONS.
IT IS THE STRINGS ATTACHED TO THE MONEY.
DRUDGERY, EVALUATION IS ONE OF THOSE WORDS LIKE
SUBPOENA AND BIOPSY THAT MOST PEOPLE HOPE DOESN'T
COME INTO THEIR LIFE.
[LAUGHTER].
AND WHEN IT DOES, THEY WANT TO GET RID OF IT AS
QUICKLY AS POSSIBLE.
OUR CHALLENGE AS PROFESSION LEADING WITH THE UTILITY
STANDARDS IS TO MOVE BEYOND THE COMPLIANCE
MENTALITY AND DEMONSTRATE WHAT WE DO MAKES A
DIFFERENCE TO PROGRAM IMPROVEMENT, TO PROGRAM
DECISION MAKING AND ULTIMATELY TO BETTER LIVES FOR
THE PEOPLE WHO ARE THE INTENDED BENEFICIARIES OF
PROGRAMS AND POLICIES.
FEASIBILITY STANDARDS CALL FOR THEM TO BE REALISTIC,
PRUDENT, DIPLOMATIC AND FRUGAL.
THE HIGHEST STANDARD IS IN THIS CATEGORY, THE COST-
FREE STANDARD.
THAT STANDARD CALLS FOR EVALUATIONS TO BE WORTH
WHAT THEY COST.IT IS A RECOGNITION THAT EVERY DOLLAR
IS A DOLLAR NOT SPENT FEEDING THE HUNGRY, NOT SPENT
HELPING THOSE THAT ARE SICK, NOT SPENT FIGHTING HIV
AIDS, NOT SPENT INCREASING ACCESS FOR THE DISABLED
OR GETTING JOBS FOR PEOPLE WHO NEED JOBS THAT ARE
DISABLED.
IT IS A JOB SPENT ON EVALUATION AND OUR COMMITMENT
AND ACCOUNTABILITY CALLS FOR US TO DEMONSTRATE THE
VALUE OF THAT MONEY TAKEN FROM DIRECT SERVICE.
PEOPLE INVOLVED IN DIRECT SERVICE FEEL THAT
COMPETITION.
THEY ARE ASKING WHO EVALUATES THE EVALUATORS?
HOW DO YOU JUSTIFY IN THE TIME OF REDUCED RESOURCES
SPENDING MONEY ON EVALUATION?
WE JUSTIFY IT BECAUSE OF FUTILITY.
NOT THROUGH COMPLIANCE, BUT BECAUSE WE HAVE TO
DEMONSTRATE THAT WE DO IS ACTUALLY USE.
.
WE GENERATE VALUE FOR MONEY BY THAT UTILITY.
PROPRIETY CALLS ON US TO BEHAVE ETHICALLY AS WELL AS
THOSE WITH WHOM WE WORK AND ACCURACY STANDARDS
HAVE TO DO WITH THE TECHNICAL QUALITY OF THE
EVALUATIONS THAT WE GENERATE DATA THAT ARE WORTH
USING.
WE HAVE MOVED INTO AN ERA OF METH THAT EVALUATION
WHERE WE EVALUATE EVASIONS FOR ONGOING LEARNING,
IMPROVEMENT AND QUALITY ASSURANCE.
ONE OF OUR PROFESSIONAL COMMITMENTS WHICH IS THE
BASIS OF MY OWN PRACTICE, BASIS OF MY WRITING, I SAW A
COUPLE OF PEOPLE CARRYING MY BOOK ALONG WITH THEM
AS PART OF THEIR WEIGHT LIFTING PROGRAM.
[LAUGHTER] IT IS HELPFUL.
I GOT AN E-MAIL FROM A COLLEAGUE WHO IS ON A TRAIN
ACROSS INDIA, HOLD UP A COPY OF THE BOOK, THANK YOU.
IT HAS GOTTEN THICKER.
THIS IS THE 4TH EDITION.

HE E-MAILED ME WHEN HE GOT TO HIS DESTINATION, HE HAD
JUST GOTTEN THE BOOK BEFORE THE TRIP AND PROVED SO
USEFUL, THEY RAN OUT OF PILLOWS AND THICK ENOUGH
FOR HIM TO USE AS A PILLOW ON HIS OVERNIGHT TRAIN
TRIP.
I TAKE MY UTILITY WHERE I CAN GET IT.

MUCH OF WHAT THAT BOOK IS BASED ON AND ITS REVISIONS,
I EVALUATE EVERY EVALUATION I DO TO FIND OUT HOW IT
WAS USED.
IF YOU ARE ENGAGED IN EVALUATION AT WHATEVER LEVEL,
THE WAY THAT YOU GET BETTER AT THIS AS
PROFESSIONALS DO IN ANY FIELD OF PRACTICE IS TO
EVALUATE HOW WELL YOU ARE DOING AT YOUR WORK.
THAT MEANS FOLLOWING UP YOUR EVALUATIONS TO FIND
OUT HOW THEY WERE USED AND THE EXTENT TO WHICH
THEY MEET THE STANDARDS AS COMMITMENT.
CHALINSKI, ELLEN IS THE GREAT PIONEER OF OUR FIELD,
SHE SET UP A SYSTEM WHERE THEY MONITOR EVERY
REPORT AND FOUND OUT WHAT HAPPENED AND PUBLISHED
THEIR OWN REPORT CARD ON WHAT HAPPENED TO THEIR
EVALUATIONS, INCREASING PRESSURE ON AGENCIES TO PAY
ATTENTION AND THEIR OWN ACCOUNTABILITY TO FIGURE
OUT WHAT WAS HAPPENING TO THEIR RECOMMENDATIONS.

WE PROVIDE OUR OWN QUALITY ASSURANCE BY ENSURING
VALUE OF EVALUATION.
VALUE IS IN THE MIDDLE OF THE WORD "EVALUATE." PART
OF OUR WORK INVOLVED IN EXPRESSION OF VALUES AND
THOSE STANDARDS ARE OUR EXPRESSION OF VALUES OF
WHAT WE STAND FOR AS A PROFESSION.

THE QUALITY ASSURANCE MOVEMENT AS YOU KNOW,
DISTINGUISHES BETWEEN QUALITY CONTROL AND QUALITY
ENHANCEMENT.
GETTING THE BASICS RIGHT, MEETING STANDARDS.
QUALITY HAS TO DO WITH EXCELLENCE AND EXCELLING.
HAVE WE MET THE MINIMUM STANDARDS OF FUTILITY,
FEASIBILITY AND ACCURACY.
ENGAGE IN WHAT CONSTITUTES EXCELLENCE.
IT IS THE CRITERIA THAT THOSE ARE A MATTER OF
JUDGMENT, ARE CASE SPECIFIC, ARE MATTER OF ENGAGING
IN DEALING WITH DIFFICULT AND CHALLENGING ISSUES AS A
COMMUNITY OF PRACTICE.
QUALITY ENHANCEMENT MOVES BEYOND SIMPLE GUIDANCE.
IT MOVES BEYOND FORMULAS, AND PRESCRIPTIONS ABOUT
DOING EXACTLY THIS.
IT TAKES US INTO THE TERRITORY OF PROFESSIONAL
PRACTICE WHERE YOU ARE FACED WITH DIFFICULT CHOICES
ABOUT GOING ONE WAY OR THE OTHER.
WE LEARN BY SHARING AT SESSIONS LIKE THIS ABOUT HOW
THOSE CHOICES GET MADE AND WHAT THE IMPLICATIONS OF
THOSE CHOICES ARE.

WE HAVE LEARNED A LOT ABOUT HOW TO INCREASE THE
UTILITY OF EVALUATION OVER THE YEARS.
ONE OF THE THINGS WE LEARNED EARLY ON WAS WHAT I
CALL IN THE FIRST EDITION AND HAVE EXPANDED ON OVER
THE YEARS AS THE PERSONAL FACTOR, WHICH IS KNOWING
WHO AN EVALUATION IS FOR AND WHAT THOSE PEOPLE ARE
GOING TO DO WITH IT.
I'LL BE DOING A SESSION THIS AFTERNOON ON UTILIZATION-
FOCUSSED EVALUATION.
BUT WHAT WE FOUND IS THAT AND THIS WAS A TOUGH
LESSON, I ADMIT.
I WAS TRAINED AS A SOCIOLOGIST.
THEY ARE TRAINED THIS IS NOT TRUE.
SOCIOLOGIST ARE TRAINED THAT PEOPLE DON'T MATTER,
ONLY POSITIONS MATTER.
THE ORGANIZATIONS ARE MADE UP WITH ROLES ATTACHED
TO THEM AND PEOPLE MOVE OUT ANONYMOUSLY, THEIR
VARY IN PERSPECTIVE THAT MANY HAVE HAD INFLICTED
UPON YOU AT SOME POINT IN GRADUATE SCHOOL.
IT IGNORES THE PEOPLE SIDE OF THE BUSINESS.

WHAT WE HAVE FOUND IS THAT PEOPLE MATTER FOR
UTILIZATION BECAUSE ORGANIZATIONS DON'T USE
INFORMATION, PEOPLE DO.

AND AN EVEN MORE STARTLING DISCOVERY FOR MUCH OF
THE WORLD, THIS IS BIG NEWS, EVALUATORS ARE PEOPLE,
TOO.

[LAUGHTER] THERE ARE A LOT OF FOLKS OUT THERE THAT
AREN'T SO SURE ABOUT THIS.
IT IS WORTH COMING TOGETHER AND REMINDING
OURSELVES EVERY SO OFTEN THAT WE ARE PEOPLE, TOO
AND THAT THE PERSONAL FACTOR EFFECTS US.
I AM GOING TO SUGGEST THAT THE IMPLICATION OF THIS AS
A BASIC PREMISE IS VALUE AND QUALITY ASSURANCE
COMES THROUGH EVALUATORS THROUGH OUR PERSONAL
COMMITMENTS AND OUR COLLECTIVE COMMITMENTS TO
MEETING THE STANDARDS.
AND SO I INVITE YOU, THOSE OF YOU WHO ARE NEW TO THE
PROFESSION AND THINKING ABOUT WHETHER OR NOT TO
TAKE ON EVALUATION AS IDENTITY AND THOSE OF YOU WHO
HAVE BEEN ON THE BORDERLINE FOR SOME TIME DECIDING
WHETHER OR NOT TO CALL YOURSELF AN EVALUATOR AND
HAVE TAKEN THE LEAP AND INCLUDE IT BRAVELY ON YOUR
RESUME, I INVITE YOU AS A PART OF THIS CONFERENCE TO
EXAM YOUR OWN COMMITMENT TO THE STANDARDS.

ARE YOU ENGAGING IN YOUR WORK IN A WAY THAT MAKES IT
USEFUL?

HOW DO YOU LIVE OUT THE STANDARDS?
THIS BECOMES PERSONAL.
ONE OF THE THINGS THAT A PROFESSION HAS BEEN
WORKING ON AND AN EXAMINATION OF THE SKILLS ARE THAT
PROFESSIONAL EVALUATORS NEED TO DO THEIR WORK.
THE CANADIAN EVALUATION SOCIETY HAS JUST LAUNCHED
THIS YEAR, A PROGRAM TO CERTIFY EVALUATORS.
AND THEY HAVE DONE IT BASED UPON THE WORK ON
ESSENTIAL SKILLS.
IT IS A PI LOT PROGRAM.
IT IS A VOLUNTARY PROGRAM.
WHAT THEY ARE HOPING IS THAT IN THE FUTURE, WHEN
RFP'S ARE PUT OUT, TENDERS ARE PUT OUT, EVALUATIONS
FUNDED, EXTRA POINTS WILL BE GIVEN TO PEOPLE WHO
HAVE CERTIFIED EVALUATORS MAKING THOSE PROPOSALS.
THEY ARE TRYING TO CREATE DEMAND FOR CERTIFICATION.
THE JAPANESE EVALUATION SOCIETY IN CERTIFYING.
AFRICAN EVALUATION IS CONSIDERING THIS.
SO THE ISSUE OF CREDENTIALING COMES UP IN ANY
PROFESSION.
RIGHT NOW, ANYBODY CAN HANG OUT THERE SHINGLE AND
CALL THEMSELVES AN EVALUATOR.
ONE OF THE REASONS, DESPITE THE FACT THAT I HAVE TO
EVERY SO OFTEN COME FROM THE GHOST TOWNS AND
MAKE A PHYSICAL APPEARANCE AROUND THE WORLD THAT I
STILL DO THAT IS BECAUSE UM, IT ALARMS ME THAT MOST OF
THE EVALUATION DONE IN THE WORLD IS DONE BY PEOPLE
WHO KNOW NOTHING ABOUT EVALUATION.
WHO DON'T KNOW THERE IS A PROFESSION, DON'T KNOW
THERE ARE STANDARDS, DON'T KNOW ANYTHING ABOUT
WHAT WE HAVE LEARNED IN 40 YEARS ABOUT HOW TO DO
GOOD EVALUATIONS.

AROUND THE WORLD, EVALUATIONS ARE DONE BY ALL KINDS
OF DISQUALIFIED AND UNQUALIFIED PEOPLE.
I URGE ALL OF YOU IN WHATEVER SETTING YOU WORK TO
CARRY THIS TORCH, THAT WE ARE A PROFESSION, THERE
ARE THINGS TO KNOW AND THINGS TO BE SKILLED IN TO DO
EVALUATIONS.
IT IS NOT ENOUGH TO BE A SUBJECT MATTER SPECIALIST.
MOST PEOPLE HIRED TO DO EVALUATION ARE SUBJECT
MATTER SPECIALIST.
IF YOU KNOW A LOT ABOUT THE SUBJECT YOU KNOW HOW
TO DO EVALUATION.
WHAT I AM SUGGESTING THESE ARE PARALLEL-KNOWLEDGE
BASIS, ONE OF THE REASONS WHY MUCH OF THE WORK WE
DO AS TEAMS.
I AM AN OLD EVALUATION GENERALIST.
I HAVE WORKED IN EVERY KIND OF PROGRAM IN THE WORLD
INCLUDING EVALUATING DISABILITY PROGRAMS.
I WORK WITH SUBJECT MATTER SPECIALISTS IN TEAMS.
I BRING TO THAT KNOWLEDGE ABOUT EVALUATION.
LET ME TAKE YOU THROUGH THE ESSENTIAL SKILLS, THERE
ARE PUBLICATION ABOUT THIS IN THE AMERICAN JOURNAL
OF EVALUATION AND WEBSITE OF AMERICAN EVALUATION
ASSOCIATION.
THE QUESTION IS HOW YOU ADD VALUE BY BEING GOOD AT
WHAT YOU DO AS PROFESSIONALS.
KNOWING YOUR ROLE AND PLAYING IT WELL.

THE ESSENTIAL SKILLS OPENING THIS UP.
SO I FRAMED THIS AS THAT ANCIENT PHILOSOPHICAL, TO
KNOW THY SELF.
TO USE THIS CONFERENCE AS A TIME, YOU HAVE TAKEN OUT
TIME TO REFLECT ON TOGETHER EVALUATION IN YOUR
PROFESSIONAL WORK TO PERSONALIZE THIS.

DO YOU HAVE THE ESSENTIAL SKILLS AT WHAT LEVEL?
AND WHICH OF THE ESSENTIAL SKILLS ARE YOU PREPARED
TO COMMIT YOURSELF TO IMPROVING AS YOU MOVE
FORWARD?
ESSENTIAL COMPETENCIES FOR PROGRAM EVALUATORS.
MY GOOD FRIEND JEAN KING, A RECIPIENT OF THE AMERICAN
EVALUATION ASSOCIATON'S PRESTIGIOUS ALVA, GUNNAR
MYRDAL PRACTICE AWARD, THE FINAL PRODUCT IS
ORGANIZED INTO 6 CATEGORIES.
IN THE INTEREST OF TIME I AM GOING TO GO THROUGH
THESE QUICKLY.
YOU CAN FIND OUT MORE IN THE PUBLICATIONS RELATED TO
THEM.
THIS IS SIMPLY TO RAISE YOUR AWARENESS ABOUT WHAT
THEY ARE AND THINK ABOUT HOW WE HAVE COME TO
CONCEPTIONALIZE THE STANDARDS.
REALLY UNSUGGESTING PARALLEL TRACKS OF DEFINING
WHAT AN EVALUATION IS.
WHAT ARE THE STANDARDS ABOUT WHAT CONSTITUTES A
GOOD EVALUATION, THE SECOND ARE THE SKILLS THAT
INDIVIDUAL EVALUATORS NEED TO IMMEDIATE THE
STANDARDS.
FIRST SKILL IS PROFESSIONAL PRACTICE KNOWLEDGE,
KNOWING AND PRESERVING THE NORMS AND VALUES
INCLUDING THE EVALUATION STANDARDS AND PRINCIPLES.

SECOND IS SYSTEMATIC INQUIRY EXPERTISE IN THE
TECHNICAL ASPECTS OF EVALUATION.
FOR MANY YEARS, THE SECOND SET OF SKILLS IS ALL THAT
PEOPLE THOUGHT EVALUATORS NEEDED.
WE HAVE ADDED 5 ADDITIONAL SKILLS IN ADDITION TO
METHOD LOGICAL.
THE 3RD IS SITUATIONAL ANALYSIS.
UNDERSTANDING AND ATTENDING TO THE CONTEXTUAL AND
POLITICAL ISSUES OF AN EVALUATION, INCLUDING
DETERMINING EVALABILITY, ADDRESSING CONFLICTS AND
ATTENDING TO ISSUES OF EVALUATION USE.
THIS IS PART OF INTENDED USE BY INTENDED USERS.
THAT SITUATION ANALYSIS.
THE 4TH IS PROJECT MANAGEMENT SKILLS.
EVALUATIONS HAVE TO BE MANAGED.
ONE OF THE THINGS I GET CALLED IN TO SORT OUT
SOMETIMES IS AN INDEPENDENT CONSULTERS,
EVALUATIONS THAT HAVE GONE BACK.
AS OFTEN AS NOT, WHERE THEY WENT BACK IS THE
EVALUATOR AND TEAMS HAD METHOD LOGICAL SKILLS,
MEMBERS OF AEA.
THEY WERE LOUSY PROJECT MANAGERS.
THEY DIDN'T KEEP UP WITH THE WORK.
COULDN'T MANAGE THE DATA, DIDN'T GET THINGS DONE,
DIDN'T GET MEETINGS FRAMED UP AND SETTING AGENDAS
AND FOLLOWING THROUGH.
THERE IS A LOT OF PROJECT MANAGEMENT THAT DOESN'T
GET TAUGHT IN GRADUATE SCHOOL ANYWHERE.

ANY OF YOU WHO HAVE BEEN IN GRADUATE SCHOOL AND
ARE AT UNIVERSITIES KNOW THE SHADOW SIDE, THAT
ANYBODY WITH A PHD CAN SERVE AS A DEPARTMENT CHAIR
OR DEAN.
THAT SHOWS UP, I AM TALKING ABOUT THE OTHER FOLKS
YOU RUN INTO, NOT ANYONE IN THIS ROOM.
THEY HAVE COME UP THROUGH THE SUBJECT MATTER
CHANGE, THERE ARE THINGS TO BE KNOWN ABOUT
MANAGEMENT.
REFLECTIVE PRACTICE, AND AWARENESS OF ONE'S
EVALUATION EXPERTISE AS WELL AS THE NEEDS FOR
PROFESSIONAL GROWTH.
PARTICIPATING IN SESSIONS LIKE THIS IS PART OF BUILDING
ESSENTIAL SKILLS, KEEPING UP TO DATE WITH THE FIELD.
KNOWING WHAT IS ON THE CUTTING EDGE.
IT IS CHANGING AND DEVELOPING.
AND FINALLY INTERPERSONAL COMPETENCE, THE PEOPLE
SKILLS NEED TO WORK WITH DIVERSE GROUPS OF STAKE
HOLDERS TO CONDUCT PROGRAM EVALUATIONS, INCLUDING
WRITTEN AND ORAL COMMUNICATION, NEGOTIATION AND
CROSS-CULTURAL SKILLS.
A RESEARCHER MAY NOT HAVE THOSE SKILLS AND MAY NOT
BE ABLE TO DEVELOP THOSE SKILLS.
NOT EVERYBODY IS A GOOD COMMUNICATOR.
YOU HAVE TO SEEK OUT COLLEAGUES WHO HAVE THOSE
SKILLS.
I RECENTLY HAD AN EXPERIENCE THAT BROUGHT THAT
HOME TO ME.
I HAD SURGERY DONE AT THE MAYO CLINIC.
I HAD ONE OF THE WORLD'S MOST ESTEEMED SURGEONS
WHICH GAVE ME CONSIDERABLE COMFORT.
BUT IT TURNS OUT THAT THIS MAN HAS ASPERGER'S
SYNDROME AND DOESN'T MAKE EYE CONTACT.
THEY HAVE ASSIGNED TO HIM A MEDICAL ASSISTANT WHO IS
AN EXTREME EXTROVERT AND EXTREMELY SOCIAL WHO IS
WONDERFUL.
SHE CALLS YOU AHEAD OF TIME.
EXPLAINS THE SITUATION OF WHAT IS GOING TO HAPPEN
WHEN I MEET WITH THE SURGEON.
SHE WILL INTERACT WITH ME AND TRANSLATE TO HIM AND
HE IS THE BEST PERSON IN THE WORLD TO BE DOING THIS.
HE DOES THIS ONE THING EXTREMELY WELL AND OVER AND
OVER.
HE IS THE PERSON YOU WANT DOING THIS IF YOU ARE GOING
TO HAVE THIS SURGERY.
HE IS NOT A WARM FUZZY.
SHE IS THERE TO DO THE WARM FUZZY AND THAT WORKED
WELL.
THAT IS AN ASSESSMENT OF THE SITUATION.

WHAT SET OF SKILLS ARE NEEDED?
MANY OF YOU WORK IN TEAMS WITH PEOPLE OF DIFFERENT
SKILLS AND CAPABILITIES.
WE NEED TO BRING THAT TO EVALUATION PRACTICE.

WE ADD VALUE BY DOING GOOD WHAT YOU DO AND WHAT
WE DO AND KNOWING YOUR ROLE AND PLAYING IT WELL.

ONE OF THE THINGS I THINK IS WORTH EMPHASIZING IN THIS
RECORD IS THE NATURE OF EXPERTISE.
I HAVE GOTTEN REALLY INTERESTED IN THE WHOLE NATURE
OF EXPERTISE.
I HAD THE OPPORTUNITY TO VISIT WITH SOME OF THE
PEOPLE TRYINGING TO MODEL EXPERTISE.
THESE ARE FOLKS DOING SOFTWARE, WRITING PROGRAMS,
TRYING TO CAPTURE EXPERTISE IN WAYS IT CAN BE
REPLICATED.
MOST FAMOUS IS THE CHESS PROGRAMS, BIG BLUE IBM
CHESS PROGRAMS.
THEY ARE TRYING TO CREATE SIMILAR PROGRAMS FOR
MEDICAL DIAGNOSIS, LEGAL MINDS, GREAT PSYCHO
THERAPISTS, GREAT ARCHITECTS.
WHAT THINKING PROCESS DO THEY GO THROUGH IN TRYING
TO FIGURE OUT THE RUBRICS?
ONE OF THE PLACES THAT IS GOING ON, THE MILITARY IS
OBVIOUSLY INTERESTED, THE AIR FORCE BASE IN DAYTON,
OHIO, THE ARTIFICIAL INTELLIGENCE COMPLEXES.
I VISITED WITH FOLKS AND TALKED WITH THEM ABOUT WHAT
THEY WERE LEARNING ABOUT WITH EXPERTISE.
THEIR DEFINITION OF EXPERTISE DOES NOT HAVE TO DO
WITH KNOWING ANSWERS TO THINGS.
EXPERTS ARE NOT CHARACTERIZED BY HAVING ANSWERS.
WHAT THEY FOUND IS, EXPERTISE IS CHARACTERIZED BY
ASTUTE SITUATION RECOGNITION.

EXPERTS KNOW HOW TO FIGURE OUT WHAT SITUATION THEY
ARE IN AND THE ANSWERS COME FROM THAT SITUATION
RECOGNITION.
THIS HAS IMPORTANT CONSEQUENCES FOR US WHEN WE
THINK ABOUT WORKING ACROSS SITUATIONS AND
EVALUATIONS.
YOU MAY HAVE SEEN FLOATING AROUND THE INTERNET THIS
NUMBER COMES UP, ESPECIALLY AROUND THE TIME OF THE
OLYMPICS, I LOVE THIS BECAUSE IT WAS A HUGELY
IMPORTANT EXPERIENCE FOR ME IN HIGH SCHOOL WHEN I
WAS A JR. IN HIGH SCHOOL.
OUR ENTIRE DAYTON, OHIO, I GREW UP THERE.
ALL THE HIGH SCHOOLS. IN DAYTON, OHIO WENT TO THE
BIGGEST ATHLETIC ASSEMBLY AT THE UNIVERSITY OF
DAYTON AND HEARD BOB RICHARDS GIVE ONE OF THOSE
WONDERFUL INSPIRING SPEECHES TO YOUNG PEOPLE.
HIS SPEECH WAS ABOUT HOW TO BECOME GOOD AT
SOMETHING.
IT WAS THE FIRST TIME I EVER HEARD THIS NUMBER.
THAT IF YOU WANT TO BE GOOD AT SOMETHING, PUT 10,000
HOURS INTO IT.
AS A WOMPIANO, I PASSED 10,000 HOURS A WHILE AGO.
I WANT THE YOUNGSTERS TO STICK WITH IT AND THERE ARE
THINGS TO LEARN AS YOU GO ALONG AND SELL WAIT THE
WISDOM AND KNOWLEDGE OF THOSE ELDERS IN THE GROUP
WHO HAVE THEIR 10,000 HOURS AND FROM WHOM YOU CAN
LEARN A GREAT DEAL.
THAT IS WHAT A COMMUNITY OF PRACTICE IS ABOUT.
WE TAKE ON A NUMBER OF DIFFERENT IDENTITIES AS
EVALUATORS.
METHODOLOGISTS, RESEARCHERS, AUDITORS, INSPECTORS,
LEARNING FICILITATORS, AND JUDGES.
THERE IS SOMETHING I WANT TO SPEND A MOMENT ON AS
PART OF REFLECTING ON OUR ROLES AS EVALUATORS,
THOSE THAT HAVE BEEN READING THE BOOK KNOW ABOUT
MY AFFECTION FOR THIS ROLE, ROLE OF THE JESTER.
THE JESTER'S JOB IN MEDIEVAL TIMES YOU SEE A PICTURE
AS A THINKER, I EVALUATE, THEREFORE I AM.
THE ROLE OF THE EVALUATOR IN MEDIEVAL TIMES,
SPEAKING TRUTH TO POWER IN A WAY IN WHICH IT COULD BE
HEARD.
THE STAKES WERE HIGH.
IF THE JESTER MISSPOKE, THE JESTER WAS KILLED.

THESE DAYS YOU JUST LOSE FUNDING OR YOUR JOB.
BUT SPEAKING TRUTH TO POWER, IS A LARGER RUBRIC FOR
RECOGNIZING THE POLITICAL CONTEXT WITHIN WHICH WE
ALL WORK, THE COMMUNICATIONS SKILLS INVOLVED, THE
CHALLENGES OF NOT JUST LETTING THE CHIPS FALL WHERE
THEY MAY BUT PART OF UTILITY IS KNOWING THE AUDIENCES
AND FIGURING OUT HOW TO MAKE THE RESULTS
COMMUNICATED IN A FORM THAT THEY CAN BE ACTED UPON.

AND SO I EMBRACE THE ROLE OF THE JESTER AND THE
CHALLENGES INVOLVED IN SPEAKING TRUTH WHEN NOBODY
ELSE WOULD TELL THE MONARCH WHAT WAS GOING ON, IT
WAS THE JESTER'S JOB TO GET THE MONARCH TO LISTEN,
TO DO THE REALITY TESTING.

JESTING IS A SERIOUS BUSINESS BECAUSE SPEAKING TRUTH
TO POWER IS SERIOUS BUSINESS.

THOSE IN POWER DON'T ALWAYS WANT TO HEAR WHAT WE
HAVE TO SAY.
THE EVALUATION OF THE GENESIS STORY MAKES THAT
POINT.
AND IN CASE A FEW OF YOU HAVEN'T BEEN INITIATED INTO
THE PROFESSION FULLY AND PROPERLY, KNOWING OUR
GENESIS STORY IS PART OF THAT INITIATION.
SO I TAKE IT AS PART OF MY RESPONSIBILITY WHEN I HAVE
THE OPPORTUNITY TO MEET WITH GROUPS LIKE THIS TO BE
SURE THAT YOU HAVE ALL BEEN APPROPRIATELY INITIATED,
EVERY CULTURE IN THE WORLD HAS AN INITIATION STORY.
IT IS ONE OF THE FEW UNIVERSALS IN THE WORLD.
AS A CULTURE, A COMMUNITY OF PRACTICE, WE HAVE A
STORY THAT TELLS HOW IT ALL BEGAN WHICH GOES LIKE
THIS.
IN THE BEGINNING, THAT IS HOW YOU KNOW IT IS A GENESIS
STORY.
WITH ME?
I KNOW IT IS EARLY IN THE MORNING.
IN THE BEGINNING, GOD CREATED HEAVEN AND EARTH.
AND GOD SAW EVERYTHING THAT WAS MADE.
BEHOLD GOD SAID IT IS VERY GOOD.
AND THE EVENING AND THE MORNING WAS THE 6TH DAY.
ON THE 7TH DAY GOD RESTED FROM ALL WORK.
GOD'S ARK ANGEL CAME IN SAYING, GOD HOW DO YOU KNOW
WHAT IS GOOD?
WHAT ARE YOUR CRITERIA?
ON WHAT DAY DID YOU BASE THAT JUDGMENT?
[LAUGHTER] JUST WHAT OUTCOMES WERE YOU HOPING TO
ACHIEVE?
AND AREN'T YOU A LITTLE CLOSE TO THE SITUATION TO
MAKE A FAIR AND UNBIASED EVALUATION?
[LAUGHTER]
GOD THOUGHT ABOUT THESE QUESTIONS ALL THAT DAY AND
GOD'S REST WAS GREATLY DISTURBED.
[LAUGHTER].
ON THE 8TH DAY GOD SAID LOUIS FER, GO TO HELL.
[LAUGHTER]
THAT'S HIS EVALUATION BORN IN A BLAZE OF GLORY, SO TO
SPEAK.
SO WHILE I HAVE BEEN TOUTING THE BEAUTY AND
IMPORTANCE OF THIS PROFESSION, I FEEL AN OBLIGATION
TO LET YOU KNOW YOU ARE A DESCENDANT FROM THE
SHADOW SIDE.
THAT IS THE WAY MOST OF THE FOLKS WE ENTER UP
ENCOUNTERING VIEW US.
SO IT IS NOT LIKE EVERYBODY IS OUT THERE SAYING YES,
HELP ME DETERMINE WHETHER OR NOT I AM DOING GOOD.
ONE OF THE COMMUNICATION CHALLENGES WE HAVE IS TO
GET PEOPLE TO EMBRACE THAT POSSIBILITY.
IN THE INTEREST OF TIME, I AM GOING TO SKIP THROUGH A
COUPLE OF SLIDES HERE.
WE MAYBE ABLE TO GET BACK TO THEM AT THE, I WAS GOING
TO TALK MORE ABOUT THE INTERNATIONAL WORK BUT LET
ME MOVE AHEAD, TOO.
THE VALUE ADDED CHALLENGE OF MATCHING THE
EVALUATION PROCESS AND DESIGN TO THE NATURE OF THE
SITUATION TO ACHIEVE INTENDED USE BY INTENDED USERS.
CONTINGENCY-BASED EVALUATION.
PART OF THE WORK YOU CAN ONLY GET GOOD AT IN
PRACTICE, WHEN YOU MOVE BEYOND THE SIMPLE RULES
AND FORMULAS.
EVALUATION OF ANY PARTICULAR EVALUATION IS A
FUNCTION OF ITS GENERIC GENERAL VALUE IN TERMS OF
STANDARDS AND PRECISE CONTEXT IN WHICH THAT
EVALUATION TAKES PLACE.
SO WE HAVE LOTS AND LOTS OF EVALUATION APPROACHES
AND MODELS.
THIS IS THE CHALLENGE OF THAT THAT COMES FROM THAT
GREAT EVALUATOR, DR. SEUSS WHO IN HIS BOOK "SNEECHS"
TELL IT IS STORY OF MRS. MCCAVE THE STORY AND SHE HAD
23 SONS AND NAMED THEM ALL DAVE.
SHE DID ANT THAT WASN'T THE SMART THING TO DO.
WHEN SHE WANTS ONE AND SAYS YOU WHO, COME INTO THE
HOUSE, DAVE, ALL 23 SONS OF HERS COME ON THE RUN.
THIS MAKES THINGS DIFFICULT AT THE MCCAVES IF YOU
ONLY HAPPEN TO WANT ONE DAVE.
THE HOUSE OF EVALUATION HAS MORE THAN 23 MODELS,
APPROACHES, TECHNIQUES, IN THE UTILIZATION CHAPTER
ON SITUATION ANALYSIS, I IDENTIFY OVER 100 DIFFERENT
MODELS OF EVALUATION, FORMATIVE, SUMMATIVE,
DEVELOPMENTAL, ACCOUNTABILITY DRIVEN, THEORY
DRIVEN, PA TIS PA TOIR, COLLABORATIVE, EMPOWERMENT.

LOTS AND LOTS OF APPROACHES.
PART OF OUR EXPERTISE IS SORTING THESE OUT BOTH IN
PROGRAM EVALUATION AND QUALITY ASSURANCE.
IT IS PART OF WHAT YOU SPEND THAT 10,000 HOURS ON IS
UNDERSTANDING WHAT WORKS IN WHAT KIND OF SITUATION
FOR WHO?
THIS COMES TO ACHIEVE SITUATION RESPONSIVENESS.
I AM GOING TO SKIP AGAIN A COUPLE OF SLIDES AS I CLOSE
OUT AND WANT TO DO QUESTIONS AND ANSWERS.
I WANT TO DO ONE FINAL SECTION ON EXECUTING.
DOING WHAT WE KNOW HOW TO DO AND KNOWING WHAT WE
SHOULD DO.

THAT IS ACTUALLY WALKING THE TALK.

I AM ARGUING WE HAVE LEARNED A LOT ABOUT HOW TO DO
EVALUATIONS THAT ARE USEFUL.
THE PROBLEM MOST OFTEN IS THAT WE DON'T DO THOSE
THINGS WE KNOW WE OUGHT TO DO.
THERE IS AS MANY OF YOU KNOW A WHOLE BIG FIELD IN
MEDICINE AROUND THIS PROBLEM.
SOME OF YOU ARE PROBABLY INVOLVED IN IT.
IT IS CALLED COMPLIANCE RESEARCH.
IT IS ABOUT WHY PEOPLE DON'T FOLLOW THE DOCTOR'S
ORDERS AND STAY ON THE MEDS AND DO THE PHYSICAL
THERAPY ALL THE WAY THROUGH.
ALL OF US ARE SUBJECT TO THAT.

I DOUBT THERE IS ANYBODY IN THIS ROOM, I CERTAINLY
KNOW IT IS TRUE OF ME DOING EVERYTHING WE KNOW WE
ARE SUPPOSED TO DO IN TERMS OF DIET, EXERCISE,
RELATIONSHIPS AND WORK, A LOT OF STUFF.
A LOT OF STUFF IN EVALUATION.
WE END UP HAVING TO PICK WHAT WE ARE GOING TO WORK
ON TO AVOID BEING BAD AT EVERYTHING.
THAT IS PART OF THE CHALLENGE OF REFLECTIVE PRACTICE.
I AM GOING TO DO A LITTLE LEFT FIELD THING HERE.
EARLY ON IN MY CAREER, I LOOKED AROUND AT THINGS
GOING ON THAT I MIGHT LEARN FROM BECAUSE I WAS
FRUSTRATED WITH THE CHALLENGES OF GETTING PEOPLE
TO PAY ATTENTION TO EVALUATION, USE EVALUATIONS.
AND AS A RESEARCHER, AS A PHD, FROM A UNIVERSITY.
I DIDN'T RESIGNATE THAT PART OF THIS IS BEING A USED
CALL PERSON.
BUT IT IS.
WE ARE TRYING TO GET SOMETHING TO BUY SOMETHING WE
THINK IS VALUABLE.
I BELIEVE PEOPLE MAKING DECISIONS ABOUT POLICIES AND
PROGRAMS AND PEOPLE RUNNING PROGRAMS NEED GOOD
INFORMATION.
THEY ARE NOT NATIONAL INFORMATION USERS, PEOPLE
OPERATE OUT OF BIAS AND PREJUDICE AND MANY WOULD
NOT BE BOTHERED WITH THE DATA AND EVIDENCE DESPITE
EVIDENCE-BASE PRACTICE BEING A MOVEMENT, TAS
CONTROVERSY MOVEMENT.
IT HASN'T BEEN EMBRACED EVERYWHERE AND CORRUPTED
IN MANY OF THE PLACES IT HAS BEEN EMBRACED.
SO I MADE PEACE WITH THAT ONE PART OF MYSELF
THINKING TO MYSELF AS A SALES PERSON AND BEGAN
LOOKING AT WHAT DO PEOPLE WHO GO TO SALE'S
CONFERENCES LEARN ABOUT HOW TO BE A BETTER SALES
PERSON.
THOSE ARE BIG CONFERENCES.

MOST OF THESE PEOPLE WORK ON COMMISSION.
THEY ARE NOT ON GOVERNMENT JOBS OR REGULAR
CONTRACTS WHERE THEY GET PAID REGARDLESS.
THESE ARE FOLKS WHO ONLY GET PAID IF THEY ARE
SUCCESSFUL.
WE HAVE HAD OCCASIONAL PROPOSALS FROM SOME OF
OUR IMMEANT PIONEERS LIKE MICHAEL SKRIBEN.
HE WORKS ON A COMMISSION BASIS DETERMINED HOW
MUCH THE EVALUATION WAS WORTH AFTER THE FACT AND
PAY ME A PERCENTAGE OF THE VALUE INCREASED IN
EFFECTIVENESS OF THE PROGRAM.
THAT IS COURAGEOUS ENOUGH TO DO THEIR WORK THAT
WAY.
PARTLY BECAUSE WE ULTIMATELY DON'T CONTROL USE.
WE ARE IN THE SITUATIONS TEACHERS ARE IN WHEN THEY
ARE ASKED TO BE ACCOUNTABLE FOR WHAT THEIR
CHILDREN LEARN.
AND THEY SAY WHERE ARE THE PARENTS. IN THIS
PICTURE?
AND THE CHILDREN'S BACKGROUND, YOU ARE GOING TO PUT
ALL THAT ON ME.
WHEN I TELL NEW EVALUATORS IS THAT WE SORT OF HAVE
TO BE A LITTLE SCHIZOPHRENIC ABOUT THIS.
WE HAVE TO ACT AS IF IT IS ENTIRELY OUR RESPONSIBILITY
WHILE AT THE SAME TIME TO MAINTAIN SANITY, IT IS AN
UPHILL BATTLE.
IT DOESN'T ALWAYS WORK.
WE HAVE TO ACT WITH GOOD INTEGRITY AROUND THOSE
THINGS WE DO CONTROL.
I GOT INTERESTED IN SALES, LOTS OF BOOKS.
THESE ARE GOOD TITLES.
I AM NOT GOING TO READ THEM TO YOU.
AROUND SALES, HUNDREDS OF THESE THINGS.
THERE ARE 3 BIG LESSONS, THAT I THINK HAVE BIG
IMPLICATIONS.
THE FIRST IS TO BELIEVE IN YOUR PRODUCT.
SECOND IS TO KNOW YOUR PRODUCT.
AND THIRD IS TO BE ABLE TO CONNECT YOUR PRODUCT TO
WHAT YOUR CUSTOMER NEEDS.
LISTEN, COMMUNICATE, AND CONNECT.

THOSE ARE THINGS THAT MAKE GREAT SALES PEOPLE.
I HAVE COME TO BELIEVE THERE ARE ALSO THINGS THAT
MAKE GREAT EVALUATORS.
TO BELIEVE IN EVALUATION, TO KNOW EVALUATION AND TO
BE ABLE TO CONNECT PEOPLE WITH EVALUATION WHICH IS
WE SAID IS NOT ALL THAT EASY TO DO.

ONE OF THE CHALLENGES I OFFER YOU IS THE ELEVATOR
SPEECH TEST.
WHAT SALES PEOPLE PRACTICE, CAN YOU SAY CLEARLY, IN
30 SECONDS WHAT YOUR PRODUCT IS, WHY YOU BELIEVE IN
IT, AND SAY IT IN A WAY THAT CONNECTS WITH THE PERSON
YOU ARE TALKING TO?
THAT IS THE LENGTH OF TIME IT TAKES AN ELEVATOR TO GO
UP 15 FLOORS UP A COUPLE OF STOPS.
MEET PEOPLE AT THE AIRPORT, THEY ASK YOU WHAT YOU
DO.
PART OF WHAT IS OF INTEREST, SOME OF YOU, PROBABLY
UM DON'T SAY THAT YOU ARE AN EVALUATOR.
THAT IS NOT YOUR PRIMARY IDENTITY.
MAY DESCRIBE YOURSELF AS A RESEARCHER OR ABILITY
SPECIALIST.
I CALL MYSELF AN EVALUATOR AND HAVE AN ELEVATOR
SPEECH FOR WHAT THAT IS BECAUSE I AM INVOLVED IN
TRYING TO PROMOTE THE PROFESSION AND THOSE OF YOU
WHO THINK OF YOURSELF AS EVALUATORS, IT IS NOT ALL
THAT EASY.
I WOULD OFFER YOU A RESOURCE ROGER MERANDA IS
DIRECTOR AT THE UNITED NATIONS.
HE STRUGGLED THIS INCLUDING WITH HIS OWN CHILDREN.
I WAS DELIGHTED THIS MORNING TO ARRIVE AND FIND OUT
WE HAVE A FATHER-DAUGHTER TEAM OVER HERE WORKING
TOGETHER.
I HAVE HAD A CHANCE TO DO EVALUATION WORK WITH MY
NOW ADULT DAUGHTER.
ROGER STRUGGLED WITH EXPLAINING EVALUATION TO OUR
CHILDREN.
WHEN HE RETIRED, HE TOOK HIS OWN MONEY AND WROTE A
BOOK CALLED "EVE OF THE EVALUATOR. ".
BOOK COST $12.
I GIVE THIS TO EVERY CLIENT ALONG WITH THE STANDARDS.
EVA IS A GIRL THAT GOES TO SCHOOL ON PROFESSIONAL
DAY AND HAS TO DESCRIBE WHAT THEIR PARENT DOES.
EVA COMES HOME IN TEARS BECAUSE SHE COULDN'T
EXPLAIN WHAT HER DAD, THE EVALUATOR DOES.
THEY HAVE A WONDERFUL DIALOGUE, HE SAYS EVALUATORS
HELP FIX THINGS.
SHE SAYS YOU ARE LIKE A CARPENTER?
NOT QUITE.
THEY HELP MAKE THINGS BETTER.
YOU ARE LIKE A THERAPIST?
NOT QUITE THAT.
EVALUATORS HELP PEOPLE IN NEED.
YOU ARE LIKE A SUPERHERO?
PRETTY MUCH.
[LAUGHTER].
THEN HE GETS HONEST ABOUT THAT.
IT IS A WONDERFUL BOOK.
EVA FINALLY FIGURES OUT WHAT HER DAD IS DOING.
SO THE ELEVATOR TEST IS A PART OF THIS.
LET ME JUST QUICKLY COMMENT ON A COUPLE OF THESE
THINGS AND MOVE TO THE Q & A SESSION.
BOB STAKE, GRAY HEAD, UNIVERSITY OF ILLINOIS, DEEPLY
COMMITTED TO RIGOROUS EVALUATION DID A TREMENDOUS
ARTICLE IN THE AMERICAN JOURNAL OF EVALUATION CALLED
BEYOND NEUTRALITY, WHAT EVALUATORS CARE ABOUT.
I OFFER THIS TO YOU AS A WAY TO THINK ABOUT WHAT WE
REPRESENT TO THE ESSENTIAL SKILLS.
BOB SAYS WE OFTEN CARE ABOUT THE THINGS BEING
EVALUATED.
THERE IS THIS MYTH THAT BEING NEUTRAL AS AN
EVALUATOR MAKES US ROBOTS, NON CARING.
MAKES US NOT PEOPLE.
BUT WE ARE PEOPLE, THE PERSONAL FACTOR, IT IS BEING
ABLE TO SAY, WHAT WE CARE ABOUT.
WE CAN CARE ABOUT DISABILITY AND DISABILITY RIGHTS.
WE ARE CELEBRATING THE 20TH ANNIVERSARY OF ADA.
NEXT YEAR IS 100TH ANNIVERSARY OF HUBERT HUMPHREY,
GREAT CHAMPION OF RIGHTS.
IT IS ALL RIGHT TO CARE.
PART OF OUR RESPONSIBILITY IS TO IDENTIFY CARING.
WHEN I GO OUT AND EVALUATE PROGRAMS, FOR EXAMPLE, I
HAVE BEEN EVALUATING HIV AIDS PROGRAMS.
WHEN I SHOW UP, I DON'T PRETEND TO BE NEUTRAL ABOUT
HIV AIDS.
MY YOUNGER BROTHER DIED ABOUT AIDS.
I SAY DON'T BE THINKING YOU ARE GOING TO GET A PASS
FROM ME.
I CARE ABOUT WHETHER OR NOT YOUR PROGRAM IS
EFFECTIVE.
THAT IS WHY I AM HERE.
THIS ISN'T JUST A TECHNICAL JOB FOR ME.
THIS ISN'T JUST A COMPLIANCE ACTIVITY.
WE CARE AND CAN CARE ABOUT THE THINGS BEING
EVALUATING.
WE AS PROFESSIONALS CARE ABOUT EVALUATION.
I AM MAKING THE CASE THIS IS A NOBLE CALLING, A
TRANSDISCIPLINE WITH HIGH STANDARDS.
WE CAN CARE ABOUT IT.
WE ADVOCATE RATIONALTY.
NOTHING TO BE EMBARRASSED ABOUT THAT.
THE WORLD IS NOT NATIONAL.
FIVE OF THE LAST AWARDS, THE FACT THAT HUMAN BEINGS
ARE IRRATIONAL ABOUT MONEY, DAH?
TELLS YOU SOMETHING ABOUT ECONOMICS, THEY HAVE TO
GIVE NOBLE PRIZES.
WE CARE TO BE HEARD.
WE ARE DISSTRESSED BY UNDERPRIVILEGED.
WE SEE GAPS IN UNDERPRIVILEGED PARTICIPANTS AND
ADVOCATES OF A DEMOCRATIC SOCIETY.
THIS IS BOB'S LIST OF THINGS WE CAN CARE ABOUT AND
STILL DO A GOOD JOB.
IT IS WHAT OUR PRODUCT IS.
YOU NEED TO FIND YOUR OWN LIST OF THINGS YOU CARE
ABOUT TO BE ABLE TO BELIEVE IN THIS PRODUCT.
I WON'T LINGER MORE ON THAT.
IT IS THE KNOWING THE KIND OF SITUATIONS THAT
EVALUATIONS AND APPROPRIATE FOR.
FINALLY CONNECTING YOUR PRODUCT TO WHAT YOUR
CUSTOMER NEEDS WHOEVER YOUR CUSTOMER IS.
THINKING ABOUT THE PEOPLE THAT USE CAN THE
EVALUATIONS AS CUSTOMERS IS A MINDSET.
I AM AN INDEPENDENT CONSULTANT.
A CLIENT OR CUSTOMER PUTS PART OF THE RESPONSIBILITY
ON US TO DELIVER SOMETHING THAT IS A SERVICE TO THEM.
SHORTLY AFTER I CHECKED IN TO THE HOTEL LAST NIGHT, I
GOT A PHONE CALL FROM THE CONCIERGE OR SOMEBODY
TELLING ME THEIR NAME, GIVING ME THEIR NUMBER AND
SAYING IF THERE WAS ANYTHING I NEEDED, HERE WAS A
NUMBER TO CALL ANY TIME THROUGH THE NIGHT, ANY TIME I
NEEDED SOMETHING, BE IN TOUCH WITH THEM.
CUSTOMER SERVICE.

LET ME STOP WITH THAT.
UM, AND I'LL CLOSE JUST BEFORE THE END OF THE BREAK
WITH ANOTHER CREATION STORY FOR YOU THAT THIS
REPRESENTS.

BUT LET ME STOP AT THIS POINT AND TAKE YOUR
COMMENTS.
SEE WHAT THIS MAY HAVE STIMULATED IN YOU, YOUR
QUESTIONS AND GET CAN A CHANCE TO INTERACT A LITTLE
BIT.
>> THIS IS JOHN MCCLUR.
>> I THINK MY QUESTION, SHORTAGES OF EVALUATORS IN
OUR STATES, NEW COUNSELORS, STAFF, REGIONAL
LEADERS THAT DON'T HAVE THE SPECIALIZATION NOR THE
SKILLS TO PUT IT ALTOGETHER, THAT ANALYSIS PIECE FOR
HOW TO BEST HELP THE CUSTOMER, KNOWING OUR
CUSTOMER.
I AM CURIOUS WHERE THE EVALUATION FIELD IS ON THIS, ON
RECRUITMENT AND RETENTION AND THOSE TYPES OF
THINGS FOR EVALUATORS AND I APPRECIATE YOUR
PRESENTATION VERY MUCH, THANK YOU.
>> THANK YOU FOR THE QUESTION.
IT IS A GREAT QUESTION.
THE FIELD IS GROWING.
IT HAS EXPANDED TREMENDOUSLY WHEN I WAS PRESIDENT
OF THE AEA IN 1988 WE HAD UNDER 1,000 MEMBERS.
THERE ARE NOW OVER 6,000 MEMBERS OF AEA.
THERE ARE THAT MANY PEOPLE ON THE AMERICAN
EVALUATION CONFERENCE IN SAN ANTONIO SECOND WEEK
OF NOVEMBER WE'LL ATTRACT ABOUT 5,000 PEOPLE.
VERY STRONG REGIONAL AND STATE ASSOCIATONS NOW
OUT IN YOUR AREA.
THERE IS SOMETHING CALLED NORTHEAST EVALUATION
ASSOCIATON HOLDS AN ANNUAL CONFERENCE.
THEY SCHEDULE THEIR CONFERENCE IN CONNECTION WITH
THIS CONFERENCE WHEN THEY FOUND OUT I WAS GOING TO
BE HERE.
THEY HAD AN INFLUENCE ON THE CONFERENCE THAT MEETS
ANNUALLY.
WASHINGTON HAS A CONFERENCE AS WELL.
TOPICAL INTEREST GROUPS, AEA HAS ORGANIZED INTO 40
SOME INTEREST GROUPS OF PEOPLE THAT SHARE
PERSPECTIVES.
PEOPLE IN EDUCATION, PEOPLE IN PUBLIC HEALTH,
QUANTITATIVE, QUALITATIVE, THERE ARE NEW SYSTEMS,
TOPICAL INTEREST GROUP.
THOSE OF YOU INVOLVED IN ADVOCACY, A WHOLE NEW
AREA.
ALL OF THEM ARE INVOLVED IN PROFESSIONAL
DEVELOPMENT CREATING ONLINE COURSES CLEARLY ONE
OF THE NEW DIRECTIONS.
NEW ONLINE MATERIALS THAT ARE AVAILABLE.
THE EVALUATORS INSTITUTE OFFERS 2 WEEKS OF COURSES
3 TIME AS YEAR IN JANUARY, IN APRIL, AND IN JULY.
IF YOU GOOGLE THE EVALUATOR'S INSTITUTE, YOU'LL TURN
UP THEIR PROGRAM.
THEY DO ABOUT 30 TOPICS.
MANY ORGANIZATIONS ARE IN THE SITUATION THAT YOU
DESCRIBED.
SOMEBODY HAS BEEN HIRED IN WITH THE SUBJECT MATTER
GIVEN AN AASSIGNMENT, GET THE MONEY VALUE TALK.
GET THEM TO FOLLOW AEA AND STATE EVALUATION
ASSOCIATONS.
THERE ARE OPPORTUNITIES.
YOU HAVE TO LOOK AROUND FOR THEM.
GO TO THE AEA WEBSITE WHICH IS EVAL.ORG THERE IS A
MENU FOR PROFESSIONAL DEVELOPMENT.
THEY HAVE UP TO DATE INFORMATION ABOUT WHERE THERE
ARE THINGS GOING ON AROUND THE COUNTRY AND WORLD.

>> BILL DOWNNY, UNIVERSITY OF ARIZONA.
I AM INTERESTED IN TERMS OF EVALUATION MOVEMENT IN
HOW YOU ARE APPROACHING PUBLIC POLICY.
WE ARE INVOLVED IN PROGRAMS THAT ARE GOVERNED BY
STATE AND FEDERAL LEGISLATION.
ARE THERE EFFORTS TO BE SURE THAT APPROPRIATE
EVALUATION IS BEING ADDRESSED IN THOSE TYPES OF
LEGISLATIVE ACTS?
>> GREAT QUESTION, BILL.
POLICY EVALUATION IS IN FACT, ONE OF THE SUBAREAS
WHERE A GREAT DEAL OF WORK IS GOING ON.
AS A TRANSDISCIPLINE EVALUATION APPLIES TO ANYTHING.
BIG FIELD OF PRODUCT EVALUATION, PERSONNEL
EVALUATION, PROGRAM EVALUATION, WHICH IS WHAT WE
ARE FOCUSSING ON.
ONE OF THE BIG AREAS IS POLICY EVALUATION.
THE METHODS TEND TO BE DIFFERENT BECAUSE POLICIES
AREN'T AS DISCRETE.
A LOT OF THAT FOCUSES UPON IMPLEMENTATION ISSUES.
HOW DO YOU GET TO THE PROGRAM LEVEL WHERE YOU CAN
SEE WHAT IS GOING ON?
I HAVE JUST WORKED ON A CASE THAT WILL BE COMING OUT
IN A VOLUME OF NEW DIRECTIONS FOR EVALUATION,
MEMBERSHIP AND AEA COMES WITH THE AMERICAN
JOURNAL OF EVALUATION AND JOURNAL CALLED NEW
DIRECTIONS FOR EVALUATION WHICH IS A THEMATIC
JOURNAL AND EVALUATING STRATEGY AS A UNIT OF
EVALUATION.
ONE OF THE CASES IS A MAJOR EVALUATION OF THE
FEDERAL GOVERNMENTS AROUND WELFARE RIGHTS THAT
THE KEL LOG FOUNDATION FUNDED.
IT IS A MODEL FOR THE CHANGES OF POLICY EVALUATION.
ONE OF THE THINGS THEY DID THAT IS INTERESTING THEY
CREATED, FUNDED, AND SUPPORTED GROUPS TO TAKE ON
THE POLICY EVALUATION.
TO DESIGN IT, INTERACT WITH POLICY MAKERS AND INCLUDE
THEM ON THE TASK FORCES AND MAKE IT A COMMUNITY
ACTIVITY, GATHER DATA ABOUT HOW DID POLICIES GET
TRANSLATED, WHAT GOT DONE, WHAT DIDN'T?
WHAT WERE THE COSTS, HOW DID THEY GET MISUSED AND
MISAPPLIED AND MADE THAT PART OF THE DISABILITY.
>> MY NAME IS MICHAEL SHOE MAKER FROM UTAH.
>> YOU ARE THE REASON I AM HERE.
GOOD TO MEET YOU, MICHAEL.
>> GOOD TO MEET YOU.
>> MY QUESTION IS ABOUT CARING NAN ONGOING
PROFESSION, CARING TAKES ENERGY FROM PEOPLE.
MY QUESTION WAS RELATED TO THOSE WHO MAY FIND
THEMSELVES ONCE IN A WHILE IN A BOUT OF NON CARING,
WHAT ADVISE WOULD YOU GIVE?
>> TREMENDOUS QUESTION, MICHAEL.
THANK YOU, AND I DO THINK THAT ONE OF THE CHALLENGES
FOR MANY EVALUATORS IS ISOLATION.

MANY EVALUATORS ARE THE ONLY PERSON WITH
EVALUATION RESPONSIBILITY IN THEIR OFFICE.
THEY GET BUSY WITH A LOT OF NUTS AND BOLTS AND
DEMANDS AND MANAGING DATA AND WRITING REPORTS AND
PUT IN LONG HOURS.
OFTEN DON'T GET MUCH FEEDBACK.
I HEAR FROM YOUNG EVALUATORS ALL THE TIME WHO GIVE
THEIR HEART AND SOUL AND DISAPPEARS INTO A BLACK
HOLE AND DON'T GET FEEDBACK OR HEAR ANYTHING.
IT IS DISCOURAGING, WEEKS AND MONTHS GO BY AND THEY
EXPERIENCE A SENSE OF POST PAR DOM.
IT IS DEPRESSING TO PUT IT OUT THERE AND CANNOT GET
ANYTHING BACK.
A COUPLE OF THINGS, ONE IS THAT WE NEED COMMUNITIES
TO PRACTICE.
THAT IS WHY I ENJOY AND WANT TO BE A PART OF MEETINGS
LIKE THIS TO AFFIRM THE IMPORTANCE OF CREATING
NETWORKS OF PEOPLE THAT SUPPORT EACH OTHER AND
UNDERSTAND AND CARE AND PROVIDE THAT CARING.
THE SECOND THING IS THERE IS WAY TOO MUCH MODESTY
ABOUT PUTTING YOURSELF OUT THERE AND WAITING FOR
FEEDBACK.
THOSE PEOPLE YOU ARE GIVING THAT TO AREN'T NATURAL
FEEDBACK GIVERS.
SOME ARE BAD FEEDBACK GIVERS.
I HAVE BECOME ASSERTIVE AS A PART OF UTILIZATION OF
CONTACTING PEOPLE AND FINDING OUT SOME OF ITS
HUMOR.
SEND THEM AN 80-PAGE REPORT.
HAVE YOU MADE IT TO PAGE 5, YET.
A WEEK LATER.
GETTING CLOSE TO PAGE 10 YET.
REALLY NEAT GRAPH ON PAGE 33, TAKE A LOOK.
I USED TO SEND PEOPLE POSTCARDS, NOW IT IS E-MAIL.
EVENTUALLY, THEY EITHER HAVE TO RESPOND SAYING STOP
BOTHERING ME.
IT IS PART OF THE UTILIZATION PROCESS.
ONE OF THE THINGS IS THAT THE WORK DOESN'T STOP
WHEN YOU WRITE A REPORT.
EXPANDING THE UTILIZATION PROCESS IS A KEY PART I'LL BE
TALKING ABOUT THIS AFTERNOON.
WE NEED THAT INTERACTION.
IT DOESN'T COME NATURALLY.
WE ARE SUITORS.
WE HAVE TO REMIND PEOPLE IT WASN'T JUST COMPLYING
WITH A DEADLINE.
I'LL SEND THEM SNIP IPS.
I WANT TO BE SURE TO HIGH LIGHT THAT.
IT IS BURIED IN THE REPORT TO GENERATE USE.
ON THE WHOLE, I TELL PEOPLE AHEAD OF TIME I AM GOING
TO DO THAT.
I TELL FOLKS THAT THEY HAVE NEVER SEEN NAGGING LIKE I
AM GOING TO DO.
[LAUGHTER].
THEY THINK THEY HAVE EXPERIENCED NAGS IN THEIR LIVES,
NOT EVEN CLOSE.
I AM THE ULTIMATE NAG.
WHY DO PEOPLE NAG?
BECAUSE THEY CARE.
THAT IS WHY I AM GOING TO NAG YOU.
I THINK THERE IS IMPORTANT STUFF HERE.
I KNOW YOU HAVE GOT A LOT ON YOUR PLATE.
I AM GOING TO NAG YOU UNTIL YOU BREAK OFF THE
RELATIONSHIP OR LEARN HOW TO DO IT AND TO HELP YOU
KNOW WHEN TO DO THE BOUNDARIES.

>> HI WE ARE FUNDED EVALUATORS, THERE SEEMS TO BE A
GROWING INTERNAL EVALUATOR ROLE.
HAVE YOU SEEN ANY MODELS OF EXTERNAL AND INTERNAL
EVALUATORS PARTNERING AND HOW WE MAY DO THAT
BETTER.
THERE ARE QUITE A FEW PEOPLE IN THE ROOM THAT WOULD
DESCRIBE THEMSELVES AS EXTERNAL EVALUATORS AND
QUITE A FEW THAT ARE INTERNAL EVALUATORS.
I AM CURIOUS HOW WE CAN WORK TOGETHER AND HAVE
THAT COLLECTIVE TEAMWORK AND EXTERNAL EVALUATORS
HAVE A LOT MORE ABILITY TO BE RELEVANT TO THE
AGENCIES.
>> GREAT QUESTION.
AS YOU NO DOUBT KNOW, THE TECHNICAL TERMS ARE IN-
HOUSE AND OUTHOUSE EVALUATORS.
[LAUGHTER].

CERTAINLY ONE OF THE IMPORTANT DIRECTIONS IN
EVALUATION IS INTERNAL/EXTERNAL COMBINATIONS.
SOME OF WHICH ARE NOT UNLIKE ACCREDITATION MODELS,
GENERATE DATA AND INTERNAL BRING EXTRA CREDIBILITY
BY SUGGESTING IT IS WORTHWHILE.
I GOT INVOLVED IN SOME OF THAT WORK IN A SCHOOL
DISTRICT THAT WAS GOING TO INTERNAL/EXTERNAL MODEL
OF EVALUATION.
THEY HAD EXTERNAL TEAM THAT WOULD COME IN AND DO A
SITE VISIT AND ADD CREDIBILITY TO THE DATA.
THEY WERE DOING A COMPREHENSIVE EVALUATION THAT
LOOKED AT STUDENT ACHIEVEMENT, STATUS OF LIBRARY,
SCHOOL LUNCH PROGRAMS, PROFESSIONAL DEVELOPMENT,
PARENT INVOLVEMENT.
THERE WAS A CONFERENCE TO WORK OUT THE ROLES OF
EXTERNAL AND INTERNAL EVALUATION.
BY THE END OF THE MORNING, ONE OF THE FOLKS
ANNOUNCED AT LUNCH THEY CHANGED THE NAME OF THIS
PROCESS.
NO LONGER THE INTERNAL/EXTERNAL, IT WAS THE ETERNAL
EVALUATION.
THEY COULD SO NO END IN SIGHT.
[LAUGHTER].
THERE ARE ADVANTAGES TO EACH PROSPECTIVE.
INTERNAL ARE CLOSER TO THE ACTION.
TYPICALLY LESS EXPENSIVE FOR INTERNAL EVALUATORS TO
DO THE WORK.
EXTERNAL EVALUATORS HAVE MORE CREDIBILITY AND
INDEPENDENCE BUT DON'T KNOW WHERE THE LAND MINDS
ARE.
IT IS A GREAT OPPORTUNITY FOR TEAMING.
IT BEGINS BY ANTICIPATING THE NEED AND BUILDING TIME
AND MONEY.
THEY NEED TO FIGURE OUT HOW TO MAKE IT WORK AND
BRING TO THE WORK AND VALUE EACH SET OF
PERSPECTIVES.
IT IS IMPORTANT TO CONCEPTIONALIZE THAT OPPORTUNITY
AND TO FIGURE OUT WITHIN ANY GIVEN CONTEXT HOW TO
MAKE IT WORK.
I THINK IT IS TREMENDOUSLY IMPORTANT THAT IT IS A PART
IN TERMS OF MICHAEL'S QUESTION, REDUCES THE
ISOLATION FOR INTERNAL FOLKS, AND HELPS EXTERNAL
PEOPLE MAKE SURE THEY ARE RELEVANT.
IT HELPS INTERNAL PEOPLE HANDLE DIFFICULT POLITICS
THEY MAY ENCOUNTER.
AND HELPS EXTERNAL PEOPLE APPRECIATE SOME OF THE
CONTEXTUAL ORGANIZATIONAL CHALLENGES THEY NEED TO
ADDRESS.
I VERY MUCH URGE YOU TO WORK ALONG THOSE LINES.
>> HI DR. PATTON.
MY NAME IS MICHAEL.
YOU TALK ABOUT THE HISTORY OF THE PROFESSION.
I AM WONDERING ABOUT TRANSPARENCY AS A MOVEMENT.
IS THAT SOMETHING THAT YOU THINK IS PUTTING MORE
WEIGHT BEHIND THE PROFESSION?
IS IT A MOVE TOWARDS MORE TRANSPARENCY HELPING THE
PROFESSION?
>> GREAT QUESTION, MICHAEL QUINN, DELIGHTED TO MEET
YOU.
TRANSPARENCIES ARE REALLY INTERESTING ISSUES
ESPECIALLY WITH CONNECTION TO THE PROPRIETY
STANDARDS AND ETHICS.
WHERE TRANSPARENCY IS HUGE AND HAS BEEN FOR SOME
TIME IS IN EUROPE.
THE EUROPEAN COMMUNITY HAS MADE TRANSPARENCY THE
NO. 1 ISSUE.
THE EUROPEAN EVALUATION SOCIETY, IN FACT IS JUST
ABOUT TO HAVE THEIR MEETING IN PROG IN A WEEK.
YOU CAN GO HERE TO PROG.
BOOK YOUR TICKETS AT NOON.
YOU WILL FIND THE THEME OF THEIR MEETINGS, THEY MEET
EVERY 2 YEARS HAVE TRANSPARENCY IN THE TITLE.
OBAMA HAS MADE IT PART OF HIS POLITICAL CAMPAIGN
PLATFORM AS YOU ALL KNOW.
THESE ARE ISSUES THAT KEEP COMING UP AND
TRANSPARENCY IS ONE OF THOSE THINGS AND WE SEE IT
WITH OBAMA, THAT ALWAYS LOOKS BETTER AND MORE
DOABLE WHEN YOU ARE ON THE OUTSIDE LOOKING IN.
WHEN YOU GET ON THE INSIDE AND START REALIZE THE
THINGS YOU'LL EXPOSE THROUGH TRANSPARENCY IT
DOESN'T LOOK SO GOOD ANYMORE.
I JUST EXPERIENCED AN EXAMPLE.
I JUST COMPLETED AN EVALUATION THAT YOU WOULD
RECOGNIZE THAT INCLUDES VERY DETAILED ASSESSMENTS
OF THEIR GRANTEES AND IDENTIFY MAJOR PROBLEMS IN
SOME OF THEIR GRANTEES.
THEY WANTED TO PUT THAT EVALUATION ONLINE.
IS IT FAIR TO THOSE NON PROFITS WORKING TO CORRECT
TOES DEFICITS TO HAVE THEIR WEAKNESSES EXPOSED IN
THE LOCAL NEWSPAPER?
WE KNOW THAT IS WHAT IS GOING TO HAPPEN TO GO TO A
MAJOR FOUNDATION'S WEBSITE AND PICK UP THE NAME OF A
LOCAL GRANT THIS HASN'T PERFORMED VERY WELL.
THAT HAS HAD INTERNAL PROBLEMS.
IN FACT, I HAVE TAKEN THE POSITION THAT FORMATIVE
EVALUATIONS SHOULD NOT BE TRANSPARENT.
THAT THE ONLY WAY TO DO GOOD FORMATIVE EVALUATION
IS NOT TO WRITE REPORTS ON THEM.
IF YOU WRITE REPORTS, THEY GET OUT.
PEOPLE THAT DON'T KNOW THE FORMATIVE DISTINCTION
WILL TREAT ANYTHING WRITTEN AS SUMMATIVE.
I HAVE EXAMPLES OF GREAT DAMAGE TO PROGRAMS THAT
WERE OPEN AND HONEST WITH THE AIM OF IMPROVING
THINGS AND THAT BECAME A PUBLIC DOCUMENT AND DONE
DAMAGE TO THE PROGRAM.
SUMMATIVE EVALUATIONS ARE DIFFERENT THINGS.
THERE ARE TECHNIQUES FOR PROTECTING SOURCES.
IT IS NOT THAT HARD WITH A COUPLE DEMOGRAPHIC DATA
TO GO BACK THROUGH THEIR GRANT LIST AND FIGURE OUT
WITHIN A CLUSTER, WHAT PARTICULAR GRANT WAS THAT
TALKED ABOUT.
REPORTERS ARE GOOD AT DOING CAN THAT.
CONNECTING THE DOTS.
I HAVE BEEN ON BOTH SIDES OF THIS, MY GOOD FRIEND
MICHAEL QUINN, ADVOCATING FOR IT AND COMING AGAINST
THE DANGER.
WE HAVE TO DO OUR OWN HIPPOCRATIC OATH, FIRST DO NO
HARM.
IN A SERVICE LEVEL IT IS EASY TO SAY EVERYTHING OUT TO
BE OUT THERE.
A LOT OF OUR FEEDBACK, I WORKED WITH A
DEVELOPMENTAL DISABILITY PROGRAM WHERE THEY WERE
INTRODUCING CONSUMER FEEDBACK OF STAFF.
THAT GOT INTO BODERLINE PERSONNEL EVALUATION STAFF.
THEN YOU GET INTO COMPARISONS AMONG AGENCIES
WHICH IS WHERE THIS IS GOING BETWEEN PROVIDERS AND
NON PROFITS.
IT IS A WONDERFUL PRINCIPLE, AN IMPORTANT ONE.
IT IS ON THE CUTTING EDGE OF ETHICAL AND POLITICAL
FIELD OF HOW TO DEAL WITH IT, BUT RESPECT PRIVACY, TO
DEAL WITH IT AND WHAT CAN BE MADE TRANSPARENT FIRST
DOING NO HARM.
EUROPEANS ARE AHEAD OF US ON THAT.
THEY HAVE A WHOLE DIFFERENT PRIVACY KIND OF
ENVIRONMENT THERE.
IT IS ONE OF THE THINGS AS A PROFESSION WE NEED TO
DEAL WITH.
WE WELCOME YOUR INSIGHTS HOW TO MANAGE THAT.
I FIND THAT A TOUGH ISSUE.
>> WE HAVE ABOUT 4 MINUTES.
ONE QUESTION FROM THE WEBCAST.
>> THIS QUESTION COMES FROM SUSAN SHERMAN, HOW DO
WE FIND THE NEED TO BUILD PROGRAMS ESPECIALLY THAT
WE DON'T HAVE PROGRAM EVALUATORS ON STAFF.
THE PERCEPTION IS SHE FINDS PROGRAM EVALUATION IS AN
AFTER THOUGHT AT THIS POINT.
>> IT IS ONE OF THE LESSONS WE HAVE LEARNED.
I APPRECIATE THE QUESTION THAT WE WANT TO AS MUCH AS
POSSIBLE BUILD IN EVALUATION FROM THE BEGINNING AND
THAT IS CERTAINLY THE IDEAL.
THE WHOLE NOTION OF EVALABILITY ASSESSMENT WAS
CREATED TO GET PEOPLE TO STOP AT THE BEGINNING OF A
PROGRAM INTERVENTION AND THINK ABOUT ARE WE READY
FOR EVALUATION?
DO WE HAVE THINGS IN PLACE AND FOR WHAT KINDS OF
EVALUATION.
IT IS ONE OF THE FORMS OF EXPERTISE THAT EVALUATORS
NEED TO BRING IS HELPING PEOPLE FOR GETTING READY
FOR EVALUATION AND GETTING BASELINES AT THE
BEGINNING.
A PART OF WHAT I URGE IS WHERE EVALUATORS ARE NOT
YET HIRED, NOT YET IDENTIFIED.
THE CONTRACT IS NOT YET PUT OUT IS TO NOT THINK THAT
YOU HAVE TO HAVE AN EVALUATOR TO ENGAGE IN
EVALUATIVE THINKING.
EVALUATIVE THINKING IS BROADER THAN EVALUATORS,
ANYBODY INVOLVED, ONE OF THE THINGS I HAVE BEEN
TRAINING ON IS TRYING TO BREAK DOWN CAN THE DIVISION
BETWEEN EVALUATORS AND PROGRAM STAFF AND
MANAGEMENT AND LEADERSHIP.

AND TO MAKE EVALUATION A LEADERSHIP FUNCTION.

EVALUATIVE THINKING, PART OF THE ORGANIZATIONAL AND
PROGRAM CULTURE.
SO IF ONE FACES ANY OF A NUMBER OF ADMINISTRATIVE
HURDLES THAT DOESN'T ALLOW YOU TO HAVE AN
EVALUATOR PRESENT, DON'T LET THAT STOP YOU FROM
DOING THE EVALUATION WORK.
PEOPLE CAN ASK YOURSELVES, WHAT IS OUR BASELINE?
WHAT IS OUR NEED'S ASSESSMENT, WHAT IS OUR LOGIC
MODEL?
GET STARTED ON THAT WORK WITHOUT WAITING FOR AN
EVALUATOR.
YOU EVENTUALLY GET WHO HAS THOSE INTERPERSONAL
SKILLS AND THEY ARRIVE AND FOUND YOU HAVE DONE A
REALLY BAD JOB, THEY WON'T TELL YOU THAT.
THEY'LL SAY WELL, I AM VERY PLEASED YOU GOT STARTED.
LET'S BUILD ON THAT EFFORT.
IF THEY COME IN AND SAY YOU HAVE REALLY MESSED THIS
UP.
THEN YOU KNOW THEY DON'T HAVE ONE OF THE ESSENTIAL
SKILLS THAT THE PROFESSIONAL IS LOOKING FOR AND YOU
MAY WANT TO LOOK RIGHT AWAY.
THANK YOU FOR YOUR QUESTIONS AND I LOOK FORWARD TO
GETTING TO KNOW YOU BETTER IN THE BREAK OUT
SESSIONS.
LET ME LEAVE YOU WITH A FINAL IMAGE AS WE CLOSE OFF.

I LIKE METAPHORS, I LIKE STORIES, AND UM, I AM GOING TO
LEAVE YOU WITH ONE OF MY FAVORITE REALITY TESTING
STORIES AND METAPHORS.
BECAUSE I BASICALLY THINK EVALUATION IS ABOUT HELPING
PEOPLE TEST REALITY.
HELP THEM SEE IF WHAT THEY THINK IS GOING ON IS REALLY
GOING ON, BRING DATA TO THAT.
AND AS WE SAID THERE ARE SOMETIMES RESISTANCE.
ONE OF MY FAVORITE EVALUATION HEROES IS DON KIOTI.
SOMETIMES CALLED THE FIRST NOVEL IN HISTORY.
IT IS THE STORY OF AN ELDER GENTLEMAN WHO IS RETIRED,
HAVE PROVENT SHL, STARTS TO READ.
HE IS DEEPLY DEPRESSED BY THE HUMAN DEGRADATION,
THE SUFFERING, THE WARS, THE CRUELTY OF SLAVERY.
AS HE READS, THE AUTHOR SAYS FROM MORNING TONIGHT,
NIGHT TO MORNING, DON KIOTI GIVES UP WHAT SERVANTI
SAYS THE MELANCHOLY OF BURDEN AND INSANITY AND
BECOMING A NIGHT RIDING THROUGH THE WORLD MAKING
THE WORLD A BETTER PLACE.
THE TENSION IN THE STORY IS DON KIOTI'S REALITY VERSES
OTHER PEOPLE'S REALITY.
THEY HAVE DIFFERENT DATA SETS.
DON SAYS THEY ARE DOING BATTLE WITH A GREAT ARMY.
OTHERS THINK IT IS A HERD OF SHEEP.
ESSENTIALLY HIS DAUGHTER AND SON-IN-LAW IS
EMBARRASSED WITH HIS RIDING THROUGH THE WORLD
TRICK HIM BACK INTO THEIR VERSION OF REALITY.
AS IF TO COMFORT HIM, THEY SAY AT LEAST YOU'LL DIE IN
TOUCH WITH REALITY.
DON KIOTI RESPONDS WITH A MONOLOGUE FROM
BROADWAY AND SAYS: IN TOUCH WITH REALITY, SEEING LIFE
AS IT IS, I HAVE LIVED FOR MANY YEARS AND SEEN LIFE AS IT
IS, PAIN, MYSTERY, CRUELTY BEYOND BELIEF.
I HAVE HEARD THE VOICES OF GOD'S NOBLEST CREATURES.
I HAVE BEEN A SOLDIER.
I HAVE SEEN MY COMRADES DIE IN BATTLE AND HELD THEM
AT THE LAST MOMENT.
NO GLORY, ONLY THE LOOK OF DISPAIR IN THEIR EYES,
QUESTIONING WHY?
I DO NOT BELIEVE THEY WERE ASKING WHY THEY WERE
DYING.
BUT WHY THEY HAD EVER LIVED.
WHO KNOWS WHERE MADNESS LIES.
TO SURRENDER ONE'S DREAMS, THIS MAYBE MADNESS.
TO SEE TREASURE WHERE THERE IS ONLY TRASH.
BUT MADDEST OF ALL IS TO SEE LIFE ONLY AS IT IS AND NOT
ALSO AS IT CAN BE AND SHOULD BE.

I THINK THAT THOSE OF US IN EVALUATION AND QUALITY
ASSURANCE WORK ARE IN A NOBLE PROFESSION WITH A
NOBLE CALLING AND THAT IS THE ELEVATOR SPEECH OF
OUR CALLING.
WE HAVE PEOPLE TO SEE WHAT IS GOING ON, TO DO REALITY
TESTING.
NOT AS AN END IN ITSELF BUT BECAUSE THAT IS THE PATH
WAY TO LIFE AS IT SHOULD BE AND COULD BE.

AND WE NEED TO HELP PEOPLE MAKE THAT CONNECTION
BETWEEN KNOWING WHAT IS GOING ON AS A PATHWAY TO
GETTING TO LIFE AS IT SHOULD BE.
I AM DEEPLY HONORED TO BE A PART OF CELEBRATING OUR
PROFESSION.
THANK YOU VERY MUCH.
[APPLAUSE]

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:0
posted:4/10/2013
language:Unknown
pages:39