7 20IS 20Audit 20Process by HC111111101345


									7. IS Audit Process Applied to
     Data Life Cycle Audit,
  e-Learning & Higher Education

      Judy Yard & Jeff Brandon

Extra Credit Project

             7. IS Audit Process   1
Conducting your own Data Life Cycle Audit

    Using an Information Technology
            Advisory Group

                7. IS Audit Process         2
            ITAG Overview
• Proven methodologies:
  – Business Technology Alignment
  – Strategic Planning
  – Business Intelligence
  – Technology Selection & Implementation
  – Technology Initiative Management
  – Business Continuation / Disaster Recovery

                   7. IS Audit Process          3
  What is the Data Life Cycle?
• The Data Life Cycle are the phases in
  which data moves through the
  organization. The different phases include
  how the organization collects, stores,
  processes and disseminates their key
• Key data defines the most critical or
  important data elements that are pertinent
  to supporting an internal organization‟s
                  7. processes.
  specific business IS Audit Process       4
  Why should you conduct a Data
       Life Cycle IS Audit?
• Eliminate duplication of work effort and
  increase overall productivity.
• Consolidate stranded information assets.
• Generate business intelligence in order to
  better facilitate customer relationship
• Support cross-departmental business
• Enhance one-on-one (personalized)
  marketing.         7. IS Audit Process       5
  Why should you conduct a Data
        Life Cycle Audit?
• Improve data maintenance and integrity.
• Helps identify data flow dependencies for
  business continuation/disaster recovery
• Identify a complete enterprise application
  integration strategy.
• Creates a vehicle to communicate
  technology needs to senior management
                  7. IS Audit Process          6
   What does the AMS have to do
              with it?
• AMS is core business (association) application
  which supports all four phases of the Data Life
   –   Streamlines data collection
   –   Primary data repository
   –   Supports transaction processing
   –   Base for reporting/business intelligence
• Identifies data integration
   – Third party applications / data sources
   – Strategic Partners
   – Web                7. IS Audit Process         7
 Enterprise Application Integration
• AMR Research (2003)
  – 45% of companies require the same customer
    to place orders through separate systems
  – 60% have limited or no integration between
    their front- and back-office systems
  – 90% struggle with customer responsiveness,
    visibility, automation and data quality
  – 20% of customer data is centrally managed

                  7. IS Audit Process        8
   How to Conduct a Data Life
          Cycle Audit
• Identify primary business functions and
  associated processes and tasks for each
• Identify departments that are associated
  with each business function
• Document the Data Life Cycle for each
  business function (and process)
  – Data Collection
  – Data Repository
  – Business Process
  – Data Dissemination                       9
                  7. IS Audit Process
    Primary Business Functions
• Membership                      • Legislative Tracking
• Committees / Boards             • Regulatory Tracking
• Meetings                        • Finance
                                  • Customer Service
• Products / Publication
                                  • Communications
                                  • Research
• Sponsorship
                                  • Foundation
• Education
• Certification
• Marketing          7. IS Audit Process                   10
   Business Function Analysis
• What are the primary responsibilities
  (business functions) for each department?
• What other departments or business
  functions are directly related with this
  business function?
• What are the routine tasks to perform the
  primary responsibilities?
• What are the standard operating
• How current is your business continuity /
                   7. IS Audit Process    11

  disaster recovery plan?
        Business Functions
• The same business function can be
  performed by multiple departments.
• Processes and tasks related to a business
  function can cross over multiple
• A business function can have multiple
  processes and tasks associated with it.

                 7. IS Audit Process      12
    Data Collection Methods
Membership Application         • Web Forms
Subscriptions                  • Email
Order Entry                    • Conversations
Meetings Registration
                                • Third-Party
Exhibit Registration
External Societies or            Partners
Chapters                        • Abstracts (Call
                                  for Speakers)

                7. IS Audit Process                 13
     Data Collection Analysis
• What key data is required to support each
  business function?
• What key data is collected during each
  data collection method identified for each
  business function?
• What is the frequency of collected key
  data based upon each data collection
• What key data is considered static, non-     14
                    7. IS Audit Process
  static or calculated?
      Key Data Mapping

                                                                    Plan Profile Report

                                                                                                                         Direct Contact
                                                                                                         Product Sales



       Organizational Key Data

Organization Name                       X              X            X                     X              X               X
Parent / Subsidiary Link                               X
Organization Address                    X              X            X
Organization Phone                      X              X            X
Organization Fax                        X              X            X
Organization URL                        X              X
Organization Type                       X              X
Organization Classification             X                           X
Products and Services                   X                           X
Number of Employees                     X              X            X
Corporate Structure                     X
Congressional District
Organization Description/Mission        X              X

                           7. IS Audit Process                                                                                            15
        Data Storage Analysis
• Where does the key data reside that supports each
  business function ( Word, Excel, etc.)?
• What format is the data?
• How does that data get accessed (standalone,
  networked, remote application)?
• Based upon the interdependencies of key data between
  business functions, does each business function have
  access to the most current and accurate data source?
• What are the key indexes for each data repository?
• How much data is being stored?

                      7. IS Audit Process                16
           Data Repository Map

  Online          Online
                  Product                          PAC

Certification   Information
                Assets                CRM/AMS     Chapters

  Press/         Legislative/
  Media          Regulatory                        FMS

                            7. IS Audit Process              17
    Challenges with Stranded
      Information Assets
• Technical Challenges
  – Multiple data sources, data types, locations
  – Data accessibility
  – Redundant data sources
• Organizational Challenges
  – Managing information
  – Maintaining data integrity
  – Creating accurate and timely reports
  – Publishing to Members, Committees or Board
                    7. IS Audit Process            18
            Process Analysis
• What process is being supported based upon
  data collected by a different business function?
• Are there manual processes which can be
  automated easily?
• What processes are supported by web-based
• What external strategic partnerships do you
  have that support internal business functions –
  how does those partnerships impact the
  process, including data collection, storage and
  dissemination?     7. IS Audit Process             19
Business Process Map
        Products                    Education

Management              Customers

                                                Legislative /

                   7. IS Audit Process                          20
    Data Dissemination Methods
• Digital Content for Sale       Web Forms
• List/Reports/Queries           Personalized Emails
• Inter/Intra/Extra Net          Publications
• Business Intelligence          File Transfer
• Forecasting
• New Product/Service
• Enhance Customer
• Knowledge Creation
                     7. IS Audit Process                21
   Data Dissemination Analysis
• What are the different tools used for
  disseminating data (report writers, data mining
  tools, etc.)?
• What format does key data get disseminated?
  –   Reports
  –   Publications
  –   Web Content
  –   PDA
  –   External Strategic Partnerships
• What other applications is data being
  disseminated to? 7. IS Audit Process              22
              Data Life Cycle Map
  Data Collection          Repositories             Processes   Dissemination
      PDA                     Mail                    Trigger    Content
                              Store                              Update

Trip Reports
CEU Verification
Issue Lobbyist

    Web                      AMS/                    Update      Analytics
    Forms                    Subset

 Meeting Registration
 Product Purchase
 Member Profile

                              Temp                    Update      Integrity
                               DB/                                Reporting
 Membership Application
                                      7. IS Audit Process                       23
                  Data Life Cycle

Data Collection   Repository            Process   Dissemination


                        7. IS Audit Process                   24
 Introduction to e-Learing Audit
 Complexity of student experience
 Problems facing evaluators
 Different research methodologies
 Audits promote evaluation
 Holistic view – why? how? when? and
  what? of evaluation

                 7. IS Audit Process    25
 Develop transferable student abilities
 Pressure within the worldwide web
 Governmental and employer pressure
 Requirement – learners to participate in a
  networked information society
 Emphasis on student-centred learning

                  7. IS Audit Process          26
 Experienced evaluators needed in
  production, usability and effectiveness of
  online delivery
 The Internet and „blended learning‟ with
  frontal learning elements now widespread

                  7. IS Audit Process          27
     Two Sides of e-Learning
 Promoted by         Can develop in ways
  government           which impoverish
  initiatives          education
 Cost effectiveness  Time & production
  of e-Learning        costs of development
 Permits tailored    Students & tutors
  education            lacking skills to use

                 7. IS Audit Process      28
     The Impact of Quality
 Assurance Assessment (QAA)
 Parallel to increased use of ICT (Interactive
  Computer Technology) is rising interest in
  quality assurance strategies
 Lighter QAA touch & emphasis on stakeholders
  – students, employers, funders, society
 Emphasis on outcomes rather than processes
 QAA looking for an institution‟s strategy

                   7. IS Audit Process            29
     Why Evaluate the Learner‟s
 Subject review & institutional audits
 Act as: educational „connoisseurs‟ with
  approaches and tools to use in order to act
  as experts informing policy
 Require: awareness and understanding of
  what is experienced
 Require: personal judgement
 Require: ability to articulate subtleties to a
  wider audience
                   7. IS Audit Process         30
   Questions to Ask yourself
 Is the student experience exceptional?
 Are there defects in the system or
 Fit for purpose?
 Value for money or adds costs?
 Transformative/Value adding?
 Students‟ perceptions?
 Effect on student numbers?
                7. IS Audit Process        31
 Metrics for e-Learning Evaluation

1. Efficiency of the teaching and
   learning process
2. Readability & usability of materials
3. Communication
4. Social relationships
5. Costs

                7. IS Audit Process       32
       Quality of the e-Learning
1. Efficiency of the teaching and learning

Accurate student data
Drop out rates
Student assessment results
Student satisfaction ratings
Training of tutors
                   7. IS Audit Process       33
      Quality of the e-Learning
1.Efficiency of the teaching and learning
To best understand the potential for
 Start with 1-2 courses
 Is this course needed by students?
 Is the assessment electronic?
 Collection of computer log data
                  7. IS Audit Process       34
     Materials and Communication
2. Readability & usability           3. Communication
  of materials                        Communication with
 Peer review of                       tutors & other
  materials                            students
 Reuse as learning                   Skills in using
 Online exercises
                                      Access from
 Study skills
 e-Library and e-journal              party
                                      Group work
                     7. IS Audit Process               35
   Social Relationships and Costs
4. Social                         5. Costs
  Relationships                    Costs per student
 Hours working online              of recruitment
 Student motivation               Training of tutors
 Student support &                Costs of
  interaction with other            electronic access
  students                         Reuse of learning
                                    materials &
                                    learning objects
                  7. IS Audit Process                36
 Approaches to Human-Computer
1. Learning theory is dominant – how
   students and their interaction with
   computers fit together
2. How students learn when using
3. Using the technology to teach in ways
   that are novel and unique to the
                 7. IS Audit Process       37
 How are we going to Evaluate the
    e-Learning Experience?
 Evaluation goals need to be clearly
 Is this a formative or summative
 How should the data be gathered?
 Should quantitative or qualitative
  methods be used?
                7. IS Audit Process     38
 How are we going to Evaluate the
    e-Learning Experience?
Methods include:
 Surveys
 Observations
 Semi-structured interviews
 Focus groups
 Cognitive walk-throughs
 Qualitative studies can provide examples of
  usability difficulties
                  7. IS Audit Process       39
  How are we going to Evaluate the
     e-Learning Experience?
Surveying students‟ views:
 detects feelings of overload
 informs necessary course adjustments
Survey methods:
 paper based with
 telephone supplementary interviews
 increased use of electronic surveys
 focus groups

                        7. IS Audit Process   40
 When to Evaluate the e-Learning
 Dependent on whether the evaluation is
  formative or summative?
 When is the student deemed to have
  completed the course?
 Will formative evaluation during the course
  engage students more?
 Harvesting evaluation information from
  student interaction with the course
                  7. IS Audit Process       41
What does the Evaluation provide?
 Audio/video record of the computer
  screen, the user‟s comments and
Baseline data to:
 make adjustments, possibly „instant‟
 for external audit
 shape e-learning strategy
 input into associated resources – library
  and information literacy provision          42
                  7. IS Audit Process
           Data Protection
 Students and tutors need to be kept
  informed about how the data they provide
  is being used
 Students and tutors retain the right to
  remove what they have provided at any
 Evaluators have to ensure secure access
  to the data
 Evaluators must get consent especially if
  using the data in a different context
                 7. IS Audit Process          43
 The University is interested in the quality
  of the student experience
 To improve that experience needs
  understanding of student and staff needs
 Here we have aimed to provide a
  framework in which to conduct evaluations
 Current work on modelling courses will
  provide data on points of concern that
  arise in blended and e-learning           44
                  7. IS Audit Process
       Preparing for an
         A-133 Audit

   Office for Sponsored Research
Risk Management and Audit Services

             7. IS Audit Process     45
       Background and
     Requirements of A-133

“Reporting Package” submitted annually includes:
      -   Financial statements
      -   Schedule of expenditures of federal awards
      -   Auditor‟s reports on internal controls
      -   Auditor‟s reports on compliance
      -   Schedule of findings and questioned costs
      -   Summary schedule of prior audit findings
      -   Corrective action plan
      -   Data collection form

                      7. IS Audit Process              46
      A-133 Audit Process

A-133 audit process involves the following:

 External groups: PricewaterhouseCoopers
  (PWC) and federal government
 Internal groups: schools, OSR, RMAS, Office
  of General Counsel, VP Finance

                   7. IS Audit Process          47
          A-133 Schedule

Month                          Schools
 July   A-133 PWC orientation meeting

Aug.    Provide support documentation for direct cost
        testing, department questionnaires to OSR

Sept.   PWC & RMAS perform direct cost & compliance
        testing, includes visits to grant administrators

                    7. IS Audit Process                 48
        A-133 Schedule, cont

Month                           Schools
 Oct.   Provide responses for additional information
        from PWC, RMAS to OSR
 Nov.   Provide responses to resolve initial findings
 Dec.   Present final draft report to Sponsored Program
        Operating Committee (SPOC)
 Jan.   PWC completes final report;
        Schools complete corrective action plan
 Feb.   “Reporting Package” submitted
                    7. IS Audit Process                 49
 A-133 Audit -
Two Major Areas

  Internal controls

 Compliance testing

      7. IS Audit Process   50
         Internal Controls

A-133 standards for grants management:

 Written policies & procedures are available
 Policies and procedures are assessed
 Operations tested for compliance to policy
 Results are reported

                  7. IS Audit Process           51
         Internal Controls

Written policies and procedures …
 Provide a system of checks and
 Provide reasonable assurance that
  management objectives are achieved
 Provide a basis for self-measurement

                7. IS Audit Process      52
Compliance Testing

Internal Controls
   - Entire University
   - Campuses / schools /
Federal Regulations
    - A-133; A-21
Program specific
    - Award terms and conditions
           7. IS Audit Process     53
Compliance Testing

   Direct cost testing
   Sub-recipient monitoring
   Cost transfers
   Effort reporting

             7. IS Audit Process   54
       Direct Cost Testing

Direct costs are …
 Costs identified specifically with a particular
  sponsored project, instructional activity, or any
  other institutional activity, or

 Costs that can be directly assigned to such
  activities relatively easily with a high degree of

                    7. IS Audit Process                55
     What Auditors Look for

Complete documentation of expenses:
   Justification of expenses
   Completed and concise business purpose
   Allowability and allocability of expenses
   Legible receipts
   Documentation completed at time of trans-
    action processing
                   7. IS Audit Process          56
          Direct Cost Areas
           Often Targeted

 Consultant expenses                      Travel
 Equipment                                Telephone
 Printing & publications                  Payroll
 Computer supplies                        Postage

                    7. IS Audit Process                 57
Federal regulations require pass through
entities to monitor sub-recipients of federal funds

 Receipt and review of technical reports
 Review sub-recipient invoices, expenses-to-
 Perform on-site visit, if necessary
 Perform “audit” option, if necessary

                    7. IS Audit Process               58
   What Auditors Look for

 Adherence to University policies
 Evidence of review of:
      - progress reports
      - invoices
 Compliance with federal regulations

                7. IS Audit Process     59
         Cost Transfers

A Cost Transfer is a transfer of an
expense to a federally funded sponsored
account that was previously recorded

               7. IS Audit Process        60
   What Auditors Look for

 Cost transfers tested against University policy
  and forms reviewed for:

     - timeliness
     - justification
     - authorization

 Large numbers of cost transfers might
  suggest a lack of internal controls
                  7. IS Audit Process               61
         Effort Reporting

 Required monthly for non-faculty on all
  federally sponsored projects
 Required annually for faculty on all
  federally sponsored projects

                 7. IS Audit Process        62
  What Auditors Look for

 Existence of effort reports
 Evidence of timely review by someone
  with first-hand knowledge
 Timely signatures, collection and filing
 Percentage of effort over 100%

                7. IS Audit Process          63
               Case Studies

   The unfriendly skies of A-133
   The charge is in the mail … a gothic tale
   Guilty as charged …
   … dangerous liaisons
   Other A-133 perennials

                      7. IS Audit Process       64
       The charge is in the

Defensive strategy
 Where possible, anticipate such expenses, include
  in budget and budget narrative/justification
 Be prepared to justify unlike circumstances
 Document circumstances, e.g.. conference calls
 Other postage expenses which could be considered
  direct: overnight shipping of samples, recruitment
  mailings, survey mailings

                     7. IS Audit Process               65
    … dangerous liaisons

Defensive strategy
 Include description of duties (as detailed as
  possible) in budget and narrative
 Descriptions should be as specific to project
  as possible
 Complete the administrative checklist
 At least 25% effort

                  7. IS Audit Process             66
     Other A-133 Perennials

 Hardly ever allowable except for travelers

 Need to document (even if only by e-mail) scope
  of duties and compensation

                    7. IS Audit Process             67
   Corrective Action Plans

 Direct cost findings need to be transferred to
  non-sponsored accounts
 Other courses of action must be proposed:
  - additional educational outreach
  - development of new policies / procedures
  - implementation of new controls
 Status report on corrective actions requested
  in following years‟ A-133 audit
                  7. IS Audit Process              68
         How to be A-133
          “Audit Ready”
Top ten things to remember about being
audited for Federal compliance:
10. Taxpayers‟ money being audited; we need
    to account for our responsibility

9. Accountability and accounting not the same

8. Reasonableness wins every time, if
                  7. IS Audit Process           69
         How to be A-133
          “Audit Ready”

7. Direct attribution and causal-beneficial
   relationships of costs must be demonstrated

6. Clear and complete documentation always
   helps; go overboard when in doubt

5. Letter-perfect compliance with internal
   policies (DS-2, Cost Transfers) and external
   rules (A-21) is essential

                  7. IS Audit Process             70
          How to be A-133
           “Audit Ready”

4.   Review top problem transactions: travel,
     postage, meals & supplies for each grant

3. Be skeptical, check out gray areas

2. Answer auditors‟ questions directly, do not
   volunteer additional information

1. Seek advice from OSR or your School
   Admin Team at any time
                   7. IS Audit Process           71

 Understand background and requirements
 Understand process and schedule
 Understand Internal Controls and Compliance
 Understand in detail “high risk” areas
 Understand impact of findings and corrective
 Understand how to be “audit ready”
                 7. IS Audit Process             72
  P.L. 106-107, Federal
  Financial Assistance
Improvement Act of 1999

        7. IS Audit Process   73
                       Current Structure

                    Audit Oversight Workgroup
Sub-recipient                                          Delinquent Audits
Monitoring                                             (HHS)
                                                   Audit Process
   A-133 Compliance Indirect Cost     FAC Users    (DOL)
   Supplement       Rate              Group
   (OMB)                              (HHS)

                             7. IS Audit Process                      74
    P.L. 106-107, Federal Financial Assistance Management Improvement Act of 1999


                   CORE TEAM

-Compliance Supplement is updated and published annually.

-Assists auditors in determining which compliance requirements
 are important to the Federal Government.

-2003 Publication can be accessed at:

-Core Team Leader: Terry Ramsey, OMB – tramsey@omb.eop.gov

                               7. IS Audit Process                                  75

            SUB GROUP

-Established to review and make improvements
 to the Compliance Supplement –

-Major revisions made to Section H., Indirect
 Cost reflected in the 2003 Publication

                 7. IS Audit Process            76
      P.L. 106-107, Federal Financial Assistance Management Improvement Act of 1999



 Ad-hoc tasks

      - Updated Standard SAC Form

      - Additional Reports Generated From the FAC
Federal Audit Clearinghouse (FAC)
Web Site:                         http://harvester.census.gov/sac/

                                7. IS Audit Process                              77
    P.L. 106-107, Federal Financial Assistance Management Improvement Act of 1999



-Single Audit Pamphlet and Brochure developed,
 published and distributed (Task Completed)

Highlights of the Single Audit Process
Single Audits Basics – Where to Get Help


                               7. IS Audit Process                                  78


- Identifying agencies policies/approaches

- Best Practices

                    7. IS Audit Process      79
                Additional Activities

• Audit Quality VS. Audit Expectations –
 White Paper submitted to OMB. Task Completed.

• Delinquent Audits –    Will be addressed in the future.

                        7. IS Audit Process                 80

• Potential restructure/expansion
 of Work Group

•Greater Alignment of future
 initiatives with E-Grants

• Address the use of the Single
 Universal Identifier with Federal
 Audit Clearinghouse information

              7. IS Audit Process    81
State Grants and Scholarships Department
        Financial Aid Audit Process

IS Audit Process
               7. IS Audit Process         82
          Financial Aid Audit Process
1.    Institutions are selected randomly for an audit

2.    SGS & HEI staff gather internal financial aid information

3.    Information is placed in an Excel spreadsheet

4.    Spreadsheet is then entered into the audit database (extraction)

5.    Listing of students who received any form of financial aid is generated for review
      by institution

6.    E-mail notification sent to audit contact with list of students attached

7.    Schools are then given roughly six (6) weeks to review the list and send in
      required documentation

8.    Documentation is then reviewed in the office for any discrepancies

9.    If discrepancies are discovered, institutions are given an additional one (1) week
      to ten (10) days to address them

10.   Audit is complete once discrepancies are cleared or a check is received
                                     7. IS Audit Process                              83
        Financial Aid Audit Process
               - Step One -

Institutions Are Selected Randomly For An Audit

 staff select schools to be audited from a list of all schools that
  participate in our financial aid programs.

 Letter sent to President of school and Financial Aid Director
  informing them of the audit

                             7. IS Audit Process                       84
     Financial Aid Audit Process
            - Step Two -
    staff gather internal financial aid information

   A list of students that participated in any State funded financial
    aid program is gathered from State Grants & Scholarships staff

                            7. IS Audit Process                          85
         Financial Aid Audit Process
               - Step Three -

      Information is placed in an Excel spreadsheet

   School Name
   Four Digit Code
   All Financial Aid Programs
   100 students for each institution listed

                               7. IS Audit Process    86
      Financial Aid Audit Process
             - Step Four -
      Spreadsheet is then entered into the audit

 Ohio Instructional Grant and Ohio Academic Scholarship audits are
  conducted on the database. All other programs are extracted via a
  manual process

 Once all programs are converted to, there will be no need for
  manual input

                           7. IS Audit Process                    87
       Financial Aid Audit Process
              - Step Five -

Listing of students who received any form of
    financial aid is generated for review by

   The extracted file is placed into an Excel spreadsheet in Social
    Security Number order

                             7. IS Audit Process                       88
     Financial Aid Audit Process
            - Step Six -

E-mail notification sent to audit contact with list of
                   students attached

                     7. IS Audit Process             89
      Financial Aid Audit Process
            - Step Seven -
    Schools are then given roughly six (6) weeks to
            review the list and send in required

Examples of documentation include:

    Admission Application
    Academic Transcripts
    Add/Drop Forms
    Schedule Registration Form
    Account Statements
    Verification of Selective Service (if not available, we can access
                             7. IS Audit Process                          90
        Financial Aid Audit Process
              - Step Eight -

Documentation is then reviewed in the office for any

Common discrepancies include:

   Students not enrolled in an Associate or Bachelors degree program
   Students not enrolled on a full-time basis
   Students drop before 100% charge date
   Financial aid awards do not appear on the account statement
   Schools fail to supply add/drop forms

                             7. IS Audit Process                        91
      Financial Aid Audit Process
             - Step Nine -
    If discrepancies are discovered, institutions are
        given an additional one (1) week to ten (10)
                   days to address them

Discrepancies are sent to schools in an Excel spreadsheet that includes
     the following information:

    Student Social Security Number
    Student Name
    Term involved (summer, fall, winter or spring)
    Discrepancy
    Possible Amount Owed

                            7. IS Audit Process                      92
        Financial Aid Audit Process
               - Step Ten -

Audit is complete once discrepancies are cleared or a
                   check is received

 A Letter is mailed to the Financial Aid Director informing him/her
  that the audit is complete. A copy of the same letter is also mailed to
  the school President

 When the audit is complete, audit results are posted to the website
  at the relevant Sponsor, State or Fed Gov.

                             7. IS Audit Process                        93
      Board of Regents

Academic IS Audit Process

          7. IS Audit Process   94
           Topics to be Covered
1. Education Quality                         2. Academic IS audit
   Processes (EQP)                              process
   • Definition                                 (reviews EQP)
   • Quality process domains                      • Origin and applications
   • Education quality principles                 • Steps in the audit process
   • Use of evidence in quality                   • Advantages of IS audit
     processes                                      process
   • The quality paradigm                         • Steps in launching a pilot
                                                    audit program
   • Quality process maturity

                            7. IS Audit Process                               95
 Education Quality Processes
 “Education quality processes” (EQP) aim
  to improve and assure the quality of
  – They apply modern quality principles, like
    those of the Baldrige, in ways that are
    understandable and acceptable to
  – They stress the education’s “fitness for
 EQP provide a framework for quality.
  – They are the “guidance system” for teaching,
                  7. IS Audit Process        96
  Quality Process Domains
• Design quality of desired learning outcomes
  – What knowledge and skills will be imparted?
• Design of curricula
  – What subjects will be taught?
• Design of educational processes
  – How will students learn?
• Design of student assessment measures
  – How will the learning be evaluated?
• Implementation quality
   Implementation quality assurance
  – How will the design goals be achieved, day in
    and day out, in the face of conflicting priorities?
                   7. IS Audit Process              97
       Quality Process Principles
 Define education quality in terms of student
  learning attainment (value added where possible).
 Focus on the processes of learning and
 Strive for coherence in curricula, learning
  processes, & assessment (sometimes called
 Work collaboratively to achieve mutual
  involvement and support (teams, teamwork).
 Base decisions on evidence wherever possible.
 Identify and learn from best practice
  (benchmark, minimize 7. IS Audit Process quality variation).
                       controllable                              98
           Use of Evidence
         in Quality Processes
 Evidence is intentional and purposeful.
 Evidence entails interpretation and
 Evidence is integrated and holistic.
 Evidence can be either qualitative or
 Evidence can be either direct or indirect.
                 7. IS Audit Process        99
     Use of Evidence (cont.)
When assessing student performance,
evidence should:
• Cover knowledge and skills taught throughout
  the curriculum
• Involve multiple judgments of student
• Provide information on multiple dimensions of
  student performance
• Involve more than surveys or self-reports of
  competence or growth by students
                  7. IS Audit Process        100
         The Quality Paradigm

Responsibility                                Process

Faculty and students         Teaching and learning itself

Faculty and chairs                  Departmental EQP
Deans                                        School EQP
President, Provost                   Institutional EQP
Governing board                   Institutional oversight
State coordinating boards,                    External
  accreditation agencies                      oversight
                       7. IS Audit Process                  101
                    Quality Process Maturity

                                                                    Mature Effort
                                                                     An embedded culture of quality
                                                                     Continuous improvement
                                               III                   Shared organizational learning
      Maturity                                       Organized Effort
       levels                                         Initiatives that are planned & tracked
                                                      Approaches that are based on principles
                                II                    Metrics & norms that are evidence-based
                                     Emergent Effort
                                      Individual initiatives
                I                     Experimentation with quality principles
                     Reaction to problems
                     Ad hoc adjustments
                     Little focus on quality principles
    No Effort                                                                      Time
     Traditional methods
     Little quality consciousness           7. IS Audit Process                                  102
    Academic IS audit process:
     Origins and Applications
 Academic audit tests EQP maturity and
  encourages improvement.
 Experience with IS audit process:
  – Pioneered by UK vice chancellors, circa 1990
  – Adapted successfully in Sweden, New Zealand,
    and Hong Kong
  – Recently adopted as the key evaluation method
    in the UK and Australia
  – Successful US pilots at WASC (accreditation)
    and the University of Missouri System
                   7. IS Audit Process         103
    Steps in the Audit Process
 Introductory workshop with auditees, 6-9
  months before the IS audit process (1 day)
 Self-study by the audited department or
  institution (15-20 pages maximum)
 Auditor selection; auditor training (1 day)
 The IS audit process visit (1 or 2 days)
 Preparation of the IS audit process report
 Follow-up session to obtain feedback and
  disseminate best practice (1 day)
                  7. IS Audit Process       104
        Introductory Workshop
 Involves a representative group of professors and
  others responsible for education quality.
 Introduces and discusses education quality
  processes and the entity‟s self-study of them.
  – Describes IS audit process as a discussion with peers
    about the entity‟s self-study and EQP generally.
 Stresses IS audit process‟s improvement
  – Better EQP benefit the entity itself and its students.
  – The effort required for improvement is modest.
                         7. IS Audit Process                 105
             The Self-Study
 Describes the unit‟s quality processes.
  – Must cover all five domains (the only “must”)
  – Identifies exemplary practices
 Reflects on how the quality and evidential
  principles contributed to the unit‟s exemplary
 Identifies domains or processes where using
  the principles could effect improvement
 Presents an improvement plan
                    7. IS Audit Process             106
 Auditor Selection and Training
 The IS audit process team
  – Target 6-8 auditors for departmental audits;
    12-14 for institutional audits
  – Mostly but not exclusively faculty
  – Include a variety of academic disciplines
 Training (1 day)
  – In-depth discussion of EQP concepts;
    exemplary and questionable practices
  – Briefings on what to look for in the SEDs, the
    audit visit agenda, and report-writing
  – Role-play on conducting an audit session
                     7. IS Audit Process             107
    The IS audit process Visit
 Department-level audits (1 day)
  – Plenary sessions with:
      departmental leadership or the whole department
      program directors and others responsible for quality
      a representative group of students
  – Two sets of small-group discussions to follow
    up on specific points in the SED
      Two auditors per session yields 3 x 2 = 6 sessions
  – Executive session
  – Exit conference
                      7. IS Audit Process                108
The IS audit process Visit (cont.)

 Institution-level audits (2 days)
  – Same as departmental IS audit process,
    except substitute relevant faculty committee
    members for program directors
  – Two more small-group slots; small groups
    sample departmental and school processes
      Two auditors per session yields 6 x 4=24
  – A follow-up plenary session with the deans
                     7. IS Audit Process           109
         Post-Visit Activities
 IS audit process Report
  – Written by the team chair
  – Checked with the department for accuracy
  – Institutional audit reports are made public;
    decide about departmental reports in advance
 Follow-up Workshop
  – Broad participation by academics
  – Discussion of the IS audit process experience
  – Presentations of exemplary practice

                    7. IS Audit Process             110
Advantages of IS audit process
 Engages faculty in interesting and practical
  dialog about education quality.
 Demonstrates that modest efforts to improve
  EQP can benefit both teachers and students.
 For institution-level audits, demonstrates that
  deans and other academic leaders can spur
  improvement in departmental EQP.
 Fuses the improvement and accountability
  agendas without threatening faculty.
 Can be done at low cost.
                   7. IS Audit Process              111
       Steps in Launching a
  Pilot IS audit process Program
 Decide whether to pilot:
  – at the departmental level, within single institutions;
  – at the institutional level, possibly with multiple
 Tailor the audit design to the chosen program.
 Conduct interviews to identify attitudes and
  issues that may arise during the pilot.
 Prepare readings and presentation materials.
 Proceed to “Steps in the audit process.”

                       7. IS Audit Process               112

To top