Quantitative Literacy Hoelter Alter

							Quantitative Literacy:
 Assessment & Practice
    Virtual OR Meeting, 2009



Flora McMartin, Broad-based
      Knowledge, Inc.
Corrine Taylor, Wellesley College &
      National Numeracy Network


                                      Page 1
Presenters

• Flora McMartin
  President, Broad-based Knowledge, Inc.

• Corri Taylor
  Director, Quantitative Reasoning
  Program, Wellesley College;
  President, National Numeracy Network

                                    Page 2
Overview of the Presentation


• What is “quantitative literacy” (QL)?

• Some background on “assessment”

• Examples of QL assessment in practice



                                      Page 3
QL is defined as

• the ability to understand and use
  quantitative measures and inferences that
  allow one to function as a responsible
  citizen, productive worker, and discerning
  consumer. (Bernie Madison)

• the ability to identify, understand and use
  quantitative arguments in everyday
  contexts. Quantitative literacy describes a
  habit of mind rather than a set of topics
  of a list of skills. (Deborah Hughes Hallett)

                                          Page 4
MAA’s QL Competencies
• Reading and understanding quantitative info in
    graphs, tables, etc.
•   Interpreting quantitative info and drawing
    appropriate inferences
•   Solving problems using logic, math, statistics
•   Estimating answers and checking for
    reasonableness
•   Communicating quantitative info – verbally,
    graphically, numerically
•   Recognizing the limitations of mathematical or
    statistical models

                                             Page 5
QL in the Social Sciences

• Nicely captured in the NCED book
  Mathematics and Democracy: The
  Case for Quantitative Literacy by
  Lynn Steen
 http://www.maa.org/ql/mathanddemocracy.html
• Statistics for data analysis in all
  social sciences
• Basic QR skills to be an informed
  citizen, voter, etc.
                                               Page 6
Assessing QL: The Ws and the H

• What to assess?
  – Students’ attitudes, behaviors; skills
  – Course materials, modules
  – Program effectiveness
• Why assess?
  – For student placement
  – For enhancement of course materials
  – For planning and accreditation
  – To test innovation             Page 7
Assessing QL: The Ws and the H
• When to assess?
  – Upon college entry and exit and later
  – Before, during, and after particular courses
• How to assess?
  – Using various kinds of “tests”
  – Using “portfolios” and other tools
  – Ensuring the assessment tools match the
    purpose and the level of QL being assessed
  – Evaluating with clear rubrics

                                           Page 8
   Some practical questions when
   considering assessment
• Is the assessment feasible (timing, logistics,
  resources)
• What is the impact of the type of institution?
   – What is the skill level of students?
• What course level(s) will be involved? (lower division,
  upper division)
• What type of administrative support is available?
   – collecting/storing/analyzing data
• What is the scope of the assessment? (Course to
  Consortium)
• Is IRB review necessary?
                                                    Page 9
Assessing Attitudes & Behaviors

• Student Attitude Assessment –
  developed at Dartmouth by J. Korey–
  Four scales: utility, personal growth,
  ability, & interest
• Office of Institutional Research –
  surveys and analyses of students’
  course-taking behavior
• Continuous feedback from
  quantitative faculty
                                   Page 10
Assessing Quantitative Skills:
Four Examples in Practice
• QR Placement Test for incoming students
  (Wellesley College)
• Test of two QR learning objectives for
  students who completed gen. ed.
  requirements (James Madison University)
• Rubric for QR in rhetoric; in sophomore
  writing portfolios (Carleton College)
• Rubric for evaluating QL skills gained over
  college career; in electronic portfolios
  (AAC&U’s QL VALUE project)
                                         Page 11
Wellesley’s QR Placement

• Assesses incoming students’ QR
  skills; weakest performers on test
  need to take basic QR course before
  enrolling in quantitative courses
• 18 open-ended questions
• Booklet with info available at
 http://serc.carleton.edu/files/nnn/teaching/wellesley_qr_booklet.pdf




                                                               Page 12
JMU’s QR Test
• Measures two specific learning objectives…
  How well students who completed gen ed:
   – Use graphical, symbolic, and numerical
     methods to analyze, organize, and interpret
     natural phenomena
   – Discriminate between association and
     causation; how to establish causation
• Test is 26 computerized, multiple-choice
  questions
  http://serc.carleton.edu/nnn/numeracyprojects/examples/32007.html



                                                                      Page 13
Carleton’s QR in rhetoric
• Examine papers in sophomore writing
  portfolios for evidence of QR
• Rubric:   http://serc.carleton.edu/files/quirk/quirk_rubric.v5.doc
• For each paper, examines
   – Potential relevance of QR (none, peripheral,
     central)
   – Extent of QR (scale: 1 to 3)
   – Overall quality of implementation,
     interpretation, and communication (scale: 1-4)
   – Problems (ambiguous words rather than
     numbers; fail to describe data collection
     methods; fails to provide comparisons)


                                                                 Page 14
AAC&U’s QL rubric

• For institutional use in evaluating and
  discussing student learning over all the
  college years
• Rubric (scale: 1-4) applied to components
  of students’ e-portfolios
• Six QL skill areas for evaluation:
  interpretation, representation, calculation,
  application/analysis, assumptions,
  communication
  http://www.aacu.org/value/rubrics/pdf/QuantitativeLiteracy.pdf


                                                               Page 15
Infusing QL throughout the
Social Science Curriculum
• NSF-CCLI funded partnership between ICPSR and Social
  Science Data Analysis Network (SSDAN)
• Purpose: Transform undergrad instruction by improving
  the teaching of QR and the way students understand
  research in the social sciences
• Activities
   – Study the impact of SSCAN and ICPSR teaching modules
     with regards to QL
   – Develop new teaching resources (American Community
     Survey)
   – Provide new instructional materials linked to ICPSR
   – Build national social science community of users of these
     resources

                                                          Page 16
Assessment Focus of “Infusing QL”
• Assessment tool/process development
  and testing
  – 8 Faculty from diverse institutions
    developing and testing tools
  – Mainly lower division introductory classes
  – Tools map to QL student learning
    objectives
• Survey of faculty use/adoption of QL
  activities

                                        Page 17
Building on the AAC&U QL Rubric

 •   Interpretation             • Communication
 •   Representation             • Find/Identify/
 •   Calculation                  Generate Data
 •   Analysis                   • Research design
 •   Method selection           • Confidence
 •   Estimation/                • Content learning
     Reasonableness               outcomes
     Checks

           Items in blue: direct map to AACU Rubric
                 Items in green: indirect map

                                                      Page 18
Using Rubrics to Plan for
Teaching and Assessing
                             Students
                Design      Experience     Assess if
 Articulate
               Learning        and       Students’ met
Expectations
                Activity   demonstrate   expectations
                             learning




                                           Page 19
Sample Assessment Plan*




                                                    Page 20
      * Adapted from Jill Bouma’s Assessment Plan
A Short Digression
• A short digression to Bloom’s
  taxonomy: a thinking skills framework
        Lower Order          Higher Order
        Remembering            Analyzing
        Understanding          Evaluating
           Applying             Creating

• Why is this important?
  – Practical: course level, student level,
    expectations
  – Methodological: qualitative/quantitative
                                              Page 21
Assessment Tools
Tool must collect data appropriate to level (Bloom)
and Learning Outcome (Rubric)
     Lower-order                 Higher-order
• Quiz                     •   Report/in-depth paper
    – Multiple choice      •   Chart/outline
    – Definition
                           •   Debate, panel
    – List
                           •   Investigation
•   Problems
                           •   Evaluation
•   Examples
                           •   Media product
•   Simulations
                           •   Art work
•   Performance
                                              Page 22
 Sample Rubric for a Paper*
   Learning
   Outcome           Excellent            Competent         Needs Work
Interpretation:   Consistently          Uses correct       Incorrect
reading and       uses correct          numbers but        numbers and
reporting         numbers; uses         makes some         frequent
frequencies;      %’s from the          mistakes with      mistakes in
properly          proper cell;          language usage     language
interpreting      correctly adds        Uses %’s from      Uses wrong %’s
numbers in        across or down        correct cell;      and can not
bivariate table   different             some problems      combine
                  categories            when               categories
                                        combining          correctly
                                        categories

              * Adapted from Jill Bouma’s grading rubric
                                                                 Page 23
Final Thoughts

• Wrap-up
• Questions from you




                       Page 24