Get documents about "From Assessment to Action Using the Delphi Technique to
Document Sample
Page 1
From Assessment to Action: Using the Delphi Technique to Encourage Faculty Buy-In
Conference Paper,
Association for Institutional Research 2007 Annual Conference
Kansas City, MO
June 2007
Sean A. McKitrick
Binghamton University
Page 2
Abstract
This paper introduces a case in which the Delphi method was used to assess critical thinking. It discusses
the reasoning behind using the method, introducing additional validity issues (consequential and
pedagogical validity in the eyes of regional accrediting organization, such as the Middle States Association
of Colleges and Schools). It provides the results of the actual study, and then focuses on some reasons why
the approach can be used for assessment, especially in areas in which student learning outcomes are
difficult to define, such as critical thinking, aesthetics, and interdisciplinary studies. There follows a
discussion of why the approach especially enables faculty discourse to occur, and concludes with some
advantages and limitations of the approach.
With regard to assessment of student learning, no one can doubt that we live in interesting times.
The United States Department of Education, specifically the Secretary’s Commission on the Future of
Higher Education, has strongly encouraged regional and national accrediting organizations to require their
member institutions to provide proof that students are learning.i Even higher education trade associations
such as the American Association of Colleges and Universities have issued statements strongly
encouraging colleges and universities to use standardized tests that move toward proving the “value added”
of a university or college education.ii And recently, the Department of Education has sanctioned accrediting
organizations for not requiring their member institutions to provide appropriate evidence of student
learning.iii
Undoubtedly, much of the pressure to demonstrate value added is welcomed by institutional
researchers and college and university administrators struggling to move the academy along a path toward
accountability. Indeed, a popular view among regional and national accreditors is that the Department of
Education is pushing toward a value added approach because fewer tax dollars are available to higher
education; thus, in order for colleges and universities to receive future funding, they will have to
demonstrate that they are using current funding effectively.iv
Accrediting organizations are also heavily encouraging institutions to demonstrate that they are
doing something with the data they collect. The problem for institutional researchers and assessment
professionals is that, given sincere efforts to collect, aggregate, and summarize assessment and other
student learning data, there is still little guarantee that program faculty will discuss the data and manage
student learning using the data submitted to them. In sum, measures of student learning data might have
high degrees of predictive validity, but little consequential validity.v
Page 3
In this paper, I address the question of buy-in. Can an assessment method be used that both
measures student learning, but maximizes the chance that faculty will use such information for program
improvement? In what follows, I describe a case study of how critical thinking was assessed using a
qualitative method of assessment, the Delphi method. I then discuss the applicability of the method in
assessing student learning, especially with regard to the pressures described above.
Case Study: Assessing Critical Thinking
Background
At a Carnegie doctoral-extensive university in the northeastern United States, the faculty are
charged by the state government to assess critical thinking, which is defined as follows: “Students will
identify, analyze, and evaluate arguments as the occur in their own or other’s [sic] work; and develop Well-
reasoned arguments.” No additional wording or specifications were provided, leaving it up to the liberal
arts faculty in composition and oral communications courses, as well as the division of academic affairs, to
define what these terms met. Assessment of critical thinking was therefore difficult to achieve because few
knew how to proceed beyond the challenges associated with unpacking vague terms such as “analyze” and
“evaluate.” In addition, the institution has had a strong cultural predisposition toward qualitative, locally-
managed assessments, and against state-mandated, standardized tests, or other ways of assessing student
learning that did not include faculty as the primary assessors, and which did not include some degree of
qualitative information, especially in the liberal arts and general education programs of the university. It
was therefore clear that one way of initially conducting assessment of critical thinking as part of the
university’s general education program was to use a discourse-oriented, qualitative technique such as the
Delphi method, described below.
Conceptual Framework
I indicated above that educational researchers and regional accrediting organizations prefer to see
student learning assessments that demonstrate value added, such as in the form of standardized tests and
other quantitative instruments. Yet also important is the use of assessment information by program and
general education faculty, many of whom are not trained in the area of quantitative analysis.vi And while
discounting quantitative, value-added approaches is well-advised given the above-described political and
Page 4
institutional pressures, focus must also be upon the effective and purposive triangulation of assessment
information to (1) reduce error due to over-reliance on one instrument or indicator; and, (2) enhance
opportunities for face validity of assessment results in the eyes of the core audience, in this case program
and general education faculty.
Suskie’s vision of pedagogical and consequential validity, in which the quality of assessment
information is judged by its effect on teaching and learning based on faculty input, therefore serves as the
conceptual framework for this study.vii This conceptual framework asserts that we must be cognizant of
positivist concerns about validity, reliability, and value added, but must also be concerned about the
legitimacy of the information in the eyes of its core audience. Including qualitative information and
considering what it has to say about student learning outcomes are just as important as more succinct,
quantitative information because these actions provide substantive, contextualized information for use by
faculty.viii
Methodology
Rationale
I chose a qualitative method to understand student performance in respect to critical thinking.
Since the purpose of the assessment was, first, to get faculty to discuss the quality of student performance
in respect to critical thinking as defined above and, second, to enable them to compare the findings of a
direct observation of student performance on critical thinking writing using a rubric, I felt that use of the
Delphi method was most appropriate for our purposes.
The Delphi method was developed in the 1950s to forecast future developments, based on the
opinions of experts who gathered for this express purpose.ix Since then, it has been used in the medical field
to produce generalizations about patient treatment options and outcomes, in teacher education to determine
appropriate pedagogies and curricula, and in several other fields of interest.x The advantage of the method
is that it involves experts who can be relied on for informed opinions about the quality of various areas of
focus. With regard to assessment, the method is a way of gathering faculty feedback after a key event (such
as the submission of final grades, the grading of key assignments such as master’s theses and doctoral
dissertations, etc.), at a distance or in person, and gaining informed opinions about strengths and
Page 5
weaknesses in student performance, especially when learning outcomes are particularly vague. It is the last
of these reasons—the vague, fairly open-ended critical thinking student learning objectives as provided to
us by the state government—that led us to use this method.
This was the best approach for us because (1) it included faculty as the primary source of
information about specific assignments and projects students have completed; (2) it generated numerical,
aggregate information enabling us to reduce information so we could understand the comparative weight of
consensus (or non-consensus) about the quality of student performance, using the above-stated student
learning outcomes as specific reference; and, (3) it contained specific information and feedback to be used
for discussion and reflection by specific parties, including the Center for Learning and Teaching, the
Institute for Student Centered Learning, the Faculty Senate and its associated committees, and the
University Libraries. These groups could then act on any information they found significant, thereby
“closing the loop” from assessment, to discussion, to recommendations for action, to the actions
themselves.xi
Procedure
This Delphi method of evaluation involves at least four steps.xii First, “experts” are identified, and
asked to answer a number of open-ended, evaluative questions. Second, these open-ended responses are
listed on a second, closed-ended survey, with an indication of how many respondents stated each. Third,
respondents rate their level of agreement with each statement on a five-point Likert scale, with (1) meaning
“strongly disagree,” and (5) meaning “strongly agree.” Four, responses with mean values of 2.0 or lower or
medians of 2.0 or lower, and standard deviations of less than 1.0 are designated items of consensus, and
reported as such in an assessment report.xiii I explain each of the above in turn:
1. Twenty-eight faculty and graduate teaching assistants who taught upper division composition
and combination composition/oral communication courses were contacted. I explained that
the purpose of the research was to gather their thoughts on strengths and weaknesses in
student performance in respect to critical thinking. Thirteen agreed to participate.
2. I sent an open-ended survey to each respondent, making sure that they were not known to the
other participants. The survey read as follows:
Page 6
Open-Ended Survey Questions
Based upon the assignments you have graded in student work in the course you taught
during this last session, what were some specific strengths or weaknesses you have
observed in student performance in respect to the following?
• Developing well-reasoned arguments (i.e., distinguishing fact from opinion,
identifying assumptions and reasons in an argument, recognizing the need for
additional information of particular types, etc.)
• Identifying, analyzing, and evaluating arguments as they occur in their own or
others’ work (i.e., finding flaws or gaps in an argument, recognizing separate
components of a reasoned argument, identifying assumptions and reasons in an
argument, etc.)
• Performing the basic operations of personal computer use
• Understanding and using basic research techniques (i.e., employing methods of
information collection and manipulation, locating and evaluating information from a
variety of sources, designing and implementing data-oriented studies for answering
research questions, etc.)
• Locating evaluating and synthesizing information from a variety of sources
13 instructors responded to this first open-ended survey and supplied numerous comments.
The comments to each question were then read several times in order to identify common
themes. In order to accomplish this task, the focus of analysis was phrases used in the
responses, which were then listed and compared for similarity. Those phrases that were very
similar were combined and tallied after several readings of the responses until it was clear that
the process would not yield further similarities or tallies, in a process called constant
comparative analysis.xiv In this way, the analysis achieves construct validity in that data were
directly related to the questions on hand, involved an exhaustive process in which there was a
high probability that all phrases would be part of the analysis through the constant
comparative analysis technique mentioned above, and responses were mutually exclusive
from other questions on the survey.xv
3. A closed-ended survey was created which, as described above, asked the anonymous
instructors to evaluate the open-ended comments on a five-point Likert scale of agreement,
with (1) meaning strongly agree, and (5) meaning strongly disagree. Ten instructors
completed the online survey, for a response rate of 77%.
Results
The results of the second Delphi round can be found in tables one through four at the end of this paper. On
each of these tables, consensus answers are those that had mean responses of 2.0 (1 = strongly agree with
the statement; 2 = agree with the statement) or lower, and standard deviation values of less than 1.0. In
respect to each of the state-mandated student learning outcomes for critical thinking, the results are
summarized in the following boxes:
Page 7
Critical thinking outcome 1: Developing well-reasoned arguments
In total, faculty made eighteen comments. When asked to rate these comments
according to degree of agreement, there was moderate consensus that students were
good at understanding general concepts as far as making correct arguments was
concerned and applying different types of information for different kinds of application.
There was also moderate consensus that students needed to improve in respect to
factually supporting their arguments or opinions.
Critical thinking outcome 2: Identifying, analyzing, and evaluating arguments
Faculty made a total of thirty comments. There was moderate consensus that students
know how to organize their thoughts in essays and written assignments, that they can
identify gaps in theoretical frameworks outlined in texts and other readings, challenge
each others’ opinions, and apply their own research and interests in their writing.
There was also moderate consensus that students had a difficult time evaluating
arguments, engaging in self-evaluation of the quality of their work, and clearly
communicating complex ideas.
Critical thinking outcome 3: Understanding and using basic research techniques
Faculty made seventeen statements. There was moderate consensus that students had
done little of their own research prior to their taking a 300-level composition or
combination composition/oral communication course, and that they have had little
practice using the library’s online databases. There was a high level of consensus that
there is a great deal of variance from student to student in respect to their
understanding of basic research techniques and using them in 300-level assignments.
There was also a good deal of consensus that student performance in respect to using
basic research techniques was dependent on how much time students put into their
papers and projects in these courses. The highest degree of consensus was that
students use Google or other search engines instead of library resources.
Critical thinking outcome 4: Locating, Evaluating, and Synthesizing Information
Faculty made thirty statements on this student learning outcome. There was a
moderate level of consensus that students needed to improve in their development of
acceptable bibliographies, and that evaluating and synthesizing information was
underdeveloped in students taking 300-level composition and combination
composition and oral communication courses. There was a high level of consensus
that students rely too much on Web pages as sources for the papers they write in
these courses.
Although the sample size for several of the questions was quite small, it was interesting to note that, in
several cases, those responses that had originally received only one mention in the initial open-ended
survey received a high degree of agreement and consensus when all respondents were asked to answer the
Page 8
second follow-up closed-ended survey. For example, table 2 indicates that only one person mentioned that
“students are better at picking out the weaknesses in the essays they read than in evaluating their own
essays,” yet received a mean of 1.63 and standard deviation of .518 in the follow-up survey. Table 3 attests
that only one respondent mentioned that “students use Google (or other search engines) instead of library
resources,” and yet was one of the four highest items of consensus on the table on the follow-up survey.
Interestingly, when the results were presented to the faculty senate organization responsible for assessment
oversight, this item caused the most discussion; all those present—experienced in one form or another in
teaching critical thinking courses—wholeheartedly agreed with this statement, and most moved to
recommend actions to counter such a trend.
Use of Data for Program Improvement
A central question for this study is, “To what extent did the assessment method lead to buy in?”
Although aggregate measures are difficult to develop, use of the Delphi method did accomplish a number
of feats necessary for faculty buy-in, and for subsequent faculty-based discussion and recommendations,
developed to further enhance student learning.
First, once the Delphi data were aggregated, it became important to summarize them in ways that
faculty and key faculty senate groups could understand. There is no question that if I had presented the data
as represented in the data tables at the end of this paper, they would not have understood what the tables
had to say, given that many are in non-quantitative, non-scientific disciplines.
Second, it became clear that contextualizing the data by focusing on what the Delphi results had to
say about faculty consensus was very helpful. In other words, taking the four points listed above, making
them into points of reference (or “talking points”), and then asking faculty members or members of the
faculty senate responsible for assessment oversight (in this case for the general education outcomes relating
to critical thinking) to make recommendations geared toward improving student performance in these
points of reference was very important.
Contextualizing the data via use of the Delphi method helped spark further specification of the
assessment process. Interestingly (and as mentioned above), much of the information contained in the
Delphi study came as no surprise to faculty and those who played a part in overseeing assessment.
Page 9
However, because the information had been systematically collected and there were clear attempts to
survey faculty about their thoughts on students’ strengths and weaknesses with regard to critical thinking,
the results were better legitimized than if a simple standardized test such as the Collegiate Learning
Assessment (CLA) or MAAP had been used. Because of this heightened degree of legitimacy, the
assessment staff was better able to articulate an already-existing, but rarely followed process at the
university, outlined in Figure 1.
Although the Delphi information in and of itself would have been insufficient from a positivist,
value added paradigm, faculty and members of the faculty senate appeared more comfortable using several
sources of assessment information. The assessment staff was able to communicate concerns about students’
acquiring critical thinking skills during their time at the university through the use of multiple assessments,
encourage discussion of these concerns at the faculty senate level, focus on recommendations, and then
bring these recommendations to various university organizations for implementation, including first-year
programs, the university libraries, and the university center for learning and teaching.
Figure 1: Triangulating assessment data, moving from discussion to action
Critical
Thinking
Delphi
Rubric Faculty Library
Survey on
Scores Survey Survey
Critical
Thinking
Faculty-based
Assessment
Organization
Center for
Learning First-Year University Undergraduate
and Programs Libraries Curriculum
Teaching Committee
Use of the Delphi method therefore had face validity in that it appeared legitimate to ask faculty teaching
300-level critical thinking courses about the quality of student learning using the state-mandated student
learning goals relevant to critical thinking. In addition, faculty concerns about standardized tests such as the
Page 10
Collegiate Learning Assessment or MAAP producing vague information about what knowledge students
were not acquiring were somewhat alleviated by providing contextualized, specific information through the
Delphi study. The method therefore also provided consequential validity in that it lent itself toward
discussion and action by faculty groups responsible for providing oversight for critical thinking and other
general education outcomes.
Discussion
Use of the Delphi method has its weaknesses. For example, in this study, the number of
respondents was low, and the response rate for the second closed-ended survey was lower than desired.
Moreover, the Delphi method did not provide baseline or value-added data. For example, it would be
difficult to follow up on this study because those who responded to the original surveys would be difficult
to recruit a second time, if only because teaching assignments change, and thus they may not teach the
same courses the next time such a Delphi study were repeated. It is also difficult to organize a study with
control and experimental groups, although it would be interesting to see if faculty teaching critical thinking
courses say similar things to those teaching upper division courses in programs or majors.
That said, use of the Delphi method provided contextualized, qualitative information that can be
used to help interpret the results of other assessments (triangulation). For instance, given a hypothetical
circumstance in which a university has randomly selected an appropriate number of students to take the
MAAP or CLA, and receives scores and sub-scores indicating that students in aggregate score in the 50th
percentile on all measures. The university might be comforted to know that it is not doing terribly, or
might be alarmed that students are not performing better than the national average, but the information
provided by CLA or MAAP can really only present a vague picture of where the university stands in
important areas of emphasis such as critical thinking, analytical thinking, or writing. Use of a qualitative
method such as the Delphi method, when carefully administered, can provide contextualized information
that might be used to interpret standardized test scores or to focus on what they can do to further enhance
student performance.
Most important, use of a qualitative technique such as the Delphi method enables information to
be both triangulated and contextualized so that faculty are enabled to discuss and act upon results. In the
case of this study, faculty focused in the beginning almost solely on the Delphi results; the assessment
Page 11
office needed to encourage them to offer other more quantitative information to balance the final
assessment of the quality of student learning of critical thinking objectives. Without doubt, the technique
enabled initial buy-in so that assessment results could be balanced between quantitative and qualitative
results.
Page 12
References
Allen, J. & Bresciani, M. J. (2003). Public institutions, public challenges: On the transparency of
assessment results. Change, January/February, 21-23.
Chioncel, N. E., Van Der Veen, Wildemeerch, D., & Jarvis, P. (2003). The validity and reliability of focus
groups as a research method in adult education. International Journal of Lifelong Education, 22,
495-517.
Council of Higher Education Accreditation. (2007). CHEA board issues resolution of support: Reiterates
Department of Education mandates concerning accreditation and learning outcomes. Retrieved
June 1, 2007, from, http://www.chea.org/pdf/CHEA_Press_Release_May_2007.pdf
Educational Testing Service. (2006) A culture of evidence: Post-secondary assessment and learning
outcomes. Princeton, NJ: Author.
Exley, C., Sim, J., Reid, N. G., Jackson, S., & West, N. (1996). Attitudes and belief within the Sikh
community regarding organ donation: A pilot study. Social Science & Medicine, 43, 23-28.
Geertz, Clifford (1987). The interpretation of cultures: Thick description toward an interpretive theory of
culture. Au taut, 217-18, 151-176.
Hunt, D. P., Haidet, P., Coverdale, J. H., & Richards, B. (2002). The effect of using team learning in an
evidence-based medicine course for medical students. Teaching & Learning in Medicine, 15, 131-
139.
Linkon, S. L. (2005). How can assessment work for us? Academe, 91(4), 28.
Lorenzetti, J. P. (2004). Transformative assessment in higher education. Distance Education Report, March
15, 2004, 3-7.
Page 13
Meyer, M. K., Conklin, M. T., & Turnage, C. (2002). School foodservice administrators’ perceptions of the
school nutrition environment in middle grades. Topics in Clinical Nursing, 17, 47-54.
Mundhenk, R. T. (2004). Communities of assessment. Change, November/December, 36-41.
North Central Association of Colleges and Schools. (2003). Assessment of student academic achievement:
Assessment culture matrix. Retrieved May 19, 2005, from
www.ncahigherlearningcommission.org/resources/assessment/assessmatrix03.pdf
Robertson, M., Line, M., Jones, S., & Thomas, S. (2000). International students, learning environments and
perceptions: A case study using the Delphi technique. Higher Education Research &
Development, 19, 89-102.
Schroeder, C., & Neil, R. M. (1992). Focus groups: A humanistic means of evaluating an HIV/AIDS
programme based on caring theory. Journal of Clinical Nursing, 1, 265-274.
Shavelson, R. J., & Huang, L. (2003). Responding responsibly to the frenzy to assess learning in higher
education. Change, January/February, 10-19.
Shively, J. (1992). Cowboys and Indians: Perceptions of Western films among American Indians and
Anglos. American Sociological Review, 57, 725-734.
United States Department of Education (2006) Retrieved June 1, 2007, from
http://www.ed.gov/about/bdscomm/list/hiedfuture/reports/final-report.pdf
Waiters, E. D., Treno, A. J., & Grube, J. W. (2002). Alcohol advertising and youth: A focus group analysis
of what young people find appealing in alcohol advertising. Contemporary Drug Problems, 28,
695-718.
Winslow, B. W. (2003). Family caregivers’ experiences with community services: A qualitative analysis.
Health Nursing, 20, 341-348.
Page 14
Wong, S. Y., & Wong, T. K. S. (2003). An exploratory study on needs of parents of adults with a severe
learning disability in a residential setting. Issues in Mental Health Nursing, 24, 795-811.
Endnotes
Page 15
Delphi Method Procedure
Recruit Participants: Focus on those who have graded similar tasks
and are connected with the same general education outcome
Design 1st Survey: Write open-ended questions about student learning
outcomes, asking respondents to write down strengths and weaknesses
Content Analysis: Use “constant comparative method,” (grouping like
responses), and tally like responses and themes
Design 2nd Survey: Report tallied responses (see above), list from top
to bottom in order of frequency, and ask respondents to rate on 4 or 5
point Likert scale.
Analysis: Report means/medians, and standard deviations. Usually,
exclude comments with sd>=1.0, and those below “agree,” according to
specifications of Likert scale
Triangulate: Use with other student learning assessment information,
including standardized test scores, etc.
Report/Discuss: Keep it simple, reporting consensus items, but engage
key audience in discussion, using Delphi results to contextualize other
provided assessment information.
i
“A test of leadership: Charting the future of U.S. Higher Education,” U. S. Department of Education,
http://www.ed.gov/about/bdscomm/list/hiedfuture/reports/final-report.pdf , downloaded June 1, 2007. Comparability of
data has apparently been the greatest issue, in the form of standardized tests, etc. That said, associations linked to higher
education, such as the Commission on Higher Education Accreditation (CHEA), are adamant that assessment be
institution-specific, and identifying standards applicable to all institutions is inadvisable, given that institutions have
different mission and value statements; see “CHEA board issues resolution of support: Reiterates Department of Education
mandates concerning accreditation and learning outcomes, http://www.chea.org/pdf/CHEA_Press_Release_May_2007.pdf
downloaded June 1, 2007.
ii
“Value added assessment: Accountability’s new frontier,” Perspectives, Spring 2006, American
Association of State Colleges and Universities, http://aascu.org/pdf/06_perspectives.pdf, downloaded June
1, 2007.
Page 16
iii
“Trade-School Unit of Major Accreditor Faces Loss of Authority, as Talks on Rules Changes Are Set to Resume, “ Chronicle of
Higher Education, May 31, 2007, downloaded on May 31, 2007.
iv
“Assessing student learning and institutional effectiveness,” Middle States Association of Colleges and Schools,”
http://www.msche.org/publications/Assessment_Expectations051222081842.pdf, downloaded May 31, 2007
v
Suskie, Linda, “What is “Good” Assessment? A New Model For Fulfilling Accreditation Expectations,” notes taken at Assessment
Institute, IUPUI, October 30, 2006.
vi
Ibid.
vii
Ibid.
viii
Ibid. Also see Sean McKitrick “The Politics of Assessment: The Successes and Risks in Using the Delphi Method to Assess Critical
Thinking,” Third International Congress of Qualitative Inquiry, University of Illinois, Urbana-Champaign, May 2-5
ix
Hunt, D. P., Haidet, P., Coverdale, J. H., & Richards, B. (2002) The effect of using team learning in an evidence-based medicine
course for medical students. Teaching & Learning in Medicine, 15, 131-139; Robertson, M. Line, M., Jones, S., & Thomas, S. (2000).
International students, learning environments and perceptions: A case study using the Delphi technique. Higher Education Research &
Development, 19, 89-102; Winslow, B. W. (2003). Family caregivers’ experiences with community services: A qualitative analysis.
Health Nursing, 20, 341-348.
x
Leape, L. L., et al., Coronary Angiography: Ratings of Appropriateness and Necessity by a Canadian panel, (2003), RAND
Corporation; Fitch, et al., The RAND/UCLA Appropriateness Method User’s Manual, (2001), RAND Corporation;
xi
Maxim, Bruce R., “Closing the loop: Assessment and accreditation,” http://delivery.acm.org/10.1145/1050000/1040232/p7-
maxim.pdf?key1=1040232&key2=9988070811&coll=GUIDE&dl=GUIDE&CFID=20204477&CFTOKEN=32209894, downloaded
June 1, 2007.
xii
See Hasson, Keeney, & McKenna
xiii
Hasson, F., “Research guidelines for the Delphi survey technique,” Journal of Advanced Nursing, 32 (4), 2000, 1008-1015; PoIll,
Catherine, “Early indicators of child abuse and neglect: A multi-professional Delphi study,” Child Abuse Review, 12, 2003, 25-40;
Broomfield, D. and G. M. Humphries, “Using the Delphi technique to identify the cancer education requirements of general
practitioners,” Medical Education, 35, 2001, 928-937; Richardson, J., “Developing and evaluating complementary therapy services,”
Journal of Alternative and Contemporary Medicine,” 7(3), 2001, 253-260.
xiii
Hein, S., D. C. Lustig, and A. Aruk, “Consumers’ recommendations to improve satisfaction with rehabilitation services,”
Rehabilitation Counseling Bulletin, 49, 1, pp. 29-39 (2005)
xiii
Kondracke, N. L., N. Wellman, and D. R. Amundson, “Content analysis: Review of methods and their application in nutrition
education,” Journal of Nutrition Education and Behavior, 34(4), (2002), pp. 224-230.
Table 1: Developing Ill-reasoned arguments
(1=strongly agree; 2=agree; 3=neither agree nor disagree; 4=disagree; 5=strongly
disagree)
Open-ended responses N Mean Median Standard
Deviation
Students need to improve in respect to factually 10 1.90 2.00 .568
supporting their arguments/opinions (7 responses)
Students simply report what they learn, rather than 10 2.30 2.00 1.252
using/applying what they learn in their assignments (4
responses)
Students are able to develop Ill-reasoned arguments (4 10 2.80 3.00 .789
responses)
Students had difficulty developing an argument (2 10 3.00 3.00 .943
responses)
Page 17
Students are good at applying general concepts to 9 2.67 2.00 1.00
specific examples or cases (2 responses)
Most students recognize the need for supporting their 10 2.80 2.50 .919
claims with additional information (2 responses)
Students struggled with taking (the) information and 8 2.38 2.00 1.061
incorporating it into Ill-written papers
Students had a hard time writing hypotheses and 6 2.33 2.50 1.211
rationale for the hypotheses
Students shoId a Iakness in analyzing research 8 2.75 2.50 .886
Students Ire Ill prepared for applying the skills of critical 10 3.20 3.50 1.135
thinking
Students Ire Ill prepared for applying the skills of 8 2.88 3.00 .835
questioning
Students Ire Ill prepared for applying the skills of 9 2.89 2.00 1.167
challenging the status quo
The majority of students appear to have a general 10 2.10 2.00 .994
interest in learning
Students are able to identify types of information 5 2.40 2.00 .548
systems used for different kinds of applications
Students are good at analyzing correctly 9 2.89 3.00 .782
Students Ire not able to combine findings from different 9 3.11 3.00 .782
disciplines
Students who had taken one or more English classes 5 1.80 2.00 .837
demonstrated a better understanding of how to analyze a
text, versus merely summarizing a text they had read
Table 2: Identifying, Analyzing, and Evaluating Arguments
(1=strongly agree; 2=agree; 3=neither agree nor disagree; 4=disagree; 5=strongly
disagree)
Open-ended Response: N Mean Median Standard
Deviation
Students are adept at evaluating arguments (2 responses) 9 3.00 3.00 .866
Students are adept at analyzing arguments (2 responses) 10 3.10 3.00 .738
The skills that come with analyzing arguments are just 10 2.70 2.50 1.059
developing for students in the cours(es) I taught
Students’ skills are just developing in respect to evaluating 9 2.56 2.00 .726
arguments
Older students Ire able to evaluate arguments more easily 9 3.00 4.00 1.732
It took a lot of effort for students to realize that they needed to 9 2.67 3.00 1.225
provide actual substantial critiques
Students Ire at a rudimentary stage in learning to write 10 2.80 2.50 1.135
Students had a hard time with critical thinking 10 2.60 2.00 .843
Page 18
Students could write decent expository papers, but getting them 7 2.14 2.00 1.069
to do further analysis was somewhat akin to pulling teeth
Students figured they would pass without putting much effort 10 2.10 2.00 1.101
into performing further analysis
Students had difficulty analyzing others’ arguments 8 2.63 2.50 .744
Students shoId a real keen sense of craft in their research 8 3.50 4.00 .756
papers, using research to back up their claims
Students took issue with the critics they cited, which shoId a 8 3.25 3.00 .886
true understanding of the subjects/issues
Students challenged each others’ opinions 7 2.29 2.00 .448
Students challenged popular opinion 8 2.63 2.50 .744
Students Ire able to identify gaps in theoretical frameworks 7 2.71 2.00 .951
outlined in the texts and other readings
Students Ire able to apply their own research and interests in 7 2.71 2.00 .951
their writing
There is not enough emphasis on critical thinking at this 10 2.10 2.00 .994
university
In general, students exhibit an acceptable level of skill in 9 2.78 3.00 1.093
respect to identifying arguments
Students are better at picking out the Iaknesses in the essays 8 1.63 2.00 .518
they read than in evaluating their own essays
Students are excellent evaluators of their peers’ work 7 3.14 3.00 .690
Students do not have an ability to offer critical feedback 7 3.86 4 .690
Students are able to understand arguments 10 2.20 2.00 1.033
Students have trouble getting beyond the most superficial 10 2.90 3.00 .876
points of an argument
Students do not read assigned materials 10 3.10 4.00 1.197
Students are proficient at pointing out a Iak thesis statement 8 2.75 3.00 .707
Students have difficulty creating an argument and developing it 8 2.13 2.00 .991
in essays
Students understand how to organize essays 8 2.50 2.00 .756
Students have been able to find the Iaknesses in their own work 10 2.70 2.50 1.059
When writing about particularly complex ideas, student have 8 2.63 2.00 .916
trouble organizing their paragraphs; they’ll allow a paragraph to
go on for a page or more because they are not sure how to break
it up
Table 3: Understanding and Using Basic Research Techniques
(1=strongly agree; 2=agree; 3=neither agree nor disagree; 4=disagree; 5=strongly
disagree)
Open-Ended Response: N Mean Median Standard
Deviation
Students’ basic understanding of basic research techniques varies 10 1.40 1.00 .516
greatly from student to student (2 responses)
Students’ ability to use basic research techniques varies from student 10 1.40 1.00 .516
to student (2 responses)
Students have some difficulty correctly citing sources (2 responses) 10 1.30 1.00 .483
Students do not care much about understanding basic research 10 2.60 2.00 1.075
techniques
Student performance depends on how much time they put into their 9 1.78 2.00 .833
papers and projects
Research sessions with librarians help students in their research 5 2.20 2.00 1.225
Page 19
Students need a refresher course on using infoLINK 8 2.00 2.00 .756
Students rarely use database searches 9 2.67 2.00 1.118
Students use Google (or other search engines) instead of library 10 1.40 1.00 .516
resources
Students are able to gather research data from a variety of sources 10 2.80 2.50 .919
Students are able to link data with their opinions to form cogent 9 2.78 3.00 .667
arguments
In general, students have done little research 8 2.38 2.00 .916
Students need to learn more about how to evaluate sources 10 1.60 1.50 .699
Students run into plagiarism problems because they do not understand 10 2.30 2.00 1.252
how to format citations
Students know how to format citations 10 3.80 4.00 .789
Students have little practice using the library’s online databases 10 2.10 2.00 .738
Students can track down book reviews, interviews with authors, etc. 7 3.29 4.00 .951
Table 4: Locating, Evaluating, and Synthesizing Information
(1=strongly agree; 2=agree; 3=neither agree nor disagree; 4=disagree; 5=strongly
disagree)
Open-Ended Responses: N Mean Median Standard
Deviation
Students need to improve in respect to synthesizing ideas 10 1.80 2.00 .422
(4 responses)
Students are able to locate information 10 2.60 2.50 .699
Students are able to evaluate information 10 2.90 3.00 .738
Students’ struggles with the synthesis of information is 9 3.33 4.00 1.658
limited to their writing
Students would be good at all of these if they devoted 10 3.00 3.00 1.247
sufficient time
Page 20
Having research sessions with librarians was helpful in 5 2.00 2.00 1.225
respect to students’ being able to locate information for
their research
Students are able to gather research data from a variety of 10 2.80 2.00 1.033
sources
Students are able to link the data they research with their 10 2.80 2.50 .919
opinions to form acceptable arguments
Locating appropriate information was underdeveloped in 10 2.30 2.00 1.059
students
Evaluating information is underdeveloped in students 10 1.80 2.00 .919
Students performed satisfactorily in respect to locating a 10 2.90 3.00 .738
variety of source materials
Students struggle with research-based papers 9 2.22 2.00 .833
Students feel they need to include everything they find to 9 2.11 2.00 1.167
the exclusion of their ideas to the extent that the paper
becomes a research dump instead of an argument on their
own
Students Ire able to develop acceptable bibliographies 10 3.60 4.00 .966
Students Ire able to perform searches using the 8 2.50 2.00 1.069
university’s library catalog
Students Ire able to evaluate internet sources 9 3.44 4.00 1.130
Students rely too much on Ib pages as sources 10 1.30 1.00 .483
Students seem to be unaware that a serious research 10 2.30 2.00 1.252
paper cannot be completed using only internet sources
Students are unfamiliar with locating and using journal 10 2.20 2.00 1.135
articles
In general, students are unfamiliar with library resources 10 2.30 2.00 1.059
xiv
Hein, S., D. C. Lustig, and A. Aruk, “Consumers’ recommendations to improve satisfaction with
rehabilitation services,” Rehabilitation Counseling Bulletin, 49, 1, pp. 29-39 (2005)
xv
Kondracke, N. L., N. Wellman, and D. R. Amundson, “Content analysis: Review of methods and their
application in nutrition education,” Journal of Nutrition Education and Behavior, 34, 4, (2002), pp. 224-
230.
Related docs
