An Introduction to ... - Humanities Graduate Studies - Curtin University by ktixcqlmc


									An Introduction to
Qualitative Research

Barbara Pini
John Curtin Institute for Public Policy
Curtin University of Technology

   Part One: Research Process and Design
   Part Two: Sampling
   Part Three: Documentary Research
   Part Four: Participant Observation
   Part Five: Interviews
   Part Six: Focus Groups
   Part Seven: Mixed Methods (Case Studies, Ethnography and Action
   Part Eight: Qualitative Data Analysis
   Part Nine: Trustworthiness in Qualitative Research
   Part Ten: Ethics and Qualitative Research

 Research Process and Design
What is Research?

   “A systematic process of critical enquiry
    leading to valid propositions and
    conclusions that are communicated to
    interested others” (McLeod, 1994).

   What are some of the key words in this definition and
    why are they important?
Research Proposal
Why do we need it?

   At Curtin University you will need to produce a
    written document outlining your research for the
    Faculty of Humanities. This is called an
    Application for Candidacy.
   Such proposals are very common in academia
    (for funding, ethics approval etc).
Components of Research Proposal






Taking A Closer Look at Methods

     What methods will you use to address the research questions?
     How many and why this many? (sampling)
     How will these methods be designed? i.e. How will the study be
      conducted? Where? How will you gain access?
     What is the justification for these methods?
     What questions will be asked and why?
     What are the limitations of these methods and how will you address
      these limitations?
     How will analysis be undertaken?
     What are the ethical concerns related to these methods and how will
      these be addressed?

        All the methodological decisions you make – i.e. how you answer each
        of the above questions should be tied to the methodological literature
        and/or the literature in your subject area.
Deciding on a methodological
   Ontology: What is the nature of the phenomena,
    or social reality, that you want to investigate?
   Epistemology: What might represent knowledge
    or evidence of the social reality that you want to
   Research area: What topic is the research
    concerned with?
   Research Question: What do you wish to
    explain or explore?
   What is the nature of things in the social world?
   For example, are you investigating:
        Bodies, subjects, objects
        Rationality, emotion, thought
        Feeling, memory, senses
        Motivations, ideas, perceptions
        Attitudes, beliefs, views
        Texts, discourses
        Cultures, society, groups
        Interactions, social relations
   Some ontologies are better matched to qualitative research
    methodology than others (e.g., social processes, interpretations,
    social relations, experiences etc.)
   What is your theory of knowledge? What are your presuppositions about
    the nature of knowledge?
   Examples of epistemological perspectives
      Positivist Perspectives (also called empiricism)
           Fundamental claim is that reality is a fixed, measurable entity that
            is external to people.
           There exist “social facts.”
           Aims to to find true, precise and wide-ranging laws of human
            behaviour which we can generalise to the population as a whole
           “If it can‟t be measured, it doesn‟t exist.”

      Social Constructionism
           Reality is constructed socially so rejection of “social facts”

           Aim is to describe the subjective and consensual meanings that
            constitute social reality.
           Understanding of social world as “local truths” which cannot be
            evaluated by external criteria
Qualitative and Quantitative
  Qualitative                           Quantitative
   Makes less use of mathematical       Employs statistics or other
    techniques.                           mathematical operations to
   Focus on interpretation by            analyse data
    researcher                           Concepts are assigned numerical
   Systematically arranging and          values
    presenting information to search     Collects a small amount of data
    for meaning in data collected         from a large number of people
   “Words, not numbers”                 Allows generalisation to wider
   Usually involves a philosophical      population
    stance that human knowledge is,
    to some extent, contextualised or
   But some form of counting is
    almost always involved in
    qualitative analysis.
Strengths of Quantitative Research

 It can deal with large numbers of cases
 It is capable of examining complex
  patterns of interactions between variables
 It can make possible the verification of the
  presence of cause and effect relationships
  between variables
Weaknesses of Qualitative
   Lack of in-depth information
   Ignores individual perspectives and experiences
   Limited with topics we know little about
   Can be built on pre-existing biases of the researcher
   The case of questionnaires:
        Language used
        Ordering of questions
        Forced response formats; what if „it depends…‟?
        Missing data
        Sampling issues
        Response rates
        Lies, lies and damn statistics; torturing your data until it confesses
Strengths of Qualitative Research
   Research done in natural settings
   Emphasis on informant interpretations and
   Seek deep understanding of informants world
      “Thick Description” (Clifford Geertz)
   Humanising research process by raising the role
    of the researched
   High levels of flexibility in research process
Weaknesses of Qualitative
   Problems of reliability - The difficulty of replicating findings
   “Subjectivity” of nature of data collection and analysis
   Observations may be selectively reported making it impossible to
    gauge the extent to which they are typical
   Risk of collecting meaningless and useless information from
   Problems of objectivity vs detachment (particularly in participant
    observation but also applies to other methods)
   Problems of ethics: Entering the personal world of the participant
   Very time consuming

Why Sample?
   1. Generalisability
     To  generalise the properties of the sample to the
       wider population
      To make conclusions regarding the wider population
   2. Pragmatic reasons
      applied research in organisations often has resource
      sampling reduces burden on resources (i.e. time and
      usually not feasible to contact the whole population
   3. Destruction of test units
      some research projects (e.g. quality testing) require
       the destruction of the items being tested e.g. testing
       cars for safety
   Important issues:
     properly  selected samples are sufficiently
      accurate in most cases to make statements about
      the population
     even when population has considerable
      heterogeneity, larger samples can provide data of
      sufficient precision upon which to base
     In quant research, the characteristics of the
      sample and actual sample size is more important
      than the relative size of the sample compared to
      the population
     In qualitative research, aim is not statistical
      representativeness, but representativeness in the
      sense of gaining access to the full range of views,
      themes or possibilities in the population
Ensuring Representativeness
   Once the decision to sample has been made, the
    researcher must identify the target population

   Must carefully define the target population so that the
    proper source from which to collect the data can be

   Population
      any complete group sharing some common set of
      can be finite or infinite
      the group we wish our research to comment upon
Types of Sampling
Probability Sampling
   Probability sampling

     facilitates generalisability
     occurs when elements in the population have an
      equal probability of being selected in the sample
     logic depends on selecting a truly random and
      statistically representative sample that will permit
      confident generalisation from the sample to a larger
     best for quantitative research
Types of sampling
Probability sampling
   How do you select a sample that will look like
    the population in terms of its demographics?

     Simple   random sampling
     Systematic sampling
     Stratified random sampling
     Cluster sampling
     Area sampling
     Double sampling
        Advantages and Disadvantages of
        Sampling Methods
   Simple random sampling
      high generalisability of findings
      not as efficient as stratified sampling

   Systematic sampling
      easy to use
      systematic biases are possible

   Stratified random sampling
      most efficient and precise
      would adequately represent strata with low numbers
Advantages and Disadvantages of
Sampling Methods
   Cluster sampling
      goal is to reduce costs of data collection
      the least reliable among all the probability designs

   Area sampling
      type of cluster sampling
      cost-effective. Useful for decisions regarding location

   Double sampling
      offers more detailed information on the topic of study
      original biases, if any, will be carried over
Types of sampling
Non-probability sampling
   Non-probability sampling
      occurs when elements in the population do not have a
       pre-determined probability of being selected in the
       sample ie non-random
      logic depends on selecting cases rich in information
       that will permit an in-depth understanding of the
       research question
      often used in qualitative research
      an un-representative sample might be a useful and
       more stringent test for a law-like hypothesis
           E.g. Does wealth = health?; sample the very rich who should
            be extremely healthy and the very poor who would be
            extremely unhealthy
           E.g. Helping behaviour and number of onlookers?; sample
            disaster situations and assess whether „law‟ holds
      Convenience Sampling

   Convenience sampling
     e.g., interviews on the street; simply asking for
      volunteers; using clients in clinical or business
     quick, convenient, less expensive
     not generalisable at all
Purposeful Sampling
   Purposeful sampling or Judgment sampling
     sometimes the only meaningful way to investigate
     Useful when you need a targeted sample
   Includes:
     Snowball sampling
        Starting with a small group and asking for further contacts

        Useful for sensitive topics

     Quota sampling
        Population is stratified and numbers within strata are decided

        Contacts are made until quotas are full

        Quotas can be proportional or non-proportional to the
Sampling Criteria
   The appropriate sample design will depend on the
    following criteria:

     degree    of accuracy required

     local   versus national project

     need    for statistical analysis

     resources    (ie time / money)
Sample Size in Qualitative Studies

   Adequacy of sample depends not so much
    on the number of cases

   Depends on the proper specification of the
    cases to be analysed

   Redundancy in information is often a sign
    that the sample size is adequate

     Documentary Research
“The good stuff of social science”
(Ryan & Bernard 2003)

   Definition
   Types of documents
   Classifications
   Advantages and disadvantages
   Analysis
   Ethics
   Activities: Journal Article and Document
  A document in its most general sense is a written
   text…Writing is the making of symbols representing
   words, and involves the use of a pen, pencil, printing
   machine or other tool for inscribing the message on
   paper, parchment or some other material
   medium…Similarly, the invention of magnetic and
   electronic means of storying and displaying text should
   encourage us to regard „files‟ and „documents‟ contained
   in computers and word processors as true documents.
   From this point of view, therefore, documents may be
   regarded as physically embodied texts, where the
   containment of the text is the primary purpose of the
   physical medium
(J. Scott 1990: 12-13)
So…this includes…
   Photographs
   Videos / film footage
   Political speeches
   Minutes of meetings
   Plays, novels
   Media sources
   Personal documents such as diaries, oral histories
   Emails
   [Visual]

     Any other suggestions?
Classification of Documents

   Primary
     Writtenor collected by those who actually witnessed
      the events they describe.
   Secondary
     These  are written after an event which the author had
      not personally witnessed.
   Tertiary
     These enable us to locate other sources. They are
      indexes, abstracts, bibliographies etc.
Other Typologies for classifying Documents
   John Scott (1990) divides documents into four
    categories according to the degree of their
     Closed
     Restricted
     Open-archival
     Open-published
   Public and Private documents
   Solicited and unsolicited documents (Some
    documents are produced with the aim of
    research in mind, whereas others would have
    been produced for alternative uses).
Advantages of Documentary Analysis

 Cost-effective
 Permanence – particularly for past
 Access is usually relatively easy
 Provides understanding of certain
  phenomena that is rich in detail and
Methodological Problems

   Authenticity of documents
   Secondary data
      When researchers use documents as a source of data, they generally
        rely on something which has been produced for other purposes and not
        for the specific aims of the investigation.
   Credibility
      What purpose was the document written to? Who produced the
        document? What was the status of the author? When was the document
   Representativeness and bias
      Is the document typical of its type? Does it represent a typical instance
        of the thing it portrays? Is it complete? Has it been edited?
   Understanding meaning
      Is the meaning of the words clear and unambiguous? Are their hidden
   Reading/legibility
   Incomplete sources
   Gaining access (restricted documents)
Analysing Documents
   Quantitative approaches
          Content analysis
   Qualitative approaches
        “Reading” the text
        An understanding of the context in which it was
        Examining symbols, hidden meanings

        What is not contained in the text? What does this
   Depictions of work in family-genre movies

     Participant Observation
 By participant observation we mean the
 method in which the observer participates in the
 daily life of the people under study, either openly
 in the role of researcher or covertly in some
 disguised role, observing things that happen,
 listening to what is said, and questioning people,
 over some length of time.

(Becker and Geer 1957: 28)
Origins and Links
   Origins in ethnography
   Linked with epistomological orientations of
    ethnomethodology and grounded theory (these methods
    entail naturalistic investigations of culturally contexted
    social processes).
   Pseudo-objective stance of the researcher has largely
    been abandoned in favour of more personal and
    subjective accounts of the participant observation
    experience (see Tedlock, 2000)
   Traditionally this method has been paired with interviews
    and document analysis, and more recently with digital
When to use participant observation

   Participant observation is especially appropriate for
    scholarly problems when:
       Little is known about the phenomenon (a newly formed
       There are important differences between the views of insiders as
        opposed to outsiders (e.g. labour unions and management)
       The phenomenon is somehow obscured from the view of
        outsiders (mental illness, family life, private interactions)
       The phenomenon is purposefully hidden from public view (crime
        and deviance, secretive groups)
Strengths of participant observation

   Natural/unobtrusive.
   Requires little more than self
   Can produce rich insights into complex realities
   Context specific and flexible
   Holistic. Can incorporate relationships between factors
    (people, settings, documents).
   Provides insight into actors‟ meanings as they see them
   Offers advantage of serendipity

(See Dennis 1993).
Limitations of participant
   Access. Limited options open to the researcher
    about which roles to adopt or settings in which to
   Commitment. Demanding method and
    significant personal resources.
   Danger (potentially)
   Reliability
   Observer effects
   Representativeness of data. Difficulty of
    generalising from data
   Ethical issues
Easy or Difficult?
This method (participant observation) is one that
  those new to social research believe they can
 undertake with ease. On first glance it appears
   to be just about looking, listening, generally
 experiencing and writing it all down. However, it
     is more plausible to argue that participant
  observation is the most personally demanding
      and analytically difficult method of social
               research to undertake
                 (May 2001: 153).
Strategies to overcome limitations

 Use multiple observers or teams
 Search for negative cases
 Spend an extended time in the field
 Use insider checking
 Use outsider checking
 Repeat observations under varying conditions
 Be meticulous in recording observations
(Alder and Alder 1994)
Participant Roles
   Complete   participant

     Enter the field under pretence or deception
     Engages fully in the activities of the group or
      organisation under investigation
     Advantages are that it can produce more
      accurate/authentic information and an
      understanding not otherwise available
     Problem of recording observations

     [Visual]

    Gold (1958)
Types of Participant Observation
   Participant as observer                   Observer as participant
           Enter the field setting with              Strictly speaking this
            an openly acknowledged                     would not be regarded as
            investigative purpose.                     participant observation
           Develop relationships with                No lasting contact with
            subjects                                   people
           Problem of „going native‟                 Focus on observation, not
            but dismissed by some                      on interaction with people
            (e.g. May 2001)                           Problem is that it does not
           May encounter hostility –                  utilise the strengths of the
            particularly in early stages               time in the field to deepen
            of research                                understanding
           Problem of
            disengagement from field
           [Visual]
Participant Roles
   Complete   observer

     Also a non-participant role
     Role completely removes the researcher from
      observed interactions
     Epitomised by laboratory experiments

     [Visual]
Stages in Participant Observation
Denzin (1989)
   Before actual field contacts and observations begin, a
    general definition of research problem is identified.
   Select field setting.
   Make initial contact and establish access.
   Collect descriptive data on setting and participants.
   Field work progressing. Informants selected, approached
    etc. Early theoretical formulations tested.
   General categories for data analysis are developed.
    Refining observations.
   Complex set of propositions developed and tested.
   Conclusion of study. Role disengagement. Writing of
Recording Observations
 Spradley (1980) and Jorgensen (1989)
  discuss initial observations as primarily
  descriptive, unfocused and general.
 After observers become more familiar with
  their setting and grasp the key aspects of
  this setting their observations will become
  more focused and selected.
Recoding Observations
   The participants: Who are the participants? How many are there? How can
    they be characterised (gender, occupations etc) Where are they situated in
    relationship to each other? Are there any key groupings or relationships?
   The tasks: What are the functions of the various groups of people? How are
    they relating in this setting? What are they doing during the key events or
    observations? Are these functions formally defined? Do individuals and
    groups have a variety of purposes for being there? Are there conflicting
    goals of various groups or individuals? What are these conflicting goals?
   The setting: Each setting has unique features. What are these? Equipment?
    Resources? Facilities? Use your senses.
   The behaviour and the outputs: How do people actually behave during the
    event? Describe this behaviour in descriptive terms. What are the specific
    movements made and activities that are carried out?
   Timing: The timing of the behaviour is described by the time it occurred, the
    time it takes, and the frequency.
   Unique causes or consequences: What unique occurrences affected the
    people, tasks, setting, behaviours, output and timing?

Cunningham (1993:141)
Video recording
   A “privileged gaze” (Atkinson & Hammersley,
   Purposes (Paterson, Bottorff & Hewat, 2005):
       Allows decontextualised sequencing of minute behaviours,
        concurrent behaviours, nonverbal behaviours and conversational
        analysis that are difficult to observe in real time
       To document the research process and check for observer effects
       To direct methodological decisions
       To enhance the validity of the researcher‟s interpretation of
   Compared to participant observation where video-
    recordings are not used, relationships less
    important for the collection of data, but more
    important for getting consent to participate
Participant Observation and Ethics

 Disguised or covert research has come
  under significant attack.
 Boundaries between covert and open
  research are not necessarily clear cut.
 Protection of informants/respondents
 Why have people shared information?
 Ethical dilemmas do not cease when you
  leave the field.

Interviews 1
   Defining Interviews
   Types of Interviews
   Advantages and
   Design questions
   Sampling issues
   Types of interview questions
   Interview skills
Defining Interviews
 A conversation with a purpose (Kahn and
  Cannell 1957:149)
 Silverman (1993) talks about us living in
  an „interview society‟
 Estimated that 90 per cent of all social
  science investigations use interviews in
  one way or another (Briggs 1986)
Types of Interviews - Structured
    Many are formally structured.
    Associated with questionnaire research (oral questionnaire); also
     used in some job interviews
    Each person asked the same question in the same way so that
     any differences between answers are held to be real ones and
     not the result of the interview situation itself.
    No deviation from question order or wording of questions.
    No adjusting for level of language.
    No clarifications or answering of questions about the interview
Types of Interviews: Semi-structured

   Questions    are normally specified, but the interviewer
    is freer to probe beyond the answers.
   Questions may be reordered during the interview.
   Level of language may be adjusted.
   Interviewer may add or delete probes.
   Allows people to answer more on their own terms, but
    still provides a structure for comparability.
   Sometimes called semi-standardised.
   Most typically used in qualitative studies (Rossman
    and Rallis 1998: 124)
Types of Interviews: Unstructured
   Includes  life-history, biographical and oral
    history interviews
   Sometimes called informal, non-standardised
   Provides qualitative depth in allowing subject
    to talk about topic within their own frame of
 “Increasingly, qualitative researchers are
  realizing that interviews are not neutral
  tools of data gathering but active
  interactions between two (or more) people
  leading to negotiated contextually based
  results” (Fontana and Frey 2003: 62).
 Impact of identities of researcher and
  participants should be considered
Advantages of interviews
     One of the most flexible/responsive methods available as
      different types of interviews can be engaged for different
      research problems.

     Ability to explore additional research questions / issues if they
      arise (semi-structured / unstructured only)

     Ability to gain rich and descriptive data; ideally suited to
      examining topics in which different levels of meaning need to
      be explored.

     Most participants will accept an interview readily. They are
      likely to be familiar with interviews.

     Ability to follow up research participants for clarification or
      further exploration
Disadvantages of interviews
    Bias and subjectivity which, in turn, affects validity and reliability
     of data

    Generalisation problem

    Process of data collection, transcribing and analysis from each
     participant time-consuming; thus, sample size generally not

    In reporting results, tendency of researchers to focus on quotes
     which are dramatic, unusual or interesting, rather than typical
Design Questions

   The recommended duration for an in-depth interview is
    one hour and a half, but may be varied according to the
    situation and respondent (Burgess 1984, p.120).

   A write-up of observations may be completed following
    each of the interviews (Burgess 1984, p.119).

    Increased rapport is likely to be facilitated through
    follow-up visits which also will improve the quality of data
    produced (Whyte 1984, p.114; Lee 1993, p.113).
What Counts as Data?
   Utterances only?
   Non-verbal aspects of the interaction?
   Written notes / tape-recordings?
   My own memories and unwritten interpretations
    of the interview?
   Diagrams, pictures, drawings, charts and
    photographs produced during the interview?
   NB: Absolute objectivity is a myth!!!
    Researchers continually make judgements
    about what to write down or record, what they
    have observed, heard and experienced and
    what they think it means (Mason, 2002).
Sampling (as per Part 2)

   Specifically, Minichiello et al. (1995, p.
    162) describe the process as it applies to
    in-depth interviewing as „selecting
    informants on the basis of relevant issues,
    categories and themes which emerge in
    the course of conducting the studies‟.
Types of Interview Questions
(Kvale 1996: 133)
 Introducing questions
     E.g. “Can you tell me about…”? Etc.
 Probing questions
     E.g. “That‟s interesting. What else can you tell me about…”?
 Specifying questions
     E.g. “Can you give me an example of…”?
 Direct questions
     E.g. “Earlier you said… How does that relate to…”?
     These may need to come later in the interview; may be slightly
       confrontational or ask for clarification of discrepant information
 Indirect questions (useful when trying to avoid social desirability bias)
     E.g. “What should someone else in that situation do…”?
 Structuring questions
     E.g. “I would now like to introduce a new topic…”
 Silence – just a nod or a pause
 Interpreting questions
     Rephrasing an answer, more speculative questions
     E.g. “So does that mean…”?; “Are you saying…”?; “Would I be right in
       interpreting that as…?”
Interview Skills
   The good interviewer needs to be attentive.
   The good interviewer is sensitive to the feelings
    of the informant.
   The good interviewer is able to tolerate silence.
   The good interviewer is adept at using prompts.
   The good interviewer is adept at using probes.
   The good interviewer is adept at using checks.
   The good interviewer is non-judgemental.
                 (Denscombe 1999:135)

      Focus Groups
   Definitions
   History
   Common uses
   Advantages and limitations
   Interviews versus focus groups
   Recruiting for a focus group
   The role of moderator
   Analysing focus group data
   Ethics of focus group research
   The future?
   Activity

 „The hallmark of focus groups is the
  explicit use of the group interaction to
  produce data and insights that would be
  less accessible without the interaction‟
  (Morgan 1988: 12).
 Kitzinger (1994: 159) „group discussions
  organised to explore a specific set of

   Originally called focussed interviews.
   Origins in the Office of Radio Research at Columbia
    University in 1941 when Paul Lazarsfeld invited Robert
    Merton to assist him in the evaluation of audience
    response to radio programs.
   Method increasingly used in social sciences and
    marketing (Catterall and Maclaran 1997; Green 1999).
   Morgan (1988) says most common form of marketing
Common uses of focus groups
   Obtaining general background information about a topic
    of interest
   Generating research hypotheses that can be submitted
    to further research and testing using more quantitative
    approaches (Stimulating new ideas and creative
   Diagnosing the potential for problems with a new
    program, service or product
   Generating impressions of products, programs, services,
    institutions or other objects of interest.
   Learning how respondents talk about the phenomenon
    of interest. This, in turn, may facilitate the design of
    questionnaires, survey instruments or other research
    tools that might be employed in a research project.
   Interpreting previously obtained research results.
Advantages of Focus Groups (Quible 1998,
Albrecht, Johnson and Walther 1993; Stewart and
Shamdasani 1990).
   time and cost efficient
   direct interaction between researcher and researched, respondents can
    qualify responses, researcher can observe non-verbals
   large amounts of rich data in the respondents‟ own words. The researcher
    can obtain deeper levels of meaning etc.
   synergistic
   flexible
   especially useful for groups with limited literacy,
   results are readily understood.
   synergism
   snowballing
   stimulation
   security
   Spontaneity
Disadvantages of focus groups
(Quible 1998, Albrecht, Johnson and Walther 1993; Stewart and
Shamdasani 1990).

   Small number of respondents and convenience recruiting limit
   Responses may be subject to group-think, especially if there are
    dominated or opinionated members. More reserved members may
    be overlooked (see MacDougall and Baum 1997).
   The open-ended nature of responses may make summarisation and
    interpretation difficult.
   Potential for moderator bias
   Cost (moderator fee, facility rental, recording and transcribing, data
    analysis, participant incentives)
   Subjects‟ conformity
Designing and Conducting Focus
 “The experience of using the focus group as
   a qualitative research method can be
   compared with that of the tightrope walker:
   when things go well there is a feeling of
   exhilaration, when they go badly….it‟s a
   long drop!”

              Pugsley 1996:126
 Interviews and Focus Groups
Focus groups are not appropriate when:

  1. Detailed probing of an individual‟s behaviour, attitudes or needs is

  2. The subject matter under discussion is likely to be of a highly
  confidential nature

  3. The subject matter is of an emotionally charged or embarrassing

  4. Certain strong, socially acceptable norms exist and the need to
  conform in a group discussion may influence response

  5. A highly detailed (step-by-step) understanding of complicated
  behaviour or decision-making patterns is required

  6. The interviews are with professional people or with people on the
  subject of their jobs.(Hawkins et al, 1994; 554-444).
Steps in Design and Use of Focus
   Problem definition/formulation of research
   Identification of sampling frame
   Identification of moderator
   Generation and pre-testing of interview guide
   Recruiting the sample
   Conducting the group
   Analysis and interpretation of data
   Writing report
Designing Focus Groups: How do you
recruit participants?
   Time-consuming
   Krueger (1988: 94) refers to „recruiting on
   Morgan (1995) says recruitment is the single
    most common source of failure he has
    encountered in focus group research.
   How to avoid problems: repeated contacts,
    offering incentives, over-recruiting. (Morgan
    1988 suggests over-recruiting by 20%).
   May be recruited by existing social networks,
    word of mouth or advertising.
Designing Focus Groups: How many people in a group?

   Literature differs but researchers highlight that size should be
    related to research topic/purpose
   Group sizes of 4 to 12 are recommended with an ideal group the
    size of 8 (Morrison & Peoples, 1999; Diloria 1994 et al.)
   6 to 9 (Garrison et al. 1999)
   Generally 8-12 individuals (Stewart and Shamdasani 1990)
   6 to 10 (MacIntosh 1993)
   Up to 15 (Goss and Leinbach 1996)
   Smaller groups may be dominated by one or two members
   Larger groups may be difficult to manage, obtain the perspectives
    of all members
Designing Focus Groups: Who should make
up your focus groups? (Sampling as Part 2)

   The issue is sample bias not generalisability (Morgan
   Typically use purposefully selected samples.
   Heterogeneous or homogenous.
   Do you want to make comparisons between different
   Key question: Would these groups normally discuss the
    topic in day-to-day interaction? (Morgan 1988).
Designing Focus Groups: How many
   Depends on approach; research questions; time and budget

   Some use only one meeting with each of several focus groups (e.g.
    Burgess 1996)

   Others use follow-up meetings (e.g. Pini 2002)

   Multiple groups of similar participants are usually necessary for data
    to be valid

   Most questions can be answered by 6 to 8 groups, although 4
    groups are adequate for some studies and 50 are needed for more
    extensive studies (Krueger 1994)

   One important determinant is the number of different population sub-
    groups required (Morgan 1988)
Designing Focus Groups: How long
should they last?

   One and a half to two and a half hours (Stewart and
    Shamdasani 1990)

   Consider moderator as well as participant
    fatigue. Keim et al (1999) study used one hour groups
    for children and found this was too long.
Designing Focus Groups: Developing a
focus group guide
   In general, keep the number of broad concepts
    examined in a focus group moderate so that
    each can be examined in detail.
   Tend to be general in nature and open-ended.
   Moderator will be improvising comments and
    questions within the framework.
   Opening question is one that everyone answers
    at the beginning of the focus group.
Designing Focus Groups: What is the
role of the moderator?
   Smith (1995) recommends two moderators for
    better control of group cohesion and more
    thorough observation of group dynamics.
   Morgan (1988: 49) favours approach he calls
    „highly nondirective focus groups‟ or what he
    says are „self-managed groups‟.
   Moderator needs to have both strong
    interviewing and observational skills (McDonald
What is the role of the moderator?
   Consider advantages of high moderator involvement:
      Can cut off unproductive discussion
      Ability to ensure all topics covered
      Can adjust discussion
   Consider disadvantages of high moderator involvement
      A biased moderator will produce data that reproduces these
      Does not allow new / unanticipated issues to emerge
   Consider advantages of low moderator involvement
      Can assess participants‟ own interests
      Participants can bring up controversial topics/topics not
       considered by moderator
   Consider disadvantages of low moderator involvement
      Relatively disorganised in content and so more difficult to
      Some topics may never come up
Analysing Focus Group Data
   A typical two hour session yields an average of 40 to 50
    transcript pages.
   Morgan (1988: 64): The group is the fundamental unit of
    analysis and the analysis should begin in a group-by-
    group progression.
   Hyden and Bulow (2003) stress the need to examine not
    only pure content, but who is saying what
   Krueger (1993) says read transcripts/summaries and:
           Consider the words
           Consider the context
           Consider the internal consistency
           Consider the specificity of responses
           Find the big ideas
           Consider the purpose of the research
Krueger (1993) Quality control in focus group
In D. L. Morgan, Successful Focus Groups, Sage, Newbury Park, 65-85

 Ten quality factors in focus group research:
  Clarity of purpose
  Appropriate environment
  Sufficient resources
  Appropriate participants
  Skilful moderator
  Effective questions
  Systematic and verifiable analysis
  Appropriate presentation
The future?
Emerging literature on virtual focus groups.

   Who and what are being researched in online
    focus groups?
   Are online groups going to replace traditional
    focus groups?
   Are respondents who they say they are?
   Do respondents in online groups really interact
    with each other?

(See Sweet, 2001; Murray 1997; Walston and Lissitz 2000. )

      Mixed Method Approaches to
      Qualitative Research
Integrative Models
   Model 1
     Quantitativeor qualitative approach is used
      independently of the other
   Model 2
     Qualitative
              approach is used to develop quantitative
      measurement scales
   Model 3
     Qualitative   approach is used to interpret quantitative
   Model 4
     Quantitative   approach is used to interpret qualitative
Specific „Mixed-Methods‟
Approaches to Research
 We have covered the 4 major
 qualitative methods, but some
 recognised methods combine these
 Case  study method
 Action research
Case Study Method
 Distinct from “a case” (the object of study)
 Features (Yin 2002)
   Single example of a phenomenon of interest
    (organisation, part of an organisation)
   May involve more than one „case‟ (multi-site study)
    but comparisons between them are a feature of the
    research (separately identifiable)
   May also only involve a single case (within-site study)
   Used in law (illustrative cases), health (unusual or
    interesting illnesses), psychology (Freud), political
    science (case reports) and business (organisations
    with defined features)
Case Study Method (cont.)
 May be qualitative or quantitative or both,
  but relies on multiple sources of evidence
  where data triangulates in a converging
 For qualitative case studies, Yin (1989)
  suggests 6 types of information:
     Observations,  interviews, audio-visual
     material, documents, archival material,
     physical artifacts
Case Study Method (cont.)
   Challenges in case study method:
     The number of cases selected – the more cases, the
      more diluted the overall analysis. Typically no more
      than 4
     The issue of Single case study research
     Choosing the case(s) – strong rationale for purposeful
      sampling strategy is important
     Deciding the „boundaries‟ of a case – how it might be
      constrained in terms of time, events and processes
     Presents general propositions but not broadly
      generalisable, but should it be?
Debates on Generalisability
   Critique of the importance and goal of
    generalisability (the discovery of laws, Lincoln &
     Attribute the belief to positivism
     Critique the view that we can produce   knowledge that
      is free of time and context
     Argue that the choice is not about searching for
      general laws OR studying the unique, but something
      in between
     i.e. stating conclusions from studying one context that
      might hold in another context, „working hypotheses‟,
      the „fit‟ between one case study and another,
      generalising not about what is, but what may be or
      what could be
   Genesis in cultural anthropology
   Argued to be not one particular method but a style of
    research that is distinguished by its objectives
       To understand social meanings and activities of people in a
        given setting
   Some overlap with participant observation as this is the
    predominant technique used. However, interviews and
    documentary methods also often utilised.
   Definition:
       “a description and interpretation of a cultural or social group or
        system” (Cresswell 1998)
   Mostly used in anthropology and sociology, but also
    health sciences, education, rarely in business
Ethnography (cont.)
   Has a number of features distinct from other
     Sees the world through the eyes of those being
      researched, allowing them to speak for themselves
     Researchers immerse themselves in the setting and
      become part of the group in which they are interested
     Aims to provide understanding of the meaning and
      importance that members of the group impart to their
      own behaviour and that of others
Ethnography (cont.)
   Key terms
     Fieldwork   – collecting data in a particular
     Gatekeepers
     Key informants
     Reciprocity
     Reactivity or reflexivity
Action Research – a „participatory
approach to enquiry‟
   Definition:
       “Disciplined enquiry (research) which seeks focused efforts to
        improve the quality of people‟s organisational, community and
        family lives” (Calhoun 1993).
   Key tenets:
       Processes are rigorously empirical and reflective (research is
       Research engages people who have traditionally been called
        “subjects” as active participants in the research process
       Research results in some practical outcome related to the lives
        or work of the participants
       Democratic, equitable, liberating, life enhancing
       Operates at intellectual, as well as social, cultural, political and
        emotional levels
Action Research (cont.)
   Has much in common with community
    development and practitioner research
   Routine is look, think, act… or observation,
    reflection, action…
   However, not linear, neat or orderly, rather
    routine can work backwards, in repetition and
    revision, can leap frog stages and sometimes
    make radical changes in direction

     Qualitative Analysis
Qualitative Analysis
   “If the sociologist or the biographer is like a detective,
    and collecting data is like detection, then analysing data
    is akin to the culminating stages of the criminal justice
    process. It has the same potential for abuse, and
    therefore requires similar safeguards. Unfortunately,
    whereas in criminal justice the adversarial roles of
    prosecution and defence can be allocated to different
    people, in qualitative analysis the analyst often has to
    play both roles”
                                                      Dey, 1993
Your approach to qualitative data analysis will be informed by your
epistemological stance (Chua, 1986)

   Three major philosophical positions in qualitative
   1. Positivist
       Evidence of formal propositions, quantifiable measures of
        variables, drawing of inferences about a phenomenon from a
        representative sample to a stated population
       Eg Content analysis – simply counting words / phrases (e.g.,
        political speeches; media articles)
       Relational content analysis – more in-depth; considers meanings
        of excerpts and the relationship between them
   2. Critical
       Main task is social critique; helps to eliminate the causes of
        unwarranted alientation
Approaches to qualitative data analysis

   3. Interpretive
        Knowledge is gained through social constructions such as
         language, consciousness, shared meanings etc. Does not
         predefine dependent and independent variables; seeks to
         understand phenomena through the meanings that people assign
         to them
        E.g. Grounded theory – inductive, theory-building approach
        Narrative analysis – preserves the story rather than fragmenting
        Phenomenological approaches – particularly concerned with
         generating meanings and gaining insights into phenomena
        Discourse analysis – assumptions and meanings underlying
         spoken language, „main line story‟
        Conversation analysis – highly specialised, based on linguistics
         (Atkinson & Delamont argue divorced from wider issues such as
         identity, interactions and social encounters)
Interpretivist Approaches
   “Interpretivist positions are concerned with how
    the social world is interpreted, understood,
    experienced, produced or constituted. While
    different versions of qualitative research might
    understand or approach these elements in
    different ways (e.g. focus on social means, or
    interpretations, or practices, or discourses, or
    processes, or constructions), all will see at least
    some of these as meaningful elements in a
    complex – possibly multi-layered and textured –
    social world”                       Mason 2002
How to „Read‟ Data
   1. Literally
     The  actual words and language used – the literal
      content of the data
     The sequence of interaction – in the case of
      interviews, who speaks when?
     In the case of visual data – style, layout, literal form
     Although these categories may be important, few
      researchers will stop here. Some argue that purely
      objective description is not possible because the
      social world is always interpreted and what we see is
      shaped by how we see it!
How to „Read‟ Data
   2. Interpretively
     Constructing   or documenting a version of
      what you think the data mean or represent
     Reading through or beyond the data
     E.g. implicit norms or rules with which an
      interviewee is operating
     Discourses that influence people
     Versions or accounts of how people make
      sense of social phenomena
How to „Read‟ Data
   3. Reflexively
     Locates  the researcher as part of the data
     Seeks to explore the role and perspective of
      the researcher in the process of generation
      and interpretation of data
     Captures or expresses the relationships
      between researcher and data
     E.g. response to a certain situation in
      fieldnotes (empathy, shock, agreement,
Stages in the Analysis of
Qualitative Data
   Stage 1: Immersion
      The researcher intensively reads or listens to material, assimilating as
       much of the explicit and implicit meaning as possible
   Stage 2: Categorisation
      Systematically working through the data, assigning coding categories or
       identifying meanings within the various segments / units of the ‟text‟
   Stage 3: Reduction
      questioning or interrogating the meanings or categories that have been
       developed? Are there other ways of looking at the data? Do some
       codes mean the same thing?
   Stage 4: Triangulation
      sorting through the categories. Deciding which categories are recurring
       and central and which are less significant or are invalid or mistaken
   Stage 5: Interpretation
      making sense of the data from a wider perspective. Constructing a
       model or using an established theory to explicate the findings of the
Making a Convincing Argument about your Data (Mason,

   Making a convincing argument will be influenced by the
    research questions you originally posed, the focus of the
    research and the kinds of data generated
   Major categories of arguments:
       1. Arguments about how something has developed – a
        meaningful process of development or a story or an
        „archaeology‟ 2. Arguments about how something works or is
        constituted – how and why social phenomena work (but not
        cause and effect)
       3. Arguments about how social phenomena compare –
        meaningful points of comparison in different contexts
       4. Arguments about causation and prediction – the effects of
        variables on each other; not widely used by qualitative
Techniques to Ensure Qualitative Data is Credible (Cresswell, 1998)

   Triangulation – checking one source of data against
   Leaving an audit trail – clear records about how the
    analysis was conducted
   Member checking – have more than 1 researcher
    conduct analysis and compare interpretations
   Checking for researcher effects – do results differ across
    researchers (e.g. focus groups with managers)
   Checking the meaning of outliers – find explanations for
    „extreme cases‟
   Searching for contradictory or negative evidence
   Replicating your findings (more difficult in qual research)
   Getting feedback from participants
   Seeking feedback from peers
Then and Now
 Coding historically done by hand – marker
  pens, cutting and pasting (scissors and
  glue), sorting and shuffling file cards
 Early to mid 1980s marked the emergence
  of basic data programs for storing and
  accessing text
 Now at least 25 different programs – some
  specifically QDA, some more general
What software can and cannot do
   Software is a tool to help analyse qualitative data. It can:
       Store transcripts / other text
       Store codes
       Search and retrieve segments of text
       Link data segments to each other, forming categories, clusters or
        networks of information
       Make notes
       Edit
       Conduct content analysis (count frequencies, sequences or
        location of words)
       Graphically map concepts
   It cannot read the text and decide what it means
   Similarly, it cannot substitute for learning data analysis
Advantages of Software for Qualitative
Data Analysis
   Consistency
     allthe places where a code or combination of codes
      applied, therefore not missing data that contradicts
      incorrect hypothesis
   Speed
     Once  program is learned and data is set up, much
      quicker than manual coding (particularly re-sorting,
      re-defining codes and creating matrices of codes)
   Graphic maps
     Helps  visualising and therefore thinking and
      theorising about possibilities and alternatives
Advantages of Software for Qualitative
Data Analysis
 Speed and ease of use can make us lazy
      Autocoding (searching key words) may encourage shortcuts
      May encourage „quick and dirty‟ research with premature theoretical
   Direct representation of hierarchical relationships (as opposed to
    „circular loops or unstructured networks) encourages hierarchical
   May tempt researchers to skip over the process of „proper‟ learning

    NB As with every methodological decision, if you DO decide to use
    software you should justify it in terms of the literature, acknowledge
    its limitations (again referring to the literature) etc.

Trustworthiness in Qualitative
   Reliability – generally understood to concern the
    replicability of research findings and whether or not they
    would be repeated if another study, using the same or
    similar methods, was undertaken.
   Validity – traditionally understood to refer to the
    „correctness‟ or „precision‟ of research.
           Internal validity: „investigating what you claim to be investigating‟
            (Arksey and Knight 1999)
           External validity: „the abstract constructs or postulates generated,
            refined or tested‟ are applicable to other groups within the
            population (LeCompte and Goetz 1982)
Examining the terms: validity and reliability

     Validity, reliability and
      generalisability have been called
      the „holy trinity‟ of the sciences
      (Kvale 1996).
     What assumptions are inherent in
      emphasising the importance of
      validity and reliability?
To use or not use the terms validity/reliability

   Some qualitative researchers (e.g. critical theorists, feminist
    theorists, post-structural theorists) have criticised the use of these
    terms in qualitative research. On what basis?
   A range of qualitative researchers have denied the relevance of
    validity/reliability to qualitative research and argued that qualitative
    research has its own procedures and processes for judging and
    attaining validity/reliability.
   There is still ongoing debate about this in the literature.
   However, many qualitative researchers utilise the terms validity and
    reliability, and describe a range of strategies to enhance both in their
   Your view about reliability/validity will depend upon your own
    epistemological, theoretical and methodological position.
Examples of moving away from the terms

   Smith (1996) argues that internal coherence
    (or lack of it) would be the most appropriate
    way of assessing qualitative research.
    Rather than being concerned, for example,
    with the representativeness of the sample,
    you should concentrate on whether it was
    internally consistent and coherent.
     Does  it present a coherent argument? Does it
      deal with loose ends and possible contradictions
      in data? Are the interpretations that the
      researcher makes warranted by the data
      presented? Does the report deal with alternative
Leininger 1994
   Credibility
        Ensuring that the researcher uses active listening, reflection and empathic
         understanding to grasp what is „true‟ to informants in their lived environment
   Confirmability
        Repeated direct participatory and documented evidence observed or obtained
         from primary sources
   Meaning-in-context
        Understanding data within holistic contexts (participants‟ contexts)
   Recurrent patterning
        Using repeated experiences, events to identify patterns etc
   Saturation
        Full immersion by the researcher in the phenomena being studied; getting „thick‟
         data to know fully what is being studied
   Transferability
        Examining general similarities of findings in similar environmental situations
Popay et al 1998
   The privileging of „subjective meaning‟ – the research illuminates the
    subjective meaning, actions, and context of those being researched.
   Responsiveness to social context – the research design is
    adaptable/responsive to real-life situations.
   Purposive sampling – the sample produces the knowledge necessary to
    understand participants‟ location in structures and processes.
   Adequate description – the reader can interpret the meaning and context of
    what is researched.
   Data quality – different sources of knowledge about the same issues are
   Theoretical and conceptual adequacy – the research describes the process
    of moving from the data to intepretation.
   Typicality – claims are made for logical rather than probabilistic
Lincoln and Guba (1990)

  credibility
  applicability
  consistency
  neutrality
Strategies for Promoting Validity and Reliability in
Qualitative Research (Merriam 2002).

   Triangulation
      Denzin (1970) identifies four types of triangulation to confirm
        emerging findings:
          Multiple investigators,
          Multiple sources of data (time, space, person)
          Multiple data collection methods to confirm emerging findings
           (observations, interviews, focus groups etc).
          Multiple theoretical perspectives. This involves using several
           perspectives to examine the same set of data. Few
           investigators use this technique.
      Janesick (1994: 214) adds interdisciplinary triangulation.
      The greater the convergence attained through triangulation the
        greater confidence in findings.
      First argued by Foreman (1948) „to establish validity through
        pooled judgement‟.
Strategies for Promoting Validity and Reliability in
Qualitative Research (Merriam 2002).

Member checks
    Taking  data and tentative interpretations back to the
     people from whom they were derived and asking if
     they were plausible.
    Also called member tests of validity and host
    Can be conducted continuously and both formally and
     informally (e.g. at the end of an interview by
     summarising data and allowing respondent to
     immediately correct errors of fact, by giving copies of
     various parts of your report to different stakeholders
     and asking for comment etc).
    Not necessarily free of bias or problems (see St.
     Pierre 1999 and Sandelowski 1993 ).
Strategies for Promoting Validity and Reliability in
Qualitative Research (Merriam 2002).

Peer review/examination
    Discussions   with colleagues regarding the
     process of study, the congruency of emerging
     findings with the raw data, and tentative
    This peer is outside the context but has some
     general understanding of the study and
     methods etc.
    Criticised by some e.g. Morse (1994)
Strategies for Promoting Validity and Reliability in
Qualitative Research (Merriam 2002).

 Researcher‟s position/reflexivity
    Critical self-reflection by the researcher
     regarding assumptions, worldview, biases,
     theoretical orientation, and relationship to the
     study that may affect the investigation.
    Keeping a journal/type of diary.
    Includes information about yourself as a
     researcher, your schedule, insights, decisions
     and justifications for decisions
Strategies for Promoting Validity and Reliability
in Qualitative Research (Merriam 2002).

Adequate engagement in data collection
    Adequate time spent collecting data so that data
     become „saturated‟.
    This may involve seeking discrepant or negative
     cases. Negative case analysis (or analytic
     induction) involves addressing and considering
     alternative interpretations of the data, particularly
     noting pieces of data that would tend to refute the
     researcher‟s reconstructions of reality.
    Prolonged engagement will build trust and
     develop rapport and the impact of your presence
     may diminish.
Strategies for Promoting Validity and Reliability in
Qualitative Research (Merriam 2002).

Attention to sampling
    Maximum    variation: Purposefully seeking
     variation or diversity in sample selection to
     allow for a greater range of application of the
    Do not suppress/ignore the deviant/the
    Allows for and opens up a range of realities
     and perspectives.
Strategies for Promoting Validity and Reliability in
Qualitative Research (Merriam 2002).

Audit trail
      A detailed account of the methods, procedures and decision
       points in carrying out the study.
      Lincoln and Guba (1985) give six categories of audit trail
           (1) raw data (interview guides, notes, documents)
           (2) data reduction and analysis products
           (3) data reconstruction and synthesis products (e.g. data analysis
           (4) process notes (journal)
           (5) materials relating to intentions and dispositions (inquiry proposal,
            journal, peer debriefing notes) (6) information relevant to any
            instrument development.
Strategies for Promoting Validity and Reliability in
Qualitative Research Cont. (Merriam 2002).
    Rick, thick descriptions
      Providing  enough description to contextualise
       the study such that readers will be able to
       determine the extent to which their situation
       matches the research context, and hence,
       whether the findings can be transferred.
      Lincoln and Guba (1985, 125) state that „The
       description must specify everything that a
       reader may need to know in order to
       understand the findings (findings are NOT
       part of the thick description).

     Ethics and Qualitative
Principles of Research Involving Human Subjects
   1. Respect for persons
       treating others as autonomous agents having rights and freedom
        not a means to an end
       free, voluntary and informed consent privacy and confidentiality
   2. Beneficence
       research should be for the good of the subject either directly or
        indirectly through benefiting society
       possible benefits are maximised and risks minimised
       impasse often develops between social good and individual
   3. Justice
       benefits and harms are to be distributed fairly
       vulnerable groups such as cognitively impaired and mentally ill,
        their above average rates of institutionalisation and their
        dependency on others, have made them a convenient subject
        pool for research
       who should participate in research poses significant challenges
        to policy formation
Key Ethical Concepts
   Protection of participant

   Informed consent

   Use of deception

   Debriefing participants

   Right to withdraw

   Privacy and confidentiality
Protection of Participants
   1. Ensure minimal risk
     must  apply the cost-benefit-ratio
     risks unlikely to be greater than any encountered in
      normal lifestyle
     must minimise negative outcomes

   2. Strategies
     obtain advice from professionals
     screen vulnerable participants
     monitor unforeseen negative events
     debrief participants about research
     conduct long-term follow-ups
     have counselling or support available
Informed Consent
   Rests on 4 elements
     competence,   information, understanding of that
      information and voluntariness
     but… cannot be established in many important areas
      of research e.g. critically ill, demented, minors
   Social contract
     restson a mutually agreed contract
     must reveal all aspects that might influence the
      decision to participate
   Strategies
     inform of the general aims of the   project
     associated costs and benefits
     consent forms
    Use of Deception
   Subjects are not given an opportunity to provide their informed consent to
    participation before data collection.
   Examples include covert observation or subject knows they are participating
    in research but not the nature of the research.
   Problems
      deprives participant of the right to informed consent
      but… providing all information is likely to influence behaviour and
         therefore results
      should be avoided if possible

   Guidelines governing deception in research:
        no more than minimal risk to subjects
        rights and welfare of the subjects will not be affected
        research cannot practicably be carried out without the deception
        where appropriate, subjects are provided pertinent information
         about the research after participation (debriefing)
Debriefing Participants
   Rationale
     traditional solution to deception problems
     participation considered an educational experience

   Strategies
     give all information needed and requested
     discuss their experience of the research
     provide contact details
Right to Withdraw
   Rights
     can withdraw consent without any penalty
     can request data be destroyed

   Controversy
     use  of captive audiences (e.g. students,
      military, prisoners, employees)
     use of incentives
Privacy & Confidentiality
   Avoid the use of sensitive questions

   Do not record names if possible

   Code questionnaires

   Warn prior to data collection what identifying information
    will be kept

   Explain confidentiality procedures
Research ethics and the Internet
   Dilemma based on three facts:
        Informed consent is not required for data to be collected from the public
         domain (naturalistic observation).
        The internet is a public domain
        Many online communications (email; discussion groups, chat rooms,
         newsgroups etc) cultivates an expectation of privacy
   The ease and attractiveness of Internet research renders the
    medium vulnerable to misuse.
   Guidelines:
      When subjects are recruited online, need secure server, secure
       protection of information during the study and removal of the records
       upon study completion
      When using data from online discussion groups, removal of any
       references to identity, web site or group, location and time of post is
       necessary for confidentiality
Research with vulnerable populations –

An EXTENSIVE methodological literature exists on
  undertaking research with specific populations
  and the ethics and practice of research with
  these populations. For example:
 Indigenous people
 Youth
 People with disabilities
 Migrants
 The aged

To top