How to evaluate Network Supported Collaborative Learning (NSCL

How to evaluate Network Supported Collaborative Learning (NSCL) Processes? A new conceptual framework Iván M. Jorrín Abellán Faculty of Education and social Works University of Valladolid, Spain Based on the work of the EMIC/GSIC group Evaluating NSCL Processes UIUC 10/12/2005 Intelligent & Cooperative Systems Research Group University of Valladolid, Spain Problem statement Evaluation of NSCL processes ASK After designing and applying to real environments an NSCL process, How can we evaluate it´s effectiveness? Evaluating NSCL Processes UIUC 10/12/2005 The work-plan of the session Overview Investigate 1- Brief introduction on CSCL systems and the evaluation problem (P - 15 min.) 2- An overview of evaluation in NSCL (P – 30 min.) 3- Evaluation in a existing real case study (S,P – 25 min.) 3.1 Presentation of the case (P – 10 min.) 3.2 How evaluation really took place (P – 15 min.) Create &Discuss &Reflect Evaluating NSCL Processes UIUC 10/12/2005 Introduction Some basic ideas  ICT (Information and Communications Technologies) in learning   The C(N)SCL field The different facets of evaluation Evaluating NSCL Processes UIUC 10/12/2005 Introduction/ICT in learning  ICT support to education (ICT-E): – A multidisciplinary approach by default – Different approaches have been tried based on a dominant pedagogical model in each period.Computer Assisted Instruction (CAI), Intelligent Tutoring Systems (ITS) and Web-based Education (WBE) Many of these technological approaches to supporting education suffered from the same problems and defects as their non-technological counterparts. Nowadays everything is e-learning: Apparent paradigm shift from teaching to learning Is it a true change or a simple use of new words and a new technology (web-based)? Evaluating NSCL Processes UIUC 10/12/2005 - – – Introduction/ICT in learning  What are the new aspects that should be studied? – learn to use ICT: P(erson) -> M(achine) – learn from machine (CAI): P <- M – learn through machine (ITS) P <-> M – learn through network (Internet) A <-> N(etwork) Maybe the Relation among learners & Technology – learn through network and persons (CSCL): P <-N-> P  CSCL or NSCL (New field in ICT-E: 1995 - …) – Computer (Network) Support for Collaborative Learning – Elements: 3Cs (Communication, Coordination, Collaboration) Evaluating NSCL Processes UIUC 10/12/2005 Introduction/The CSCL field   NSCL can be seen as an „emerging field’ (Koschmann, 1996b) for research in educational technology that focuses on the use of „information and communications technology (ICT) as a mediational tool within collaborative methods (e.g., peer learning and tutoring, reciprocal teaching, project- or problem-based learning, simulations, games) of learning‟ NSCL interest centres on “how collaborative learning supported by technology can enhance peer interaction and work in groups, and how collaboration and technology facilitate sharing and distributing of knowledge and expertise among community members”.  Theories: – social constructivism, situated cognition, activity Evaluating NSCL Processes UIUC 10/12/2005 Differences between NSCL and other approachs  Promote learning: the purpose is not just to perform a task but to promote learning during and after the collaborative interaction. Typically, collaborative learning involves two or more peers, with shared learning goals, who interact in a technology-mediated learning environment in order to acquire new capabilities/skills or enhance understanding. Evaluating NSCL Processes UIUC 10/12/2005 Differences between NSCL and other approachs  Enable collaboration: collaboration is distinguished from simple interaction between participants. Collaboration involves more than communication; more than the simple exchange of ideas, information or material. Many distance/e-learning technologies support information exchange but not collaboration. Collaboration implies working together, on the creation and improvement of (cognitive) artefacts (see e.g. Bereiter, 2002). Evaluating NSCL Processes UIUC 10/12/2005 Differences between NSCL and other approachs  Promote collaboration: It isn‟t enough for an NSCL system to allow collaboration. The design of tasks and technology needs to promote and support collaboration. This is because: (a) commitment to NSCL arises from a belief in the potential of collaboration to enhance learning, (b) realisation of this potential can be impeded by a number of factors, including (c) the fact that individualisation and competition are powerful forces at work in the Evaluating NSCL Processes world. UIUC 10/12/2005 Differences between NSCL and other approachs -NSCL is even more ambitious than previous approaches of ICT-support in education. -It is therefore more difficult to evaluate the effectiveness and efficiency of NSCL activities -Nonetheless, all actors involved in NSCL processes, – from policy makers to everyday practitioners – need to have evidence of whether, how and when expected improvements in learning take place. -Significant effort is required to provide systematic evaluation of innovative projects, the specific experiences within an action/research framework, the new NSCL systems developed, and so on. Evaluating NSCL Processes UIUC 10/12/2005 Introduction/Aspects of evaluation •Due to inherent difficulties in performing evaluation,very few systematic and complete studies have been reported in the literature •Although there is an evident conviction in the field that recommendations have to be based on real findings of evaluation studies in real environments, Evaluating NSCL Processes UIUC 10/12/2005 Introduction/Aspects of evaluation Evaluation is a process or activity that is intended to form judgements about something. These judgements are judgements of value – whether something is good or bad; fit for purpose or not; an improvement or not; whether a is better than b; whether something or someone or some project has achieved the goals it set out to achieve; whether some intervention has had the desired effect, or no effect, or some undesirable and unintended side effects, etc. Evaluating NSCL Processes UIUC 10/12/2005 Introduction/Aspects of evaluation Evaluation activity, and evaluation reports make most sense within a defined community – such as a community of teachers and educational policy makers (at local, national or international levels). The point to understand about evaluation is that it needs to (a) operate within a shared framework and (b) recognise the different needs of the main stakeholder groups. Without some kind of shared framework there are no possibilities for shared understanding. Evaluating NSCL Processes UIUC 10/12/2005 Introduction/Aspects of evaluation        Perspective Users involved in evaluation Objective of evaluation Evaluands (things and processes to evaluate) Techniques to be used for evaluation Data employed for evaluation Conclusions from the evaluation experience Evaluating NSCL Processes UIUC 10/12/2005 Introduction/Focus of evaluation  Evaluation – in an educational setting – in the NSCL domain – of NSCL systems S co pe S E d u ca tio n a l S e ttin g A pproaches co pe E d u ca tio n a l Educational Setting S e ttin g NSCL to o l Learning objectives ... NSCL to o l P articipants a ) E d u ca tio n a l a p p r o a ch b ) T e ch n o lo g ica l a p p r o a ch Educational Approach Technological Approach Evaluating NSCL Processes UIUC 10/12/2005 Introduction Evaluation: under which perspective?  Take into account context or obtain an objective view of an objective world   Use evaluation to enhance or to give credit … Evaluating NSCL Processes UIUC 10/12/2005 Introduction Evaluation: to be done by and for whom?     Funding institutions Researchers in the field Teachers / Educational practitioners Designers / Developers of NSCL systems Evaluating NSCL Processes UIUC 10/12/2005 Introduction Evaluation: for what purpose?  Control degree of learning  Provide feedback for professors  Assess impact of ICT in education  Perform general educational research Evaluating NSCL Processes UIUC 10/12/2005 Introduction Evaluation: on what? A learning resource  A learning tool  A course  A teaching strategy  A learning environment  An innovation project …  Evaluating NSCL Processes UIUC 10/12/2005 Introduction Evaluation: with what technique? Questionnaires  Interviews  Analysis of examination scores …  Evaluating NSCL Processes UIUC 10/12/2005 Introduction Evaluation: which context and conclusions? Limitations of the evaluation  Positive and weak points detected …  Evaluating NSCL Processes UIUC 10/12/2005 The work-plan of the session Overview Investigate 1- Brief introduction on CSCL systems and the evaluation problem (P - 15 min.) 2- An overview of evaluation in NSCL (P – 30 min.) 3- Evaluation in a existing real case study (S,P – 25 min.) 3.1 Presentation of the case (P – 10 min.) 3.2 How evaluation really took place (P – 15 min.) Create &Discuss &Reflect Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL Some ideas  Evaluation in NSCL is complex and there is no real guide to the eventual evaluators    A framework for evaluation of NSCL could be useful Context and the complexity of educational processes were taken into account An overview of existing resources and findings was produced Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL A generic evaluation process Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL The 3P Model for learning Presage Process Product Student - prior knowledge - abilities - preferred ways of learning - values, expectations Perceptions Task processing Teaching context - curriculum - teaching method - classroom climate - assessment Nature of outcome - structure - detail Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL 3P model for evaluation of NSCL Learning outcomes Presage factors task design (were task designs appropriate to learning objectives, learning situation and learner needs?) organisational forms (were there good foundations for the creation of convivial learning relationships?) technology and resources (were appropriate learning resources & technology made available?) students (did students have the appropriate knowledge, skills and attitudes?) Learning processes/ activity engagement (did the students engage with the tasks set?) cognitive (what has been learned?) affective (did the students feel positive about the experience?) technology (was the technology robust and fit for purpose?) relationships (did students help each other? did the teacher(s) intervene appropriately?) assessment regime (was the assessment regime aligned with the intended learning?) System outcomes teacher time (is this commitment of teacher time/energy sustainable?) technical (does the techology need improving?) Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL The proposed framework  It is based on an analysis of what more and less experienced evaluators and other stakeholders need, by way of structure and guidance with respect to the conduct of an evaluation in the NSCL area.  We deliberately chose not to present just another cookbook guide to evaluation. Instead, our framework is based on a deeper analysis of user needs and consists of what we have come to call a „skeleton + guidelines‟. Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL The proposed framework •Framework: -Skeleton: What has to be considered -Guidelines: How to perform it -Visualization model Objective: To provide some paths to eventual evaluators, together with recommendations The skeleton part of the framework provides a conceptual and methodological structure on which evaluation approaches can be supported. In a sense, it defines the problem space of „NSCL evaluation‟. But such a skeleton can be hard for users to understand and interpret – hence the need for guidelines and examples. Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL Skeleton and visualization  Axes of evaluation: • Scope (S) • Perspective (P) • Techniques (T) • Method (M) • Experience (E) Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL The skeleton (1/7) (S) Scope (Context of evaluation) (SG) General Information • Name of organisation, Contact Details, URL, Date & Place (SP) Premises (SP1) Theoretical framework • Foundations (SP2) Roles (SPR1) Evaluators • • • • • Number, Experience, Availability, Competencies / formal qualifications Teaching Style (Subject-centred, Student-centred), Abilities in the use of ICT (High, Medium, Low) Worried about students´ tutorisation Competencies / formal qualifications Evaluating NSCL Processes UIUC 10/12/2005 (SPR2) Teacher An overview of evaluation in NSCL The skeleton (2/7) (SPR3) Student • • • • • • Previous Knowledge (High, Medium, Low) Learning style (Active, Reflective, Theoretical, Pragmatic) Expectations (Positive, Negative, None) Capability to assume strong workload Abilities in the use of ICT (High, Medium, Low) Competencies / formal qualifications (SPR4) Analysis of the institutional and organizational context • • • Scenario (Technological, Non technological) Climate (Competitive, Collaborative) Grade (Primary, Secondary, University, Others) (SP3) Conditions of use • • Size of the group to evaluate (Small (2-10), Medium (10-100), Large (100)) Process extension (Unique action, Medium term, Long term) Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL The skeleton (3/7) (P) Perspective (P1) Evaluation objectives • • • As a support to research To improve the educational practice To monitor the progress of something, and allow timely adjustments to be made A learning resource A learning tool A course A teaching strategy A learning environment An innovation project An innovation programme An evaluation process Evaluating NSCL Processes UIUC 10/12/2005 (P2) Evaluands • • • • • • • • An overview of evaluation in NSCL The skeleton (4/7) (T) Techniques (T1) Previous analysis techniques • • • • • • • • • • • • Expert review Document analysis Resource questionnaires Heuristic Evaluation Assessment tests Benefits – defining and quantifying Checklists Sociograms/network analysis Focus groups Pre and post testing Interviews, Journals Log files Analysis Evaluating NSCL Processes UIUC 10/12/2005 (T2) Data collect techniques An overview of evaluation in NSCL The skeleton (5/7) • • • • • • • • • • • • • • • Observation (participant and non-participant) Sampling Self-completion questionnaires Costing techniques Cost-effectiveness Split screen video Transcript analysis Supplemental observation Content Analysis Nominal Group Technique Designing Experiment Viewing Analysis Sociograms/network analysis Statistical modelling Triangulation vs. combination of moments and techniques of evaluation Evaluating NSCL Processes UIUC 10/12/2005 (T3) Triangulation techniques An overview of evaluation in NSCL The skeleton (6/7) (M) Method (Data&Findings) (M1) Objective data • • • • • Data collection procedure Type and format Analysis method Theoretical framework for interpreting the data Summary of analysis & interpretation (M2) Interpreted data (M3) Recommendations based on findings Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL The skeleton (7/7) (E) Experience (E1) Observed advantages/strengths (E2) Observed disadvantages/weaknesses (E3) Limitations of the study (E4) Meta-level use of the evaluation study • • • • Ethical sensitivity Role of the evaluator Generalizability Validity and Reliability Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL Guidelines and examples  Eventual users of framework – Researchers in the field, practitioners, institutions, designers and developers of tools    Different requirements for each type of users A large “evaluation space” defined by the skeleton Paths are necessary in order that users navigate at the evaluation space Evaluating NSCL Processes UIUC 10/12/2005 An overview of evaluation in NSCL Guidelines and examples  Each skeleton item is translated to a question and a set of recommendations and interdependences Example for researchers: – SP3: Which is the size of the group to evaluate and what is the process extension? – Small or medium groups and a medium-long process are recommended – High interdependences with SPR1 (evaluators), SPR3 NSCL Processes … (students) Evaluating UIUC 10/12/2005  The work-plan of the session Overview Investigate 1- Brief introduction on CSCL systems and the evaluation problem (P - 15 min.) 2- An overview of evaluation in NSCL (P – 30 min.) 3- Evaluation in a existing real case study (S,P – 25 min.) 3.1 Presentation of the case (P – 10 min.) 3.2 How evaluation really took place (P – 15 min.) Create &Discuss &Reflect Evaluating NSCL Processes UIUC 10/12/2005 Evaluation of existing case studies Presentation  Took place at the University of Valladolid – Run by the GSIC/EMIC team – Project or problem based formal learning – Blended or hybrid settings (face-to-face and distance) – Learning and instructional design based on wellknown “collaborative learning flow patterns” (CLFP) • Jigsaw • Pyramid Evaluating NSCL Processes UIUC 10/12/2005 Evaluation of existing case study Presentation      Context: Second course, Faculty of Education Course: ICT in Education Participants: 40 Duration: semester (40 sessions of 1 hour) Task: produce report “impact of ICT in education” and an educational webquest to be applied in a real context    Authenticity: Real context NSCL systems: Synergeia, Quest Other ICT tools: MS Office, browser, etc. Evaluating NSCL Processes UIUC 10/12/2005 Evaluation of existing case study Presentation/Structure of the Course Jigsaw for the theoretical part of the course Mediated by Virtual Tools -Synergeia -Quest PBL for the practical part of the course To produce a report on the “impact of ICT in education” Evaluating NSCL Processes UIUC 10/12/2005 To produce an educational webquest to be applied in a real context Evaluation of existing case study Presentation/Mediational Tools/ Synergeia Evaluating NSCL Processes UIUC 10/12/2005 Evaluation of existing case study Presentation/Mediational Tools/ Synergeia Students Functionality •Uploading, archiving and versioning files •Downloading files •Searching the web and saving bookmarks •Annotating and rating own and other´s files •Collaborative Maptool Teachers Functionality •Registering people •Defining Courses and groups •Seeding content •Defining folder structure •Creating Knowledge building areas •etc •Chat/ Instant messaging •Collaborative Agenda •Knowledge building areas/forum Evaluating NSCL Processes UIUC 10/12/2005 Evaluation of existing case study Presentation/Mediational Tools/ Quest QUEST : allows for the design of questionnaires by a teacher or researcher, and their presentation as webbased forms to the students. QUEST collects the results automatically and converts them into different formats: RTF files for their use with Nud*IST, spreadsheet files for quantitative data analysis, and XML files representing the interactions for SNA. We should point out that these automatic conversions of data are simple, but very necessary for improving the efficiency of the whole process. Evaluating NSCL Processes UIUC 10/12/2005 Evaluating NSCL Processes UIUC 10/12/2005 Evaluating NSCL Processes UIUC 10/12/2005  To enforce collaboration, teachers require each group of students to compare in their reports with the solutions proposed by others, using the tables Evaluating NSCL Processes of answers provided UIUC 10/12/2005 The work-plan of the session Overview Investigate 1- Brief introduction on CSCL systems and the evaluation problem (P - 15 min.) 2- An overview of evaluation in NSCL (P – 30 min.) 3- Evaluation in a existing real case study (S,P – 25 min.) 3.1 Presentation of the case (P – 10 min.) 3.2 How evaluation really took place (P – 15 min.) Create &Discuss &Reflect Evaluating NSCL Processes UIUC 10/12/2005 Method requirements Mixed method of evaluation  Integrate context  Study of real situations  Participants‟ point of view – Ethnographic data sources – Qualitative analysis – Automatically recorded data – Quantitative analysis – Tools  New forms of interaction  Scalable and efficient processes  Visualisation processes  Participatory aspects – Social network analysis Evaluating NSCL Processes UIUC 10/12/2005 Evaluation of existing case studies Mixed evaluation method Phases Observations Preparation Throughout the experience Daily work Face to face interact. Scheme of categories Focus Groups Questionnaires Previous concepts (individual) Analysis methods Automatic data Initial Sociometries Quantita tive Qualitati ve SNA Conclusions After milestones • After milestones • Critics about the project Event logs End of project Final Final Sociometries Data sources Evaluating NSCL Processes UIUC 10/12/2005 The work-plan of the session Overview Investigate 1- Brief introduction on CSCL systems and the evaluation problem (P - 15 min.) 2- An overview of evaluation in NSCL (P – 30 min.) 3- Evaluation in a existing real case study (S,P – 25 min.) 3.1 Presentation of the case (P – 10 min.) 3.2 How evaluation really took place (P – 15 min.) Create &Discuss &Reflect Evaluating NSCL Processes UIUC 10/12/2005 Create  After having explain in class the followed steps in order to create the proposed evaluation framework, i would like you to find a real experience in wich you can apply it. Although the framework was designed to evaluate NSCL processes, you can choose any educational experience at any level, in wich technology is use as an educational support. This practical application might be the start point of a discussion about the usefulness of the proposal. Discuss  Advantages & Disadvantages of the evaluation framework  Improvements  Relation to Pragmatic Technology  Relation to Participatory Action Research Evaluating NSCL Processes UIUC 10/12/2005 Wrap-up The final ideas      What was all this about ? What are the main ideas and problems? Is the proposal of TELL good enough for real new case studies? Should we continue the work in studying how to enhance learning using ICT? Should we try to put all these ideas in practice? Evaluating NSCL Processes UIUC 10/12/2005