AERA '07 TPS Paper

Document Sample
AERA '07 TPS Paper Powered By Docstoc
					                                                                                          Tech Proficiency 1


                      Assessing the Technology Proficiency of College Faculty

                                           Corenna C. Cummings
                                            Deborah L Kalkman
                                             Jason Underwood
                                           Nancy DeFrates-Densch
                                              William Boelcke
                                             Agnes E. Conway

                                         Northern Illinois University

        This study describes a self-report questionnaire originally developed in 1998 for use in a
        College of Education to assess the technology proficiency of pre-service teachers. The
        original survey was revised to improve the validity and reliability of item and scale
        scores; reflect technological advances in software, hardware, and the internet; and better
        capture the breadth of technical proficiency of its respondents. The current version of the
        survey has been used since Fall 2003 with both pre-service teachers and college faculty.
                                            f
        This study reports the reliability o the nine subscales of the instrument; the technical
        proficiency of college faculty; and changes in that technical proficiency over three years.


                                                PERSPECTIVE

At a time when computer access and use in K-12 schools is approaching “saturation” with nearly
universal teacher and student access to computers and computer technology (Ely, 2002), and school
administrators more than ever demanding technology-savvy teachers (Albee, 2003), increasingly, the
focus of teacher preparation programs has been to cultivate skill in using technology and integrating it in
the classroom (Jonas, 2004; Paige, 2004; Vannatta & Beyerbach, 2000). While implementing programs to
increase these abilities, one common finding was that, in addition to technology skills courses, it was also
important for pre-service teachers to have “experiences in which the integration of technology with
teaching and learning is modeled” (Mims, 2006) that requires the same set of skills, abilities, and attitudes
be present in teacher preparation faculty. Included in the publication of the latest version of the National
Educational Technology Standards (NETS) for Teachers (ISTE, 2000) are guidelines that call for “teacher
education faculty…skilled in using technology systems and software…and model[ing] effective use as a
part of coursework” (ISTE, 2000). These elements highlight a critical link between technology proficient
teacher candidates and technology proficient faculty, indicating that the need for professional
development of teacher preparation faculty relative to technology is crucial.

Given the need for faculty proficiency in technology, the ability to assess this proficiency also becomes
quite important. Often, self-report measures of technology proficiency are instruments of choice for
assessing technology skills, abilities, and attitudes (Vannatta & Beyerbach, 2000; Zhao et al., 2002;
Howland & Wedman, 2004). The results of such surveys can be used in at least two very important ways:
analyzing gaps for which instruction is a solution and developing professiona l learning experiences to
match; and measuring growth, either at the individual or institutional level. The repeated use of
proficiency surveys can also serve as a continuous needs assessment, allowing programs to make
adjustments where necessary to address the particular needs of specific groups of learners (Mims, 2006).

This study describes the current version of a self-report questionnaire originally developed in 1998 for use
at a large Midwestern university to assess the technology proficiency of pre-service teachers. The original
survey was revised to improve the validity and reliability of item and scale scores; reflect technological


Paper presented for discussion at the annual meeting of the American Educational Research Association, April 2007,
Chicago, IL.
                                                                                          Tech Proficiency 2

advances in software, hardware, and the internet; and better capture the breadth of technical proficiency
of its respondents. The current version of the survey has been used since Fall 2003 with both pre-service
teachers and college faculty. This study reports on the use of the current version with college faculty.
Research questions include:
     • How reliable are the survey’s nine subscale scores?
     • What is the technical proficiency of college faculty in specific areas?
     • How has the technical proficiency of college faculty changed over three years?


                                                   METHOD

Participants. Ninety-three college faculty at a large Midwestern university participated in the study.
Eighty-three of the participants were faculty in the College of Education; 71 were tenured or tenure-track
faculty, 18 were adjunct faculty, and 4 were pre-service teacher clinical supervisors; 44 had taught at the
college level for nine or more years; and 63 had K-12 teaching experience. All study participants were
involved at some point, and to varying degrees, with a federal technology grant designed to promote the
integration of technology into pre-service teacher education.

Data. The Technology Proficiency Survey (TPS) is a 106-item, self-report questionnaire, assessing
respondents’ perceptions of their knowledge and skills in nine technology-related areas:
    • Spreadsheets: The 11 items on this subscale assess knowledge and skill in entering and
       manipulating data, formatting cells, creating and using formulas, printing, and graphing.

    •   Word processing: The 19 items on this subscale assess knowledge and skill in opening, printing,
        and saving documents; formatting pages; creating lists and tables; creating and using templates;
        mail merging; inserting graphics, and using spell check, the Thesaurus, and reviewing features.

    •   Presentation Software: The 17 items on this subscale assess knowledge and skill in creating and
        modifying text and multimedia slides, setting up slide shows, modifying slide masters, and
        printing slides, notes, and handouts.

    •   Web Skills: The 11 items on this subscale assess knowledge and skill in using web browsers and
        navigating the web, conducting efficient web searches, authoring web pages, using an e-mail
        account, downloading and installing software and updates, using an FTP client, and working in a
        networked environment.

    •   Multimedia: The 13 items on this subscale assess knowledge and skill in scanning files, using
        digital still and video cameras, and creating and using audio, video, and graphics software.

    •   Handhelds: The 6 items on this subscale assess knowledge and skill in using basic handheld
        features and Documents-To-Go software.

    •   Social, Ethical, and Legal Issues: The 5 items on this subscale assess behaviors associated with
        the modeling and promotion of technology-related social, ethical, and legal considerations.

    •   File Management: The 12 items on this subscale assess knowledge and skill in installing
        computer software, organizing and storing electronic files, creating CDs and DVDs, and working
        with compressed files.

    •   Databases: The 12 items on this subscale assess knowledge and skill in creating fields, entering

Paper presented for discussion at the annual meeting of the American Educational Research Association, April 2007,
Chicago, IL.
                                                                                          Tech Proficiency 3

          and sorting records, conducting queries, creating forms and reports, and printing queries, forms,
          and reports.

Respondents indicate on a 5-point Likert-type scale the degree to which they agree with statements, with
1 indicating strong disagreement and 5 indicating strong agreement with the statement. Each statement
begins with “I am able to ….” The complete scale is available in the attached appendix.

Procedures. Participants completed an online version of the survey either prior to attending a semester-
long technology workshop or as part of a technology integration project. The survey results were used to
design instruction and provide technical support appropriate to the knowledge and skill level of the
participants.


                                                   RESULTS

Subscale reliabilities. Internal consistency estimates were computed for each subscale using coefficient
alpha. Alphas range from .78 to .99. With one exception, all alphas are greater than .91, indicating high
internal consistency for eight of the nine subscales. Table 1 reports the alphas, means, and standard
deviations for all subscales across all semesters of data collection.

Table 1

Internal Consistency Estimates, Means, and Standard Deviations for the Technology Proficiency Survey
Subscales (N = 93)

Subscale                                                        α                M                SD

Spreadsheets (11 items)                                         .96             3.89              1.10

Presentation (17 items)                                         .96             4.32              0.84

Web Skills (11 items)                                           .78             4.46              0.51

Databases (12 items)                                            .99             2.76              1.30

Multimedia (13 items)                                           .97             3.34              1.21

File Management (12 items)                                      .92             4.12              0.98

Handheld Computers (6 items)                                    .97             3.44              1.51

Social, Ethical, and Legal Issues (5 items)                     .94             4.24              0.82

Word Processing (19 items)                                      .91             4.52              0.55


Technology Proficiency of College Faculty. Overall, faculty perceived themselves to be most proficient
in using the web and word processing, and least proficient in using databases, multimedia, and handheld
computers.



Paper presented for discussion at the annual meeting of the American Educational Research Association, April 2007,
Chicago, IL.
                                                                                          Tech Proficiency 4

Change over Time. Nine one-way analyses of variance were conducted to identify statistically
significant changes in mean subscale scores across the four semesters data was collected: Fall ’03 (n =
20), Spring ’04 (n = 16), Spring ’05 (n = 33), and Fall ’05 (n = 24). The ANOVA was significant for six
of the nine subscale means: spreadsheets, F(3, 89) = 4.00, ρ = .01; presentation software, F(3, 89) = 8.27,
ρ = .00; multimedia, F(3, 89) = 6.74, ρ = .00; file management, F(3, 89) = 33.13, ρ = .00; word
processing, F(3, 89) = 2.96, ρ = .04; and handheld computers, F(3, 89) = 9.85, ρ = .00. The strength of
the relationship between the semester the survey was administered and the mean subscale scores was
assessed by η2 , with the semester accounting for 12%, 22%, 19%, 55%, 9%, and 25% of the variance of
the spreadsheets, presentation software, multimedia, file management, word processing, and handheld
computers subscale mean scores, respectively.

Follow-up tests were conducted to evaluate pairwise differences among the means. Because variances
among the four groups were not equal, Dunnett’s C test was used to conduct post hoc comparisons.
Means and standard deviations for the four semesters, are reported in Table 2. Tests indicate significant
differences between the Fall ’03 and Fall ’05 means for all subscales, between Fall ’03 and Spring ’05 for
presentation software, multimedia, file management, and handheld computers, between Fall ’03 and
Spring ’04 for file management, and between Spring ’04 and Fall ’05 for handheld computers.


Table 2

Pairwise Comparisons for Subscale Means by Semester

                                              Fall ‘03       Spring ‘04       Spring ‘05              Fall ‘05
Subscale                                          M              M                M                      M
                                                (SD)            (SD)             (SD)                   (SD)
Databases                                        2.19           3.01             2.77                   3.04
                                               (1.20)          (1.41)           (1.22)                 (1.34)
File Management                               2.82 a b c       4.07 a           4.51 b                 4.69 c
                                               (0.55)          (0.80)           (0.80)                 (0.50)
                                                    ab              c
Handhelds                                      2.50            2.69             3.67 a                4.43 b c
                                               (1.45)          (1.62)           (1.29)                 (1.01)
Multimedia                                     2.52 a b         3.06            3.53 a                 3.96 b
                                               (1.13)          (1.41)           (0.99)                 (1.04)
Presentation Software                          3.76 a b         3.98            4.52 a                 4.75 b
                                               (0.92)          (1.23)           (0.50)                 (0.41)
Social, Ethical, and Legal Issues                3.97           3.96             4.30                   4.58
                                               (1.10)          (1.04)           (0.53)                 (0.57)
Spreadsheets                                    3.30 a          3.64             4.08                  4.30 a
                                               (1.30)          (1.35)           (0.86)                 (0.80)
Web Skills                                       4.32           4.23             4.51                   4.68
                                               (0.53)          (0.68)           (0.39)                 (0.42)
Word Processing                                 4.25 a          4.44             4.58                  4.71 a
                                               (0.59)          (0.61)           (0.52)                 (0.45)
a
  indicates significant mean differences using Dunnett’s C procedure for pairwise comparisons
b
  indicates significant mean differences using Dunnett’s C procedure for pairwise comparisons
c
  indicates significant mean differences using Dunnett’s C procedure for pairwise comparisons
d
  indicates significant mean differences using Dunnett’s C procedure for pairwise comparisons




Paper presented for discussion at the annual meeting of the American Educational Research Association, April 2007,
Chicago, IL.
                                                                                          Tech Proficiency 5

Significant and non-significant ANOVAs and significant pairwise differences among the means may be
explained by respondents’ involvement in specific grant-supported professional development activities,
thus supporting the content validity for the subscale scores. For example, the three nonsignificant
ANOVA subscales, databases, file management, and social, ethical, and legal issues were not
substantially represented in the content of the professional development activities while the content
represented by the six subscales with significant ANOVAs were substantially represented in the
curriculum.

The high subscale scores and low standard deviations for word processing and web skills reflect the
relatively high level of proficiency of participating faculty in these areas by Fall ’03 and their access to
the web and word processing programs. Handheld computers were not widely available early in the
implementation of the surveys and were gradually worked into the faculty technology professional
development between Fall of 2003 and Fall of 2005.


                                                DISCUSSION

The TPS yields reliable scores in nine technology skill areas. This instrument may be used for needs
assessment in the development of professional development programs matched to participant skill level as
well as providing for differentiation of instruction based on individual need. Further study should be
conducted using other populations, such as students and K-12 teachers to determine if the scale is
appropriate for use with these populations.


                                                REFERENCES

Albee, J. J. (2003). A study of preservice elementary teachers' technology skill preparedness and
        examples of how it can be increased. Journal of Technology and Teacher Education, 11(1), 53-
        71.
Ely, D. P. (2002). Trends in educational technology (5th ed.). Syracuse, NY: ERIC Cle aringhouse on
        Information & Technology.
Howland, J., & Wedman, J. (2004). A process model for faculty Development: Individualizing
      technology learning. Journal of Technology and Teacher Education, 12(2), 239-262.
ISTE. (2002). National educational technolo gy standards for teachers. Eugene, OR: International Society
        for Technology in Education (ISTE).
Mims, C., Polly, D., Shepherd, C., & Inan, F. (2006). Examining PT3 projects designed to improve
       preservice education. TechTrends, 50(3), 16-24.
Phillips, R., Cummings, R., Lowe, K., & Jonas-Dwyer, D. (2004). Rethinking flexible learning in a
         distributed learning environment: A university-wide initiative. Educational Media International,
         41(3), 195-205.
Vannatta, R. A., & Beyerbach, B. (2001). Facilitating a constructivist vision of technology integration
       among education faculty and preservice teachers. Journal of Research on Computing in
       Education, 33(2), 132-148
Zhao, Y., Pugh, K., Sheldon, S., & Byers, J. L. (2002). Conditions for classroom technology innovations.
       Teachers College Record, 104(3), 482-515.




Paper presented for discussion at the annual meeting of the American Educational Research Association, April 2007,
Chicago, IL.
                                                                          Tech Proficiency 6


                                              APPENDIX

                 Internal Consistency Estimates, Means and Standard Deviations
                          for the Technology Proficiency Survey (N = 93)

Sub-scale and Items                                                α      M      SD
Word Processing Sub-Scale (19 items)                               .91   4.52    0.55
I am able to …
    1. use a word processing program to type documents                   4.93    0.29
    2. print documents I have created                                    4.96    0.21
    3. open documents                                                    4.98    0.15
    4. save documents to a hard disk and a diskette                      4.94    0.23
    5. use a spell checker feature                                       4.96    0.26
    6. use the thesaurus feature when editing documents                  4.75    0.70
    7. change the font (or typestyle) for a document                     4.96    0.21
    8. change the margins of an entire document                          4.70    0.78
    9. change the margins of only a part of the document                 4.66    0.72
    10. change the line spacing within a document                        4.92    0.35
    11. set and change the tabs within a document                        4.52    0.85
    12. create a bulleted list                                           4.84    0.54
    13. insert headers and footers                                       4.67    0.86
    14. create a table                                                   4.76    0.71
    15. insert graphics into a document                                  4.46    0.99
    16. use document templates                                           4.17    1.25
    17. create a document template                                       3.38    1.44
    18. use the review feature (track changes/markup)                    4.18    1.21
    19. perform a mail merge                                             3.01    1.38
Spreadsheets (11 items)                                            .96   3.89    1.10
 I am able to …
    1. use a spreadsheet program                                         4.13    1.21
    2. enter text into cells in a spreadsheet                            4.39    1.06
    3. enter numbers into cells in a spreadsheet                         4.39    1.01
    4. create and use formulas                                           3.48    1.39
    5. print an entire spreadsheet                                       4.26    1.23
    6. print specific sections of a spreadsheet                          3.74    1.38
    7. insert rows or columns into a spreadsheet                         4.03    1.33
    8. use functions like SUM and AVERAGE                                3.81    1.40
    9. use drag and drop techniques to move, copy, or fill cells         3.78    1.43
    10. create a graph or chart within a spreadsheet                     3.20    1.45
    11. print a graph or chart from a spreadsheet                        3.37    1.47
                                                                               Tech Proficiency 7


Sub-scale and Items                                                      α      M     SD
Presentation Software (17 items)                                         .96   4.32   0.84
 I am able to …
    1. use presentation software                                               4.76   0.68
    2. enter text into slides                                                  4.79   0.68
    3. change the appearance and placement of text                             4.72   0.74
    4. select different slide types or layouts                                 4.76   0.72
    5. change an existing slide’s layout to a new one                          4.61   0.88
    6. create text bullet lists                                                4.72   0.73
    7. add clip art to slides                                                  4.61   0.92
    8. add audio or video to slides                                            3.92   1.35
    9. select and change the slide background design                           4.55   0.96
    10. modify the slide master                                                3.80   1.33
    11. set-up slideshow to loop continuously                                  3.67   1.44
    12. reorder the slides in an on-screen slide show                          4.37   1.17
    13. use transitions between slides in a slide show                         4.13   1.29
    14. animate objects and text within a slide                                3.68   1.46
    15. create hyperlink to other slides, external files, or web pages
        within a slide                                                         3.77   1.47
    16. print full size slides for transparency                                4.61   1.00
    17. print miniature slides for handouts                                    4.59   1.02

Databases (12 items)                                                     .99   2.76   1.30
 I am able to …
    1. create a table in design view                                           2.82   1.41
    2. create field names                                                      2.95   1.45
    3. enter records in design view                                            2.95   1.46
    4. copy, paste, and delete records and fields                              3.04   1.52
    5. sort records (ascend, descend)                                          2.94   1.50
    6. create and use a query as a wizard                                      2.72   1.39
    7. customize a query in design view                                        2.60   1.32
    8. create and use a form us a wizard or design view                        2.56   1.33
    9. customize a form in design view                                         2.56   1.34
    10. create and use a report using a wizard or design view                  2.64   1.36
    11. customize a report in design view                                      2.64   1.40
    12. print queries, forms, and reports                                      2.87   1.49
                                                                              Tech Proficiency 8



Sub-scale and Items                                                     α     M      SD

Web Skills (11 items)                                                   .78   4.46   0.51

 I am able to …
    1. navigate through web pages                                             4.91   0.47
    2. access web pages by typing the URL                                     4.94   0.23
    3. find information efficiently using a search engine                     4.77   0.58
    4. print web browsers                                                     4.89   0.35
    5. create bookmarks or Favorites in a web browser                         4.80   0.58
    6. author web pages                                                       3.50   1.39
    7. use an email account                                                   4.92   0.46
    8. send, receive, and save email attachments                              4.99   0.11
    9. download and install software applications, patches, updates           4.18   1.36
    10. use FTP                                                               3.29   1.49
    11. work in a networked environment                                       4.11   1.13

Multimedia (13 items):
                                                                        .97   3.34   1.21
Audio, Graphics, and Video Hardware and Software
 I am able to …
    1. create graphics files with a scanner                                   3.60   1.42
    2. scan a paper document into a digital text                              3.72   1.43
    3. use a digital still camera                                             4.16   1.21
    4. create graphics using graphics software (e.g. Paint,
        Illustrator, Fireworks)                                               3.14   1.47
    5. adjust brightness, contrast, and colors of images                      3.59   1.40
    6. crop, resize, rotate, and change the resolution of images              3.72   1.34
    7. distinguish among graphics file formats (e.g. bmp, jpg, png,           3.47   1.40
        gif)
    8. determine when to use different graphics file formats                  3.06   1.42
    9. distinguish among wav, mp3, cda, and midi audio formats                2.88   1.51
    10. create digital audio files vinyl records, audio tapes, or CDs         2.89   1.45
    11. distinguish among avi, mov, mpg, and stream video formats             2.78   1.52
    12. use a digital video camera                                            3.59   1.51
    13. create digital video files from VHS or Beta videotapes                2.65   1.56
                                                                           Tech Proficiency 9



Sub-scale and Items                                                  α     M      SD

Computer File Management (12 items)                                  .92   4.12   0.98

 I am able to …
    1. start up software already installed on the computer                 4.85   0.63
    2. install new software that I have purchased                          3.58   1.76
    3. copy files from hard disk to floppy, floppy to floppy, etc.         4.78   0.65
    4. move files between disks                                            3.98   1.67
    5. create file folders                                                 4.73   0.75
    6. organize files into folders                                         3.95   1.66
    7. delete files                                                        4.78   0.73
    8. remove file folders                                                 3.98   1.65
    9. “burn” a CD                                                         4.14   1.32
    10. “burn” a DVD                                                       3.12   1.63
    11. identify the functional difference between read only,
        writable, and rewritable CDs an DVDs                               3.92   1.35
    12. zip and unzip compressed files                                     3.70   1.46

Handheld Computers (6 items)                                         .97   3.44   1.51
 I am able to …
    1. use the basic management features of a Palm OS handheld             3.73   1.60
    2. synchronize data entered on handheld and on desktop by              3.71   1.63
       “Hotsyncing”
    3. locate Palm OS applications online for use on the handheld          3.31   1.62
    4. install applications on the handheld by “Hotsyncing”                3.34   1.62
    5. use “Documents-to-Go” or a similar program to interoperate
       with MS Office products                                             3.19   1.68
    6. transfer applications and documents using the infrared port
       (“beaming”)                                                         3.29   1.63
Technology-Related Social, Ethical and Legal Issues (5 items)        .94   4.24   0.82
 I am able to …
    1. model and teach legal and ethical practice related to               4.16   1.01
       technology use
    2. apply technology resources to enable and empower learners
       with diverse backgrounds, characteristics, and abilities            4.31   0.91
    3. identify and use technology resources that affirm diversity         4.20   0.94
    4. promote safe and healthy use of technology resources                4.32   0.78
    5. facilitate equitable access to technology resources for all         4.19   0.86
       students

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:8
posted:3/18/2012
language:
pages:9