Running head: USABILITY OF VIRTUAL MANIPULATIVES 1
Usability Testing of Virtual Manipulatives by User Profile
The University of Texas at Austin
College of Education
EDC 385G Design/Strategies for New Media
Dr. Min Liu
April 25, 2010
Running head: USABILITY OF VIRTUAL MANIPULATIVES 2
Usability Testing of Virtual Manipulatives by User Profile
Universal design (UD) was originally developed for architectural design so that
individuals with physical disabilities would have greater access to physical spaces inherently
open to the individuals without disabilities. UD is the foundation for the design, development,
and implementation of assistive technology (AT) in education so that students with physical
disabilities can gain greater access to the general education setting (Elder-Hindshaw, Nelson,
Manset-Williamson, & Dunn, 2006; Crow, 2008). A consequence of the inclusion model in
educational reform (i.e., where students with atypical needs have a right to be included in general
education), UD is gaining acceptance for designing curriculum and educational environments for
all typical and atypical students (Scott et al., 2003).
Usability of Online Learning
The last twenty years have provided a seemingly unlimited advancement of affordances
in educational materials, especially with online access. Theoretically, the technology is available
to design, develop, and implement tools needed to achieve a universally accessible general
education. Unfortunately, the research shows that UD is not consistently followed in online
learning (Grabinger, Aplin, & Ponnappa-Brenner, 2008; Pisha & Coyne, 2001; Scott et al.,
The needs of online learners with physical and cognitive disabilities require alternative or
enhanced features so that access is available and/or unfettered. Students with physical disabilities
can greatly benefit from screen readers, descriptions of figures and diagrams, layouts that have
organized headings, available closed captions, text captioning for video or other forms of multi-
media, limited synchronous activities, and additional time to complete asynchronous activities
(Crow, 2008). Research has found that designing universal access can improve the usability of
Running head: USABILITY OF VIRTUAL MANIPULATIVES 3
the technology for all learners. This would include limiting content to the appropriate level for
the target user, providing choices in features for accessibility and providing live tech support. In
addition, tools should be dynamic, tactile and intuitive at any age and for any purpose
(Simoncelli & Hinson, 2008). Grabinger et al. (2008) found that the most effective online layouts
and functionalities met the requirements of 5 symptomatic needs of cognitive disabilities
(attention and memory, language, executive function, problem solving and reasoning, and social
function) through 3 lenses (recognition, strategic, and affective). Online materials that follow the
key elements of usability (information, interface and interaction) were found to be effective for
students of all ages, abilities and purpose because the interfaces were organized, non-cluttered
and included the freedom of choice in features (like text-to-speech). More importantly, it has
been found that the most effective platforms for both students and teachers are flat in design
where the interaction has limited external links and layers for searching of materials (Keeler and
A great deal of research and development is dedicated to online or computer-based
manipulatives in primary and secondary mathematics. Although this research reports on the
effectiveness of the tools in terms of academic achievement, it does not provide evidence as to
whether or not the design of the platforms on which these tools reside are useful in terms of
interaction, interface and information. Thus, this study will examine whether or not the 3 most
popular sites for virtual math manipulatives (VMs) are easy and intuitive to navigate and use for
students of various ages. Furthermore, we will examine the usefulness of these sites to teachers
who are attempting to plan to use a VM in their lessons.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 4
Since the reenactment of No Child Left Behind (2006) and the Individuals with
Disabilities Education Act (2004), educators have been held to a higher standard for instructional
materials, particularly for those in digital form. Teachers must use tools, activities and lessons
that are based on or are themselves scientifically proven methods of instruction (Pisha & Stahl,
2005). As a result, researchers have studied the essential components of websites that meet this
standard in addition to outlining instructional design principles that will guide developers. In
other words, they have identified what works and doesn’t work. However, these researchers did
not include usability in their studies (Reimer & Moyer, 2005; Underwood et al., 2005).
Established usability testing for math websites or virtual math manipulatives is almost
nonexistent. Therefore, principles on testing, data collection and analysis by Moggridge (2007)
and Rubin and Chisnell (2008) (among many others within these same standards) were
referenced and/or implemented as a theoretical foundation for the following methodology.
Underwood, Hoadley, and Lee (2005) and Reimer and Moyer (2005) collectively
reported that the 3 most prominent websites for virtual math manipulatives are the National
Library of Virtual Manipulatives (http://nlvm.usu.edu/), the Math Forum Math Tools
(http://mathforum.org/mathtools/) and the Shodor Foundation Project Interactive
(http://shodor.org/interactive/index.html). We used 3 search engines on April 2, 2010:
www.google.com, www.bing.com and www.yahoo.com. Our search consisted of three layers:
general, age and purpose. For a general search, we used the keywords “math manipulatives,”
“virtual manipulatives” and “virtual math manipulatives” in all 3 search engines. Searching by
age, we used the keywords “preschool,” “middle school” and “teacher.” And a search based on
Running head: USABILITY OF VIRTUAL MANIPULATIVES 5
purpose consisted of the keywords “activities,” “tools,” “lessons” and “lesson plans.” The results
consistently listed the NLVM, Math Forum and Shodor websites in the top ten.
Three individuals participated, varying in age, computer competency levels and
occupation. The first to participate was a male preschool student age 3 years 5 months and of
Asian-Caucasian ethnicity; the second participant was a Hispanic-Caucasian female student age
13 years enrolled in middle school TAG classes (in particular, algebra 1 in 7th grade); the last
participant was a Caucasian female teacher at age 32 who taught special education and holds a
bachelor’s degree. All three have average to above-average proficiency with computers, online
activity and virtual manipulatives (self reported). It should be noted here that all 3 participants
are related to the researcher.
All 3 participants completed the study at the researcher’s home on the same computer in
a secluded den. The Mac Mini computer was connected to a high definition 20+ inch color
monitor and high speed Internet. Participants could use the keyboard, touch pad and/or
traditional mouse to navigate the 3 sites. A webcam and microphone was attached to the monitor
of the computer, which was used in conjunction with Camtasia to record each session.
Each session was conducted one at a time conducted with one participant sitting at the
desk with the computer while the other participants were not in the home. In each session, the
participant was debriefed about the study, using language relevant to the participant’s age and
education. He/she was also instructed not to answer any questions or perform any task that made
him/her feel uncomfortable. In addition, each participant was informed that the data and results
Running head: USABILITY OF VIRTUAL MANIPULATIVES 6
collected from the study would not be shared with anyone except the professor and other
researchers in the class.
Pre-questionnaire. Participants were verbally asked questions about their comfort level
with computer hardware, computer software, Internet use and virtual manipulatives. Each
participant was asked to rate his/her comfort level on a scale of “1” to “5,” “5” representing very
comfortable or on a scale of “no” or “yes,” depending on his/her age. The researcher did ask
unscripted, follow-up questions for clarifications of responses.
Data collection. The researcher logged onto the tools, activities or lesson plans section
(depending on the occupation of the participant) for each of the 3 sites (NLVM, Math Forum and
Shodor Foundation) one at a time in the same order. He then instructed the participant to search
(excluding the preschooler), find and click on a link of a manipulative that he/she either found
interesting or was currently studying/teaching. Guidance was given if the participant: 1)
exhausted the search on his/her own and did not receive satisfying results; or 2) needed help with
troubleshooting the technology and/or with navigating the website. The researcher sat in a chair
next to the participant and watched the same screen. No judgment was given on what the
participant chose as a virtual manipulative and reassurance was given with statements like
“There is no right answer.” No other instructions were given as far as search techniques or
navigation within the site. The researcher did use hand-on-hand guidance for the preschooler if
he had difficulty navigating the site, and did ask follow-up questions for areas of difficulty.
Notes were taken by the researcher on how the participant searched for the topic, on ease
of navigation and on difficulties experienced when executing the tasks, all within the site up to
exploring with the VM. Data on ease and difficulty with the VM was only noted if it pertained to
an overall usability issue that was exhibited or could be relevant to the host site.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 7
Post-questionnaire. Once the 3 sites had been explored, each participant was asked to
verbally state which site was liked best and why (excluding the preschooler), to explain which
site(s) was easiest to find the virtual manipulative pertaining to the topic in mind (excluding the
preschooler), and to rate overall satisfaction (using the scale previously mentioned) of each site.
The researcher did ask follow-up questions for clarifying responses.
Data and Results
Each participant provided a different perspective on good and ineffective design of the
interface, information and interaction of the websites. This occurred due in large part to the
participant’s age and purpose, but also due to unforeseen actions and uses that he/she attempted.
Technology Competence and Experience
Each participant was able to respond to the pre-questionnaire. The results are shown in
Table 1 below. Overall, the participants were fully capable of completing the tasks of the study.
Experience levels warranted physical and technical guidance for the preschooler, but the effects
of that assistance by the researcher are negligible.
Participant by Comfort Level Comfort Level Comfort Level Comfort Level
Occupation with Hardware with Software with Internet with VM
Preschool 5 5 3 5
Middle School 3 4 5 4
Teacher 4 4 5 4
Table 1. Some of the participants were unfamiliar or not capable of using the scale. Thus,
comparable words were used to represent the ratings on the scale. For instance, the participant
was given a choice between “No,” “I think so,” or “Yes.” The results have been transformed to a
scale of 1 to 5, with 5 representing “very comfortable.”
Navigating through the Tasks
Overall, each participant found it difficult to find the tool or activity he/she had in mind
on all 3 sites. Each website had a different layout and search capabilities. Figures 1, 2 and 3
below show the home page for each site. In Figure 1, a user in the NLVM site must move the
Running head: USABILITY OF VIRTUAL MANIPULATIVES 8
mouse pointer over the boxes by category and age group and click on the box that is highlighted
in white; if he or she wanted to search by a particular topic, the user would have to notice the
search box on the upper right area of the screen and realize the box to the left of the label
“Search” in intended for input. This is a different design than that of the Math Forum site, shown
in Figure 2, where the search features are prominently displayed at the top of the page utilizing a
keyword search engine or a drop down menu by age, category and type of materials (i.e., tool,
activity, or lesson). The Shodor Foundation home page uses a tabular layout where the user may
search by keywords, category or type of materials; however, there are no search capabilities by
In addition, the teacher found it to be particularly challenging to find pertinent lesson
plans for the topic and virtual manipulative sought. The NLVM home page required the teacher
to click through the same levels as the students, then click on the buttons at the top of the screen
to access standards and teacher notes (which are available to all users). The Math Forum and
Shodor Foundation home pages allowed the teacher to “one-click” her way to lessons, but had to
stumble upon or be guided to the drop-down menus that enable the functionality.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 9
Figure 1. NLVM home page.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 10
Figure 2. Math Forum home page.
Figure 3. Shodor Foundation home page.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 11
Preschool student. The preschooler was not required to search for a topic, which the
researcher chose (counting whole numbers in pre-kindergarten), due to experience and cognitive
capabilities. However, he was asked to find and click on a virtual manipulative he found
interesting, of skills he was capable. The easiest navigation for him of the 3 sites was the Shodor
Foundation (Figure 3); this may have been greatly influenced by the fact that he was currently
learning how to read an analog clock and the first virtual manipulative in the results list was
“Clock Arithmetic.” He had slightly less difficulty navigating the results list in the NLVM site
(Figure 4), where the pictures (although small) drew his attention to a counting manipulative.
The least useful site for the preschooler was Math Forum; the list had no graphic icons, required
an unstable “over” movement with the mouse pointer in order to reveal detailed information
(Figure 5), and often led to irrelevant or dead links. The researcher had to assist the participant in
choosing an appropriate link on this site.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 12
Figure 4. NLVM results list for pre-kindergarten and number and operations.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 13
Figure 5. Math Forum results for pre-kindergarten and number sense.
There are few points to note about the actual VMs and their usability for the preschooler.
First, none of them had automatic voice over (VO) or the option to have VO that could be
verbally describing the tasks. Shown in Figure 6 is the VM “Let’s count!” on the Math Forum
site, which has written directions but no intuitive action. The participant did not know what to do
and randomly clicked on different areas of the screen until he became frustrated. Second, the
layout of the counting numbers VM in “Clock Arithmetic” in the Shodor Foundation site (Figure
7) had buttons in close proximately to each other and with small real estate for clicking. This
made it difficult for this participant to navigate the VM, which is contrary to his previously
demonstrated abilities and frustrated him to point where he stated, “I can’t do it,” for subsequent
tasks. The researcher at that point used hand-on-hand guidance until his confidence returned.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 14
Figure 6. Counting VM on Math Forum.
Figure 7. Shodor Foundation VM on counting numbers.
Middle-school student. The middle school student was capable of using the search
engines or given filters. She indicated that her current studies in math touched on graphing of
parabolas. She was then instructed to find a VM that would help her in that topic. She found the
matrix of categories and ages easy to navigate; although, she clicked on the broad category of
algebra instead of using the cross reference of “Algebra” and “9-12” and had to look through
over 100 topics in order to find one on graphing parabolas. She did have considerable difficulty
Running head: USABILITY OF VIRTUAL MANIPULATIVES 15
with the Math Forum site, first inputting keywords in the search box on the top left of the screen
and then having difficulty finding a category that matched her topic in the “Browse” drop-down
menus (Figure 8). The search resulted in no relevant links, links that had little relevancy or VMs
that required cash payment or an unsupported platform (e.g., Java Applet). This become so futile
that the researcher had to stop her after 10 minutes and redirect to any VM of interest, not
necessarily one that was pertinent to her topic of study. Her easiest navigation and search was in
the Shodor Foundation site, which had a tab of “Algebra” and large, descriptive graph icons that
allowed her to scroll down and find the “GraphIt” VM.
Figure 8. Math Tools search engine.
Figure 9. Math Forum search results for the middle school student.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 16
Figure 10. Shodor Foundation search results for the middle school student.
Teacher. The purpose, as directed by the researcher, for the teacher was to navigate
through the sites and find lesson plans supporting the VMs on a current topic taught (in her case,
multiplying polynomials). As indicated by the teacher, a lesson plan would consist of standards,
notes to the teacher and instructions on how to facilitate use of the VM. In addition, the lesson
plans should not be apparent and readily available to the student. Given these constraints, both
the NLVM and Math Forum sites failed the usability test for teachers. As seen in Figures 11 and
12, the teacher notes are clearly marked at the top banner of the screen and are accessible to the
students. In addition, the participant commented that the notes were neither to the depth nor in
the form normally written in instructional materials. When she clicked on the instructions button,
the site gave the same directions given to the student with no answers or further guidance (Figure
13). Finally, when she clicked on the standards button, the site hyperlinked to a page on the
NCTM website, which was not specific to the VM and too general for her to fully understand the
goal of the materials.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 17
Figure 11. NLVM screen layout of a VM on multiplying polynomials.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 18
Figure 12. NLVM teacher notes on multiplying polynomials.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 19
Figure 13. NCTM standards for secondary algebra, hyperlinked from NLVM.
The Math Forum, when the teacher was able to find a readily available lesson on a topic,
was just as unclear and shallow as the lessons on the NLVM site. Searching by keywords or
browsing by topic and/or resource type yielded no or very little results. Unfortunately, she was
not able to find a lesson on multiplying polynomials; instead she found a lesson on addition and
Running head: USABILITY OF VIRTUAL MANIPULATIVES 20
subtraction of polynomials. As seen in Figures 13 and 14, this provided very little resources; the
one resource that was available consisted of a hyperlink to another site that required registration.
This participant expressed a desire to stop on that site and the researcher allowed her to move on
to the Shodor Foundation site.
Figure 14. Math Forum results for both student tools and teacher lessons.
Figure 15. Math Forum hyperlink to lesson on polynomial operations.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 21
On the Shodor Foundation site, the teacher was able to quickly ascertain that the topic on
multiplying polynomials was not available by clicking on the “Algebra” tab and scrolling down
the page. She did comment on the sparseness of resources. She then clicked on a comparable
topic and found it difficult to locate any link to lesson plans. The researcher then directed her to
the “Jump To:” menu at the top of the screen; the teacher immediately found “Lessons” and
clicked on the plan for Venn diagrams (Figures 16 and 17). She verbally reported that the results
list was clear, easy to read, informative and not cluttered. This satisfaction of the Shodor
Foundation site continued after exploring the actual lesson plan on Venn diagrams. According to
her, the site was organized and relevant to the information customary in lesson plans (Figure 18).
Also, she gave high marks on the standards drop-down menu, mainly because she could examine
detailed standards for Texas within the same page of the lesson plan (Figures 19 and 20).
Figure 16. Shodor Foundation “Jump To:” drop-down menu.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 22
Figure 17. Shodor Foundation site results on lessons.
Figure 18. Lesson plan page on Venn diagrams
Running head: USABILITY OF VIRTUAL MANIPULATIVES 23
Figure 19. Standards drop down menu on Shodor Foundation site.
Figure 20. Texas standards imbedded on the same screen as the lesson plan.
Once the tasks were complete, each participant was asked which site he/she liked best
and why (excluding the preschooler), to explain which site(s) was easiest to find the virtual
manipulative (excluding the preschooler), and to rate overall satisfaction (using the scale
previously mentioned) of each site. Table 2 shows that the Shodor Foundation had the highest
marks by all participants; this was confirmed after a review of the data by the researcher, noting
Running head: USABILITY OF VIRTUAL MANIPULATIVES 24
the level of enthusiasm and lack of frustration when compared to their experiences on the other
sites. The site with the lowest marks was the Math Forum site; this was also confirmed by a
review of the data, noting a greater frustration level with finding VMs or lesson plans pertinent
to their current topic. In addition, the participants spent the most time on the Math Forum site,
with an average of 12 minutes of the average total session time of 20 minutes. This time was not
spent exploring, but in searching for relevant tools, activities and/or lesson plans. The two eldest
participants also indicated, as answers to the first 2 post-questions, that the Shodor Foundation
site was the easiest to use because of the smaller number of levels needed to navigate, good use
of graphic icons, clear, concise and organized layout, and pertinent and intuitive information.
Participant by NLVM Math Forum Shodor
Preschool 5 5 5
Middle School 4 2 4
Teacher 1 1 5
Table 2. Some of the participants were unfamiliar or not capable of using the scale. Thus,
comparable words were used to represent the ratings on the scale. For instance, the participant
was given a choice between “I didn’t like it,” “Maybe,” or “I liked it.” The results have been
transformed to a scale of 1 to 5, with 5 representing “I liked it.”
This study on virtual math manipulatives explored the usability of 3 of the most popular
sites with regard to age and purpose of the user. The researcher had a general procedure for
collecting data, instructing participants on the given tasks and analyzed the data based on
recorded digital video. The goal of the study and its research questions was to determine the ease
of use and barriers to access. Conclusions would be based on principles of instructional and
universal design: unfettered and flexible access for all individuals. Based on the data collected on
the use by the aforementioned 3 participants, all 3 sites have severe flaws in their design, which
Running head: USABILITY OF VIRTUAL MANIPULATIVES 25
may create barriers to individuals with physical or cognitive disabilities, limitations due to
developmental age, or specialized needs.
Recommendations for Higher Degree of Usability
In order to provide a more universally designed site for virtual manipulatives, the
following parameters are recommended:
1) Provide a simple yet clearly labeled search matrix, such as the one found on the
NLVM and Math Forum sites.
2) Depending on the age and/or abilities of the user, text, graphics and sound should
be readily enlarged and available.
3) Sites should clearly separate purpose of use; thus, students should be prevented
from gaining access to the teacher materials.
4) Online materials should be organized in the same manner as on paper in
traditional platforms (i.e., teacher edition books) but with the dynamic
affordances of technology.
5) The use of drop-down menus should be limited since they have a tendency to hide
the features and functions available, and buttons should be large and spaced
reasonably from each other to allow those individuals with limited physical range
to click on the desired button.
6) Outside, external links should be avoided when possible so that users feel there is
cohesiveness to the site.
7) Navigation should be tabular (e.g., breadcrumbs) with text and graphic icons so
that users of any age or ability can easily go back and forth in their search and use
of VMs. However, no individual should have to click through more than 3 levels
Running head: USABILITY OF VIRTUAL MANIPULATIVES 26
(i.e., pages, actions or portals) in order to arrive at his/her desired tool, activity or
8) Unnecessary graphics (i.e., site trademarks and ads) should be eliminated since it
reduces available real estate for large icons and concise, precise information.
9) To the point it does not detract from a simply organized layout, the home page of
the site should be just as inviting and exciting as the VMs themselves.
10) Organized headings that can embed expanding and collapsible information should
be the cornerstone of the overall layout.
Other than a need for a larger sample size and control group, limitations to the results of
this study included possible bias due to the relationship between the researcher and participants.
The capabilities of the preschooler and subsequent guidance required to complete tasks may have
some bias effect on the results; also, he did not answer all questions in the pre- and post-
interviews, which essentially provides incomplete and incomparable data to the data of the other
participants. Furthermore, a limited script and use of follow-up questions, although designed to
simulate the environment the user would experience independently with the sites, could be
source of compounding factors. Perhaps more training and guidance, arguably what school
districts provide, on the features and functionality on each site would garner more favorably
Overall, educational websites lack usability and universal design. They are often rushed
to production without the labor-intensive features that require websites to be accessible. As a
result, teachers, parents and students are less likely to use the tools that are necessary to complete
Running head: USABILITY OF VIRTUAL MANIPULATIVES 27
technology-math standards. As seen with the teacher’s navigation in the study, most sites require
3 to 4 clicks before arriving at needed tool, activity or lesson plan. This scaffolding is a barrier to
those with disabilities or an abhorrence of online materials. Until designers consistently develop
online sites and materials that meet the needs of as many users as reasonably possible, the
affordances of VMs and like technology will be accessible to only a small, elite set of learners.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 28
Crow, K. L. (2008). Four types of disabilities: Their impact on online learning. TechTrends:
Linking Research & Practice to Improve Learning, 52(1), 51-55.
Elder-Hinshaw, R., Nelson, J. M., Manset-Williamson, G.,& Dunn, M. W. (2006). Engaging
older students with reading disabilities: Multimedia inquiry projects supported by reading
assistive technology. Teaching Exceptional Children, 39(1), 6-11.
Grabinger, R. S., Aplin, C. & Ponnappa-Brenner, G. (2008). Supporting learners with cognitive
impairments in online environments. TechTrends: Linking Research & Practice to
Improve Learning, 52(1), 63-69.
Keeler, C. G. & Horney, M. (2007). Online course designs: Are special needs being met?
American Journal of Distance Education, 21(2), 61-75.
Moggridge, B. (2006). Designing Interactions. Cambridge, MA: The MIT Press.
Pisha, B. & Coyne, P. (2001). Smart from the start: The promise of universal design for learning.
Remedial & Special Education, 22(4), 197-203. Retrieved September 30, 2008 from
Academic Search Complete database.
Pisha, B. & Stahl, S. (2005). The promise of new learning environments for students with
disabilities. Intervention in School & Clinic, 41(1), 67-75. Retrieved September 29, 2008
from Education Full Text database.
Reimer, K. & Moyer, P. S. (2005). Third-graders learn about fractions using virtual
manipulatives: A classroom study. The Journal of Computers in Mathematics and
Science Teaching, 24(1), 5-25.
Rubin, J. & Chisnell, D. (2008). Handbook of usability testing: How to plan, design and conduct
effective tests. Indianapolis, IN: Wiley Publishing, Inc.
Running head: USABILITY OF VIRTUAL MANIPULATIVES 29
Simoncelli, A. & Hinson, J. M. (2008). College students' with learning disabilities personal
reactions to online learning. Journal of College Reading and Learning, 38(2), 49-62.
Underwood, J.S., Hoadley, C. & Lee, H.S. (2005). IDEA: Identifying Design Principles in
Educational Applets. Educational Technology Research and Development, 53(2), 99-112.