Automated Assessment of Sketched Diagrams

Automated Assessment of Sketched Diagrams Presentation by: Tracy A. Hammond Brandon C. Paulson Joshua B. Johnston Joshua M. Peschel Department of Computer Science Texas A&M University – College Station BACKGROUND An example of Sketch Recognition… http://srl.csdl.tamu.edu/videos/mecheng.avi PRESENTATION OUTLINE Today we will be discussing: (1) (2) (3) (4) (5) Background Information Geography Education Application Sentence Diagramming Application Engineering Education Application Summary and Ongoing Work PRESENTATION OUTLINE First we will be discussing: (1) (2) (3) (4) (5) Background Information Geography Education Application Sentence Diagramming Application Engineering Education Application Summary and Ongoing Work BACKGROUND Graphics are pervasive throughout education, business, engineering…almost everywhere! BACKGROUND Diagrams are a natural way to describe a variety of information. So what is this? • Body: Polygon, center (2.79, -3.98), points (1,-2.4)  (4.89,-4.96)  (4.71, -5.37)  (.8, -5.56)  (0.69, -2.66) • Body: Polygon, center (2.29, -1.91), points (2,0, -1.1)  (2.89, -1.72)  (2.47, -2.04)  (3.21, -2.18)  (2.77, -2.72)  (1.37, -1.9) • Body: Circle, center (2.49, -2.43), radius .35 • Body: Circle, center (1.78, -1.99), radius .3 • Pinjoint: center (1.83, -1.95) • Pinjoint: center (2.54, -2.39) • Anchor: center (1.6,-4.36) • Gravity on BACKGROUND Diagrams are a natural way to describe a variety of information. Which is more intuitive? BACKGROUND Computer-Aided Drawing (CAD) systems provide a lot of power for creating diagrams… ● Wide variety of information ● Understood by computers ● Can be animated ● Quick feedback But… DISCUSSION PREMISE There is something special about a blank sheet of paper… ● ● ● Natural expression Drawing freedom Sketcher is creator (i.e. active learning) ● Visualization BACKGROUND Computer-Aided Drawing (CAD) provide a lot of power…but… ● systems ● ● ● Not freeform drawing Requires training Time better spent learning content Restrict creativity BACKGROUND A simple equation… Sketch Recognition = Creativity + Function where Creativity = Freeform drawings Function = CAD system understanding BACKGROUND Question: What is Sketch Recognition? “STAR” Answer: Identification of hand-drawn shapes. BACKGROUND In the context of real-time educational assessment… Assessment is time-consuming. Solutions: ● ● Scantrons Online multiple choice questions? Sketched diagrams are often omitted from realtime assessment processes… BACKGROUND Sketched diagrams are often omitted from realtime assessment processes… But… Testing and feedback are important… ● ● ● Testing combined with study is more effective than studying alone [Roediger] Real-time feedback is crucial to the learning process CAD systems provide real-time feedback and thus provide functional visualization BACKGROUND The benefits of Sketch Recognition: ● ● ● ● More natural design that encourages creativity Enhance student learning through animated application and real-time feedback Reduce teacher time by automating answer corrections and question generation Enhance teacher feedback by real-time collation of grades showing mistake themes BACKGROUND A simple equation… Sketch Recognition = Creativity + Function where Creativity = Freeform drawings Function = CAD system understanding BACKGROUND Is Sketch Recognition hard to implement? ● ● ● Requires programming expertise Often a lengthy build time involved Definitely impractical for instructor But, does it have to be this way…? No! We have made significant progress to overcome these barriers…now for some fun. PRESENTATION OUTLINE Now we will be discussing: (1) (2) (3) (4) (5) Background Information Geography Education Application Sentence Diagramming Application Engineering Education Application Summary and Ongoing Work SHAPE LEARNING APPLICATION ● ● ● Children learn through sketching Address children through kinesthetic and tactile learning Recognition with PaleoSketch SHAPE LEARNING APPLICATION http://srl.csdl.tamu.edu/userstudy/children.shtml GEOGRAPHY APPLICATION ● ● ● Real-time assessment of geographic skills Allows corrections Can be easily expanded GEOGRAPHY APPLICATION http://srl.csdl.tamu.edu/geography.shtml PRESENTATION OUTLINE Next we will be discussing: (1) (2) (3) (4) (5) Background Information Geography Education Application Sentence Diagramming Application Engineering Education Application Summary and Ongoing Work SENTENCE DIAGRAMMING And now another demonstration… http://srl.csdl.tamu.edu/videos/sentenceDiagramming.wmv PRESENTATION OUTLINE Now we will be discussing: (1) (2) (3) (4) (5) Background Information Geography Education Application Sentence Diagramming Application Engineering Education Application Summary and Ongoing Work ENGINEERING APPLCIATION Readiness Assessment Tests (RATs): ● ● ● ● ● Formative assessment tools (incremental) Usually to test one or two specific concepts First introduced by Michaelson1 for business Introduced into engineering in early 1990s Basically a pop quiz with a two-way purpose 1Michaelson, L.K., W.E. Watson, J.P. Cragin and L.D. Fink. 1982. Team Learning: A Potential Solution to the Problems of Large Classes. Exchange: Journal of Organizational Behavior, 7:13-22. ENGINEERING APPLCIATION Readiness Assessment Tests (RATs): ● ● ● ● Standard medium is paper-based Feedback delays caused by need for grading Classroom content therefore hard to adjust Students may miss critical knowledge points ENGINEERING APPLCIATION Readiness Assessment Tests (RATs): ● ● ● ● ● Some near real-time RATs developed2 Scantrons used and graded in class Course management systems available BlackBoard and WebCT could be used Features and cost may not be practical 2Yost, S., D.J. Lane and G. Blanford. 2005. 2006-2306: RATS: Students Working in Teams, Do They Really Benefit? In Proceedings of the 2005 ASEE Annual Conference, Portland, Oregon. ENGINEERING APPLCIATION Our goal for this system is to enhance learner understanding within a large, two-semester freshman engineering course at Texas A&M University. ENGINEERING APPLCIATION The flow of information can be described as: ENGINEERING APPLCIATION The quiz interface for the learner: ENGINEERING APPLCIATION The graphic results output displayed to all: ENGINEERING APPLCIATION But it’s not quite real-time, is it…? Sketch Recognition ENGINEERING APPLCIATION Sketch Recognition can be used for creating quiz content in real-time… http://srl.csdl.tamu.edu/videos/STRAT_Final.mov PRESENTATION OUTLINE Finally we will be discussing: (1) (2) (3) (4) (5) Background Information Geography Education Application Sentence Diagramming Application Engineering Education Application Summary and Ongoing Work SUMMARY AND ONGOING WORK Over 30 applications built so far across a myriad of domains… ● Mechanical engineering, electrical circuits, military course of action diagrams, finite state machines, music notation, math notation, device creation, flow charts, games, chemistry diagrams, Japanese kanji, many more… SUMMARY AND ONGOING WORK Over 20 applications have been built by students in HCI / sketch recognition courses as a prototyping tool… http://srl.csdl.tamu.edu/videos/kanji.avi SUMMARY AND ONGOING WORK In summary… ● ● Looking for collaborators interested in implementation! Please visit our website: http://srl.csdl.tamu.edu SUMMARY AND ONGOING WORK SRL Members… Dr. Tracy Hammond, Brandon Paulson, Josh Johnston, Katie Dahmen, Brian Eoff, Pankaj Rajan and Aaron Wolin Funding… ● NSF: Developing Perception-Based Geometric Primitive-Shape Recognizers to Empower Instructors to Build Sketch Systems in the Classroom ● Automated Assessment of Sketched Diagrams Questions or Comments? For more information contact: Dr. Tracy Hammond hammond@cs.tamu.edu Department of Computer Science Texas A&M University – College Station