Intelligent Diagrams: An Overview of the Adaptive Diagram Research Project
Kim Marriott, Bernd Meyer, Peter Stuckey (Uni. of Melbourne) Adrian Bickerstaffe, Tim Dwyer, Nathan Hurst, Aidan Lane, Cameron McCormack, Linda McIver, Peter Moulder, Pritika Sanghi, Kieran Simpson, Peter Sbarski, & Michael Wybrow
Traditional view of documents
Traditional view is print-media based
Static collection of markings on a fixed size piece of paper
Content (text and illustrations) already fixed Graphic designer determines the precise layout and graphic qualities such as fonts However the Web has changed this…
1
Adaptation to Viewing Environment
Device capabilities are more and more diverse ranging from mobile phones to wall-mounted displays
Need to vary the layout to better utilize different size displays
2
Adaptation to User Needs
Need to vary the layout to handle different text size (e.g. larger fonts or different language) Support tactile graphic version for blind users
3
Interaction
For mobile devices need different layout and semantic zooming to allow interactive exploration by expanding and collapsing nodes
4
Animation
We need to support animation But do something reasonable when the media is static
5
Access to semantics
We would like
semantics-based search and retrieval semantic adaptation to context support audio-only views of diagrams for blind users etc
Population of Oceanic Countries Estimate for 2000 in millions
Australia Papua New Guinea New Zealand Fiji 18.9 4.8 3.9 0.8
6
Web view of documents
Need to adapt layout to viewing context and user needs Should support interaction and animation Access to semantics Content may be dynamic, generated from database etc Graphic designer cannot precisely specify appearance Need authoring tools which support this new kind of document This is true for all content: text, diagrams, equations, video etc
7
Adaptive Diagrams Research Project
Aims to provide the technical foundation for this new view of documents Extend web standards to better support interaction and adaptation Access to diagram and document semantics Layout of diagrams and other visual languages Text and document layout Authoring tools Look at these from algorithmic, software engineering and HCI perspective
8
Underlying Ideas
Layout is the problem of finding a position and size for various graphic objects which
Satisfies structural & designer specified geometric relationships
Contained inside, don’t overlap etc Reduce number of edge crossings etc
Maximises readibility & aesthetics
This is a constrained optimisation problem!
9
Underlying ideas (Cont.)
Layout as constrained optimisation Constraints allow the designer to declaratively specify what the design rules for layout are Objective function provides a measure of the aesthetic quality Models both static layout and dynamic layout say as a result of user interaction Unifying approach to text, diagram and document layout
10
Underlying ideas (Cont.)
Sue Blogg CEO
Daryl Rogers Head Marketing
Sue Fraser CFO
John Yeo Head ofTechnical Services
Tripartite view of a diagram Visual representation Abstract syntax Semantics
11
Sample Projects
Layout of diagrams
IPSep-CoLa (with Tim Dwyer & Yehuda Koren (AT&T)) Constraint SVG (with Cameron McCormack & Bend Meyer) Interpretation of diagrams using CIDER (with Bernd Meyer, Tony Jansen & Vivek Ramakrishna) Optimal table layout (with Nathan Hurst, Peter Moulder & David Albrecht) Constraint-based diagram editor (with Michael Wybrow, Peter Stuckey, Linda McIver & Kieran Simpson)
Extend web standards to support adaptive, interactive web documents
Access to diagram semantics
Text and document layout
Authoring tools
12
Project 1: IPSep-CoLa
In most applications network diagrams (graphs) have application specific layout constraints. Current approaches either cannot handle these, or are brittle and use algorithms specialized for a particular constraint. We wanted to develop a single, generic technique for graph layout with constraints.
13
Separation Constraints
Separation constraints have the form U+g ≤ V where U and V are horizontal or vertical node positions and g is the minimum gap between them. They can model a wide variety of application specific layout constraints:
Directed edges Alignment and distribution, layers Fixed position Containment Orthogonal ordering of nodes Non-overlap of nodes and/or clusters Non-overlap of edges and edge labels
14
The Main Idea
Force-directed graph layout approaches use ``energy minimization’’ to find a layout They are the most commonly used technique for drawing unstructured graphs They use unconstrained optimization techniques to minimize an objective function that captures aesthetic criteria. We modify the approach to use constrained optimization techniques which handle separation constraints.
15
The IPSep-CoLa Algorithm
We wish to minimise the stress function
stress(X) = " w ij (|| X i # X j || #dij ) 2
Functional majorization solves this iteratively, by in turn fixing each dimension and moving the nodes in the other dimension For each dimension it minimizes a quadratic objective that ! bounds the stress function. We modify functional majorization so that the quadratic objective is minimized subject to the separation constraints in that dimension. We use a gradient projection algorithm which relies on fast, incremental projection on to a set of separation constraints.
16
IPSep-Cola Summary
The approach works well and can handle layout of thousands of nodes in seconds It provides the first, practical generic approach to network layout with layout constraints We believe it is the best technique for layout of graphs with directed edges (better than the standard specialised algorithm of Sugiyama et al)
Ideal for Bayesian nets
It is well-suited for interactive layout since the algorithm is incremental and the objective function modifiable.
17
Project 2: Constraint SVG
XML file XSLT SVG file
XML file XSLT SVG file
SVG is the W3C standard for Scalable Vector Graphics. It provides standard graphic elements and transformations. Element attributes must have concrete values. SVG provides uniform rescaling but does not provide more sophisticated adaptive layout. It does not handle font-size changes well
18
Adaptive SVG (CSVG)
width x
b1
x
b2
To provide adaptation of SVG documents we have proposed extending SVG by adding one-way constraints Formulas instead of concrete values are given for the graphical attributes of the objects. Like a spread-sheet or Visio formula These are evaluated at display-time to determine layout.
19
CSVG: Adapting to text size
Default Different language Different font size
XML file
XSLT
SVG file
Fichier XML
XSLT
Fichier SVG
XML file
XSLT
XML file
Compute box width (= maximum of text widths) Compute location of boxes Compute start and end point of arrow Compute arrow label position (= above middle of arrow)
20
CSVG: Adapting to viewing canvas
Desired layout Bit narrower: Use differential scaling Much narrower: Use an alternate layout
XML file XSLT SVG file
XML file
XSLT
SVG file
XML file XSLT SVG file
Negotiate actual viewing canvas size Use conditional in formula “scale = if … then … else … ” to compute layout
21
CSVG: Semantic Zooming
Viewer interactively toggles “expand/collapse” flag for diagram elements Compute size of each element based on “expand/collapse flag” Compute layout of elements
22
Authoring Tool
Need an authoring tool which generates adaptive SVG high level representation of diagram allowing access to semantics Graphic designers don’t want to write formulas or logic
23
Constraint Compilation
Authoring Tool SVG One-way constraints Browser One-way Constraint Solver Search Engine
Layout Engine(s)
Multi-way Constraint Solver
XML/RDF
The constraints used in the authoring tool need not be one-way Our user studies show they should be multi-way In theory more powerful multi-way constraints can be compiled into one-way constraints.
24
Project 3: Automatic interpretation of diagrams using CIDER
Earlier web image formats such as GIF, PNG, JPEG do not support access to diagram semantics (although ALT tags and surrounding text may help) SVG considerably better: text in diagram is accessible. However there is still no interpretation of the non-textual diagram elements. Two possible approaches Semantic description exported by (customised) graphics editor in tandem with the SVG. Semantic description generated automatically from SVG document The first approach is much easier, but the second is more general
25
Diagram Interpretation Tools
We have built one tool which takes a barchart specified in SVG and generates its semantics in RDF And another tool which interprets labelled graphs
26
Grammar-based Interpretation
Our tools use a grammarbased approach.
The following pseudo-grammar defines the (visual) language of barcharts Barchart → Axes Bars Captions Axes → Xaxis Yaxis Xaxis → horiz_line text Yaxis → vert_line text Bars → ε| rectangle Bars
27
Grammars for diagrams
Key differences to string grammars are
Sets (or multisets) of symbols are rewritten rather than sequences of symbols. Geometric relationships between symbols are important rather than just adjacency. Context sensitivity is often required rather than just context-free grammars.
We use constraint-multiset grammars a kind of attributed multi-set grammars (Anderson).
28
Constraint Multiset Grammars
Constraint multiset grammars use production rules to rewrite (multi)sets of symbols. The symbols have associated attributes which encode geometry and also the semantics. Rewriting is controlled by constraints over the attributes of the symbols in the right-hand side of the production. Constraints can be over symbols as well as over symbol attributes.
29
Final State
T S:state → C1,C2:circle,T:text where ( C2.mid = C1.mid && Check C2.radius < C1.radius && applicability T.mid = C1.mid ) { S.mid = C1.mid; S.radius = C1.radius; Compute attributes S.name = T.text; S.kind = ”final” }
C2 C1
Name
30
Normal State
T X C1 Name S:state → C1:circle,T:text where ( not exist C2:circle where (C1.mid = C2.mid) && T.mid = C.mid ) { S.mid = C.mid; S.radius = C.radius; S.name = T.text; S.kind = “normal” }
C2
31
CIDER
Application Program CMG specification Incremental Parser
CIDER
The tools use CIDER a parser generator (like YACC) for constraint multiset grammars This was designed to help implementation of intelligent diagramming environments
32
Project 4: Table layout
Adaptation requires automatic sizing of table columns and rows HTML tables are
not powerful enough do not give good layout if they contain reasonable amounts of text or compound cells
Not surprising since automatic layout of tables is NP-hard
33
Table layout tool
Example table laid out with Mozilla
Example table laid out with our approach
We have built a number of tools that provide fast, good table layout
34
Table Layout as Constrained Opt.
Table layout is a 2-D combinatorial optimisation problem
Structural constraints
W = C1 + C2 +C3 H = R1 + R2
W C1 R1 H R2 Text 1
Heading 1
Containment constraints
C1 ≥ H1width R1 ≥ H1height C2+C3 ≥ H2width R1 ≥ H2height C1 ≥ T1width R2 ≥ T1height C2 ≥ T2width R2 ≥ T2height C3 ≥ T3width R2 ≥ T3height
C2
C3
Heading 2
Text 2
Text 3
Designer constraints: C2 = C3 Design style: the layout should (in decreasing order of importance)
ensure W ≤ page width minimize H minimize W
35
Text in Tables
The quick brown fox The quick brown fox The quick brown fox
The quick brown fox
The main difficulty in table layout is that text can take different configurations This is why table layout is NP-hard
36
The Main Idea
Our key idea: use a continuous approximation Cellheight * Cellwidth ≥ Cellarea This leads to a convex non-linear continuous optimisation problem This is solvable in polynomial time using conic programming We have also developed a specialised active set based algorithm. We then use heuristics to find a solution to the original discrete problem from the solution to the continuous approximation
37
Project 5: Constraint-based diagram editor
The original motivation for constraint-based graphics was for diagram editors (Sketchpad by Sutherland, 1963) Current diagram editors provide one-way constraints at best The project is to develop a constraint-based graphics editor and to show that constraint-based layout makes it more usable Developed Dunnart which is instrumented to facilitate usability experiments The longer term aim is to transfer ideas from Dunnart to Inkscape a widely used open source SVG editor which group members contribute to.
38
Dunnart
Dunnart provides
Persistent multi-way alignment and distribution constraints Interactive object-avoiding connector routing Directed edges Interactive constrained layout (using IPSep-CoLa) Container shapes Text shapes
And is intended to provide
The user interface is carefully designed based on ongoing user testing 39
Conclusion I
The web and other types of computer-mediated communication requires a new view of text and diagrams, one which supports sophisticated presentation, interaction and retrieval. It should allow universal access from different devices by people with different needs Such behaviour will rely on
Sophisticated. adaptive layout and explicit encoding of the semantics
40
Conclusion II
Automatic layout is increasingly important
Intelligent manipulation in authoring tools Static layout to visualise abstract data Adaptive layout of documents for the web All kinds of information: text, diagrams, equations, widgets etc
Automatic layout is really a constrained optimisation problem Most computer scientists (much less graphic designers) don’t know much about constrained optimisation
Use ad hoc techniques to solve these problems Don’t believe they are practically solvable Don’t separate specification of what makes a good layout from the algorithm used to compute it.
Great opportunity for those who know about constrained optimisation and a little bit about graphic design … 41