Two Handed Drawing
The Design and Evaluation of a Two-handed Drawing Program
A thesis submitted in conformity with the requirements
For the degree of Masters of Science
Graduate Department of Computer Science
University of Toronto
© Copyright by Jade Rubick (2000)
The Design and Evaluation of a Two-handed Drawing Program
Jade Rubick
Master of Science
Graduate Department of Computer Science
University of Toronto
2000
This thesis describes the design of a two-handed drawing program, T4, which explores the extent to which a range of two-handed techniques can provide an alternative to the traditional graphical user interface. T4 builds upon the foundation laid by the T3 system (Kurtenbach, Fitzmaurice, Baudel, Buxton 1997), which enumerated three design goals for this interface paradigm: keep the users' attention on their work, maximise screen used for artwork, and increase the quality of input. We describe the integration of a range of techniques, new and old, that aim to fulfil these goals. We also conduct a user study, which suggests that these techniques are readily learnable, integrate smoothly, and provide a viable alternative to traditional techniques.
In particular, we find the digital-tape, A-Line, and Graffiti techniques to be compelling possibilities. The other techniques (keypad, slicing, Ligne Claire, and a selecting Magic Lens) show substantial benefits, but could benefit from further research.
To conclude, we present a possible improvement to the Toolglass, which addresses user feedback, called the On-Demand Toolglass.
Acknowledgements
Symbol Technologies Industrial Design Group
Bill Buxton
Ronald Baecker
Ravin Balakrishnan
George Fitzmaurice
Gordon Kurtenbach
Paulo Pacheco
Guillaume Fraysse
Azam Khan
Christie Johnson
Table of Contents
1 Introduction..................................................................................... 1
1.1 Design Goals 2
1.2 Thesis Organisation 2
1.3 Terminology Used 3
2 Literature Review.............................................................................. 4
2.1 Two-handed Input 4
2.2 Novice and Expert Behaviour 6
2.3 Marking-based Interaction: Marking Menus 7
2.4 Focus and Attention 8
2.5 Interaction Techniques 10
3 Context: T3................................................................................... 20
4 Integration and Design...................................................................... 25
4.1 Technological Environment 25
4.2 Interaction Paradigm 26
4.3 Integrated Additions 26
4.4 Selecting and Revealing Numerical Attributes of Graphical Objects 27
4.5 Soft Keypad – Using a Keypad to Specify Object Attributes 30
4.6 Graffiti – Using Handwriting to Specify Object Attributes 33
4.7 Digital Tape Drawing 40
4.8 Ligne Claire Curve Modification 43
4.9 Slicing Technique For Modifying Shapes 45
4.10 A-Line Technique 46
5 User Study..................................................................................... 57
5.1 Rationale for user study 57
5.2 Methodology 58
6 Results......................................................................................... 62
6.1 Selecting and Revealing Drawing Attributes 62
6.2 Using a Soft Keypad to Change Object Attributes 63
6.3 Using Graffiti to Change Object Attributes 63
6.4 Digital Tape Drawing 65
6.5 Ligne Claire Toolglass 71
6.6 Slicing Technique 72
6.7 A-Line Technique 72
7 Future Work................................................................................... 74
7.1 On-Demand Toolglasses 74
7.2 Other Future Directions 77
8 Conclusion.................................................................................... 79
9 Appendix A: User Study Notes............................................................ 81
9.1 User 1, Session 1, 12/16/99 81
9.2 User 1, Session 2, 12/17/99 83
9.3 User 1, Session 3, 12/22/99 84
9.4 User 2, Session 1, 12/20/99 86
9.5 User 3, Session 1, 12/20/99 87
9.6 User 4, Session 1, 12/21/99 87
9.7 User 4, Session 2 , 12/22/99 87
9.8 User 5, Session 1, 12/21/99 88
9.9 User 6, Session 1 , 12/22/99 89
9.10 User 7, Session 1, 12/23/99 89
10 Appendix B: Technological Environment............................................. 92
10.1 OpenGL 92
10.2 Wacom Tablet 92
10.3 Windows NT 93
10.4 Visual C++ 6 and MFC 93
11 Appendix C: Additional Design Issues................................................ 94
11.1 Drawing Space 94
11.2 Colouring 94
11.3 Input Decisions 95
12 Appendix D: Selected Algorithms...................................................... 96
12.1 Ligne Claire Algorithm 96
12.2 A-Line Technique 97
13 Appendix E: User Study Findings.................................................... 100
13.1 Digital Tape Drawing 100
13.2 Toolglasses 102
13.3 Marking Menus 107
13.4 Sketching And Templates 108
13.5 Line Width 108
13.6 Tablet / Tools 108
13.7 Task 109
13.8 Two-Handed Interaction Issues 110
13.9 Overall 110
13.10 Feature Requests 113
14 Appendix F: Implementation History................................................ 114
15 Appendix G: Design Goal Evaluation Details...................................... 122
16 Bibliography.............................................................................. 124
Table 5‑1: User experience levels and computer usage. 58
Table 5‑2: Subject session length and frequency. 59
List of Figures
Figure 2‑1: Hierarchical marking menu, with two selection methods, (a) and (b) 7
Figure 2‑2: Toolglass interaction. 9
Figure 2‑3: Graffiti numbers are very similar to Arabic numbers. 11
Figure 2‑4: Object creation using the two-handed stretching technique. 12
Figure 2‑5: Tape artist drawing with physical tape. 13
Figure 2‑6: Tape drawing of a car interior using traditional tape 13
Figure 2‑7: Digital tape tool upon a projection wall. 14
Figure 2‑8: Editing a curve with successive editing strokes (in grey) 14
Figure 2‑9: Comparison of slicing technique and point and click technique. 15
Figure 2‑10: Rotating a graphical object using the Turntable technique. 16
Figure 2‑11: Using the turntable tool 17
Figure 2‑12: The pivot point is specified by placing the "anchor" at the base of the arrow. 18
Figure 2‑13: The arrow is grabbed at the tip, at point A, and rotated by moving the mouse. 18
Figure 3‑1: T3 prototypes set-up, using two pucks on two Wacom tablets 21
Figure 3‑2: A screenshot of the T3 interface. 22
Figure 3‑3: A view of the Toolglass. 23
Figure 3‑4: Object creation using the two-handed stretching technique and Toolglasses. 23
Figure 3‑5: The menu bar acts as a marking menu. 24
Figure 4‑1: Selecting and revealing object attributes. 28
Figure 4‑2: The Toolglass as a selection mechanism. 28
Figure 4‑3: Graffiti Toolglass, showing a screen dense with vertices. 29
Figure 4‑4: Graffiti Toolglass, showing feedback for the selected line. 30
Figure 4‑5: Early iteration of the keypad Toolglass. 31
Figure 4‑6: Sketch of one possible solution to soft-keypad obscuration problem. 31
Figure 4‑7: Revised Toolglass, with the user entering values for the side's new length. 32
Figure 4‑8: Revised Toolglass, showing the result after the user clicks the enter button. 32
Figure 4‑9: a conceptual example of using Graffiti to change object attributes 33
Figure 4‑10: The Graffiti Toolglass. 34
Figure 4‑11: The strokes for backspace or erase (A) and for execute or enter (B). 34
Figure 4‑12: The Graffiti Toolglass, in length change mode. Here the value 279 is not self-revealing. 37
Figure 4‑13: Graffiti orientation issue. 38
Figure 4‑14: An older version of the digital tape tool, with three cursors. 40
Figure 4‑15: The digital tape tool. 41
Figure 4‑16: Lines and curves being drawn in Adobe Illustrator 7.0 for the Macintosh. 42
Figure 4‑17: The digital tape drawing tool. 42
Figure 4‑18: This diagram shows the user making a curve. 43
Figure 4‑19: When the left button is not held down but the puck moves, a green line provides feedback. 43
Figure 4‑20: Ligne Claire Toolglass with the user making a mark. 44
Figure 4‑21: Occasionally, the algorithm produces unexpected results. Error! Bookmark not defined.
Figure 4‑22: Expected outcome after the Ligne Claire mark. 45
Figure 4‑23: Slicing technique. 45
Figure 4‑24: The initial state. 48
Figure 4‑25: The goal state 48
Figure 4‑26: Invoking the A-line tool to the square by clicking through the Toolglass. 48
Figure 4‑27: Positioning the hands over the points on the object to be aligned, a and b. 48
Figure 4‑28: Selecting points a and b. 49
Figure 4‑29: Aligning by moving hands to points A and B, respectively. 49
Figure 4‑30: Completion on release of the puck button. 50
Figure 4‑31: Rotation around a pivot which is one of the initial selection points. 51
Figure 4‑32: Rotation around a pivot which is not part of the object. 51
Figure 4‑33: Hand & Object Movement. 53
Figure 4‑34: Translation through non co-linear motion by both hands. 54
Figure 4‑35: Translation by same-direction simultaneous movement of hands. 54
Figure 4‑36: Correcting object position by constrained dragging. 55
Figure 4‑37: Constrained dragging of other objects. 56
Figure 6‑1: Examples of difficult curves with the digital tape. 66
Figure 6‑2 detailed drawings can be difficult for users at first. Scaling in helps to alleviate this problem. 67
Figure 6‑3: User 2 was able to draw pictures of this complexity after 90 minutes. 70
Figure 7‑1 Near-current version of Toolglass 75
Figure 7‑2 Concept sketch of a new Toolglass 75
Figure 7‑3: The two cursors in the default mode. In this mode, the cursors act like digital tape tools. 76
Figure 7‑4: On-demand Toolglass 76
Figure 7‑5: sketch of on-demand Toolglass's marking menus. 77
Figure 11‑1 Wacom tablet, with a puck and stylus, the setup used for T4 92
Figure 11‑2 Puck for the Wacom tablet 93
Figure 12‑1: Scaling in T4 94
Figure 13‑1: Like-This or More-Like-This Curve algorithm. 96
Figure 13‑2: A and B represent the new locations of the cursors, and C and D the previous positions. We want to move our object to a position between A and B. 98
Figure 13‑3: We rotate around pivot point P to match the slope of the two cursors, A and B. 98
Figure 13‑4: We then translate along the vector from P to the line AB. 98
Figure 13‑5: the cursor is mapped to the constraint line. 99
Figure 14‑1: Narrowing the area in which cursor motion is interpreted as erase may decrease erase problems. 100
Figure 14‑2: Users tried to click through the button (labelled A) instead of seeing that as a mode, and swiping through the shape. 104
Figure 14‑3: Toolglass without any distinction between classes of buttons 105
Figure 14‑4 revised Toolglass, separating functionality by group and shading, and optimising layout for right-handed interaction 105
In this thesis, we describe the design and evaluation of a two-handed drawing program called T4. This work is a design exploration, motivated by the goal of creating drawing interaction that is more "natural", easier to learn and use, faster, and in general more effective for the artists. This design studies the integration of a range of techniques, some new, and others known but not commonly used. The goal is to explore the extent to which these techniques can work together to provide a compelling alternative to the current, ubiquitous graphical user interface (GUI).
The techniques we present are described in detail in the Literature Review chapter, on page 4. Briefly, however, the techniques we explore include the following:
• A method for selecting and revealing the numerical attributes of graphical objects.
• Using a keypad to specify the attributes of graphical objects. We use a soft keypad, similar to a telephone keypad, to change the attributes of drawn objects, such as the length of a line segment.
• A similar technique using handwriting to change object attributes. We use Graffiti, an alphabet similar to English but different in that each letter can be created with one stroke, to specify attributes of graphical objects.
• A "digital-tape" technique, which allows for the creation of perfectly straight lines and smooth curves with a seamless switch between the two.
• A technique for editing curves by drawing over them, rather than manipulating control vertices (CVs).
• A slicing technique for modifying shapes. We use this method for cutting objects in half, deleting objects, and for adding or removing vertices from an object.
• A new alignment technique, called A-Line. We present an alignment tool that allows alignment and distribution along any line.
1.1 Design Goals
These techniques represent a broadening of the vocabulary with which the artist can express their goals. However, the method in which this vocabulary is presented is equally important. In the process of designing and evaluating the T4 prototype, a number of design goals were used as reference points, to determine the effectiveness of the interaction styles used. These design goals are as follows:
• Keep the users' attention on their work
• Maximise screen used for artwork
• Increase the quality of input
Note these design goals are identical to the design goals for the original "T3" project, described by Kurtenbach, Fitzmaurice, Baudel, and Buxton (1997).
1.2 Thesis Organisation
Our design was affected by research on the techniques presented earlier, but it was also motivated by a collection of higher-level research on two-handed input, workflow and expert behaviour, marking based interfaces, and focus and attention. Both the research on these techniques and the higher-level research are presented in detail in a Literature Review.
T4 builds upon the work on T3. T3 was a simple "next-generation" MacDraw-like application, used for creating two-dimensional drawings. Due to its importance as a precursor to the T4 prototype, T3 is reviewed in a separate chapter, Context: T3.
We then present a description of the design, the problems faced during the integration and design process, and the choices we made in resolving these problems, in the Design and Design Rationale chapter.
We follow with a User Study that returns some tentative conclusions about the effectiveness of the techniques we explore. The results are very positive for two-handed input, but indicate that a lot of careful design and continued research are necessary.
Finally, we present our conclusions and discuss future directions for research.
1.3 Terminology Used
For simplicity, we often refer to right-handed or left-handed instead of the more correct but cumbersome terminology of dominant hand and non-dominant hand.
T4 is built upon a great body of previous research. The most important research in this area is on two-handed input, novice and expert behaviour, marking based interaction, and focus and attention. In this chapter, we will review relevant aspects of this literature. In addition, we will explore the research on techniques that have been developed in light of these ideas.
2.1 Two-handed Input
Psychology has historically focused on our two-hands as competing for dominance in our brain. This is reflected in our usage of the terms, right-handed and left-handed. The question you are asked is if which hand you use for writing, eating, and brushing your teeth.
What this viewpoint misses is that most manual activities are essentially bimanual. When we write a letter, paint a picture, play a guitar, lift a heavy object, read a book, type on the computer, eat with a knife and fork, drive a car, swing a bat or club, shoot a rifle or bow and arrow, or pull a rope, we are using both of our hands. Some of these might not be immediately obvious – for example, why writing with a pen is a two-handed task. However, when we observe people writing, they invariably use the left hand to orient and position the paper they are writing on. In fact, one informal study found that writing speed decreased by 30% when subjects were forced to use only one hand to write (Guiard 1987).
We will not cover the historical views of handedness and bimanual manipulation, which are amply covered in Guiard (1987; and Guiard and Ferrand 1995). However, research has progressed greatly in this area in the last twenty years.
We should first clarify the different forms on manipulation humans engage in. First of all, a division should be made between symmetric and asymmetric manipulation. Examples of symmetric manipulation are lifting a barbell, and pushing on a wall. In these examples, the movements are similar or the same in both hands, and they are either happening simultaneously or out of phase.
Asymmetric manipulation is by far the most common, and has come to prominence in HCI research through the work of Guiard. Generally, asymmetric bimanual manipulation can also be termed co-operative bimanual manipulation, because the two-hands take on different but complimentary roles to accomplish a single task.
Guiard introduced the Kinematic Chain (KC) model, which describes asymmetric bimanual manipulation as being like two abstract motors in a serial chain, which cooperate in a kinematic sense similar to the way the arm and hand do (Guiard 1987; Guiard and Ferrand 1995). Some principles of the model are as follows:
• Non-dominant hand precedence – the non-dominant hand initiates a gesture that involves two hands. Thus, for example, when a person sits down to write, the movement in initiated by the left hand positioning the paper, followed by the right hand writing upon it.
• Asymmetric scales of motion – the frequency of movement for the right hand is greater than the left; hence, the right hand tends to make finer, more detailed movements, while the left makes larger movements.
• Dominant to Non-dominant Spatial Reference – This is shown in the example of writing on a sheet of paper, where the left hand sets up the context for the right hand. It is also demonstrated in the case of a painter's palette, where the painter dips the paintbrush in the palette, which is positioned by the left.
2.1.1 Two-handed Input in the Computer World
It wasn't until 1986 that anyone formally investigated how computers could take advantage of both hands (Buxton and Myers 1986). In this study, two experiments showed reduced time performance when the subjects used both hands as opposed to one hand. This paper opened up a new area of research in human-computer interaction.
Subsequent studies refined the understanding of two-handed interaction, and Guiard's KC theory became better known in the research community. As an example of the refinements subsequent research produced, Kabbash, Buxton, and Sellen showed that two-handed input had to be developed carefully (essentially, with an understanding of Guiard's KC model), or two-handed input could result in worse performance (1994). The same paper showed significant advantages of two-handed techniques that took advantage of asymmetrical bimanual motor and cognitive skills.
2.2 Novice and Expert Behaviour
Buxton summarises the differences between expert and novice user performance in his 1986 paper on chunking and phrasing. The level of granularity at which novice and expert users view a system is different. A novice user pays great attention to every detail of the system, and their behaviour may best be described as "problem-solving". In contrast, expert users have acquired the skills to automatically execute the low-level details, and their attention is focused on the task at hand (Buxton 1986). For example, if we imagine a novice and an expert user of a bicycle, the novice's attention is consumed with the mechanics of riding, and they are trying to "figure out" the bicycle. The expert bicyclist, however, is focusing on much higher-level details, such as where they are going, and perhaps the way in which they pedal, fast or slow. Buxton points out that a goal in designing computer systems can be to minimise the transition time between novice and expert behaviour (1986).
2.3 Marking-based Interaction: Marking Menus
Marking menus were developed in response to the problem of marks not being self-revealing with their functionality. They were originally described in 1991 [J1](Kurtenbach and Buxton 1991) and refined in successive papers (Kurtenbach and Buxton 1993).
Figure 2‑1: Hierarchical marking menu, with two selection methods, (a) and (b)
There are several things that are remarkable about marking menus. First of all, the user has a self-revealing system, similar to normal menus, which they can browse, as shown in Figure 2‑1(a). It thus furthers recognition rather than recall. Second, as the user begins to learn the location of various items in the marking menu, they can make the stroke without waiting for the menus to come up (Kurtenbach and Buxton 1991), as in (b). This is one of the best examples of an interaction technique in which the novice behaviour trains the user for expert behaviour. This is in contrast to many other expert-level techniques, such as hot-keys, which require learning a completely different type of interaction in order to increase proficiency with the system.
Marking menus can be a simple stroke (split into n directions), or they can be hierarchical. Research suggests an upper bound of eight items in each level as a ceiling (Kurtenbach and Buxton 1993), but this depends on what error rates are acceptable and other factors.
2.4 Focus and Attention
Traditional GUIs focus on visual perception. This has allowed them to leverage many existing graphic arts skills in the design of visual interfaces; however, it may also have drawbacks (Harrison, Ishii, Vicente, Buxton 1995). As Harrison, Ishii, Vicente, and Buxton point out, an alternative is to focus on attention. This has the added benefit of looking at how layers and depth affect the user, and reflects a focus on what the user is actually noticing rather than what is there. For this reason, it may be a more user-centred approach. This concept is especially important when we consider interfaces that take advantage of transparency.
2.4.1 Toolglasses & Transparency
One such interface is the Toolglasses and Magic Lenses interface, an idea that came out of Xerox PARC, and reflected this focus on attention (Bier, Stone, Fishkin, Buxton, Baudel 1994; Bier, Stone, Pier, Buxton, DeRose 1994). Toolglasses are a two-handed technique in which the non-dominant hand controls a semi-transparent tool palette. The palette can either be a lens that reveals extra information about the underlying object (a Magic Lens) or an actual tool palette (a Toolglass). Bier, Stone, Fishkin, Buxton, and Baudel present many types of Toolglasses (1994) but generally, the mechanism is click-through. This is shown in Figure 2‑2. The user controls the Toolglass with their non-dominant hand, and the cursor with their dominant hand. Assume there is an object with an attribute "C". The user moves their non-dominant hand to position the Toolglass over the object. Next, or simultaneously, they move their dominant hand to position the cursor over the Toolglass and the object. The user it then presses a button on their dominant hand and the attribute "B " is transferred to the object, replacing the value of "C".
(a) (b) (c) (d)
Figure 2‑2: Toolglass interaction.
The user positions the Toolglass over the object to be modified,
clicks through the Toolglass on to the object, and changes that attribute from C to B.
The original Toolglasses paper describes object creation like a cookie cutter approach. The user places the Toolglass over the location they want the new shape. When they press the button, the new object is created in that location.
In experiments with the Toolglass techniques, the Toolglass techniques proved themselves to be markedly superior speed-wise to other two-handed and one-handed techniques (Kabbash, Buxton, and Sellen 1994). This is probably due to their ability to leverage proximity and maintain focus on the artwork, requiring little time to refocus, in line with Harrison, Ishii, Vicente, and Buxton's suggestions (1995).
2.5 Interaction Techniques
Whereas the research we previously summarised represented higher-level, theoretical frameworks within which to guide design, the following techniques represent forays into possible designs. We explore researched techniques for selecting and revealing values of object attributes, using a keypad to specify those same attributes, using handwriting to specify these graphical attributes, "digital-tape" drawing, curve editing, a slicing technique, and an alignment technique.
2.5.1 Selecting and Revealing Numerical Attributes of Graphical Objects
This technique has not been previously researched, but is described in Selecting and Revealing Numerical Attributes of Graphical Objects, on page 27. It is based upon the research on Toolglasses and Magic Lenses, as described in Toolglasses & Transparency, on page 8.
2.5.2 Using a Soft-keypad to Specify Graphical Attributes
This technique has not been previously researched, but is described in Soft Keypad – Using a Keypad to Specify Object Attributes, on page 30.
2.5.3 Using Handwriting To Specify Graphical Attributes
Using handwriting to specify the attributes of drawn objects is a new technique based in research on pen-based input. Goldberg and Goodisman developed the first computer-based unistroke alphabet for text entry (1991). A unistroke alphabet is an alphabet in which each letter can be made in a single stroke. Goldberg and Goodisman's alphabet was designed for text-based entry using a stylus, and was inspired by work by Buxton, Sniderman, Reeves, Patel, and Baecker on a unistroke alphabet for scoring music (1979). The time required to learn Goldberg and Goodisman's alphabet was significant, but the speeds were much greater than a later system developed by 3Com for the Palm Computing platform. This system, Graffiti, proved to be very popular because of its similarity to the English alphabet. Mackenzie demonstrated that users with no familiarity of the Graffiti system could achieve 86% accuracy after one minute study of the system, and 97% accuracy after five minutes of practising with the system (MacKenzie and Zhang 1997). These results are for the 26 letters of the alphabet.
Figure 2‑3: Graffiti numbers are very similar to Arabic numbers.
For performance, we restrict our view to numerical entry, as shown in Figure 2‑3. Graffiti performance is about 16.7 words per minute to 17.6 words per minute, and its accuracy ranges from 87% to 94%. Of course, performance varies depending on the recogniser used, experimental conditions, and many other factors. This performance is definitely slower than on a soft keyboard (30 words per minute, with 1.2% errors) (MacKenzie, Nonnecke, Riddersma, McQueen, and Meltz 1994).
2.5.4 Digital Tape Drawing
Digital tape drawing builds upon a
two-handed stretching technique first developed by Krueger (1991), and refined
by Leganchuk (1996). This interaction is described in
Figure 2‑4.
Figure 2‑4: Object creation using the two-handed stretching technique.
The user moves two cursors on the screen, as shown in (a). When they desire to make a shape, they initiate the creation, perhaps by the use of a button on one of the input devices. The interface displays a shape defined by the position of both cursors, as in (b). the user stretches out the shape to the desired location and shape, as in (c), and then lets up on the button, placing the object, as in (d).
Note this allows for translation and