A Dynamic Guide for MultiTouch Gestures

Arpege helps users learn and execute chord gestures on multi-touch surfaces by providing finger by finger Feedforward (guidance) and Feedback in a playful and efficient way. Arpege decomposes each chord in an Arpeggio, as a piano player would decompose a chord note by note. An experiment suggests that Arpege took no more time than a conventional “cheat sheet”, enabled efficient learning of chord gestures, and resulted in almost three times fewer errors. Techniques such as Arpege enables users to take advantage of the expressivity of abstract multi-finger chord gestures in real-world applications, offering a rich and powerful form of interaction on multi-touch surfaces.

Keyset Stenograph
Chord gestures consists of the simultaneous placement of multiple fingers on a touch-surface. These gestures are inspired by piano chords and stenographs (Right). The Keyset (Left) is the first attempt to use such gestures to interact with computers. Engelbart demonstrated the device in his 1968 Demo, along with the mouse. While chord gestures are a very efficient way of interacting with a computer, they suffer from a very steep learning curve. Arpege helps users to learn and execute such chord gestures.

How does Arpege Work?

Arpege Operating Principles

In this example, the user wants to issue a cut command, a three-finger chord gesture. a) A double tap produces finger prints, with labels for possible commands. b) He places his ring finger on the cut command circle; paste disappears. c) He then places his middle finger on the next circle; copy disappears. d) He completes the gesture by placing his index finger on the final circle, which executes the cut command.

Arpege Feedforward

Arpege guides the user to learn and execute a chord gesture by decomposing each chord in its Arpeggio. First, to invoke a "cut" command for example, the user first looks for the command label around the displayed finger prints. As the user begins to follow the path, Arpege is updated to guide the next finger placement until the user reaches the last finger placement and releases all the fingers to invoke the command.

Arpege Feedback

Feedback provided by Arpege gives the user information about how the current input is interpreted. In this example, placing the finger on a position which is common to the cut and copy command results in two alternatives. After placing a second finger, the path to the copy command disappears because recognition became impossible at this point.


Arpege: Design and Learning of multi-finger chord gestures.
O. Bau, E. Ghomi and W. Mackay. Report #1533, LRI. February 2010. [pdf]

Team and Credits

Arpege was developed in collaboration with Emilien Ghomi in 2009 while at In|Situ| INRIA, Paris, France.