How to optimize switch virtual keyboards to trade off speed and accuracy

An algorithm to solve the speed/accuracy trade-off

We describe the design problem in a general way because it demonstrates the common aspects of keyboard design for many different situations. An instance of the virtual keyboard design problem consists of an integer set \(\mathcal {I}=\{1,2,\dots,N\}\) that refers to characters and an integer set \(\mathcal {K}=\{1,2,\dots,N\}\) that refers to keyboard locations. The design task is to assign the characters to keyboard locations. Let F _i be the frequency of the i-th character in a given text corpus. These character frequencies might be estimated from general databases, or might be based on the specific type of text entered by a specific user.

First consider the time needed to reach a location on the keyboard (speed of entry). Let S _k be the number of cursor steps required to reach location k from the start of the cursor path. For example, for the row–column cursor path shown in Fig. 2, to reach position 12 (supposing the keys are indexed in sequence from left-to-right and top-to-bottom), it will first take two steps to reach the second row, and then four steps to reach the fourth column in row 2. Therefore, in this cursor path, the total number of steps to reach position 12 is S ₁₂=2+4=6. Note that the value of S _k depends on the cursor path; different cursor paths may lead to quite different values of S _k even for the same key position. In addition, we let D be the duration that a cursor will stay on each step of the path. Therefore, the time to reach position k is D×S _k .

Now consider the accuracy of reaching a location on a keyboard. Let P _k be the probability of the user making an error while trying to guide the cursor to position k. It is reasonable to imagine that P _k increases as D decreases (with a slower cursor the user is less likely to make an error), but the exact nature of the relationship needs to be measured or modeled. We will measure and model the relationship in later sections.

which gives the average error probability across all character entries.

In general, C _t and C _e trade off each other. For example, placing the most commonly used characters near the beginning of the cursor path will decrease C _t , but if those locations are error prone, then C _e will increase. We suggest that a practical way to trade off speed and accuracy is to identify a user-defined acceptable error rate, ε. We then describe the problem as assigning characters to the keyboard so that C _e ≤ε and C _t is minimized.

where the indices 55–64 for the first coordinate correspond to the numerical characters 0–9 and the indices 55–64 for the second coordinate indicate the last ten locations on the keyboard. Figures 1, 2, 3, and 4 (and Additional file 1: Movie 1, Additional file 2: Movie 2, Additional file 3: Movie 3, and Additional file 4: Movie 4) show keyboards that follow these constraints.

Constraint (4a) seeks to minimize the average time needed to reach items on the keyboard. Constraint (4b) ensures that each character can be assigned to only one position on the keyboard. Constraint (4c) ensures that each position in the keyboard layout contains only one character. Constraint (4d) determines the arrangement of the numerical characters as mentioned above. Constraint (4e) ensures that the average error rate is no larger than the user-identified acceptable error rate ε. Constraints (4f) and (4g) are variable-type constraints that ensure the objective functions remain true to their definitions.

As an initial check on the MIP approach, we took the S _k , F _i , D, and P _k terms as defined by Francis and Johnson (2011) and used Gurobi Optimizer 5.6 (Gurobi, 2013) to solve the MIP problem. The resulting solution of characters assigned to locations on the keyboard was very similar to what was produced by Francis and Johnson (2011) with a hill-climbing algorithm. However, the MIP algorithm was much faster. It took approximately 3 seconds to find a solution while the hill-climbing algorithm took approximately 3 h to generate essentially the same keyboard layout. Furthermore, the MIP algorithm is guaranteed to find an optimal keyboard layout that satisfies the acceptable error rate condition (if one exists), while the hill-climbing algorithm yields a keyboard layout that may not be optimal.

For any given keyboard design task, most of the terms in the MIP model are readily available: the cursor duration, D, can be taken as a given value for any instance; the character frequencies, F _i , can be estimated from a text corpus; the number of cursor steps, S _k , can be calculated for a given cursor path; and characters assigned to fixed positions, \(\mathcal {G}\), can be readily created as needed. The only terms that remain to be identified are the error probabilities for different keyboard positions, P _k . In the next section, we describe how to estimate these terms with a behavioral study.

Estimates of error probabilities

This section demonstrates one way to estimate the P _k terms that are needed to drive the optimization approach described above. P _k corresponds to the probability that a user guiding the cursor to key k on a switch keyboard makes an error some place along the cursor path to that keyboard location. Ideally, these values would be guided by basic research on how quickly and reliably humans respond to dynamic stimuli. Although the literature on reaction time, timing, and predicted movements is enormous (Jensen, 2006; Posner, 1978), we were unable to identify published work that provides a model framework for characterizing the probability of correct responses for the kind of task involved in using a switch keyboard. Speed/accuracy trade-offs are commonly studied in areas such as traditional typing (Yamaguchi, Crump & Logan, 2013), but there the users largely control their actions rather than time their action to coincide with a stimulus event (the cursor being over an appropriate section of the keyboard). Likewise, the switch keyboard task seems related to simple reaction times, but effective use of the keyboard involves planning a precisely timed action rather than quickly responding to a stimulus. Such plans may occur well before the cursor covers a relevant part of the keyboard. Perhaps the area of basic research closest to the actions involved in a switch keyboard are measures of coincidence timing (Smith & McPhee, 1987). Even here, though, the fit is not perfect, as the cursor movements are highly learned and initiated by switch keyboard users. Ultimately, we suspect that switch keyboard users memorize a planned set of timed actions that are learned for commonly used characters with a specific cursor duration (for example, see the user video at http://www.assistiveware.com/a-pivotal-role-in-the-household).

Here we describe an experiment that estimates these error probabilities for the different locations of the keyboard for several different cursor durations. These estimates will both contribute directly to the optimization methods described above and provide basic research into the accuracy of a sequence of timed actions. For the design task, the ideal experiment would estimate these probabilities from the same kind of individuals who will ultimately use the switch keyboard. In many cases, the ultimate users are patients with severe motor disabilities and their performance with a switch device might be quite different from a non-patient population. To complicate matters further, performance on the switch keyboard should be estimated after users have had substantial experience using the switch keyboard; otherwise the estimated probabilities will become inaccurate as users improve with additional practice.

With these constraints in mind, it seemed impractical (and perhaps unjustifiable) to ask patients to participate in a tedious experiment to measure performance on a switch keyboard until we had demonstrated the validity and value of the overall optimization approach. Thus, rather than using locked-in patients, we recruited seven students from Purdue University to complete up to 25 experimental sessions. The use of normal subjects rather than patients with motor disabilities restricts our conclusions about which keyboards should be used for a given person, but this was always going to be a conclusion of this kind of study. The optimized keyboard design for a given patient will depend on the characteristics of the user, their familiarity with the device, the type of text they enter, and their acceptable error rate. Since our intention is to demonstrate an optimization algorithm that can accommodate these individual characteristics, gathering data from normal students is appropriate (and much easier). Of course, interesting optimization outcomes may also be derived by specifically studying the characteristics of patients.

Subjects

Seven student participants used a button-style switch device (Origin Instruments, 2013) that is similar to a computer mouse, but specially designed for people with motor disabilities that use switch keyboards. A participant could attend at most two sessions each day (in the morning and afternoon, respectively). Before the formal experiments, a 1 h tutorial was given so that the participant could practice and get familiar with the keyboard and the switch devices. To motivate participants to perform their best, each participant received a base of $5 for each session and a reward of $0.025 for each correct character entry. On average, participants earned around $45 for a session.

Modeling of error probabilities

To promote optimal switch keyboard designs, we wanted to use the empirical data to develop a model that could predict performance for different cursor paths and different cursor durations. With that goal in mind, we focused on modeling a user’s ability to trigger the switch device at the appropriate times. Each successful trial in the row–column keyboard involves two such triggers: one for selecting the row containing the target and one for selecting the column/key within the selected row. An unsuccessful (error) trial may have one correct or incorrect trigger selection (for selecting a row) but may not have any selection for a column (because there is no need if an error were already made by selecting the wrong row). On some error trials, there may be no selection at all (e.g., the cursor moves past the row containing the target). We hypothesized that trigger timing performance would be a function of the cursor duration and the number of cursor steps from the cursor start or last trigger (e.g., the row or column number).

The same type of probabilities can be used to predict performance for a quadrant cursor path, where the predicted correct entry probability to reach position k would be π _quadrant(k) π _row(k) π _column(k) and the predicted error entry probability is 1 minus that product. Similarly, for a linear cursor path, there would be a single opportunity to make an error with probability 1−π _k and for the binary cursor path there would be six opportunities to make errors with associated probabilities.

where s indicates the number of cursor steps since the last selection action (e.g., the quadrant, row, or column to be selected). The β weights were estimated from the experimental data using logistic regression to produce β=[−1.85,21.20,0.41]. We also considered a model with an interaction of the cursor duration and the number of cursor steps, but it made the nonsensical prediction that performance deteriorates when the cursor duration increased beyond 400 ms.

To explore the behavior of the model, these parameter estimates were then entered into Eq. (6) to produce estimated correct selection probabilities of \(\hat {\pi }_{s}\) for s=1,…,8 and cursor durations D=0.1, 0.125, 0.150, 0.175, and 0.2 seconds. For the row–column keyboard, the predicted probability of a correct entry is the product of the selection probabilities for the row and column of a target key. To get a sense of how well the model matches the experimental data, Fig. 8 plots the proportion of correct entries for the experimental data against the model-predicted proportion of correct entries for the five experimental cursor durations. Here, only those conditions (key location and duration) with 100 or more trials are included because the model cannot be expected to match the proportions for poorly measured cases. The error bars indicate one standard error for the experimental data (for many points the error bars are smaller than the symbols). Ideally, the points would fall on the diagonal line from bottom left to top right, and while the model is not perfect, the predicted performance does correlate somewhat with the experimental data, r=0.57.

To understand the model’s behavior further, Fig. 9 displays the predicted correct entry percentages in the same format as in Fig. 7. Unlike the data, the model has no missing conditions, but it generally behaves similarly as the data. In particular, performance deteriorates as the cursor duration decreases. For a cursor duration of 200 ms, there is little variation in performance across the different keyboard locations. For shorter cursor durations, the model predicts improved performance for locations reached with late selections.

Although far from perfect, our overall impression is that the model does a good enough job of accounting for the experimental data that it is fruitful to explore how it contributes to the optimization of switch keyboards. In particular, the model predicts performance for cursor durations and cursor paths that were not measured in the experiment. Hopefully, future research and theories will lead to better model predictions and thus better optimized switch keyboard designs.

Creating optimized keyboards

Note that the correct entry probabilities in Eq. (6) depend on the value of the cursor duration, thus the average error rate also depends on the cursor duration. The cursor duration is restricted by the display capabilities of computer hardware. We supposed that a computer monitor refreshes the display at 100 Hz, which means that the shortest cursor duration would be 10 ms and that longer cursor durations would be multiples of 10 ms (covering multiple frames of the display). For a fixed acceptable error rate, we considered all possible cursor durations between 10 and 1,000 ms, in steps of 10 ms. For each of these cursor durations, the MIP algorithm identified the optimal placement of characters to keys to satisfy the acceptable error rate and minimize average search time. The optimal keyboard is the one with the cursor duration that produces the shortest entry time and satisfies the acceptable error rate.

Optimal row–column keyboards

We start by considering a row–column keyboard. Figure 10 shows the character arrangements for optimal keyboards that satisfy different acceptable error rates. According to the entry error model, both of these keyboards perfectly match their respective acceptable error rate. Not surprisingly, the optimal cursor duration depends on the acceptable error rate. To satisfy the ε=0.1 acceptable error rate, the cursor duration must be rather long, D=0.19 seconds (190 ms), while a much shorter cursor duration can be used when the acceptable error rate is ε=0.5, D=0.02 seconds (20 ms).

To make the average entry time small, one might expect that the optimal design places the most frequent characters early in the cursor path and sets the cursor duration long enough to ensure that the error probabilities for these characters do not become too high. Indeed, this strategy is used for the ε=0.1 condition. Here, the most common characters (space, “t”, “e”, “a”, “o”, and “i”) are located along the first few rows and columns of the keyboard. Less frequent characters, such as “&”, “%”, and “]”, are placed at locations that require more cursor steps. This strategy is a good one because the most frequent characters are reached with few steps, which tends to minimize average entry time.

However, an alternative approach is used for the ε=0.5 condition. Here the cursor duration is set to be very brief, which means that there is a high probability of making an error for the first few rows and columns. High-frequency characters (“s”, “o”, “e”, “a”, and “t”) tend not to be placed on those error-prone locations and instead are shifted to later rows and columns that have a lower probability of error. The large number of cursor steps needed to reach these frequent keys is offset by the short cursor duration, so the average entry time is low.

Generally, for a fixed acceptable error rate, a decrease in the cursor duration leads to a decrease in average entry time. However, this relationship is not always the case; sometimes increasing the cursor duration led to a decrease in the average search time because an increase in the cursor duration alters the probabilities of making an error and thereby enables a reconfiguration of the characters that leads to faster entry. This kind of effect highlights the need to consider a range of cursor durations and to identify the optimal character assignments to best satisfy the accuracy and speed goals of an optimized keyboard design.

Optimal keyboard designs for other cursor paths

Figures 11, 12 and 13 show the properties of the optimal designs for keyboards using a linear, quadrant, or binary cursor path, respectively. As shown in Fig. 11, a linear cursor path, where the cursor scans over individual keys, generally uses the shortest possible cursor duration (0.01 seconds) and assigns characters to different keys to alter the average error rate and average entry time. For the ε=0.5 condition, the optimal keyboard produces an average error rate of 0.35, which is lower than the acceptable error rate of 0.5. This mismatch is because the cursor duration is already at its smallest possible value, and the most frequently used characters are already assigned to their most error-prone locations (e.g., the top row). It simply is not possible for this cursor path to further trade off accuracy for speed for this text corpus.

Figure 12 shows two optimal quadrant keyboards, where the cursor first moves over different quadrants and then rows and columns within a selected quadrant. Both keyboards tend to place high-frequency characters in the first two quadrants (top-left and top-right quadrants). They differ in the placement of low-frequency characters and in the cursor duration, with the keyboard requiring a low acceptable error rate having a cursor duration nearly twice as long as the keyboard requiring a high acceptable error rate.

Figure 13 shows two optimal binary keyboards, where the cursor alternates between remaining halves of the keyboard to zero in on the desired key. Here, a change in the acceptable error rate from 0.1 to 0.5 hardly alters the assignment of characters to keys (only the “m” and “p” characters switch positions), which makes some sense because different key positions are nearly equivalent due to the way the cursor focuses in on a target key. For example, with a binary cursor path, 20 of the 64 keys are reached with nine cursor steps, while an additional 30 keys are reached with eight or ten cursor steps. With such homogeneity, changing the key assignments for a binary cursor path often has very little impact on the average entry time. In a similar way, key assignments have only a modest effect on the error entry rate. Thus, the main variable controlling speed and accuracy for a binary cursor path is the cursor duration, which is longer for smaller acceptable error rates.

A user of a switch keyboard wants to identify the best possible keyboard design for the type of text entry they will use, and for an acceptable error rate. As described above, the optimal design depends on the cursor path, so users should compare the optimal keyboard designs for different cursor paths and choose the best. Figure 14 summarizes the properties of optimal keyboard designs for different cursor paths as a function of the acceptable error rate. Critically, Fig. 14 a indicates that for all but the highest acceptable error rates, the linear cursor path produces a much smaller average entry time than any other cursor path. This advantage for the linear cursor path is because it involves a single switch action, so there is only one opportunity for the user to make an entry error. In contrast, the row–column cursor path always involves two switch actions, and the quadrant and binary cursor paths have more switch actions and therefore more opportunities for an entry error. The linear cursor path leverages this advantage by making the cursor duration extremely brief and assigning characters to keys in a way that satisfies the acceptable error rate.

It is notable that the optimal solution for a linear cursor path does not always look like a very good keyboard design. For example, the keyboard in Fig. 11 b places the space character at the first key. The logistic model predicts that a user has a 0.77 probability of making an entry error at this key, so this most commonly used character is unlikely to be entered correctly. Such a property seems to make the keyboard essentially unusable, but this limitation simply reflects the requirements imposed on the design. A keyboard with an acceptable error rate of 0.5 is, indeed, unlikely to be a usable keyboard. The problem is not with the design process but with the design specification. There is a similar problem for the keyboard in Fig. 11 a, which satisfies an acceptable error rate of 0.1. Here, the “k” character is on the second key, which has a predicted error rate of 0.7. Clearly, someone entering the word knock on such a keyboard faces low odds of success. But for the provided text corpus (the quotes used in the experiment), the letter “k” is rather uncommon (Table 2), so this type of entry is uncommon. One could impose additional constraints on the design process, such as to ensure that every key has an acceptable error rate in addition to an acceptable average error rate. The optimization process would be similar to what is described above, but the additional constraint would result in quite different optimized keyboards.

Effect of a different text corpus on optimal keyboard design

The optimal keyboard design also depends on the distribution of character frequencies F _i , which are defined by the text corpus that is appropriate for a given user. These frequency distributions may vary widely depending on what kind of material is being entered into the keyboard. For example, a user who writes novels or poems may rarely use characters such as “[” or “}”, while a user who writes Python computer programs would commonly use characters such as “i” and “f” as well as “(” and “*”. To explore the effect of the text corpus on optimal keyboard design, we identified the frequency of 64 characters from an early version of Python code that was used to calculate the optimized keyboard, and we compared the resulting keyboards to the keyboards designed for the quotations used in the experiment. Table 2 shows the frequencies of the characters included in our Python code. To ensure that all Python-related characters were part of the keyboard, the characters “{” and “}” in the original keyboard were replaced by the characters “#” and “ ^″”.

Figure 15 shows optimal keyboards using a linear cursor path for the Python text corpus. Like the linear cursor path keyboards for the quotes, the cursor duration is at the smallest possible value, 0.01 seconds. Given the differences in character frequencies, it is not surprising that the optimized assignment of characters to keyboard positions varies with the text corpus, as a comparison of Figs. 11 and 15 demonstrates. Nevertheless, the same basic principles underlie each optimized keyboard. For a low acceptable error rate, the high-frequency characters are placed a bit down the cursor path so that they have a low probability of an entry error and low-frequency characters are placed at the beginning of the cursor path. For a higher acceptable error rate, the high-frequency characters are placed at the beginning of the cursor path, where the keys have the highest predicted entry error probability and are reached most quickly.

The average entry time seems to be faster for the keyboards optimized for the quotes text than the keyboards optimized for the Python code text. This difference reflects the distribution of frequencies across the characters in the text corpus. Among other things, the Python computer code has 294 uses of numerals, while the quotes text uses numerals only ten times. Since the numerals are all fixed at the end of the linear cursor path, they take a relatively long time to reach.

For completeness, we also investigated the relationships between average entry time, cursor duration, average error rate, and the acceptable error rate for different cursor paths, using the Python code text corpus. These relationships are very similar to the results obtained for the quotes text corpus shown in Fig. 14.