The Secrets of "Input"

Today, I'd like to talk about the theme of input using keyboards. I've been involved in developing UIs (user interfaces) for smartphones and other devices. When it comes to keyboards, I'd like to start by hearing from Mr. Yasuoka, who has also written a book about the "QWERTY layout," the most common layout on Japanese PCs.

To put it simply, the QWERTY layout came about by chance. Some books state that it was invented by Christopher Latham Sholes, but the truth is that it was patented and became widespread through the trial and error of various people in the 19th century.

Actually, the typewriter was originally more of a "receiving device" or "output device" than an input device. This is because the first typewriter was invented to transcribe Morse code faster than handwriting. The profession of a typist was also created to document (output) Morse code.

Later, in the latter half of the 19th century, when a mechanism that distinguished between uppercase and lowercase letters was invented, it became a tool for shorthand, directly typing what people said. The QWERTY layout was completed around this time. However, it was still difficult to type at the speed of speech, and it was reportedly used to make fair copies of shorthand notes. It gradually spread in this way.

It seems there were quite a few different keyboard layouts until the Typewriter Trust was formed in the 1890s. Many records remain. The QWERTY layout was just one of them, so there's no particularly rational reason for its dominance.

Currently, in the field of UI, a lot of research is emerging that suggests non-QWERTY layouts are better. With smartphones and tablets becoming commonplace and more opportunities to input with fingers or a pen, people are saying that the keyboard layout is difficult to type on. For example, proposals have been made at academic conferences to place frequently used combinations like T and H, or L and Y, next to each other.

However, everyone finds it troublesome to learn a new layout, so as a result, the software keyboards on smartphones and tablets still follow a QWERTY-based layout.

So, with the first typewriters, was it a case of input and output being almost simultaneous?

I believe they started being used as "input devices" in the 1940s, when people began connecting teletypes to computers.

Using a teletype, which was originally a communication device, to replace Morse code was not such a strange idea; it was a matter of outputting what was typed directly. It was a hassle to print just to make a fair copy of a handwritten document, so the typewriter was invented when a machine was needed that could convert text into something resembling movable type.

I've never used a typewriter, but I imagine the keys are very heavy. Is it true, as mentioned in your book, Mr. Yasuoka, that people in the past could type 170 words per minute?

Apparently so. It seems they devised various techniques to type faster, such as not pressing the keys all the way down, adjusting the force to the bare minimum needed for printing, and modifying the key actuation force.

When you think of a typewriter, you probably picture the "front-strike" type, which prints from above the paper, but this mechanism appeared in the 20th century. Before that, the common method was to strike the type from below. This used the weight of the type bar to ensure a solid impression, but it had the drawback that you couldn't see if you made a mistake. Furthermore, the keys were heavy because they had to lift the type from bottom to top, so it's amazing that people could type 120 to 130 words per minute with them.

Was the goal for typewriters at that time an operability that would allow typing at the speed of human speech?

It seems so for stenographers. However, a drawback of old typewriters was their loud typing noise. It was apparently too noisy to type clackingly next to someone speaking, so they couldn't be brought to dictation sites. The method used was for a stenographer to take notes and then type them up later.

On the other hand, in places where typewriters were used to convert Morse code into text, a device called a "sounder" was invented, and it seems they could type while listening to the telegraph, even if the key-press sound was somewhat loud. However, Morse code has a slow transmission speed, so no matter how fast you typed, it was at most about 40 words.

In my daily practice, I input information into electronic medical records while listening to patients, but I can't type simultaneously. There's also the pressure to type as quickly as possible, so I often make mistakes. Correcting them is also a hassle, so I've been wishing for a better Japanese input method.

Japanese input requires conversion, so it has different circumstances from input in Western languages.

In addition to that, Japanese has romaji input and hiragana input methods. I'm a romaji user, but I hear, Mr. Ogawa, that you've been using a "thumb-shift" keyboard, which inputs hiragana directly, for a long time.

The first word processor I bought after entering university was a Fujitsu with thumb-shift. As I continued to use it, I couldn't let it go, and I've been a "thumb-shifter" ever since. However, since thumb-shift is only supported by a limited number of keyboards, I now use an emulator (software that converts a standard JIS kana keyboard layout to thumb-shift).

The reason I go to such lengths to keep using it is that typing with thumb-shift feels incredibly good. It might be similar to the difference between a violin and a piano. With a violin, only the left hand is used to control the pitch, but a piano is played with both hands. Thumb-shift also uses all fingers for input, and the tactile feedback is just...

By the way, for a thumb-shifter, the J, K, and L keys on a QWERTY layout are in the most accessible position for the right hand, and are assigned "to," "ki," and "i" in order of frequency. Since these keys are hardly ever used in romaji conversion, I find the layout to be very irrational.

I think the frequency of K use is quite high in romaji input, though. As for L, more students have recently been using it for small characters in class, but I had to correct them for deliberately typing "kya" as "KILYA" (laughs).

Mr. Yasuoka, did you also research Japanese keyboard layouts?

I have looked into it. When I entered university around 1984 or '85, the keyboards in the lab were about half "thumb-shift layout" and half "JIS kana layout," so it was inconvenient if you couldn't use both.

Was romaji conversion in the minority at that time?

The keyboard market share was about half and half. Around that time, I was researching katakana layouts, and the current JIS standard kana layout, which starts with "ta-te-i-su-ka-n-na-ni-ra-se," is actually a layout proposal from the "Kanamojikai" (a private organization advocating for kana-only text) from when the kana typewriter key layout was standardized in the 1950s.

The proposal from the former Ministry of Communications and Posts was a layout designed for telegraphy, which grouped frequently used characters on the keys operated by the index finger. However, it was abolished because it was inconvenient, lacking small kana characters. While researching this history, I came to realize that determining a key layout based on frequency is a recipe for failure.

Mr. Masui, you're the developer of flick input for smartphones, but do you use flick input yourself?

Actually, I'm not very good at using it myself. However, many people find hiragana input more convenient, and flick input seems to be quite popular among them.

People who are fast at flick input are really fast. Watching their hand movements, they type at an incredible speed.

Predictive input has become standard now, regardless of whether it's on a feature phone or a smartphone. Many people probably feel that even inputting with a finger or a pen is slow. I think more people are finding it faster to select from candidates than to input characters and symbols directly. In the US, it seems very few people use predictive input, so I think we're in an interesting situation where new input methods are still likely to emerge.

On the other hand, on PCs, many people still feel it's faster to input the entire reading before converting, so predictive input hasn't become very widespread yet. It seems to be valued by people with hand disabilities or those who are not good at typing.

I use predictive conversion when entering electronic medical records. Candidates are displayed based on history, so it's very convenient that frequently used items appear at the top.

Mr. Ogawa, are you using a medical-specific input method?

I have the ATOK medical dictionary installed. However, what always gives me trouble is converting homonyms. I wish that kind of function were optimized for medical use, but general terms often appear as candidates.

That might just be a poorly made system (laughs). Even in regular Japanese, it's common for the system not to distinguish between "shiyou" as in specification and "shiyou" as in usage when you type it.

There was talk before about creating an input system for medical records, but it never materialized, perhaps because users can use it without issue if they each optimize their Japanese input dictionary.

It's a bit of a hassle to select a displayed candidate with the mouse when I have to type in a hurry. I sometimes wish there were a technology that would let me select characters just by looking at them. Is the appearance of such an input system still a long way off?

Research on eye-gaze input has been going on for decades, but older eye-tracking devices were inaccurate and expensive. However, recently, relatively cheap and high-precision ones have become available, so it might be put into practical use eventually.

I've been calling it "predictive input," but to tell the truth, this system is actually "search." It's just pasting the search results from a dictionary into the input field, so if there's a good medical dictionary search system, you can just combine it.

I see. So the accuracy of the input depends on the performance of the dictionary's search system.

In the old days of PCs, "multi-segment conversion," where you'd type a long sentence and convert it all at once, was popular. At the time, companies competed on its accuracy, but now it's considered faster and more accurate to input gradually with a predictive input system that uses a flexible search function.

The drawback of long-text input was that it was hard to see at a glance where a conversion error occurred.

Exactly. If there's a conversion error, you have to go back and fix it. That was a hassle, which is why we shifted to the current input method.

I use flick input on my smartphone. Did the idea of inputting "a, i, u, e, o" by flicking up, down, left, and right come from a lot of trial and error?

Actually, rather than inventing flick input, I combined things that already existed. You know pull-down menus, right? That's a method of selecting from a list of candidates arranged vertically. But there's another method called a "pie menu," where you select by the direction you move the mouse or pen. The advantage is that you can select a menu item by the direction of movement after the menu is displayed.

I thought that by combining a pie menu with predictive dictionary input, it could be used for Japanese, and that's how that system was born. I'm often called the developer, but the reality is that I implemented and sold an existing technology.

Mr. Masui, before you were involved in developing flick input, were you also developing keyboard input?

I once prototyped a flexible dictionary search system, and that's when I realized that search and input are the same thing. Since a dictionary search system can retrieve words surprisingly easily, I thought about combining this function with an input system, and the result was predictive input.

I didn't feel there was any particular problem with the keyboard layout, so I usually use a QWERTY with romaji input, and a keyboard from a cheap PC is totally fine for me. I use a US layout keyboard, but now there's software to change the layout, so it's not like I absolutely have to use this specific one.

So, you approached it from the perspective of designing a dictionary search system and thought about an easier input method from there.

That's right. As long as I can search for a word quickly, that's all that matters, so the operation method can be a pen, voice, or gestures.

Apparently, the number of people building their own keyboards has been increasing recently. Did you know there's a specialty shop in Akihabara?

No, I didn't know that.

That shop deals exclusively in custom-built keyboards. They sell kits where you can freely arrange the keys and control them with a microcontroller to make them usable just like a commercial keyboard.

It's like a keyboard with a built-in CPU. As long as you can write the program, you can freely change not only the layout but also which key produces which character or symbol. So, various enthusiasts are making all sorts of keyboards. The shop seems to be thriving, which shows that there are a lot of people with very specific preferences.

I'm the opposite of Mr. Masui; I'm particular about the stroke depth and key actuation force. Once, people around me recommended Topre keyboards, and when I looked into them, I found you could choose various weights. I bought and tried a model with an actuation force of about 35 to 40 grams, and it reduced the strain on my wrists and elbows, and I made fewer typos. The subtle angle of your hands can change typing accuracy, so I realized it's important.

What kind of keyboards do you two usually use?

I use the one built into my laptop. I basically like keyboards with a shallow stroke that I can type on without moving my fingers much.

For that reason, I also change the position of the return key myself. I don't like having to stretch my pinky finger to hit it in its usual position, so I've remapped the semicolon key to be the return key. This way, I can do most things without moving my fingers much.

Doesn't the lack of a semicolon cause problems when you're writing programs?

These days, there are many programming languages that don't require semicolons.

The guitar virtuoso Eric Clapton is sometimes called "Slowhand" because his finger movements are so exquisite. That efficient movement is what's cool.

So you like that kind of keyboard mastery, Mr. Masui.

That's right. I want to type with a slow hand, like Clapton. I like the idea of looking like my fingers are barely moving, but actually typing characters at incredible speed.

Conversely, a typewriter involves rather dynamic movements, doesn't it?

It's mechanical, after all. It originally printed using the force of human fingers, so the stroke could never be zero. But I do wonder why the stroke wasn't eliminated even after they became electric.

When PCs first came out, there were no such things as membrane switches (thin, sheet-like switches made by printing circuits and contacts on film and layering them), and physical switches that wouldn't react unless pressed a certain distance were the norm. But even now, there's a strong sentiment that having a stroke is better.

The fact that there's still demand for deep-stroke keyboards even after technical constraints have disappeared is probably a matter of preference.

I'm fine with a shallow stroke like on a laptop, but with the cover-style keyboard for Microsoft's "Surface," I get anxious about whether I'm actually pressing the keys. I find it a bit difficult to use with no tactile feel at all.

What about the MacBook? For me, that's just about right.

The MacBook is fine, but I'm not a fan of software keyboards that you tap on a screen like on a tablet. It might be different if there were a pen, but tapping the display with my finger just doesn't feel right. I'd be okay with it being a bit light as long as there's some tactile feedback.

I don't like the clicky-clacky typing feel of laptops. The IBM "ThinkPad" was good because it had a relatively deep stroke for a laptop, but with shallower ones, I'm afraid of getting tendonitis. That's why I use an external keyboard of my own even with a laptop.

Actually, for my UNIX machine for programming, I use a keyboard with a deeper stroke, but the feel I want from it is different from what I want in a keyboard for Japanese input.

When you watch old American movies, you see people who type quickly, clattering away with just their index fingers. Do people like that still exist?

Apparently, Bill Gates is like that.

Really!?

We have students like that at the university these days, too. When I watch them write programs from behind, they type very quickly, but they only use two or three fingers. There are many students like that.

Do they type while looking at the keyboard?

No, they can mostly type without looking. When I observe liberal arts classes where they don't write programs, there are quite a few students from the smartphone generation who aren't used to keyboards. After they use one for a while, some start saying it's easier to type on than a smartphone. Once they get used to it, they seem to feel it's faster than flick input.

Mr. Masui, your story about remapping the return key to the semicolon key was shocking. Don't you ever make mistakes?

I don't make mistakes on my own keyboard, but since I customize everything, it's a problem when I use someone else's device. By the way, Mac and Windows have software to replace the input system, but Chromebook doesn't.

On my Chromebook, I use my own IME (Input Method Editor) written in JavaScript. It replaces the default Google input system, but I can't recommend this to the average person.

I use both a JIS keyboard for daily use and a US keyboard for programming, and what gives me trouble is inputting symbols. The positions of the double quote, "@," "+," and "*" are different on these two, and I get them mixed up. Mr. Ogawa, since you use thumb-shift, do you ever have that problem?

I don't use symbols very often in electronic medical records.

But you must need "@" to type emails.

I switch modes with the emulator for any symbols I'm missing. Sometimes the characters on the screen keyboard and what's actually typed are different, though.

How do you input "@" with thumb-shift?

On the original keyboard (Thumb TouchTM), the enter key is split into two vertical rows, and the bottom one is "@." With the emulator, if you switch to half-width input, you can input "@" from the key to the right of P, just as it's printed on a JIS kana keyboard.

People who have been writing programs on keyboards for a long time used things like Sun workstations, so their bodies have become accustomed to only using US keyboards. US keyboards are becoming less common recently, so we're feeling a bit marginalized.

Exactly.

Even within the English-speaking world, the UK keyboard layout is different from the US keyboard.

It's different in the US and the UK?

Yes, it is. If you carelessly buy a keyboard at a computer shop in the UK, you'll be in for a surprise.

On a UK keyboard, the "@" is to the right of the "P."

France has a different layout too, right?

France is unique. It's an AZERTY layout.

It's interesting that just because they use the same alphabet doesn't mean the layout is common.

The hardware industry is tough because standards differ by region. That's why Steve Jobs made the iPhone with a software keyboard. It wasn't because touchscreens look cool; it was a matter of cost.

But what I thought was truly amazing about Jobs was that he said, "Don't you think a touchscreen looks cooler?" Thanks to that, everyone was fooled, weren't they?

Today's keyboards have a lot of mysterious keys that I don't know what to use for. I think they're remnants of IBM mainframes, but it's strange that they're still around.

I agree. I always think we don't need this many keys.

It's because they were assigned different functions during OS transitions. Young people who don't know this history misunderstand them as being for that purpose, and what's worse, it's even written on Wikipedia, so they believe it even more.

The Tab key was originally used for creating tables on a typewriter, but now it's thought to be for jumping to the next column in an editor.

I really wish they would get rid of the Caps Lock key. Why is it still there?

Caps Lock is also a remnant of the typewriter. When uppercase and lowercase were first distinguished, there was only a Lock key. You'd press it to switch to uppercase, but it wouldn't switch back when you released it; you had to use a separate key (Unlock) to return to lowercase. Later, in the 1890s, a spring was added, so releasing the shift key would return you to lowercase. But to avoid inconveniencing people accustomed to the old Lock key, it was left in the form of Caps Lock.

What era was that?

Roughly from 1900 to 1905.

We don't need it anymore. Thinking there must be some software that uses it, I once asked on Twitter, "What do you use Caps Lock for?" The most common answer was, "To undo it when I accidentally press the key." Nobody knew what it was for.

There are probably a lot of people like that. In that sense, the Alt key is also a special key. It's from IBM, and if I recall correctly, it was for assigning functions other than the "Ctrl + ○○" defined by ASCII (American Standard Code for Information Interchange, standardized by the American Standards Association in 1967). You'd use it in combinations like "Alt+A," "Alt+B," and so on...

Were there menus like that on IBM mainframes?

The PCs of our time didn't have them anymore. It was originally a key on the IBM terminal itself, and I think it was created for some necessary function when connecting mainframes to PC-type terminals.

I also don't understand the point of having 12 Fn (function) keys.

This is what happens when you add things and then don't take them away.

TV remotes are the same way. Once a feature is added, it can't be removed. That's common to all products.

In that respect, the small number of buttons on the Apple Remote and Fire TV Stick was revolutionary. They must have rethought it from scratch. Maybe we should do the same with today's keyboards.

Was part of the reason smartphones went with touchscreens to reset that trend?

But the keys on smartphones will probably increase too. Someone will arbitrarily add a feature thinking it would be nice to have. It's really a problem when that happens.

What do you think the future of input will be? We talked about how the original purpose of the keyboard was to type at the speed of speech. I was wondering if the role of the keyboard will end as voice input like Siri becomes more widespread.

The accuracy of voice input is improving. Apparently, the recognition rate is higher the faster you speak. If the accuracy improves further, it will be effective for creating book manuscripts directly from spoken content. In fact, the economist Yukio Noguchi, who wrote "Super" Organizing Method, apparently writes his books using voice input.

However, the drawback of voice input is that it's difficult to use when other people are around. Also, as you'll find if you try it, it's not that easy to speak without making mistakes.

With voice input, it's a hassle that you can't correct mistakes with your voice.

A master like Mr. Noguchi can write so many books precisely because he can speak without making mistakes. But I think the recognition rate has improved astonishingly in the last three or four years.

Dictation has always been a rational method for the writer as well. If a machine can do it accurately, it would certainly boost writing productivity.

In the field of medical sciences, there used to be a position called a "Beschreiber" who would write down what a professor said.

I think voice input is a machine doing that job instead, but while it might be fine for writing on a blank slate, the current electronic medical record format has various frames like prescription and test sections shared with nurses and pharmacists. I feel it would still be difficult for voice recognition to distinguish between these differences.

There are issues with formatting like that, and it also seems technically difficult for Japanese to distinguish between hiragana, katakana, and kanji. Even when typing on a keyboard, I always think having three types of characters is just unreasonable.

From a reader's perspective, I think having various types of characters actually makes it easier to read. It allows for skim-reading.

It's true that there's an advantage in being able to pick out words at the word level.

With voice input, will you have to manually correct kana and kanji?

I wonder what will happen. Will AI do its best to correct it...? However, it's difficult to distinguish between "Keio" as in the university, "Keio" as in the railway, and "KO" just by saying "keio." It would be great if the machine could judge by context, but that won't always be the case.

Differences in input methods between English and a country's native language, like we just discussed, exist in other countries too, right?

Yes, everyone is creating input systems for their respective languages through trial and error. Even in German, besides the standard Roman alphabet, there are assignments for "ä," "ü," and "ö."

It's more complex in cultures that use non-Latin alphabets. Most of Asia is like that, and it's especially tough for the Sinosphere, including us Japanese.

This is because kanji input can only be distinguished by "sound" or "shape." Input by sound is limited in variety, and there are countless homonyms. People in China and Taiwan use ten thousand different kanji, but searching by sound alone is impossible, so the method is to have kanji components written on the keyboard from the start and select from them.

Which method is most common for Chinese input?

Looking at students from mainland China, "Pinyin," which can be typed on a Roman alphabet keyboard, seems to be common. But students from Taiwan use "Wubi," which inputs based on the shape of the kanji.

Neither of those is very old, right?

That's right. Both methods appeared in the 1990s. Using Wubi by sticking it on the keyboard seems to be common among students from Taiwan and Hong Kong.

Why don't they both just use Pinyin?

I'm not sure about Hong Kong, but in Taiwan, a system called Bopomofo (one of the phonetic symbols for Chinese, mainly used in Taiwan, also called Zhuyin Fuhao) was already in use, and the input method is different. The correspondence with the alphabet is not as strict as in China, so a unique input method was created.

Thailand has a Thai keyboard, which has Thai characters and corresponding alphabet letters on the keyboard. Korea also has a method called "2-set," where Hangul and its corresponding alphabet letters are on the keyboard for input. Both are similar in that they don't use the Latin alphabet.

Hangul, in particular, has only 21 vowels and 14 consonants, so the number of keys is limited. Moreover, the pattern is consonant + vowel + final consonant (patchim), so the input method is also simple.

With Hangul, you just have to input the consonant and vowel in order, and you don't need to convert, so it must be easy.

We've covered a lot of topics, but I'm still sad that thumb-shift will eventually disappear.

You might want to stock up on them now (laughs).

Fujitsu doesn't manufacture them anymore, so I guess the only way is to develop emulators among fellow enthusiasts and keep it going on a small scale. As long as the emulator is compatible with the current OS, I want to keep using thumb-shift.

Do you know about the TRON keyboard created by computer scientist Ken Sakamura? There was someone who loved it so much that he wanted to use it for the rest of his life, so he had a mold made and produced about 20 of them himself. I'm sure there are people like that for thumb-shift, too.

And there's the trend of building your own these days.

Exactly. You can put a CPU in the keyboard itself and even program it, so it's no problem at all. In fact, you might even be able to do the conversion on the keyboard.

Conversion on the keyboard?

Even if the OS doesn't support it, you can just build the kanji conversion system into the keyboard. You'd still need a display, but we live in an age where that's possible.

Input has truly become diversified. With the spread of voice and eye-gaze input, will input devices continue to expand even further?

At conferences on computer-human interfaces, papers on input are presented every year. A great idea might suddenly emerge one day.

Is it possible that a new idea like flick input will emerge?

Whether it becomes popular or not is another question, but there's a certain number of researchers, and it's an era where hardware is easy to make, so there's no doubt that things like eye-gaze and voice input are gradually becoming more user-friendly. I hope people will watch over their development warmly.

The QWERTY layout was also a product of chance, so I'm certain that new devices and corresponding input methods will emerge. However, people are reluctant to let go of what they've learned, so we won't know if it will catch on until the next generation.

Input methods have changed a lot in the last 20 years, yet PC windows and menus haven't changed in about 40 years. When you think about it that way, it feels like input still has more potential.

But I hope the number of keys doesn't increase any further (laughs). It's clearly beyond what a human can remember.

But I'm excited to see what kind of input will come out next.

I'd like to be involved in it too.