Disclaimer: This post does not constitute an endorsement of any Microsoft service, technology or product by Stephen Hawking. The purpose of this post is to discuss Intel’s communication platform for individuals with disabilities and it’s technical details. Pictures are used purely for informational and demonstrational purposes.

I’m typing this sentence with my face. And no, I didn’t somehow smash my face onto a keyboard with laser-like precision. I used Intel’s ACAT, or Assistive Context-Aware Toolkit, an open source platform developed in C# using .NET 4.5 and Visual Studio 2012 at Intel Labs to allow people with disabilities to communicate with ease, even in very constrained situations, like Stephen Hawking’s,

As we all know, the venerable physicist Professor Stephen Hawking is unable to talk as he is afflicted with ALS and thus relies on a computer system to communicate. In 2011, his condition was deteriorating so badly that he could best communicate at a rate of only 2 words per minute. He reached out to Alan Moore at Intel and asked if Intel could come up with new technology to help his plight.

Intel was happy to oblige. For 3 years, Intel Labs worked in close collaboration with Hawking to acutely address his needs and in January of 2014, they announced the first stable release that Hawking would use to replace his decades-old speech system. There was a near instant 10x improvement to all common tasks such as conducting a web search or opening a Word document. Fast-forward a few months and now Intel has open sourced the whole platform to allow researchers and hackers to broaden its use for people with disabilities of all backgrounds.

Exploring ACAT

Intel’s custom PC for Hawking, with the ACAT software in view.

When a friend sent Intel’s announcement to me, I couldn’t help but hunker down and immediately start tinkering with the ACAT software. Contrary to my initial expectations, it was very to install and use. It suffices to just run the installer on your PC and within moments, the ACAT suite is fully accessible to you.

The principle in which ACAT operates is very simple. There is essentially only one ‘switch’ and you can trigger a sequence of these switches to do anything. Hawking operates this switch with his cheek. An infrared beam from Hawking’s glasses tracks his cheek and a twitch from it breaks the beam and triggers the switch. With a webcam, we can emulate Hawking’s method of communication as well.

ACAT Vision detecting my face through the webcam and monitoring my cheek twitches.

A portion of the software called ACAT Vision detects your face and monitors your cheek for twitches. When it detects one as shown in the above picture, a switch is triggered. On screen you have the alphabet laid out in columns as well as some editing and navigation options, kind of like a virtual keyboard. There is a yellow box that flashes and alternates between the top and bottom halves of the laid out alphabet and options. You trigger the switch when the yellow cursor box includes the letter or option that you desire to use. Then the box shrinks to the size of a row of letters and goes through each row in the previously selected half and waits for you to twitch your cheek to trigger a switch and select the row. Once the row is selected, the box shrinks to the space of a letter and it goes through each letter in the previously selected row until you twitch to signal the correct letter.

This process is repeated until you have a few letters for the predictive word completion to choose a word for you. After a while of using the system, the world completion will start to understand the context of your phrases and will suggest words based on your previously typed words. For example, when Hawking types ‘t’, it offers to complete the word as ‘the’ and then follows up with a word like ‘black’ and then based on black it will suggest the word ‘hole’.

The ACAT Talk app narrating my cheek twitched sentences, just like the real Hawking!

While the primary intention of ACAT was to allow Hawking to communicate, it does more than just that. It’s actually a comprehensive system that allows Hawking to control the entire operating system. If he selects the ‘M’ character on the bottom right of his menu, he access the menu, from where he can choose to access many options such as the ‘New File’ dialog or the ‘Launch Application’ dialog. From the ‘Launch Application’ dialog he can start a Google search or use Internet Explorer (though Intel should really set him up to use Edge). The principle of navigation is still the same, the yellow box cursor alternates on the visual keyboard and awaits cheek twitches.

The ‘Launch Application’ menu. To select an application to launch, the corresponding number on the keyboard must be selected. For example, to access Notepad, 7 must be selected.

Once a desktop app is selected, a navigation context menu shows up. From there the relevant options appear to support the navigation of the app. For example, once ‘Google’ is selected, an Internet Explorer browser window pops up. The yellow box cursor cycles through the context menu. An option like ‘Search/URL’ can be chosen to then allow the browser bar to be used for searching or accessing URLs.

Well, what are you waiting for? You can try all this out for yourself. Visit the software's Github release page to get the installer. Make sure to check out the user guide, where you can learn advanced navigation techniques like the radial laser scanner, To make a fork and incorporate your own contributions, check out the Github repository. Happy hacking! (or should I say Happy Hawking? )