Editorial: “Let the mind-reading begin“
Understand how the brain turns thoughts into words and you can make machines that read minds
AS YOU begin to read this article and your eyes follow the words across the page, you may be aware of a voice in your head silently muttering along. Most of us do it when we talk, listen, write letters or emails. The very same thing is happening to me as I type these words: a private internal narrative is shaping the words before I commit them to text.
Advertisement
What if we could tap into another person’s inner voice? Thinking of words does, after all, create characteristic electrical signals in our brains, and decoding them could make it possible to piece together our thoughts. Such an ability would have phenomenal prospects, not least for people unable to communicate as a result of brain damage. But it would also carry profoundly worrisome implications for the future of privacy. It’s time to start thinking about these things, because the technology to access those signals deep within the brain is fast becoming a reality. Welcome to the brave new world of mind-reading.
It would not be the first time that we have eavesdropped on the brain’s signals. The first scribbled records of electrical activity in the human brain were made in 1924 by a German doctor called Hans Berger using his new invention – the electroencephalograph (EEG). It uses electrodes placed on the skull to read the output of the brain’s billions of neurons.
By the mid-1990s, our ability to translate the brain’s activity into readable signals had advanced so far that people could move computer cursors using only the electrical fields created by their thoughts. The electrical impulses such innovations tap into are produced in a part of the brain called the motor cortex, which is responsible for muscle movement. To move a cursor on a screen, you do not think “move left” in natural language. Instead, you imagine a specific motion like hitting a ball with a tennis racket. Training the machine to recognise which electrical signals correspond to your imagined movements, however, is time-consuming and difficult.
And while this method works well for directing objects on a screen, its drawbacks become apparent when you try using it to communicate. At best, you can use the cursor to select letters displayed on an on-screen keyboard. Even a practised mind would be lucky to write 15 words per minute with that approach. Speaking, we can manage 150.
Matching the speed at which we can think and talk would lead to devices that could instantly translate the electrical signals of your inner voice into sound produced by a speech synthesiser. Such an application would be a godsend for severely paralysed people, or those who are awake and fully conscious but are unable to communicate – so-called “locked-in syndrome” – who could once again engage in effortless conversations.
A new approach
This effortlessness, however, requires a significantly different approach. You need to focus only on the signals coming from the brain areas that govern speech. In 2008, neuroscientist Philip Kennedy at Neural Signals in Duluth, Georgia, tried to do just that by closing in on the small, face-specific part of the motor cortex, which controls the mouth, larynx and tongue. Here, it is possible to select only the muscle signals that are created when a person speaks.
For example, saying the word “bee” requires the mouth to execute very specific movements. First the lips must be pursed to form the initial “buh” sound, then the lips must stretch and the tongue press against the top row of molars to express the sound “ee”. All of these movements originate in the facial motor cortex, and so by tapping the signals before they mobilise their target muscles, you can work out what the person intends to say.
Kennedy and Frank Guenther, a neuroscientist at Boston University, read these signals in a person with locked-in syndrome by implanting penetrating electrodes in his facial motor cortex. It worked: they were able to identify three vowel sounds, and feed these signals to a speech synthesiser that broadcast the sounds in real time. The work was a first step in deriving the consonants and vowels that make up whole words. Kennedy thinks that by using this method he and Guenther could have a sample of 100 or more useful words for this patient within five years (PLoS ONE, vol 4, p 8218).
The method is not ideal. For one thing, it is specific to one patient. Perforating a person’s brain with an electrode is also not something to be undertaken lightly. And finally, while the move from using general motor signals to using language-specific signals is a significant step, it is not mind reading.
Real mind reading requires some way to intercept the signals before they hit the motor cortex. That means researchers have to search other parts of the brain for the source of language. Fortunately, they have a good idea where to look.
The translation of thoughts to language in the brain is an incredibly complex and largely mysterious process, but this much is known: before they end up in the motor cortex, thoughts destined to become spoken words pass through two “staging areas” associated with the perception and expression of speech.
The first is called Wernicke’s area, which deals with semantics – in this case ideas based more in pure meaning than in linguistics, which can include images, smells, or emotional memories (see diagram). Damage to Wernicke’s area can result in the loss of semantic associations: words can’t make sense when they are decoupled from their meaning. Suffer a stroke in that region, for example, and you will have trouble understanding not just what others are telling you, but what you yourself are thinking. Trying to speak results in garbled, nonsensical juxtapositions that neuroscientists call ““.
The second is called Broca’s area, agreed to be the brain’s speech-processing centre. Here semantics are translated into phonetics and, ultimately, word components. From here the assembled sentences take a quick trip to the motor cortex, which activates the muscles that will turn the desired words into speech. Injure Broca’s area, and though you might know what you want to say, you just can’t send those impulses, effectively rendering you mute.
When you listen to your inner voice, two things are happening. You “hear” yourself producing language in Wernicke’s area as you construct it in Broca’s area.
Outing your inner voice
The key to mind reading seems to lie in these two areas, but they had remained largely unexplored for good reason. Neuroscience was limited by the two main technologies that could read electrical signals in real time. Penetrating electrodes pick up with high fidelity the signals emitted by the few neurons they touch, but they inevitably carry risks.
Traditionally, the only alternative has been a surface EEG taken from electrodes sitting on top of the skull. Unfortunately, the bone blocks and jumbles the brain’s electrical signals, so while inexact signals can be used to deduce broad muscle movements, finding meaningful information in tiny areas like Broca’s and Wernicke’s requires much greater precision.
By 2004 neuroscientists had turned to a technique called electrocorticography (ECoG), which involves opening the skull and placing arrays of electrodes onto the top of the brain, just beneath the skull but without actually penetrating the brain (). ECoG has been used for several decades to locate the source of seizures in people with epilepsy.
Armed with ECoG, several researchers set about looking for the inner voice. Last year, Bradley Greger at the University of Utah in Salt Lake City successfully used ECoG to detect the electrical signatures of whole words – finer detail than anyone had ever obtained. By placing arrays of electrodes over both the facial motor cortex and Wernicke’s area, Greger and his colleagues detected the words “yes”, “no”, “hot, “cold”, “thirsty”, “hungry”, “hello”, “goodbye”, “more” and “less” as they were being produced in the brain of an awake, aware person with epilepsy.
Like Kennedy’s group, Greger says, “we were going after a constant set of words”, specifically chosen to be useful to locked-in people. But unlike Kennedy, Greger was looking beyond the motor cortex. Though most of the signals still came from the facial motor cortex, this work nevertheless marked the first ever excursion into the brain’s language centres ().
Promising as it is, this approach does not scale up well. Like Kennedy’s word-mapping technique, it requires a new signal to be learned for each new word you add to the lexicon. English contains . And though this was the first instance of monitoring Wernicke’s area, it still relied largely on the facial motor cortex.
There might be another way. “You could go after more fundamental building blocks of language,” says Greger. These building blocks are called phonemes and the English language has about 40 of them – the “kuh” sound in “school”, for example, the “ah” in “blah”, the “sh” in “shy”. Every English word contains some subset of these sound components. Decode the brain signals that correspond to the phonemes, and you would have a kind of Rosetta Stone to unlock any conceivable word at the moment you think it.
Where would you find these phonemes? They would be much more likely to exist in Broca’s area or Wernicke’s area, the precursor regions to verbal motor signals.
In fact, tantalising clues from functional MRI scans had hinted at the presence of (Cerebral Cortex, vol 19, p 2156). So last year Eric Leuthardt at Washington University School of Medicine in St Louis, Missouri, and his colleague Gerwin Schalk positioned ECoG patches over the language regions of four fully conscious people. Sure enough, it was possible to detect the phonemes “oo”, “ah”, “eh”, and “ee” ((Journal of Neural Engineering, vol 8, p 036004). The neural signatures of those sounds in the motor cortex looked the same whether the participants imagined they were speaking them or actually spoke them aloud. However, phonemes that were strictly imagined – sounds that the thinker was not intending to speak aloud – produced radically different signatures.
Leuthardt had been on the hunt for phonemes, but he had stumbled across something much better. Spoken phonemes activated both the language areas and the motor cortex. But imagined speech – that inner voice – boosted the activity of the neurons in the Wernicke’s area. “This imagined speech, or inner voice, was especially easy to pick up in the classical language areas,” he says.
“The imagined speech, or inner voice, was especially easy to pick up in the classical language areas”
“Part of the way we process the world is through that internal dialogue,” says Leuthardt. And he was able to listen in on it. He had effectively read their minds. “I would call it brain reading,” he says. “It is literally taking your thoughts and translating them to the outside world.” Leuthardt’s work has impressed other researchers. “This is the first time anyone has been able to get these kinds of signals out of the language areas,” says Greger. “It’s really impressive.”
The real-time speed at which Leuthardt was able to detect the specific phonetic signals implies that it could be done as we think the words. We could then read signals off an ECoG at the instant a person thought them. “It should be possible to produce real-time speech,” Kennedy says.
To arrive at whole words, Leuthardt’s next step is to expand his library of sounds. He initially thought pursuing vowel phonemes would be far easier than consonant type phonemes because the mouth movements are so different, whereas transitioning between “k” and “t” barely causes the mouth to move. However, that is not necessarily a problem. It’s too early to discuss the research findings, but progress is happening faster than he had anticipated, and Leuthardt is very happy with the results. “We’re burning through [phonemes] fast,” he says.
For now, this research is primarily aimed at improving the lives of people with locked-in syndrome, but the ability to explore the brain’s language centres could revolutionise other fields. The consequences of these findings could ripple out to more general audiences, who might like to use extreme hands-free mobile communication technologies that can be manipulated by inner voice alone; consider a device that could someday give you your own private telepathic link. For linguists, it could provide previously unobtainable insights into the neural origins and structures of language. For example, linguists currently have only a hazy understanding of the neurobiological processes that create language from thought.
Leuthardt’s next project is a collaboration with international researchers to find out how the production of phonemes translates across different languages. If linguistic thoughts activate the same groups of neurons in people across cultures and languages, that might imply that humans are in some way hard-wired to communicate using these types of sounds.
Leuthardt is not, however, content to stop at the inner voice. He is also looking further, to see whether semantic understanding can be mapped between vastly different cultures: which groups of neurons fire, for example, at the idea of “dog” as thought by an American and a Chinese person? “We want to see how far we can reach,” Leuthardt says. “Let’s see what semantic representations look like.” If it were possible to generalise these semantic images and use them for communication instead of language, universal translators might be within reach. Indeed, knowing what someone is thinking without needing words at all would be functionally indistinguishable from telepathy.
Is the world ready for mind reading? Or is it the stuff of Orwellian nightmares – a technology that makes it possible to read your deepest, darkest secrets? Delve far enough into the brain’s idea-making processes, after all, and it should be possible to probe thoughts that were never intended for sharing. Where Kennedy’s method relied on speaking or the intent to speak, Leuthardt’s technique can pick up those thoughts without the participant ever intending to speak them.
For now, this difference is academic. This version of mind-reading requires brain surgery. But if less invasive techniques could harness the same information, all bets would be off. If that were to happen, it would be important to ensure that there was a mechanism to stop information being extracted without the subject’s cooperation.
In fact, such a mechanism already exists. If we always said every word we were thinking, the consequences would be disastrous. Instead, we separate public from private with a kind of self-censoring mechanism, though the point at which this mental brake comes into play during speech generation is unclear, says Leuthardt. It’s an important distinction, because if the brake exists only in the motor cortex, information accessed at the language centres would be unfiltered.
What do you think? Are we ready to breach the last bastion of privacy – our own skulls – to reveal the things we have so far kept safely locked away in the darker corners of our heads? No, don’t say anything. I’ve already read your mind.
