The Big Life Fix: A Microsoft director has helped a girl rediscover her lost memories

A 10-year-old girl who struggles with severe memory loss following a car accident has revealed how technology created by a Microsoft director has kept alive her dream of becoming a dentist.

Aman Kular, who suffered brain damage after the crash nearly three years ago, was struggling to remember what her teacher said at school. Now, a new speech recognition program, developed by an Innovation Director at Microsoft Research Centre in Cambridge, is enabling her to keep up with classwork.

The technology, called Study Sparks, uses artificial intelligence to convert the teacher’s words to text on a laptop screen in real-time and allows Aman (top, with mum Rupinder and BBC presenter Simon Reeve) to read what was said as many times as she likes.

“It has helped me a lot,” Aman said. “I used to find that, compared with other people in class, I couldn’t keep up, I couldn’t remember certain things. Now, as well as having my own brain, I have Study Sparks as well. It’s easy to use.”

Aman and Study Sparks featured in a special episode of The Big Life Fix on BBC2 (above). The programme brings together eight of the UK’s top inventors, including Microsoft Innovation Director Haiyan Zhang, to create life-changing solutions for people. In the first series, which aired in December 2016, Zhang created the Emma Watch, a device that helped a woman with Parkinson’s control her tremors.

This recent episode, in aid of Children in Need, saw Zhang travel to Birmingham to help Aman.

Zhang said: “Aman’s accident was really tragic, she has done incredibly well since then, re-joining school and resuming her family life. She was in need of new technology tools to overcome some of the lasting memory loss from her accident. AI tools are becoming more democratised and easily accessible, in this case we were able to use the power of AI to power Aman’s memory.”

Her mum, Rupinder, had seen a video clip of the Emma Watch and wanted to work with the Microsoft employee.

“I saw the Emma Watch clip and was blown away by how Haiyan had come up with the watch,” Rupinder said. “I contacted the show’s researchers about the watch and they invited us to be on the new show.

“I really wanted Haiyan on board, so I was over the moon with that. I thought that if anyone could help it would be her. I saw the compassion she had with Emma [Lawton, who owns the Emma Watch] and you knew Haiyan cared. You can tell she wants to help people; she’s doing it because she cares and wants to fix something.”

Since Aman’s accident, which was so serious that doctors initially didn’t think she would survive, the youngster has found it difficult to form new memories. For example, in the TV programme, Aman doesn’t remember Zhang arriving at her home or recognise family photos.

Rupinder admitted she is worried about how her daughter will cope as she grows up.

“For her long-term future, I’m worried,” she told the show’s presenter, Simon Reeve. “At the minute I pretty much do everything for her. But things like going to secondary school and getting a job, you need a degree of memory, even for the simplest jobs. She’s quite worried about that herself.”

Reeve added: “Before the accident, Aman was doing well at school. Now she struggles in class and needs the help of a full-time teaching assistant.”

As well as feeling frustrated in lessons, Aman often feels dizzy and needs regular breaks.

“It’s heartbreaking,” says Zhang, who was filmed coding the software for Study Sparks over the course of a few weeks.

“What I’m trying to do is capture her lesson but capture it in a really simple way so that Aman can bookmark it and automatically record notes. I need to train an artificial intelligence algorithm to turn it into text. That’s not very easy to do. If we get three words wrong in a sentence, Aman is going to be completely lost.”

Zhang created a prototype device that was wirelessly connected to a microphone worn by Aman’s teacher. As her teacher spoke, the same words appeared on Aman’s laptop screen. Although the technology correctly captured most of the words, it wasn’t accurate enough for Aman’s needs. Zhang subsequently improved her design, making the transcription feature more accurate and adding a video function, which allowed Aman to record parts of her lessons.

The device was a success. Asked to rate it out of 10, Aman gave it an “11”.

“It made me feel like I could do it on my own. I know I can bookmark things and then listen to what the teacher is saying. This is now my teaching assistant.”

Rupinder agreed. “We are involved in the Child Brain Injury Trust and so many people have asked us if it is available to buy. Aman is very lucky to have this.”

However, Zhang didn’t stop there. She wanted to help Aman at home, too, and set about creating a program that would display pictures and play recordings of her family describing where they were when the photo was taken.

The result was Memory Sparks, a personalised, interactive photo album. Zhang also built an app so Aman’s family can upload photos and recordings to it from anywhere in the world.

“I want to give Aman the ability to relive something. Not just remember it but relive it,” Zhang said.

Read how Haiyan created the Emma Watch

Aman tried it out at home, with her family around her. She struggled to remember the first picture, until she clicked on the audio recording. “I remember now! It was when we had breakfast, and we were waiting in the lobby,” she said, with a huge smile on her face.

Zhang was delighted it worked, too. “When I see you remembering some of those stories, I’m so happy it can help you.”

Zhang will appear in the next series of The Big Life Fix, which will be broadcast on BBC2 early next year.