Seeing AI can now tell people with blindness which banknote they are holding

Microsoft’s Seeing AI app, which helps people with blindness by narrating the world around them, can now recognise currencies, colours and handwriting.

The free program, which was released in the UK last month, uses artificial intelligence to recognise objects, people and text via a phone or tablet’s camera and describes them to the user. Since it was launched in the US earlier this year, the app has been downloaded more than 100,000 times and has helped users with over three million tasks.

It has now been updated with a host of new features to make everyday tasks easier for the visually impaired, Microsoft revealed today during an AI Summit in San Francisco.

Seeing AI will now recognise UK, US, Canadian and Euro banknotes; the colour of objects such as clothes; if a lightbulb or battery pack is switched on so users don’t touch them when they’re hot (the app will also alert you with a corresponding audible tone); and handwritten text to help users “read” greetings cards from loved ones.

Other new features include:

  • Reading documents aloud without additional voiceover, with synchronised word highlighting
  • Changing text size when looking at documents
  • Customising the voice and the speed at which it talks
  • Showing a list of new features whenever the app is updated

According to the Royal National Institute of Blind People (RNIB), more than two million people in the UK are blind or have sight loss.

Everything you need to know about AI

Cameron Roles, a university lecturer at the Australian National University College of Law and Director with Vision Australia, Australia’s largest vision and blindness charity, said: “I absolutely love Seeing AI. If my children hand me a note from school or if I pick up a book, I can use Seeing AI to quickly capture that text and just give me a very brief instant overview of what’s on the document.

“I think we’re coming into a really exciting time. The growth in artificial intelligence, augmented reality, self-driving cars… I feel that it’s a great time for all of us in society.

“Technology can be such an enabler of good and such an enabler for people to shrink the world, for the world to come closer together, and for people to be able to achieve so much more than they ever could without it.”

Seeing AI is now available in 35 countries across the world.

The update was revealed during a Microsoft event in California that featured Harry Shum, executive vice president of Microsoft AI and Research.

A person using the Seeing AI app to recognise a friend

Seeing AI recognises objects, people and text via a phone or tablet’s camera and describes them to the user

The company also announced a new partnership with Reddit, to bring information from the discussion website to Bing in the first search engine tie-up of its kind.

Bing will also get new intelligent search capabilities to efficiently help connect users with the information they need. When searching online, you will now see multiple perspectives on a search topic, answers that have been verified across numerous sources and data translated into easy-to-understand concepts. Users can also use Bing’s object recognition technologies to search for information in images.

Meanwhile, Cortana – Microsoft’s intelligent personal assistant – can now provide an overview of new emails in your inbox to allow you to choose the one you would like her to read. She will tell you whether specific people, such as your boss, have sent you an email, and let you reply just by using your voice. If asked, Cortana can set up an appointment with someone in your contact list, find the best time to meet with someone in your company and check your schedule.