Amazon’s Alexa isn’t just using voice to provide assistance to users. A new feature called “Show and Tell” enables the tech giant’s AI to use its vision to help blind and low vision customers identify items through the use of an Echo Show.
Amazon began rolling out the feature, detailed in this blog post, to U.S. customers on Monday. Users can hold an item up to an Echo Show camera and ask, “Alexa, what am I holding?” and using computer vision and machine learning, the voice assistant can help identify common household packaged goods such as canned or boxed foods.
Amazon’s Alexa for Everyone team collaborated with blind customers as well as blind Amazon employees to develop the technology. User Stacie Grijalva, a mechanical engineer who lost her sight later in life, is featured in the blog and video above.
“It’s a tremendous help and a huge time saver because the Echo Show just sits on my counter, and I don’t have to go and find another tool or person to help me identify something. I can do it on my own by just asking Alexa,” Grijalva said.
Amazon worked with the Vista Center for the Blind and Visually Impaired in Santa Cruz, Calif. Grijalva, who is the assistive technology manager there, is shown helping others use the Echo Show to identify pantry items.
“The whole idea for Show and Tell came about from feedback from blind and low vision customers,” said Sarah Caplener, head of Amazon’s Alexa for Everyone team. “We heard that product identification can be a challenge and something customers wanted Alexa’s help with. Whether a customer is sorting through a bag of groceries, or trying to determine what item was left out on the counter, we want to make those moments simpler by helping identify these items and giving customers the information they need in that moment.”
Show and Tell will be available on first and second generation Echo Show devices.
The accessibility feature follows other Alexa innovations such as “Tap to Alexa” which allows customers to interact with Alexa without using voice; “Alexa Captioning” for real-time text captions of Alexa responses; the ability to have Alexa to speak faster or slower simply by asking; and more.