People posed in front of the Amazon Go sign as testers tried out the new service. (GeekWire Photo / Nat Levy)
The first Amazon Go store, in Seattle, is open only to the company’s employees for now, prior to a scheduled public opening early next year. (GeekWire Photo / Nat Levy)

As an online retail giant, Amazon can use its engineering prowess and the vast capabilities of the digital world to track and analyze how customers use its site.

So it’s only natural that the company, in its first brick-and-mortar convenience store, would want to create a real-world version of the same thing.

The new Amazon Go store logs customers in when they walk through the door, knows instantly when they pick an item up from the shelf, automatically tallies what they put in their carts or shopping bags, uses past purchases to improve their shopping experience, and automatically charges their accounts when they walk out — no need for checkout lines — making it nearly frictionless to buy something.

In short, it’s a physical manifestation of Amazon.com.

The 1,800-square-foot store is also a new test of our willingness to trade privacy for convenience in the real world. We already leave digital footprints online. Are we ready for an e-commerce system to follow us down the aisle at the neighborhood market?

That’s one of the fundamental questions to be answered by the new store, unveiled by the company this week in its hometown of Seattle.

For now, Amazon Go is just for Amazon employees, as the company prepares for a public opening early next year. Amazon explained the basics in the video above and an online FAQ. But past patent filings by Amazon engineers and executives, some of whom ended up leading the Amazon Go team, go into much greater detail about their vision for a new type of store.

A store full of sensors

3-3

For starters, their vision involves a wide array of sensors.

This diagram appears in a pair of related patents by Amazon executives, which describe a variety of high-tech methods of tracking products, employees and customers through a facility such as a distribution center or a retail store. Here is the description of this chart from one of the patent filings.

“Generally, the materials handling facility 200 may include one or more image capture devices, such as cameras 208. For example, one or more cameras 208 may be positioned in locations of the materials handling facility 200 so that images of locations within the materials handling facility can be captured. In some implementations, the image capture devices 208 may be positioned overhead, such as on the ceiling, to capture images of users and/or locations within the materials handling facility. In addition, in some implementations, one or more cameras 208 may be positioned on or inside of inventory locations. For example, a series of cameras 208 may be positioned on external portions of the inventory locations and positioned to capture images of users and/or the location surrounding the inventory location. Likewise, one or more cameras 208 may be positioned within the inventory locations to capture images of items stored in the inventory locations and/or images of objects (e.g., a user’s hand, items) moving into and/or out of the inventory locations. Cameras may also be positioned at or near transition areas to capture images of users as they enter or pass through the transition area.”

Another diagram from the patent filing goes into greater detail.

4

“Generally, the inventory management system 150 may include input components 311, output components 301 and computing resource(s) 203. The input components 311 may include an image capture device 308 (e.g., camera), microphone 314, antenna 316, sensor (not shown), scale (not shown), light curtain (not shown), a volume displacement sensor (not shown) or any other component that is capable of receiving input about the surrounding environment, from the user of the portable device and/or from the portable device. The output components 301 may include a projector 310, a portable device 306, a display 312, an antenna 316, a radio (not shown), speakers 313 and/or any other component that is capable of providing output.”

The filing explains the practical benefits of deploying this system of sensors and data in a retail store or a distribution center.

“For example,” it says, “if the inventory management system cannot determine if the picked item is a bottle of ketchup or a bottle of mustard, the inventory management system may consider past purchase history and/or what items the user has already picked from other inventory locations. For example, if the user historically has only picked/purchased ketchup, that information may be used to confirm that the user has likely picked ketchup from the inventory location.”

The filing also outlines how the system could follow users through the facility.

“For example, a camera may capture an image of the user that is processed using facial recognition to identify the user. In some implementations, one or more input devices may collect data that is used to identify the user when the user enters the materials handling facility. The position of the user may also be monitored as the user moves about the materials handling facility. When the position of the user enters or passes through a transition area, the identity of the user is known because the position of the user and the identity of the user has been maintained.”

Another portion of the filing explains how the system could capture an image of someone’s hand as it crosses the plane of a shelf, analyzing the skin color to help identify the person’s hand, and determine whether the person was removing or placing an item on the shelf.

Of course, this is just a patent filing. It’s not clear which of these approaches the company will be using in the new Amazon store, and which might be reserved for use in an Amazon distribution center. Also unclear are the extent of the customer tracking technology being used in Amazon Go, and the related privacy policies and practices. Amazon declined GeekWire’s request to try out the store this week.

Amazon SVP Steve Kessel has returned to Amazon after a long sabbatical to work on a "super secret" project.
Amazon SVP Steve Kessel

However, the gist of the patent filing matches the Amazon Go FAQ: “Our checkout-free shopping experience is made possible by the same types of technologies used in self-driving cars: computer vision, sensor fusion, and deep learning. Our Just Walk Out technology automatically detects when products are taken from or returned to the shelves and keeps track of them in a virtual cart. When you’re done shopping, you can just leave the store. Shortly after, we’ll charge your Amazon account and send you a receipt.”

Two of the inventors listed on the latest version of patent filing, Dilip Kumar and Gianna Puerini, have updated their LinkedIn bios to note that they are Amazon vice presidents. The third inventor on the latest patent filing, Steve Kessel, is an Amazon senior vice president who returned from a sabbatical last year to lead a “super secret” project, now believed to be Amazon’s high-tech physical retail initiatives.

A new frontier for privacy

So who cares if the store’s technology follows you through the aisles? We asked a variety of technology and retail experts whether privacy concerns about Amazon’s approach are likely to create pushback, either in customer acceptance or regulation. Or is it already too late for that?

“It may be too late for that, at least among the younger demographics, although, on the whole, Millennials haven’t shown much concern about privacy issues to date,” said Tom Caporaso, CEO of Clarus Commerce, in an email. “Grocery purchases are also less likely to cause concern than, say, pharmacy purchases might, so consumers could very well choose to use the technology in some places but not others.”

Larry Arnstein, vice president of business development and solutions at Seattle-based RFID technology company Impinj, pointed out one of the realities of the digital economy: “We tend to be willing to give up information to get good service,” he said.

One area where the store could help Amazon is in product recommendations, said Nick Manzo, the Global Omnichannel Lead of product information network 1WorldSync.

“Because the company would know exactly what consumers picked up and put back, or even the walking patterns of customers in store, Amazon would have a whole new set of consumer data to add to targeted promotions and ads,” Manzo said. “For some consumers, this may be a layer of intrusion that sparks a conversation on privacy issues, but for many, the difference in convenience may be worth it.”

Like what you're reading? Subscribe to GeekWire's free newsletters to catch every headline

Job Listings on GeekWork

Find more jobs on GeekWork. Employers, post a job here.