Trending: Amazon employee at Seattle-area warehouse tests positive for COVID-19
Computer vision expert Harry Shum leads Microsoft’s artificial intelligence initiatives as the executive vice president of the company’s AI and Research group. (GeekWire Photo / Todd Bishop)

Microsoft’s first mission statement envisioned a computer on every desk and in every home, but Bill Gates also had another goal: that computers would someday be able to see, hear, communicate and understand humans and their environment.

More than 25 years and two CEOs later, Microsoft is betting its future on it.

“We truly believe AI is this disruptive force, even though it’s not new,” said Harry Shum, the executive vice president in charge of Microsoft’s AI and Research group, in an interview with GeekWire. “The recent progress is just enormous. We certainly have seen that through our own products and engagement with customers. We also feel we have a very strong point of view about how we take AI to the next step.”

Microsoft CEO Satya Nadella formed the Microsoft AI and Research group one year ago this month as a fourth engineering division at the company, alongside the Office, Windows and Cloud & Enterprise divisions. The move reflects Nadella’s belief in “democratizing AI,” making it available to any person or company, and radically changing the way computers interact with and work on behalf of humans.

One way to measure Microsoft’s AI bet: In its first year of operation, the AI and Research group has grown by 60 percent — from 5,000 people originally to nearly 8,000 people today — through hiring and acquisitions, and by bringing aboard additional teams from other parts of the company.

Bill Gates and Satya Nadella. (Microsoft Photo)

The creation of Microsoft AI and Research also underscores the intense competition in artificial intelligence. Microsoft is gearing up to compete against the likes of Google, Amazon, Salesforce, Apple, and countless AI startups and research groups, all of them looking to lead the tech industry in this new era of artificial intelligence.

Microsoft wants to show that it’s leading and not falling behind in artificial intelligence, but competitors are also moving aggressively, said Rob Sanfilippo, research vice president at the independent Directions on Microsoft research firm.

“Microsoft has made advances, but so have IBM, Apple, Amazon, Google, Facebook, and others,” he said. “Arguably, in the consumer space, Amazon leads in AI mindshare — more people are acquainted with Alexa than Cortana. Microsoft is looking to avoid missing giant opportunities as it did with mobile and social media, so it is giving its AI strategy a lot of attention and resources.”

Microsoft has advantages and disadvantages in this quest. For example, the company’s Windows Phone business hasn’t come close to competing with iPhone and Android, which means its virtual assistant Cortana is relegated to third-party app status on the most popular smartphone platforms. However, Cortana is native to Windows 10, which is now on more than 500 million devices.

The company also has the advantage of large amounts of data, the raw material of machine learning, through its Bing search engine and, more recently, its $26 billion acquisition of LinkedIn, the largest deal in Microsoft’s history. Products such as Office 365 also provide a unique distribution channel for Microsoft’s AI features.

Lili Cheng, corporate vice president in the AI and Research group, with Irving Kwong, lead program manager for Microsoft Cognitive Services. (GeekWire Photo / Todd Bishop)

Then there’s Microsoft Research, founded more than 25 years ago based on Gates’ original vision. It’s stocked with computer scientists and engineers who have spent years pursuing breakthroughs in areas such as deep neural networks, computer vision, machine translation and other fundamental underpinnings of artificial intelligence.

The idea behind AI and Research is to get those researchers working side-by-side with product teams to move artificial intelligence advances — some of them in the works for years or decades — out of the labs and into new and existing products.

People inside the group point to early progress from this approach. In one example, a researcher’s new method for getting computers to recognize human emotion was released as an API for Microsoft cloud customers in a matter of weeks rather than languishing for months or longer after the publication of an academic paper.

“We’ve had this dream for a long time — that systems could be smarter and model the way you think,” said Lili Cheng, a longtime Microsoft researcher and engineer who now leads the company’s AI developer platform as a corporate vice president in the AI and Research group. The company’s leaders believe that aligning the researchers and product groups will allow that to happen faster.

Microsoft changed the vision statement in its annual report this year to read, in part, “We believe a new technology paradigm is emerging that manifests itself through an intelligent cloud and an intelligent edge where computing is more distributed, AI drives insights and acts on the user’s behalf, and user experiences span devices with a user’s available data and information.”

Artificial intelligence is one of the key topics of Nadella’s upcoming book, Hit Refresh. We’ll also be asking Nadella about AI as part of his upcoming appearance at the GeekWire Summit.

Flurry of AI activity

During the past year, Microsoft has introduced new artificial intelligence and machine learning features and services in products including Office, Bing, Azure and programmable AI chips for the company’s data centers. The company has also released standalone AI programs, such as a Seeing AI app that helps visually impaired people better sense and understand the world around them.

After early triumphs and struggles with chatbots, Microsoft has been rolling them out around the world. Shum says the goal is to have a bot in every country with more than 100 million people.

Microsoft Photo

And in a surprise move that reflects Nadella’s pragmatic approach, Microsoft announced a deal with rival Amazon to connect Cortana and Alexa, their voice-activated AI assistants. The news also illustrated their respective strengths: Amazon in consumer technology, and Microsoft in enterprise technology.

Shum hinted that the initial announcement between Microsoft and Cortana might not be the end of the AI collaboration between the Seattle-area tech giants, saying that he sees more opportunities for them to work together. “The world is so big,” he said. “This is the beginning with Alexa and Cortana.”

Microsoft is pushing to incorporate AI more deeply into Office with features such as PowerPoint Designer, which analyzes the content placed on a slide to suggest an optimal layout. Another project called Presentation Translator presents subtitles of the presenter’s live speech, translated into 60 languages, with the ability to create a custom speech model for better accuracy by analyzing the text of the PowerPoint slides.

Microsoft’s Custom Vision Service running in Core ML on Apple’s iOS 11, using a custom data model to automatically recognize a pineapple. (GeekWire Photo / Todd Bishop)

Another major AI initiative is Microsoft Cognitive Services, which offers APIs for developers to put elements of artificial intelligence into their apps — a key part of the company’s plan to “democratize” artificial intelligence.

Examples include a new feature, released just last week, which allows developers to export a custom data model to work offline with Apple’s iOS 11 Core ML machine learning framework, letting apps use Microsoft’s Custom Vision Service to recognize images even when not online.

“We actually export into Core ML and then you can download it into whatever app you have and then you can actually start using AI models at the device level, not connected to the cloud,” explained Irving Kwong, lead program manager for Microsoft Cognitive Services, showing how to use the technology to recognize pineapples during a demo last week inside a Microsoft office tower in Bellevue, Wash.

More artificial-intelligence announcements are expected from Microsoft next week at the company’s Ignite conference in Orlando, where Nadella is delivering the keynote address on Monday morning.

More challenges ahead

Even with all the activity, Microsoft and the field of AI have a long way to go.

“While this is very exciting, I think people might get confused that most AI problems are solved. That’s definitely not true. I want to caution everyone — we’re still very early in this AI thing,” Shum told the audience at the recent opening of the GIX U.S.-China tech institution. “Computers today can perform specific tasks very well, but when it comes to general tasks, AI cannot compete with a human child.”

Talking with GeekWire in his Microsoft office last week, Shum acknowledged that parts of the company’s artificial intelligence business are still very nascent, as well. “With many of those applications today, we are not overly thinking about making money (or) being profitable too soon,” he said. “We think we’re still early in terms of the right user experience. I think the next couple of years will be very important for us.”

After starting out with teams such as Bing, Cortana, Microsoft’s Information Platform Group, ambient computing and robotics, Microsoft AI and Research has continued to grow. Internal additions include the Azure Machine Learning team, headed by Joseph Sirosh, corporate vice president of Microsoft’s Cloud AI Platform, which previously was in Microsoft’s Cloud and Enterprise Division, reporting to Scott Guthrie.

Microsoft has also built up its AI capabilities with strategic acquisitions, including the natural language scheduling startup Genee and deep learning startup Maluuba. Deep learning expert Yoshua Bengio, who leads the Montreal Institute for Learning Algorithms and was a Maluuba adviser, is now an adviser to Microsoft and Shum.

Eric Horvitz leads Microsoft Research AI. (GeekWire Photo / Todd Bishop)

This summer, the company formed a new team inside Shum’s organization called Microsoft Research AI, led by longtime artificial intelligence researcher Eric Horvitz, to bring together the company’s top talent in core areas such as machine perception, learning, reasoning and natural language.

“We’ve largely built what I would call wedges of competency — a great speech recognition system, a great vision and captioning system, great object recognition system,” said Horvitz, who is known for projects such as the virtual animated assistant that greets visitors at his door. “We’ve managed to do incredible work with those wedges.”

But now the company is looking to bring all of that together in a quest for the elusive goal of general machine intelligence. “We really want to pursue the understanding of the mysteries of the human intellect,” Horvitz said.

Addressing another major issue facing AI, Microsoft has formed an internal group called “Aether,” which stands for AI and ethics in engineering and research. The group includes a representative from each Microsoft division, reporting to Nadella and the senior leadership team. Horvitz, who is part of the group, said it has been meeting about every two weeks to help the company form standards and best practices for artificial intelligence.

One of the issues the company will grapple with: leveraging the huge amounts of data available through Bing and LinkedIn. Shum acknowledged the limits to how Microsoft can use and present the connections among that data.

“When you think about the Microsoft Graph and Office Graph, now augmented with the LinkedIn Graph, it’s just amazing. It will take some good product sense to bring those things together,” Shum said. “We have to think through the rights to the data, whether it belongs to a company or an individual, and what the user shared and what the company would like to keep. Those things are good product design questions. But we’re very excited about LinkedIn and we are already working very closely.”

If the company can figure out those issues, it could be in unique position, said Samir Diwan, a former Microsoft senior program manager who is now CEO of the startup Polly, which makes chatbots for polls and surveys in Slack, Microsoft Teams and Google Hangouts Chat.

“When we think of research getting closer to product, we frequently think that product teams will be able to take better advantage of the research to deliver more compelling products,” Diwan said. “This might be the first time in history where things are inverted — a strong objective for bringing research closer to product is for research to take advantage of the large amounts of data being generated through product usage.”

Diwan explained, “That’s why I think to a large degree, the true wins at Microsoft of bringing research and product teams closer are years down the line. Over time, that research ideally feeds back into products creating a nice symbiotic relationship between the two.”

As we walked out of his office last week, Shum assessed Microsoft’s prospects in AI. “The new platform is emerging. It will take a little time, but directionally, I think many people see that,” he said. Now, the question is whether Microsoft “can really execute with differentiation,” he said, concluding, “We feel pretty good about our chances here.”

Like what you're reading? Subscribe to GeekWire's free newsletters to catch every headline


Job Listings on GeekWork

Executive AssistantRad Power Bikes
Find more jobs on GeekWork. Employers, post a job here.