MJFChat: Ask anything Cortana

The transcript is posted below:

We are entering the second year of our twice-monthly interview show on Petri.com that will cover topics of interest to our tech-professional audience. We have branded this show “MJFChat.”

In my role as Petri’s Community Magnate, I will be interviewing a variety of IT-savvy technology folks. Some of these will be Petri contributors; some will be tech-company employees; some will be IT pros. We will be tackling various subject areas in the form of 30-minute audio interviews. I will be asking the questions, the bulk of which we’re hoping will come from you, our Petri.com community of readers.

We will ask for questions a week ahead of each chat. Readers can submit questions via Twitter, Instagram, Facebook and/or LinkedIn using the #AskMJF hashtag. Once the interviews are completed, we will post the audio and associated transcript in the forums for readers to digest at their leisure.

Our next MJFChat, scheduled for Monday, January 6, is all about Cortana, Microsoft’s personal digital assistant. My special guest is Andrew Shuman, Vice President of Cortana. We want you to submit your best questions ahead of our chat.

Despite what you may have read on the Interwebs, Cortana is not dead. (Nor is she “just resting.”) For the past year-plus, Microsoft has been working to remake Cortana as a productivity aide rather than a standalone assistant like Alexa or Google Assistant. Microsoft is integrating Cortana with search, Office apps and data and other AI tools in the name of helping make users more productive.

If you’ve got questions about what’s new and what’s next for Cortana, Shuman is your go-to. If there are any specific topics or scenarios you’d like him to cover, make sure to chime in ahead of time.

Also: If you know someone you’d like to see interviewed on the MJFChat show, including yourself, send me a note at [email protected]. (Let me know why you think this person would be an awesome guest and what topics you’d like to see covered.) We’ll take things from there….


Mary Jo Foley: 00:01 Hi, you’re listening to the Petri.com MJFChat show. I am Mary Jo Foley, aka your Petri.com community magnet and I am here to interview tech industry experts about various topics that you, our readers and listeners want to know about. Today’s MJFChat is going to be all about Cortana and Microsoft’s new and hopefully improve strategy around it’s personal digital assistant and my special guest today is the corporate vice president of Cortana, Andrew Shuman. Welcome Andrew, and thank you so much for doing this chat with me.

Andrew Shuman: 00:43 Glad to be here.

Mary Jo Foley: 00:44 So we’ve heard at least for a year, probably more that Microsoft’s been working to reposition Cortana from a standalone assistant like Alexa and Google assistant to more of a productivity / assistant aide. Details on what that meant have been somewhat scarce. I knew based on talking to Andrew before that it involved enabling Cortana to handle more complex quote, turn by turn interactions, especially interactions focused on making business users more productive. But I want you to explain the Cortana strategy shift in your own words, Andrew, because I think you do a really good job of explaining this. So just to open the conversation in a very kind of succinct way, how would you say Microsoft is thinking differently about Cortana these days?

Andrew Shuman  01:40 Thanks, Mary Jo. Yeah, I think you introed it well. I think the thing that we have been thinking the most about is how to better achieve on some of these kind of daunting promises that the assistant landscape offers end users. I kid people that I’ve, I’ve never worked on a project where the expectations are so much higher than the day in day out reality. But the promise is so exciting. I mean, the first time that you set a reminder with your voice or you get music playing when your hands are full, it’s very exciting. Right? And that combination of natural language, voice control, the ambient device are all really, really exciting and an eyeopening. For us, we started thinking a lot more about how an assistant could truly help you and that in order for an assistant to truly help you, the first thing is it has to really know you and know you super well.

Andrew Shuman: 02:32 That was a big, a big change in our kind of thinking about the problem. If you think about a real life assistant, of course they get to know you, they understand your family, where your kids go to school, how you like your calendar organized, which people you meet with frequently. And as we kind of zeroed in on that, we thought a lot about, okay, how do we go after those users that really have embraced the Microsoft ecosystem? And thus the ones that we know the most about. And if you think about using Office 365 every day, they’re constantly, you know, showing us kind of how they spend their time, what their tasks are about the people they’re collaborating with most frequently, the documents. And projects they’re working on. And that, that part is really fundamental to how we think about an assistant moving forward.

Mary Jo Foley: 03:17 Okay, that’s good. So when I, when I think about this now, I kind of do a split in my head between Cortana being focused on business users and Cortana focused on consumers. And that may not be correct, but I have a reader, Reese Cooper who said he’s very interested in Cortana’s consumer future. And I know he’s echoing a lot of what I hear from readers and listeners because they understand to some degree that you guys are doing this as a business productivity focus now and embedding Cortana into like the Office 365 apps. But they also want to know that this has a consumer future too. If it does.

Andrew Shuman: 04:01 Absolutely.

Andrew Shuman: 04:09 This is a very interesting point of view and we really think about our users, um, as having come fundamental needs that span across their whole life.That’s an important thing to consider. We think that people managing their time, managing their tasks, managing their communication, keeping track of all the projects they’ve got going on is something that truly is a whole life problem. You don’t just stop doing that when you’re at home with your family. You don’t stop doing those things when you’re at work managing your family. The worlds are very interwoven. We do believe in a world where there’ll be lots of assistants and certainly there are things like online shopping, which Microsoft is not going to participate in, but we want to be a really good partner, with other assistants that can do that kind of thing. But we do think that really focusing our efforts on those users that are truly managing their time, their communications, the documents and projects they’re working on within Microsoft is a place that we can add the most value to those users.

Mary Jo Foley: 05:29 So it was a good way to think about it then is prosumers being the focus?

Andrew Shuman: 05:35 I think the main thing is to think about those universal needs that Microsoft Office 365 can really delight users on or Microsoft three 65 can delight users, which is about helping you manage your time, manage your projects, manage things that you’ve got going on. And that’s where I think the power of natural language is really powerful. If you think about something like a calendar situation, it’s actually a lot to say, I have an appointment next week at three o’clock with my doctor than it is type and search and tab through dialogues. And that’s the kind of thing that an assistant can enable. I actually chose a more personal or home kind of scenario by saying doctor, but of course I’ve could’ve said, you know, Mary Jo and I are meeting next, next Monday at noon and we should need to set aside time for that. There are these moments where it really is easier to say things than to do them with today’s software. If you think about mobile experiences, which is where we’re really starting from, that’s where something like outlook mobile, we think we can take a real leapfrog approach by, by thinking about that, how that intelligence, that natural language, the speech can all come together there.

Mary Jo Foley: 06:43 Okay. Speaking of mobile, I’ve been getting a lot of questions about Microsoft’s recent decision to remove the Cortana app on Android and iOS for people who are outside the U S in fact, Windows Central. Zach Boden just asked me that today on Twitter and he said, why are they doing this? We want to be productive out here too. So yeah, if you could talk a little about the thinking there and why, why outside the U S and why remove that app and that experience on Android and iOS?

Andrew Shuman: 07:17 Absolutely. Well first off, there’s some practicalities there. The reason we did outside the U S was because inside the U S people are still using that app to control their surface headphones or their invoke speakers, which are available in the U S and we need to continue to do that. The more broad point of view is that we want to be better focused on being where users already are, you know, with a hundred plus million weekly active users of outlook mobile, a growing number of millions of users on teams mobile. We think that those places are places where we can help people in situ and you don’t need to go off to an assistant to do something when you want to do it within the application that you’re already kind of using day in, day out. People on their mobile phones are checking outlook multiple times a day. It is a hub of their experiences and that’s where we think an assistant will really add the most value to users. It’s an important part of it. We’ve also done some work in windows and that windows work will continue to be international too. That’s a great place where that’s more of a hub like experience on your PC. But on mobile devices, we think it’s really important to, to be embedded within those Microsoft featured products that people are using.

Mary Jo Foley: 08:30 Okay. That’s, that’s kind of where I thought you were going because I think, I think that approach makes a lot of sense. Instead of having to switch back and forth and say, okay, am I using the assistant now or am I staying in Outlook or am I staying in Teams? You can just use it where you are. Right. Kind of access it without, without having a call on it. Even.

Andrew Shuman: 08:49 Exactly. And I think about Cortana as really being a glue across all of the M 365 products and services. Just like Microsoft account or Microsoft search. It’s a capability that you come to expect whether you’re inside any of the applications we have and that’s the future we see for us.

Mary Jo Foley: 09:08 Hmm, makes sense. So is there any thing in the future that’s going to be something like Cortana as a service on iOS or Android? Or is it all about being embedded in applications and other existing services that are on those devices?

Andrew Shuman: 09:25 Well, I think one of the things that we’ve seen with Apple in particular is that as they’ve expanded the Siri landscape to let you do Siri commands, that’s a nice way to tie Siri together with an application experience. So for example, with our new feature in outlook called play my emails, I can set up Siri to easily say, Hey Siri, play my emails and it’ll go ahead and launch outlook and launch the voice experience. So that’s an example where I think you are seeing a little more of what you turn to like a Cortana service. It’s interplaying with all these other assistants. That’s also where we think we can do a lot of interesting work with Amazon and continue to use that partnership in ways that are interesting to have assistance handing off to one another.

Mary Jo Foley: 10:10 That’s is a perfect segue into the next question that I was asked today, which was from Kenneth Moody and he said, how is Microsoft going to better align Cortana and Alexa for productivity specifically in the home? So he’s looking for this more seamless experience between the two. Not where you have to ask one assistant to access another, but more just in a way that things like to do an outlook and calendar will work when you’re using the device. I keep using the wrong word. I keep saying the word.

Andrew Shuman: 10:48 Sorry about that. Yeah.

Mary Jo Foley: 10:50 So yeah, he’s, he’s asking about how and when will we see better integration between the different digital assistants.

Andrew Shuman: 10:59 Well, the, the Amazon partnership is a great one and I’m one that I’m really excited that we’ve got ongoing, it does have some kind of hurdles as you kind of have these two companies working together on these emerging platforms and how they can communicate with one another. So for example, Kenneth is kind of pointing out that wake word thing where you say, Hey Alexa, and then you say open Cortana and we are absolutely aligned with Amazon on wanting to really lower that barrier and make it simpler and simpler to invoke skills more based on your intense and how that can come together. We don’t have anything thing to announce today, but I definitely echo that kind of concern and an area that I think we absolutely want to go after. And it’s an area that I think we’ve seen a lot of great partnership, at the highest levels of our two companies.

Mary Jo Foley: 11:44 I’m going to ask you about the Invoke now and I know you guys are not the developers of the Harman Kardon Invoke speaker, but you have a partnership with them and that device has been in the market for a while. So a lot of people noticed recently that there were some problems when they were using their Invoke speaker. Like I, I’ve noticed this too because I have one and it seemed for a little while, like it was just a one off kind of thing, you know, like getting Skype to connect to Cortana problems when I’m trying to use it to add a new calendar item or things like recognize and commands that it once did recognize. But it doesn’t seem to anymore. So I and other readers and listeners are curious if you can talk about the future on that device in particular and what should people expect? I mean, is it just starting to get old and that’s the way of the world or are there things that you’re doing on the back end that could improve the experience for people who still have the speaker people, I guess just want to know what should they expect on that front?

Andrew Shuman: 12:50 Well, I’m really glad we did the Invoke, I mean it was an important way for us to learn about doing ambient devices and important starting point for us to understand how we can kind of get our hands around the assistant landscape and I to still have an invoke at home and still use it regularly. There’s one in both of my kids’ rooms and we’re happy to understand any problems people are having because it’s a fully supportive product. It should work fine for people. I think as I mentioned to you when we were, we were chatting, at the ignite conference, we continue to want to support that and we think there are good opportunities moving forward. I think there’s going to be much more of a lens where we think a lot about how a device like that can delight users around productivity.

Andrew Shuman: 13:32 And so around things like managing their time, managing projects they’ve got kind of coming up or to do lists, places where an ambient device can really fit into the M 365 ecosystem is how I think about that. We’ve had some starts with, with the invoke, but we also need to recognize that M365 is based 100% on real deep trust trust. That’s at this enterprise level of security. And that’s where we’ve done a lot of work as we’ve embedded in outlook and other places to truly move to a fully eyes off training model. Not to get too in the weeds here, but, but historically many of these assistants have tended to have human judges that look at different kind of utterances and things to, to decide if they were right or wrong. And with our system, we actually embrace eyes off training, which means we train only with some donated data and internal employee data, at Microsoft and then use that to generalize to the public. And that’s a huge change for something like the Invoke speaker.

Mary Jo Foley: 14:35 Hmm. The other thing I remember, I think you and I talked about this at Build last year was something I hadn’t even thought about. But right now if somebody walks into your house and says, tell me what my meetings are tomorrow. It doesn’t matter if it’s you or not you, it will say what your meetings are. And in terms of compliance work, I know that Microsoft’s done. That’s a big no, no. Right. It needs to be tied to the person whose schedule it really is.

Andrew Shuman: 14:59 Absolutely. I mean, if again, you think about your mobile phone, you’ve got a fully trusted device with two factor off, a great microphone, a good speaker that’s available right now in your pocket and for 100 million users. And that’s, that’s just a very exciting space to explore and go big on. You’re right there, there are a lot of IT managers and there’s really a lot of people right, who are not comfortable with the idea that anyone in their house could send an email on their behalf or look at their calendar or things like that.

Mary Jo Foley: 15:29 Mm. So Kenneth Moody who gave us another question about the Invoke, he asked if Microsoft has ever thought about doing something like an Invoke type speaker for the office. And he said, wouldn’t this be a cool accessory for things like a Hurface hub or just using it standalone, but like in a meeting room. Is that a place Microsoft wants to play and or partner or is it not really where you’re focusing your focus more on phones?

Andrew Shuman: 15:57 Well, we have to get going with getting more usage and engagement. And seeing people use the system so they can understand where voice and natural language can truly add value. And that’s where going after those phone users is really paramount and are our top priority. But we do think that there’s an interesting place as we were alluding to, I’ll just take windows as an example, like where windows could be simpler and certain tasks and productivity sort of scenarios by using your voice or even by typing a command, go again to the kind of remind me next week to, to, to schedule time with Mary Jo. That’s a kind of thing that’s really easy to say, even easy to type, but it kind of hard to go do and navigate with all tab and window clicks and things like that. So I do very much believe that a so-called ambient experience of voice forward experience in the workplace, will be really exciting for us.

Andrew Shuman: 16:52 And an area that we’ll come back to again and again and we’re going to start there. And I think we’ll start to see that in a lot of places. You know, we’ve already announced some Teams integration. We haven’t yet shipped that. But that’s an example of where things could start to head. Then I think windows is another place where, you know, as I’m walking into my office in the morning, how do I kind of get up and going just using my voice. That could be powerful. Lastly, I’ll mention the headphones.The new surface headphones and the revs of them are all examples where I think in a productivity environment, you know, kind of get going and calling or meeting with people, uh, can be very powerful with your voice. So I echo kind of optimism and I think we will come back to that. But in the meantime, I think about those, like in your car when you’ve got your outlook mobile going and now you can triage your email. The play my emails. That’s a really powerful feature and a feature that’s really gravitating for a lot of users in an interesting way.

Mary Jo Foley: 17:50 know, I’m excited to have that come to Android. So, you know, I’m not an iOS user, but I’m hoping it’ll be soon.

Andrew Shuman: 17:57 Yeah, we’re actively working on the Android version. Absolutely.

Mary Jo Foley: 18:01 Nice. So I have to ask this question because it comes up anytime you do any kind of Q and A about Cortana or Bing. And it’s always the same question from everybody outside of the U S when will we get support for other languages? Is there any hope? People, people really want to use it but they are just like, I’ve been waiting and waiting and I need other languages beyond English. So just today, reader Andrew Reynolds said Cortana’s even disabled on PCs outside the US even if you select language and region set to the US, so people are desperate.

Andrew Shuman: 18:42 So on windows we will continue to support the markets we support in Windows over the coming year. We do want to think about how as we learn a lot in the US how those can go to global markets, but we also, I point you back to being a little more humble and a little more managing expectations. We have a lot to learn and a lot to think about with users and make sure the experience is great and we want to start that in the, in in the U S experience, especially on the mobile devices. So you’ll continue to be able to do the simple windows commanding and things like that, but we definitely with the productivity stuff will take a little longer as we work through. But rest assured, I mean we are a global company or a company that has invested a lot in data centers around the world and in security and privacy around the world. And we will, we will head in those directions. But with something as nascent as a natural language experience voice first, I think we need to be thoughtful and humble as we, as we move out.

Mary Jo Foley: 19:43 Hmm. Okay. I have to throw him one of my own questions because I’m letting everyone else ask all the questions. So I’m super interested in where you’re at with some of the things you showed us a sneak preview of at build where we talked about some of the work that semantic machines, which Microsoft bought has been doing with you around the full duplex slash turn by turn kind of capabilities. What are we going to hear more about how that’s progressing?

Mary Jo Foley: 20:12 Yeah. And so for, for your readers, I encourage everyone to look at the Build. I mean your listeners. I encourage everyone to look at the Build video, which does show a really interesting kind of full dialogue, full multi turn where you’re referencing history in the conversation. You set up an appointment at a Starbucks and then you say, I wonder what the weather will be like there. And then maybe I need, do I need an umbrella? Like there’s all these kinds of nuances that semantic machines has brought to us that basically boil down to letting us understand rich semantic knowledge of the conversation and the, and being able to reference simply and naturally also think about being able to scale out to more and more skills and capabilities and it really easier way. And we are continuing to work really hard with those guys and thinking a lot about how we can have more to show to end users than just a video. So that’s definitely an area of deep investment for us. We don’t have anything go now or, or talk about right now, but we do think that that’s a very promising area. And it does bring the next level of truly natural language where you speak even more naturally. I mean, it’s just another, you know, 10 X better with that experience.

Mary Jo Foley: 21:24 Yeah. I just, I just thought it was there recently and I said to my speaker, what’s the weather today and what’s it going to be tomorrow? I followed it up with, and of course it doesn’t know what you just meant because there’s no continued like thread there so far. So I think that’ll be great when that is there, it’ll make it feel like you actually are talking more in a more natural conversational way than we are now.

Andrew Shuman: 21:49 Yep. Absolutely.

Mary Jo Foley: 21:50 All right, Andrew, we’ll thank you so much for taking the time to talk all about Cortana. It’s been really good and very much appreciated.

Andrew Shuman: 21:58 Great. Thank you Mary Jo, and I’m always happy to talk more, so look forward to followups.

Mary Jo Foley: 22:04 Great. And for everyone else who’s listening to this, all you MJF chat readers and listeners, I’ll soon be posting information on Petri so you can see who my next guest is going to be. Once you see that you can start submitting questions on Twitter and other social media channels where the guests and in the meantime, if you or anyone you know might make a good guest for one of these kinds of chats, please don’t hesitate to drop me a note. All my contact information is available on Petri.com. Thanks again.