Widespread Criticism of Microsoft Productivity Score is Unfounded

Intrusive Usage Data Exposed in Microsoft 365 Admin Center

Over the past few week or so, a bunch of news reports have emerged about the new Microsoft 365 Productivity Score feature (Figure 1), which was announced at the Ignite 2019 conference before achieving general availability in October 2020. In the Microsoft blog, CVP Brad Anderson said: “Productivity Score focuses on two areas – the Employee Experience and the Technology Experience, which provide visibility into how your organization works, insights to identify where you can make improvements, and actions you can take to update skills and systems so everyone can do their best work.

Image 1 Expand
Productivity Score
Figure 1: Productivity Score analyzes different aspects of tenant activity (image credit: Tony Redmond)

 

That’s not quite the view taken outside Microsoft. The Guardian newspaper in the UK said that the toolallows managers to use Microsoft 365 to track their employees’ activity” while Business Insider led its report with the assertion that the tool “lets your boss track how much you use email, Teams, and even whether you turn your camera on during meetings.” It seems like most reports picked up on the views of Austrian researcher Wolfie Christl, who called Microsoft 365 a “full-fledged workplace surveillance tool,” something explicitly denied by Microsoft CVP Jared Spatero who emphasized that “Productivity Score is not a work monitoring tool.”

Privacy is Important

Privacy is an important topic that needs to be taken seriously. The complaint in the 2018 Data Protection Impact Analysis (DPIA) for the Dutch Government is one example of how Microsoft has failed to do enough to protect user privacy at times. I still think that not enough has been done to anonymize user data in some places, notably in the information sometimes exposed in Office 365 audit records. It’s right to focus on privacy and ask questions about the topic when new software appears, so let’s consider if the Productivity Score has a problem.

Usage Data and the Microsoft Graph

Two salient points seem to have been ignored by most commentators. First, and most important, the usage data presented by the Productivity Score feature has long been available through Microsoft Graph APIs. Organizations can access the data to generate their own analysis of what people do. It’s relatively common, for instance, to report usage to identify underused accounts that don’t need expensive licenses. I have a PowerShell script to grab data from the Graph and analyze the data to report inactive accounts. Many third-party ISV products like Quadrotech Nova and CoreView consume the Graph data to create reports and graphics about user activity.

Second, most of the user-level data exposed in Productivity Score has been available in Office 365 usage reports for many years. Productivity Score is an opt-in feature, and a tenant administrator can turn off the detailed user-by-user view (what Microsoft calls “people experience insights”) through a setting in the Reports section of the Microsoft 365 admin center (Figure 1). As Wolfie Christl correctly notes, the setting to show people experience insights is activated by default. It would be better if this was also an opt-in choice.

Image 2 Expand
Admin Center Anonymize Data
Figure 2: Settings to control usage data in Office 365 reports (image credit: Tony Redmond)

 

While you’re reviewing the option for reports, consider anonymizing the data used for other Office 365 usage reports. This won’t stop administrators being able to get to the data if needed, but it will stop accounts assigned the Reports Reader role seeing user data.

Productivity Score Data is at a High Level

It’s entirely possible for managers to see information about different aspects of user activity, but not at the level anticipated in news reports. Knowing the number of meetings someone attended with video enabled is all very well but is meaningless unless placed in context. Perhaps their network connection didn’t tolerate the higher bandwidth requirement (common when working for home). Perhaps they were listening to a briefing instead of participating actively in a discussion. And perhaps they simply prefer to not use video. I doubt any manager worth their salt will rush to harass an employee because they have a low ratio of video enablement for Teams meetings. People are judged by their effectiveness and output and not by how often they turn on video in online meetings.

The same is true for the number of messages sent or read, documents created, and so on. No insight is revealed about the type of messages, how long someone spends creating a document, the number of paragraphs written, and so on. Instead, we learn about how many people use OneDrive for Business or SharePoint Online to store their files, who prefers to use “cloudy attachments” for email, and who shares documents externally. All interesting data points, but all available for years through the Graph and Office 365 audit records (here’s an example of using the audit log to analyze sharing behavior). Reading the commentary, it seems like many of the concerns raised seem to be founded on incomplete knowledge, perhaps because the writers have never investigated the data available to tenant administrators through the Graph and other sources.

Microsoft’s Productivity Score is much more about understanding how some elements of work happens inside Microsoft 365 apps at an overview level. It’s not detailed enough to understand how anyone works.

Microsoft’s Fascination with Data

To me, Productivity Score simply puts lipstick on a data pig. It’s part of Microsoft’s growing fascination with data signals gathered from user activity. Some of the fascination is driven by technological desire to deliver better services, such as making search more intelligent by using data gathered in the Microsoft 365 substrate.

The dark side of the Productivity Score is shaming organizations into doing better with functionality they pay for through monthly licenses. The most altruistic view of this is that Microsoft is proud of its products and simply wants people to realize benefits such as saving 104 minutes weekly through better meeting habits (doubtful in some organizations). Microsoft engineers have told me that people just don’t get the value of the way they use telemetry gathered from user interactions with applications to refine experiences and increase reliability. That’s certainly a valid perspective, albeit one very inward-focused.

It’s laudable when Microsoft uses data to improve applications. It only becomes a problem when customers think that Microsoft is becoming creepy and abusing its access to data which belongs to them. Usage data falls into that category and I don’t like it when Microsoft tries to force their view of what tenants should do by creating features like the ill-fated attempt to create Groups for managers and their direct reports.

An even less charitable view is that Productivity Score is just a way for Microsoft to convince organizations to spend more on higher-priced licenses to drive its annual revenue per user (ARPU), a much beloved data point for CFO Amy Hood. I don’t think that this is true, but I can understand how some believe it.

What’s clear though is that Microsoft started on this journey when they acquired Volometric in 2015. This led to the increased use of Graph signals to power features like MyAnalytics, Workplace Analytics, Cortana’s daily briefing message, the Outlook insights add-in, and meeting insights. Delve is another place where Graph signals are used to generate insights about documents people might be interested in reading. Microsoft’s desire to exploit usage data in meaningful ways to inform and improve applications won’t go away.

Update (December 1): in a response to the public criticism of Productivity Score, Jared Spataro published a new blog to announce that Microsoft is removing user names from the product and making it clearer that the intent is to help organizations adopt technology rather than measure individual user productivity.

A Pretty Interface, but Nothing New

Apart from creating some new graphics, I don’t see anything in Productivity Score to warrant the criticism heaped on the feature. Everything is summarized nicely in a generic sort of way, much like the same data is presented in Microsoft 365 usage analytics (the Power BI app available to tenants). ISVs do the same job with the same data to deliver their own take on what’s important to understand about how an Office 365 tenant functions.

No matter how you report the data, usage reports must be put into organizational context and fully understood before they deliver anything that’s useful and actionable. Time will tell if the Productivity Score proves to be anything more than a mildly interesting new take on an old topic.