Analyzing Delayed Roll-Outs for New Office 365 Features
How Microsoft Advises Tenants About New Features
During a recent discussion, I was asked why Microsoft seems to publish so many incorrect roll-out dates for new Teams features. While my gut feel was that it certainly seems like some Teams features are delayed, some evidence was needed to prove or disprove the case.
When Microsoft is ready to deploy a new feature to tenants, it posts a notification in the Microsoft 365 message center for the tenants affected by the change. Each notification has a unique identifier, like MC212453 (the new Teams calling and meeting experience). In fact, some updates never warrant a notification, usually because Microsoft deems the update to be too minor to justify telling customers about the change. Leaving that point aside, let’s assume for the purpose of this exercise that what shows up in the message center is a good guide to the deployment of new features.
As I write this on November 11, the message center shows 205 notifications for my tenant. Depending on the geography your tenant is located in, the products used, and licenses, you might see a different number because notifications are issued to relevant tenants, not all.
Each notification has a start or publication date. It also has an end date, or when the notification is removed from the message center. Some notifications are marked as high importance, meaning that they might have a significant effect on users. Administrators can archive notifications as they wish, usually if a notification is deemed to be dealt with or unimportant, and the notification then disappears from the default All active messages view
When a feature is delayed, Microsoft republishes the notification and adds a prefix of “(Updated)” to the title. Although the metadata for notifications include a last updated property, that doesn’t seem to be populated when notifications are republished. However, the text of the notification usually contains the republication date. This format is obvious in the notification shown in Figure 1.
Varied reasons exist for why a feature is delayed. One thing you’ll never see reported is that “we found too many bugs.” On the other hand, Microsoft is fond of citing “To ensure the best possible experience for our users we are delaying some of our deployments to reduce the amount of change flowing into the services.” This covers myriad sins.
Some notifications persist for a long time without changes being made. The oldest notification in the message center dates from March 2019 (MC175274). Microsoft has extended its visibility several times to make sure that admins know about product lifecycle dashboards. In most cases, notifications aren’t visible for long after a feature is deployed.
Given that there’s only 205 notifications, it should be easy to analyze the data and discover which apps have most delays. A manual process would grab the data from the message center and put it in Excel. There’s no method to download the information from the message center, but you can exploit the synchronization to Planner feature because Planner supports export to Excel. This has an advantage in that the export includes all notifications since synchronization was established with Planner (858 in my case), so you get access to older data. The problem is that some duplicated tasks occur in the Planner data (Figure 2) when Microsoft publishes updates to notifications.
Microsoft is working on the synchronization between Planner and the message center to eliminate duplication and improve the data in what is already a very useful tool.
Using the Service Communications API
I decided that the best approach was to use the Graph Service Communications API to download the information shown in the message center. The requirements for an app to use the API to access service data is listed online and are similar to other apps you might have used with the Graph. For instance:
- Working with Planner data through the Graph.
- Fetching Azure AD sign-in data from the Graph.
- Using the Graph with Teams and PowerShell.
The PowerShell script I used can be downloaded from GitHub. It grabs the data, does some massaging to extract dates from text and set some flags, and creates an output list. A separate list is then generated for the updates. Sixty of the notifications had a prefix of “(Update),” meaning that 29.27% of all notifications in this sample have updates.
Analyzing the Data
When I looked at individual workloads, I found that Teams generated the highest number of notifications (because it shipped most new features in the sample set). At the same time, the new Teams features have the highest percentage of updates, but not by a lot over SharePoint Online. Table 1 shows the raw data based on the workloads noted in the notifications.
Table 1: Percentage of updates issued for new features by different Office 365 workloads
Figure 3 shows details of some of the delayed Teams features together with the number of days between the original publication and the date of the last update. In fact, some of the notifications tagged for Teams (like MC217339 covering shortcuts to shared folders in OneDrive for Business) are only slightly related to Teams, so some adjustments might be necessary to the raw data reported in Table 1. To arrive at definitive results, you’d need to read the text for each notification to assess just how connected it is to a particular workload.
What We Learned
Microsoft communicates change information about Office 365 apps to customers better today than it has ever done in the past. The data is available, and it tells us that apps like Teams and SharePoint Online do delay a sizable percentage of new features past the original deployment date. Microsoft could do a better job of announcing software when it’s ready, but it’s preferable to releasing software when it’s not ready, which would be much worse.