Measuring the Impact of Teams on Email Traffic

TeamsVersusEmailHero

Email or Teams

It is natural that when someone like me who has been involved with email for so long argues a case like “Teams won’t replace email,” people will say “well, he would say that, wouldn’t he.” After all, we all show some form of personal bias in the technology choices we make.

But I am a data-driven guy, so to back up my assertion, I reviewed the available data to prove the case. Or at least, to discover what the data revealed.

Getting Email Usage Data

Before reviewing the data, we should talk about what data is available. In the past, and even now for on-premises servers, message tracking logs are the source of email traffic data. All versions of Exchange have generated message tracking logs, and many articles are available to explain how to interpret and parse the logs using tools like Microsoft’s LogParser utility to generate statistics.

In the Microsoft cloud, the Graph reigns supreme. Although Exchange Online servers still generate message tracking logs and those logs hold interesting and useful data, the usage reports in the Office 365 Admin Center and the usage data made available for public consumption are based on the Graph reporting API.

There is both goodness and badness in this situation. It is good to have a single definitive source of Office 365 data. What’s not so good is the relative inflexibility of the API. For example, the Graph reporting API is happy to return data for 30 days of email traffic, but not 31 days or 28 days. That’s why some ISVs continue to mine data from sources like the message tracking logs.

In any case, the Graph is a good source for email traffic data. If my assertion is correct that Teams will not replace email, but can reduce the volume of email, we should see a reduction of email traffic in the period following the introduction of Teams within a tenant.

Measuring the Impact of Teams

Introducing Teams is one thing; people using Teams in an intelligent and productive manner is quite another. It takes time and some training to make people productive. In other words, do not expect miracles to happen after users see the Teams icon show up in the Office 365 app launcher and receive invitations to join a couple of teams. If you do not give guidance to help people use a tool effectively, they will revert to habit and use email.

In terms of measurement, it is best to analyze data several weeks after people start to use a new tool to cut out the first burst of enthusiasm and all the “Wow, we’ve got Teams” messages from the equation. We need some data from the period when Teams was not in general use to be the yardstick for comparison. Finally, the comparison periods must be comparable, such as two weeks that do not include holidays or other factors that might influence traffic.

When analyzing data in this instance, it seems sensible to focus on sent messages. Although the app supports external users, Teams is a very inward-focused application and is unlikely to affect the number of inbound messages that arrive via email, especially traffic from other domains.

In my case, the company that I work with most closely began to use Teams on October 16. I am a guest user in that tenant. Allowing two weeks for things to settle down, I reviewed my email usage for the week of November 6 and compared it to earlier activity, specifically the week of October 2. I did not travel during these weeks, so they gave me an “apples to apples” comparison.

Applying a Lens to Usage Data

The Email Activity report in the Office 365 Admin Center is not granular enough to focus on a week-by-week analysis for selected users. The report also lags by a few days. Possibly because of a glitch, on November 13, the latest data available when I looked at the data was for November 5 (Figure 1).

Office 365 Mail Usage Report
Figure 1: Office 365 Mail Activity report (image credit: Tony Redmond)

Another way of looking at the same time is through the Office 365 Adoption content pack for Power BI, which is still in preview but available to all Office 365 business tenants. After a dip in August (vacation) and a rise in September (Ignite), the Email activity report shows a drop-off in email received and sent in November 2017 (Figure 2).

Office 365 Power BI Content Pack
Figure 2: Office 365 Adoption content pack email activity graph (image credit: Tony Redmond)

Teams Usage Reports

Earlier today, Microsoft announced that Teams usage reports are now available in the Office 365 Admin Center (Figure 3). The new reports detail user activity and a breakdown by devices going back to 3 July 2017, which is probably when Microsoft began gathering the data. Interestingly, I see no data for guest user activity and only tenant accounts are listed in the detail.

Teams Usage Report
Figure 3: Teams Usage report in Office 365 Admin Center (image credit: Tony Redmond)

Unfortunately, most of my Teams activity is as a guest user in other tenants, so knowing what is happening inside my own (small) tenant is not helpful in this case.

Getting Specific

The email usage reports give us a general feeling of email traffic patterns within a tenant, but because I wanted to look at a selected group of users (just me!), the usage reports are not precise enough for my purpose. One way of probing deeper is to download the usage data from Office 365 and use Excel to slice and dice it as you wish. Alternatively, you can use one of the ISV reporting products available for Office 365, Because I know it best, I used Quadrotech’s Office 365 reporting tool to examine my personal traffic pattern.

After I focused in on the data I wanted, the picture was clear. If users move internal discussions to Teams, you see a distinct and measurable impact on email traffic volume. In my case, the number of daily outbound messages dropped from a range of 45-95 in the week of October 2 to 15-29 in the week of November 6 (Figure 4). Essentially, I moved back-and-forth email to Teams. The remaining email was to people I could not reach through Teams, needed protection, or contained more thoughtful (some say verbose) interactions.

Quadrotech Radar Reporting Email Analysis
Figure 4: Email usage declines as Teams takes effect (image credit: Tony Redmond)

Note the huge peak in traffic during the week of the recent Ignite 2017 conference in Orlando. As much of that traffic centered around industry contacts that I wanted to meet during the conference, none of this traffic would have moved to Teams. The point here is that traffic can only move if people share the same space within Teams. The value of email is that you can send a message to anyone once you know their address.

Another thing to consider is that email volume is influenced by situation. For example, if I am on the road, I am less likely to use Teams because it does not support offline access.

The MyAnalytics View

The MyAnalytics developers recently enhanced their dashboard with a view showing trends of user activity in the buckets they measure. One of the buckets is the hours spent processing email (both inbound and outbound). Not everyone has MyAnalytics as you need Office 365 E5 or a separate add-on license. If you do, MyAnalytics gives a time-based method to measure the impact of Teams on email for individual users.

MyAnalytics user trends
Figure 5: MyAnalytics Trends (image credit: Tony Redmond)

MyAnalytics has a six-week moving timeline. Figure 5 shows my timeline from 8-October to 12 November. Over this period, my time spent on email declined from 11.5 hours to 7.4 hours. Teams might have taken the “gained” time, but it is impossible to know because MyAnalytics does not yet measure Teams activity.

A Rigged Result

Of course, this data gives a rigged result and your mileage will vary. I had a good idea what the data would show because during the reporting period I deliberately used Teams whenever possible to conduct conversations. Accepting that I knew the likely results, what I wanted to show is how to use the data available to administrators to measure the influence of Teams within their tenant.

Although Teams will change the volume of email traffic in a tenant, the important thing to realize is that it is impossible to move all communications to Teams. Any message that needs to go outside the coverage of Teams must go via email, as do anything that needs protection (encryption). The case I made that Teams can reduce the use of but not replace email seems true.

Follow Tony on Twitter @12Knocksinna.

Want to know more about how to manage Office 365? Find what you need to know in “Office 365 for IT Pros”, the most comprehensive eBook covering all aspects of Office 365. Available in PDF and EPUB formats (suitable for iBooks) or for Amazon Kindle.