Microsoft Closes Outlook Copy-On-Write Flaw with Exchange Online Fix
Outlook Bug Clamped Down on by Exchange Online
On January 14, I reported how a flaw in Outlook desktop compromised Exchange Online Native Data Protection. In a nutshell, users whose mailboxes were on hold were able to remove attachments from sent or received messages if Outlook was configured in cached Exchange mode. This undermined the guarantee of Exchange Native Protection that any attempt to remove data from mailboxes on hold would be captured and available for eDiscovery.
After experimenting with several novel theories, such as an Outlook failure to synchronize properly, the Exchange development group concluded that the best solution was to deploy a server-side fix to prevent any further possibility of data loss. The deployment of the fix is now complete across Office 365.
Assessing the Impact
Although it’s good that Microsoft has closed the gap, they have not said how many attachments might have disappeared from eDiscovery. It’s a difficult question for Microsoft to answer, mostly because they don’t have data to quantify any loss of data, if that in fact occurred.
The problem of measuring data loss is hard. We know that the problem only surfaces in specific circumstances when on-hold mailboxes are accessed by Outlook configured in cached Exchange mode and the mailbox owner (or a delegate) removes an attachment from a sent or received message. Depending on who you ask, this set of conditions might be considered rare or common.
Say Goodbye to Traditional PC Lifecycle Management
Traditional IT tools, including Microsoft SCCM, Ghost Solution Suite, and KACE, often require considerable custom configurations by T3 technicians (an expensive and often elusive IT resource) to enable management of a hybrid onsite + remote workforce. In many cases, even with the best resources, organizations are finding that these on-premise tools simply cannot support remote endpoints consistently and reliably due to infrastructure limitations.
Easy to Put Mailboxes on Hold
Asserting that the combination of on-hold mailboxes and Outlook in cached mode is rare might be true when measured against the full set of Office 365 tenants, but it runs into problems in the enterprise. Placing mailboxes on-hold has become increasingly common since the in-place holds first became available in Exchange 2010. Indeed, Microsoft makes it easy for organizations to apply holds to Exchange Online mailboxes and provides a substantial 100 GB quota for the Recoverable Items folder to store deleted and modified copies of on-hold items.
Many Enterprise Mailboxes are on Hold
The last public data released by Microsoft for Office 365 said that 200 million users are active at least once a month. That figure is from October 2019 and is higher now because Office 365 grows at between 3 and 4 million users per month. Most Office 365 users have an Exchange Online mailbox and Exchange Online is the largest workload. An account must have at least an Office 365 E3 license before a hold can be applied. Microsoft doesn’t release data about the distribution of licenses across the Office 365 user base, but it’s reasonable to assume that roughly half of the base, or 100 million mailboxes, belong to large enterprise tenants. These users are likely to have E3/E5 licenses and are the likely target group whose mailboxes might be on hold.
Let’s say that 25 million of those enterprise mailboxes are on hold. Although OWA is a fine client, my belief is that most enterprise mailboxes are accessed with Outlook desktop configured in cached Exchange mode. For the purposes of argument, let’s 20 million on-hold mailboxes belong to Outlook users running in cached mode.
Bug Around for Over a Year
The next thing to ponder is how long this bug existed in the wild. Given that it doesn’t occur in Exchange 2016 or Exchange 2019, it’s probable that the bug crept in some time after Microsoft split the code and on-premises code bases in mid-2018, meaning that the bug might have existed for a year or more.
We therefore have a situation where the owners of millions of on-hold mailboxes could have removed attachments using Outlook for over a year. This sounds like it could be a big problem.
Stay Calm and Think
But before we conclude that the world of Exchange is faltering, users who removed attachments to hide evidence of some malfeasance must have done so in hope rather than expectation of success. They didn’t know that the bug existed, and no trace would be left. Removing the attachment just seemed like a way to cover their tracks. In some respects, removing an attachment is an act of desperation because the possibility is always there that the attachment is present and can be found in some other mailbox.
The truth is that no one knows how many times people removed attachments from on-hold Exchange Online mailboxes and succeeded in deleting data. Microsoft doesn’t track user activity at this level of detail. We don’t know if any data loss occurred in error or by purpose. We don’t know what mailboxes or tenants might have been affected. And apart from looking for another mailbox that might have a copy of a message with the attachment intact, no one has any way to recover data loss if it happened because Microsoft uses Exchange Native Data Protection to avoid the need to take backups.
Taking all that we know into account, I don’t think much data loss occurred because of this bug. Yes, a big number of users were exposed to the potential of data loss, but in the real world how many people whose mailbox are on hold decide to remove attachments? Perhaps some do and maybe a few did. The real number is likely a very small percentage of the overall Exchange Online mailbox count. However, we don’t know.
It’s disappointing that the bug appeared in Outlook. It’s disappointing that Exchange Online didn’t include code to stop clients ignoring the tenets of Native Data Protection. And it’s disappointing that Microsoft testing (or lack thereof) allowed such a bug to exist in the wild for over a year. But software is software and bugs exist. The key thing now is to hope that Microsoft learns from the situation and takes steps to bulletproof Native Data Protection for Exchange Online. And no, they won’t take backups…