Outlook Flaw Compromises Exchange Online Native Data Protection

Problem Exposed When a User Accidentally Deleted an Attachment

A January 9 post on Spiceworks noted a problem when a user accidentally deleted an attachment from a message they had received. The mailbox was on hold, so the administrator tried to exploit the copy-on-write protection feature in Exchange Online Native Data Protection to find and recover the original item from the Versions subfolder of Recoverable Items in the user’s mailbox. No trace could be found.

Microsoft’s documentation for copy-on-write protection says:

“If a user who is placed on In-Place Hold or Litigation Hold modifies specific properties of a mailbox item, a copy of the original mailbox item is created before the changed item is written. The original copy is saved in the Versions subfolder. This process is known as copy-on-write page protection.”

Obviously, the original message with the attachment should have been in the Versions folder and retrievable using an Office 365 content search. It wasn’t, so some alarm bells sounded, first in the MVP community when several MVPs reproduced the problem, and then inside Microsoft as customer support tickets flowed in.

Microsoft’s respond is described in Office 365 Service Health Dashboard advisory EX199762 (Figure 1). The notice is disingenuous in that it does not clearly describe the impact and severity of the issue in terms of compliance and data governance.

Reproducing the Problem

The issue is easily reproducible. Create a message with an attachment. Send the message to any other recipient. The sender then uses Outlook desktop configured in cached Exchange mode to alter their copy of the message by removing the attachment (one of the documented properties of messages that trigger copy-on-write protection) and saves the item. If copy-on-write protection works, Exchange saves the original copy of the message in the Versions sub-folder.

The first step is to check that the mailbox used for testing is on hold.

Get-Mailbox tredmond | fl litigation*
LitigationHoldEnabled  : True
LitigationHoldDate     : 9 Jan 2020 09:45:28
LitigationHoldOwner    : [email protected]
LitigationHoldDuration : Unlimited

Next, create a Word document containing a unique phase. The exact phase doesn’t matter; we’re only going to use it to search for the message. Create a new message and attach the Word document. And send it to someone else. The message is now in a sent state in an on-hold mailbox, meaning that Exchange should apply copy-on-write protection to track any modification.

To make it easy to find the message, move it to an empty folder (I used one called Testing). Now run a search to establish that the message with its attachment can be found.

Search-Mailbox -Identity tredmond -EstimateResultOnly -SearchDumpster -SearchQuery "Ballykeen 115"

RunspaceId : f36f7d7c-2bcc-4f16-9362-0f1796dba751
Identity : TRedmond
TargetMailbox :
Success : True
TargetFolder :
ResultItemsCount : 1
ResultItemsSize : 31.71 KB (32,470 bytes)

Check that there is one message in the folder.

Get-MailboxFolderStatistics -Identity Tredmond |?{$_.Name -eq "Testing"}|ft name, itemsinfolder, foldersize     
                                                                            
Name    ItemsInFolder FolderSize
----    ------------- ----------
Testing             1 31.71 KB (32,470 bytes)

With our test message in place, we now use Outlook to make a change to force copy-on-write to kick in. Open the message and remove the attachment (Figure 2). Save the message when prompted. Outlook then synchronizes the modification back to the server folder.

Outlook desktop is the only Outlook client that allows users to remove attachments from sent messages. You can’t do this with OWA or Outlook mobile.

Checking that Outlook Has Removed the Attachment

Synchronization of changes between Outlook and Exchange typically happens very quickly. To be sure, you can wait 30 seconds before checking the size of the message in the Testing folder. As we can see, the item is smaller because the attachment is missing.

# After attachment removed
PS C:\temp> Get-MailboxFolderStatistics -Id Tredmond |?{$_.Name -eq "Testing"}|ft name, itemsinfolder, foldersize

Name    ItemsInFolder FolderSize
----    ------------- ----------
Testing             1 16 KB (16,381 bytes)

Now search the mailbox using the same search query to see if we can find the attachment. If copy-on-write works, the attachment should be found in the Versions subfolder. However, in this case, Search-Mailbox (or an Office 365 content search) cannot find the item.

Search-Mailbox -Identity tredmond -EstimateResultOnly -SearchDumpster -SearchQuery "Ballykeen 115"

RunspaceId       : f36f7d7c-2bcc-4f16-9362-0f1796dba751
Identity         : TRedmond
TargetMailbox    :
Success          : True
TargetFolder     :
ResultItemsCount : 0
ResultItemsSize  : 0 B (0 bytes)

Verifying the Versions Subfolder with MFCMAPI

Outlook clients cannot access the Recoverable Items structure. To check what’s in the Versions subfolder, we need to configure Outlook in online mode and then run the MFCMAPI program to open the mailbox, navigate to the Versions subfolder, and open its contents table. If everything works, we should see the original copy of the message (the subject is Important doc). However, no trace of the item is visible (Figure 3).

The problem does not appear when Outlook desktop is configured in online mode. If you repeat the same steps in online mode, the copy of the original message is captured in the versions folder.

A Serious Outlook Bug

This is a serious bug. Although the mailbox was on hold, a user was able to remove an attachment from a message without Exchange noting the fact. Copy-on-write protection failed.

Pleading that the feature works when Outlook runs in online mode is no comfort. Most Outlook desktop clients connected to Exchange Online are configured in cached Exchange mode to avoid interruption from network glitches. In any case, a bug is a bug, and a bug in an area that affects data governance and compliance is of huge concern.

The sole consolation for compliance managers and eDiscovery investigators is that any attempt to remove incriminating data by deleting an attachment for messages sent inside a tenant is unlikely to succeed. Copies of the message exist in mailboxes for tenant recipients and is available for eDiscovery from those locations (unless everyone removes their copy). Intact copies of compromised messages might also be backed up or journaled to an external system. The problem is more severe when attachments are removed from messages received from outside the tenant as only one (compromised) copy might be available.

Outlook Desktop and MAPI over HTTP

The underlying bug is in the way Outlook for Windows implements the MAPI over HTTP protocol, which Microsoft released in 2014 as a more robust replacement for the Outlook Anywhere (RPC over HTTP) protocol. The Outlook code bypasses the copy-on-write protection when attachments are removed from sent messages. Microsoft removed support for RPC over HTTP in October 2017, so it’s likely that the majority of Outlook clients connect to Exchange Online with MAPI over HTTP. Some RPC over HTTP might still exist on-premises.

Several questions arise:

1. What versions of Outlook are affected? The assumption is that all versions which support MAPI over HTTP contain the bug, including Outlook click to run and Outlook 2019.
2. Why does the server not enforce copy-on-write protection in the Store? This would safeguard against client bugs like this.
3. What business need is answered by Outlook desktop being able to remove attachments from sent messages when neither OWA nor Outlook mobile allow this action?
4. Can apps written with Exchange Web Services or the Graph API avoid copy-on-write protection?

A prior example of a similar problem occurred in 2014 when OWA failed to preserve folder deletions for mailboxes on litigation hold.

At this point, we know that the problem affects Exchange Online and the latest version of Outlook (click to run) but It’s unclear exactly when that this bug surfaced and what combinations of Exchange and Outlook are affected. Some reports indicate that the problem doesn’t exist in Exchange 2016 CU14, which raises the question of why Outlook behaves differently when connected to Exchange Online.

Outlook Undermines Exchange Online Native Data Protection

We also do not know how many times users have removed attachments from their on-hold mailboxes accidentally or on purpose. We do not know how many eDiscovery searches have been compromised by missing information. What we do know is that bugs like this undermine Microsoft’s Native Data Protection strategy for Exchange Online.

Native Data Protection means that Microsoft depends on software features in Exchange Online to make sure that data is never lost or corrupted. This bug demonstrates that data can be lost under the aegis of Native Data Protection. And that data can never be recovered because Microsoft does not take backups for Exchange Online. Every backup ISV who has told Office 365 tenants for years that they should use a third-party backup service to secure data in their Exchange Online mailboxes now has a huge reason to justify their claims. The flaw in that argument is that backups depend on the data they stream from Exchange Online. If data is missing, the backups are also incomplete.

All in all, this is a big mess that Microsoft needs to fix quickly. We await future developments.