Microsoft Updates How Exchange Online Processes License Removals Without Warning; Chaos Ensues
[Update October 31 2016: Microsoft’s description of the problem and their resolution is now posted on the EHLO blog]
It’s common to find that large Office 365 tenants create their own account provisioning tools to fit in with existing processes and procedures for maintaining user accounts. In the enterprise world, the need might arise to align Office 365 accounts with other internal systems. For instance, the creation of a new employee record in a HR system might lead to the creation of an Office 365 account and the assignment of whatever license is required for the employee to work. In the academic world, provisioning is a big deal because of the amount of churn in the student population that occurs at regular intervals. Thousands of accounts might have to be provisioned or deprovisioned quarterly.
Exchange Online and Licenses
Recently, Microsoft made a change to the actions taken when a license for Exchange Online is removed from an Office 365 account. Previously, license removal resulted in a disconnected mailbox. A disconnected mailbox can be recovered by linking the mailbox back to an AAD account. Now, the mailbox remains in the database, but its previous owner is blocked from signing into Exchange Online. If necessary, access to the mailbox can be regained by reassigning a license to the Office 365 account.
On the surface, the change seems reasonable. However, some bugs in the implementation meant that disconnected mailboxes retain their proxy addresses, meaning that Exchange continued to deliver inbound messages to the mailbox. Although this might seem like a good thing because any new email sent to the user is captured in the mailbox and is therefore available if the mailbox is subsequently recovered, holding on to the proxy addresses means that the addresses cannot be reused with other mail-enabled objects. Another bug meant that users could continue to access the mailbox after the license was removed. Finally, the mailbox remained visible in the GAL.
What an Administrator Needs to Decide
The intended behavior is that if a license is removed from a mailbox, the mailbox essentially enters a limbo state where it exists but cannot be accessed or participate in email delivery. It’s up to the tenant administrator to decide whether the mailbox should be:
- Permanently removed (hard-delete). In this case, the mailbox can no longer be recovered. You can permanently purge a mailbox from Exchange Online by running the Disable-Mailbox command. For example:
[PS] C:\> Disable-Mailbox -Identity JoanDoe -PermanentlyDelete
- Retained for a short period in case it needs to be recovered (soft delete). Mailboxes in this state are removed from Exchange after 30 days, at which time they can no longer be recovered. A mailbox is soft-deleted if the user’s Office 365 account is removed.
- Retained indefinitely (inactive). A mailbox becomes inactive if it is under a litigation or in-place hold for compliance or regulatory purposes and is then deleted. Exchange recognizes that the hold exists and retains the mailbox for as long as the hold applies. During this time, the mailbox can be recovered or restored. When the hold is removed, the mailbox is permanently deleted.
The approach is eminently reasonable. It is designed to provide tenants with control over their data without running the risk that a temporary condition, such as removing and reassigning a different Office 365 license to a user, could cause data to be lost. Disconnected mailboxes work, but they are designed to allow for short-term recovery of mailboxes deleted in error instead of a method to retain mailboxes in a consistent and predictable way.
The proxy address bug caused some disruption for the provisioning system used by a large U.S. university, all of which was highlighted in different online forums. Software bugs are a fact of IT life and Microsoft has backed the change out to fix the bugs. Once updated code is available that is of sufficient quality, the change will appear inside Office 365 again.
The Importance of Communication
However, buggy software is not the real issue. No one would ever complain if the change had been flagged well in advance to allow tenants to consider the ramifications of the new approach so that they could test whether the change affects tenant-specific code, such as the provisioning system used by the university that hit the problem.
An update to a TechNet page says:
Previously, in Exchange Online, if you removed an EXO license thehttps://technet.microsoft.com/en-us/library/dn186233(v=exchg.150).aspx%20says: user was kept, but the mailbox user was transformed into a mail user and the mailbox was moved to the recycle bin. You could then recover the mailbox within the 30 days’ time limit. This has now changed in Exchange Online. If you remove a user’s license, the user mailbox will no longer be able to sign in and use Exchange Online or Office 365. The user mailbox will remain in Exchange Online until it is deleted, permanently removed or purged by the Office 365 admin. You can reassign a license to the user and make the mailbox active again.
The Dark Side of the Cloud
Apart from the text inserted into TechNet, no steps were taken to advise customers that a pretty significant change was in the works. Microsoft introduced the change into production without so much as a “by your leave.” Few scan TechNet for updates on a regular basis, as it’s a source of knowledge rather than a news channel. The net result was chaos and unhappiness as customer code stopped working.
Situations like this illustrate the loss of control companies hand over to cloud providers. Even in the fast-paced world of evergreen software, it would be nice if developers paid some consideration to how changes they make might affect real-life systems.
In this case, a well-intended alternation in behavior failed due to a combination of buggy software and poor communications. It’s not a mix that anyone wants to see, least of all Microsoft.
Follow Tony on Twitter @12Knocksinna.
Want to know more about how to manage Office 365? Find what you need to know in “Office 365 for IT Pros,” the most comprehensive eBook covering all aspects of Office 365. Available in PDF and EPUB formats (suitable for iBooks) or for Amazon Kindle.