Microsoft Migrates Exchange Public Folders to Office 365 Groups

office365

On the Way to Groups

In May 2017, Microsoft made a second call for customers to help test a migration toolset to move public folders to Office 365 Groups. Testing has progressed since and Microsoft is approaching the point of making the tools available. In my view, it is likely that we will see an announcement about availability happen at the Ignite conference later this month. This description is based on the code made available by Microsoft to beta customers.

Migrating Public Folders to Office 365 with MRS

Microsoft’s approach to migrating public folders to Office 365 Groups uses migration batches and the Mailbox Replication Service (MRS). When it first appeared in Exchange 2010, MRS replaced an older mechanism for moving mailboxes from earlier versions of Exchange. The big advance was asynchronous movement, meaning that users could continue working while MRS moved their mailboxes. Asynchronous mailbox moves have proven their worth as we move into the era of 100 GB mailboxes.

Today, MRS can:

  • Move mailboxes from old versions of Exchange.
  • Move mailboxes between databases/servers to rebalance workload, including cross-forest moves.
  • Migrate mailboxes from on-premises servers to and from Exchange Online, including migrations from IMAP4 servers.
  • Move old public folders to modern public folders.
  • Move public folders from on-premises servers to Exchange Online.

Because Exchange Online stores public folders in public folder mailboxes and MRS already supports access to pubic folder mailboxes, it is natural to extend MRS and give it a new migration target in Office 365 Groups. The source public folders can be on Exchange 2010 (SP3 RU8 or later), Exchange 2013 (CU15 or later) or Exchange 2016 (CU4 or later) on-premise servers, or the folders can already be on Exchange Online. If you want to move public folders from on-premises servers, you need a functioning Exchange hybrid configuration in place.

Because this is an Exchange-driven process, you can only move public folders to Office 365 Groups that store conversations in an Exchange Online mailbox (Outlook Groups). You cannot move public folders to Yammer Groups.

Submitting Migration Jobs to MRS

To move public folders to Office 365 Groups, the same kind of processing steps occur as for other migration batches. You build a CSV file to give MRS a list of public folders to process and their destination groups, which you must create to act as targets before the migration begins. The CSV file is very simple and consists of two columns specifying the folder to move and the target mailbox belonging to the Office 365 Group.

FolderPath,TargetGroupMailbox
\Email\Archive for Exchange Discussions,[email protected]

In this case, MRS will connect to the public folder \Email\Archive for Exchange Discussions (you must give the full path to the folder) and move its content to the Office 365 Group with the primary SMTP address [email protected].

When the CSV file is complete, you pass it to MRS for processing as a parameter for a migration batch using the New-MigrationBatch cmdlet.

[PS] C:\> New-MigrationBatch -Name PFCalendar -CSVData (Get-Content c:\temp\pfmigrate.csv -Encoding byte) -PublicFolderToUnifiedGroup -Autostart

You can only move one public folder to an Office 365 Group if an existing migration batch already uses that group as a target. In other words, you cannot combine content from a set of public folders into a group in a one-short operation. However, if you want to consolidate some folders into a single group, perhaps to combine content from a set of folders belonging to a department, you can do this by running multiple migration batches, each targeting the same group.

Moving Content

MRS picks up the job and connects to the source public folder to pull data that it inserts into the target group. This process continues until MRS moves all data from the public folder into the group. While moves are in progress, you can check them through the migration section of EAC or by running the Get-MigrationBatch cmdlet. When MRS finishes processing a batch, you can download a complete move report from EAC (below). As you can see from the extract from the move report, the steps to move a public folder are identical to those when MRS moves a mailbox.

25/08/2017 19:35:05 [VI1PR0401MB2208] '' created request.
25/08/2017 19:35:08 [VI1PR0401MB2208] '' allowed the request to be created with mismatching source and target mailboxes.
25/08/2017 19:35:18 [DB6PR04MB3047] The Microsoft Exchange Mailbox Replication service 'DB6PR04MB3047.eurprd04.prod.outlook.com' (15.1.1124.9 caps:37FFFF) is examining the request.
[Content removed]
25/08/2017 20:49:53 [DB6PR04MB3047] Copy progress: 11784/12824 messages, 1.251 GB (1,343,427,971 bytes)/1.339 GB (1,438,125,414 bytes), 0/0 folders completed.
25/05/2017 20:53:06 [DB6PR04MB3047] Source mailbox information:
Regular Items: 12856, 1.347 GB (1,445,921,717 bytes)
Regular Deleted Items: 0, 0 B (0 bytes)
FAI Items: 5, 18.29 KB (18,728 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
25/08/2017 20:53:06 [DB6PR04MB3047] Target mailbox information:
Regular Items: 12990, 1.367 GB (1,467,751,450 bytes)
Regular Deleted Items: 0, 0 B (0 bytes)
FAI Items: 5, 19.83 KB (20,310 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
25/08/2017 20:53:06 [DB6PR04MB3047] Copying messages is complete. Copying rules and security descriptors.
25/08/2017 20:53:07 [DB6PR04MB3047] Verifying mailbox contents...
25/08/2017 20:53:12 [DB6PR04MB3047] Mailbox contents verification complete: 1 folders, 12824 items, 1.339 GB (1,438,125,414 bytes).
25/08/2017 20:53:12 [DB6PR04MB3047] Copy progress: 12824/12824 messages, 1.339 GB (1,438,125,414 bytes)/1.339 GB (1,438,125,414 bytes), 0/0 folders completed.
25/08/2017 20:53:13 [DB6PR04MB3047] Request is complete.

After years of using Office 365, my tenant does not have many public folders left. However, MRS moved an old archive for a mailing list holding 12,824 items (1.339 GB). MRS took 78 minutes to move the folder, including time to setup the move and then process all the items. I also moved a public folder with a calendar, which had a lot less data (148 KB), as you might expect with a calendar.

No Incremental Synchronization

I think most migrations will be one-off operations where tenants will transfer public folders in bulk to Office 365 and then switchover a selected set of those folders to Office 365 Groups. This is the simplest and cleanest way to migrate, but it means that everything must be ready and that is sometimes not possible. For instance, you might have a public folder used as a repository for inbound service requests and need to make sure that processing of service requests continues to work with Office 365 Groups before you switch.

However, unlike regular mailbox moves or public folder migrations, this kind of migration does not support incremental synchronization. Therefore, if you want to copy, test, and then switchover, you must remove the migration batch and resubmit it to MRS for processing. In this case, MRS ignores any item that is already present in the target group and only copies new and updated items from the source public folder.

Post-processing

When a migration job finishes, the source public folders still exist and a copy of their content is in the destination Office 365 Group. Mail items are in the Inbox folder of the group mailbox and become group conversations. Calendar items are in the calendar folder and become part of the group calendar.

You need to perform two other tasks to complete the move. First, you run the LockAndSavePublicFolderProperties.ps1 script (provided by Microsoft) to lock the public folder (set the folder to be read-only rather than locking the entire public folder hierarchy) so that users cannot add any further information. The same script copies essential properties like the SMTP address of the public folder over to the new group so that if anyone sends email to the public folder, Exchange will redirect the messages to the group.

[PS] C:\> .\lockandsavepublicfolderproperties.ps1 -mappingcsv 
c:\temp\pfmigrate.csv -backupdir c:\temp

Creating c:\temp\PfLockdown_summary.log
Exporting public folder permissions..
Successfully saved permissions to c:\temp\PfPermissions.csv
Mail Properties of mail enabled public folders are successfully saved to c:\temp\PfMailProperties.csv
SMTP Addresses of mail enabled public folders are successfully saved to c:\temp\PfToGroupMapping.csv
Locking down migrating public folders by removing permissions..

FolderName           User                 AccessRights
----------           ----                 ------------
French Public Hol... Default              {Reviewer}
French Public Hol... Anonymous            {None}
French Public Hol... Marc Vigneau         {Reviewer}
French Public Hol... Ben Owens (Busine... {Reviewer}
French Public Hol... Tony Redmond         {ReadItems, CreateSubfolders, FolderContact, FolderVisible}
Migrating public folders are successfully locked down with ReadOnly permission.
Public folder lockdown is complete.

It is wise to perform all stages of the migration, especially switching properties, at a time when public folder usage is low, including when users might send messages to the folders. There will be a certain period when the email address for the public folder is being switched over to the group and you want to minimize the possibility of message non-delivery.

The second task is to transform the permissions that exist for the migrated public folders into group membership. Public folders have a wider range of access types than the owner/member split available for Office 365 Groups, so this is a matter of moving public folder owners, publishing editors, and publishing editors to become group owners while anyone else with access to the group becomes a member.

[PS] C:\> .\addmemberstogroups.ps1 -mappingcsv c:\temp\pfmigrate.csv 
-arepublicfolderslocked $true -backupdir c:\temp
Reading permissions from file, c:\temp\PfPermissions.csv.
Adding members and owners to the group, '[email protected]' based on the permission entries of public f
older, \France\French Public Holidays.
WARNING: Default permission for the public folder '\France\French Public Holidays' is 'Author', but only users with explicit permission entries were added as members to the group [email protected]'. Please change the privacy setting of the group to 'Public' if you need it to be accessible to everyone.
All the users with explicit permissions (except None, FolderVisible, and CreateSubFolders) to access input public folders have been added as Owners/Members to the respective group successfully!
Note: Please use the following cmdlet to add new members to the group if required.'Add-UnifiedGroupLinks -Identity  -LinkType [Owners | Members] -Links '

At this point, you have fully migrated the public folder and you can instruct users to access the new group instead. All the old content exists in the group and any new email will arrive there. After a period when you are sure that you do not want to revert to the old public folder, you can remove the public folder.

Reversing the process uses another Microsoft script (UnlockAndRestorePublicFolderProperties.ps1), but the big downside is that there is no way to transfer content added to the Office 365 Group since the switchover happened back to the original public folder. In other words, some data might be lost.

Some Issues to Think About

Using MRS to move public folders is a logical approach for Microsoft to take. It does not make sense to recreate another migration tool when MRS exists. Although the approach enables Microsoft to give tenants a way to move public folders to Office 365 Groups, some downsides exist.

First, only public folders holding mail and calendar items are candidates for migration. If you use public folders for applications or tasks, you will have to wait (or look for another solution).

Second, this method is not for a complete migration like the way that you can move all public folders from Exchange on-premises to Exchange Online. Instead, this is something that you use to move carefully-selected public folders to Office 365 Groups. That’s sensible because most public folders are not great candidates to become Groups. Transforming old and tired data that no one cares about to a new collaboration platform will not make the data any more valuable.

Third, MRS is not good at logging what happens when it drops “bad items” encountered in public folders. Public folders go back to 1996 and many of the items held in public folders are old (and decrepit). Because of their age, it is likely that old clients created these items at a time when Exchange was more accommodating of items with missing or malformed MAPI properties. These items might be discarded as “bad” when MRS moves data to Office 365 Groups, especially because the default bad item limit for the migration batch is zero, meaning that MRS halts a migration batch if it meets any item that it considers malformed or is unable to process.

To force a batch to complete, you can increase the BadItemLimit parameter when running New-MigrationBatch to submit a new job, (or Set-MigrationBatch for an existing job), but if you do this, MRS will drop the bad items and you will not know what has happened.

Last, once you move public folders to Office 365 Groups, you create the need for people to use either OWA or Outlook 2016 to access the new groups. Older versions of Outlook that can access the public folders do not support Groups.

Manual Processing

Microsoft’s migration process is intensely manual. You must:

  • Find suitable public folders to move to Office 365 Groups and note their properties (identity and public folder mailbox).
  • Create Office 365 Groups as the destination.
  • Build CSV files for migration batches.
  • Submit migration batches to MRS.
  • Check that MRS processes the batches successfully.
  • If necessary, perform some incremental synchronizations.
  • Perform post-processing to lock old public folders and transfer properties to Office 365 Groups.
  • Eventually, remove the source public folders.

Although none of the steps are difficult, this is a process that could quickly become tiresome. All of which means that these tools are most suitable for one-off movement of selected public folders.

The Big Time

Although it is always great to have some free tools, Microsoft’s migration method is probably not the best choice for anyone faced with the task of migrating tens of thousands of public folders. Time is money and automation becomes more important as the number of public folders to process increases. In these circumstances you will consider tools like Quadrotech Public Folder Shuttle, Binary Tree E2E Complete, or BitTitan MigrationWiz. For the cost you pay, look for advantages like streamlined processing, better error handling, extra intelligence, and faster data transfer.

Quadrotech offers a free public folder analysis service that runs a tool against your public folder hierarchy to generate a report giving an insight into the data that lurks inside your public folders. Even if you plan to use Microsoft’s free migration toolset, the analysis report might just save you a lot of work to figure out what folders you should move.

Follow Tony on Twitter @12Knocksinna.

Want to know more about how to manage Office 365? Find what you need to know in “Office 365 for IT Pros”, the most comprehensive eBook covering all aspects of Office 365. Available in PDF and EPUB formats (suitable for iBooks) or for Amazon Kindle