Coming Soon: GET:IT Endpoint Management 1-Day Conference on September 28th at 9:30 AM ET Coming Soon: GET:IT Endpoint Management 1-Day Conference on September 28th at 9:30 AM ET
Office|Office 365|SharePoint Online

When Technology Fails: Woes With SharePoint Online Restore this Library

In April, Microsoft launched the Restore this library feature for SharePoint Online document libraries. I like the functionality very much because users can restore files without administrator intervention or, even worse, the need to ask Microsoft to restore a complete site collection from backup.

The SharePoint Online Restore this Library Feature

Microsoft built the Restore this library functionality using the experience gained with a similar feature called Restore your OneDrive that’s available to OneDrive users with an Office 365 subscription. In both cases, the technology depends on knowing when changes occurred and the existence of previous versions of files that can be used to roll back. The versions generated automatically for Office documents stored in SharePoint Online and OneDrive for Business ensure that a high degree of granularity is available for a restore, but this isn’t always true for other formats. It all depends on how files are saved in SharePoint (for instance, PDFs edited in the Adobe cloud are versioned properly).

Using Restore this Library

Restore this library is available in the settings (cogwheel) menu of document libraries. When invoked, SharePoint queries the Microsoft Graph to retrieve a list of changes made to the library over the last four weeks. The user can then browse the list and decide what changes to undo (Figure 1).

Sponsored Content

Say Goodbye to Traditional PC Lifecycle Management

Traditional IT tools, including Microsoft SCCM, Ghost Solution Suite, and KACE, often require considerable custom configurations by T3 technicians (an expensive and often elusive IT resource) to enable management of a hybrid onsite + remote workforce. In many cases, even with the best resources, organizations are finding that these on-premise tools simply cannot support remote endpoints consistently and reliably due to infrastructure limitations.

SharePoint Online Restore this library
Figure 1: Using the SharePoint Online Restore this library feature (image credit: Tony Redmond)

A set of precanned queries for yesterday, one week ago, and three weeks ago are also available to select sets of changes, or you can use a slider to go back to any point up to 30 days ago.

After selecting the set of changes to undo, click Restore and SharePoint reverses the changes. Depending on how many changes there are to process, this could take some time.

When Things Go Wrong

On July 30, an Office 365 tenant in Asia experienced a problem when a workstation was infected with the Format ransomware. The infection spread to SharePoint and encrypted thousands of documents in a Projects subfolder in a document library. No other folder was affected.

The first action taken by the administrators was to rescue files deleted by the infected user account from the site recycle bin. This recovered about 10,000 files, but over 32,000 files in the folder remained encrypted.

No problem. When a file is encrypted, a new version is generated, so it should be possible to use Restore this library to discard all the changes made by the ransomware. Before doing anything, the administrators synchronized all the files in the document library (both good and bad) to another PC, broke the synchronization link, and deleted all the infected files in the synchronized copy to create a set of “good” files as a backup.

Then they started the Restore this library function to roll back the changes wreaked by the infection. At this point, the infection had happened eight days prior, so they needed to use the slider to find the point in the set of changes just before the infection started. The first issue appeared when the slider became slower and slower as it navigated the list of changes. After many hours of struggling to move the slider (and several restarts), the administrators got it to 9:35AM on August 1.

Even though this point was two days after the infection started, they decided to proceed with the restore, which worked and rolled back all changes before 9:35AM on August 1. Unhappily, as you’d expect, many encrypted files remained in the folder.

Further attempts to roll back changes prior to August 1 also failed and the administrators eventually had to resort to choosing the three-week restore option in the knowledge that this would remove many good files. However, they felt that they had no choice but to get the document library back to full working order in the hope that the lost files could be recovered from other repositories or recreated.

Lessons Learned

This tale of woe revealed several issues surrounding the Restore this library feature.

  1. It struggles with very large document libraries. Large document libraries tend to generate lots of change activities. SharePoint Online looks for those change activities to give administrators the chance to select the restore point. It seems like the slider used in the SharePoint GUI struggles with large number of activities by slowing to a sluggish crawl before stopping.
  2. More precanned restore points (for example, three days, five days, and two weeks) would help by avoiding the need to use the slider. In fact, it would be best to allow the administrator to enter a date and time for SharePoint Online to use the restore point.
  3. Restore this library processes everything for a document library. You can’t set the scope to a specific folder.
  4. Restore operations to roll back tens of thousands of changes take time. Better feedback is needed for administrators so that they know exactly what point a restore has reached. The screen used by SharePoint Online is pretty, but that percentage progress doesn’t increase often large quantities of files are being processed (Figure 2). For audit purposes, an emailed list of operations processed by the restore would also be appreciated.
Restore this library progress
Figure 2: Pretty graphics, but no real information about what the restore is doing (image credit: anonymous customer)

New Technology Takes Time to Bed In

Restore this library is new technology and it will take time to experience the full range of experiences that customers can throw at it. In this instance, the restore worked after hours of effort to get to an August 1 restore point, but the slider couldn’t go back far enough to do the full job. Imperfect feedback to administrators is something Microsoft can quickly fix. I hope the same is true to improve the scalability of the Restore this library feature. After all, SharePoint Online supports up to 30 million files and folders in a document library. Enough said.

BECOME A PETRI MEMBER:

Don't have a login but want to join the conversation? Sign up for a Petri Account

Register
Comments (9)

9 responses to “When Technology Fails: Woes With SharePoint Online Restore this Library”

  1. <p>Couldn't you also put a 3rd party backup solution as well rather than just relying on Microsoft? Examples being Avepoint, Veeam, etc.</p>

    • <blockquote><a href="#16425"><em>In reply to morbrosit:</em></a><em> You could. If you wanted to spend money on such a solution. As this situation proves, sometimes a third-party solution can come to the rescue (if a backup existed), but it would be nice if the OOTB software worked as you'd expect it to.</em></blockquote><p><br></p>

      • <blockquote><em><a href="#16430">In reply to Tony-Redmond:</a></em></blockquote><p>I definitely agree with you on that one. Worst part is actually testing the backups. I deal with smaller businesses and I can tell you no one verifies their backups.</p>

    • <blockquote><a href="#16427"><em>In reply to bluvg:</em></a><em> I believe so. I think a user uploaded an infected document to SharePoint.</em></blockquote><p><br></p>

      • <blockquote><em><a href="#16429">In reply to Tony-Redmond:</a></em></blockquote><p>Wow, thanks for the coverage. I thought Microsoft said that an infected document uploaded to SharePoint would not execute and spread further. And that there were protections to stop infected files from uploading via sync in the first place. Do we have any more insight on how these protections were ineffective in this case? It's a pretty scary single situation and sends worrying vibes for other organisations.</p>

        • <blockquote><em><a href="#16431">In reply to collabguy:</a></em></blockquote><p>I'm not exactly sure how it would spread once uploaded, which is why I'm curious. I don't know what mechanism it could use to spread itself, so that seems unlikely to me. I think the more likely scenario is that they synchronized it locally, the execution affected local files, then they were synchronized back. If there was a way to hijack the browser or Word or something like that to spread, that would be a novel attack as far as I'm aware. But I don't think it's possible for an uploaded document to spread by itself with no client-side interaction. If that's true, that would be deeply disconcerting.</p>

          • <blockquote><a href="#16432"><em>In reply to bluvg:</em></a><em> I believe the victim had synchronized the folder to their PC and the infection happened to the local copies which were then synchronized to SharePoint.</em></blockquote><p><br></p>

Leave a Reply

Tony Redmond has written thousands of articles about Microsoft technology since 1996. He covers Office 365 and associated technologies for Petri.com and is also the lead author for the Office 365 for IT Pros eBook, updated monthly to keep pace with change in the cloud.