Diving Deeper into Office 365 Audit Data with Cloud App Security
Strange Entries in the Office 365 Audit Log
After I wrote about using data from the Office 365 audit log to analyze sharing behavior, fellow MVP Vasil Michev pointed out some strange audit entries related to sharing activities that deserved some further investigation.
The entries show up as UserLoggedIn events. Normally, these events include the name of the user or guest who signs into a tenant, but in this case the events have “Unknown” as the user identifier. A quick scan of the event log in my tenant revealed 72 entries of 5,000 log-in events registered as coming from the mysterious Unknown user. The Search-UnifiedAuditLog cmdlet returns a maximum of 5,000 events, so there might have been more had I cared to check further.
Ingestion from Many Sources
The Office 365 audit log ingests records from many different workloads. During this process, the records are normalized to make sure that all events have some common fields. A set of schemas govern the common fields created for all workloads and the fields specific to each workload. This is a sensible approach because the audit information generated for a document-centric event such as the creation of a new document is different to an event like the creation of a new group.
The downside of the normalization process is that some interesting information captured by a workload might not make it into the Office 365 audit log. And that’s just what happened here. This becomes important when we investigate events that might be evidence of suspicious activity, like an attempt to penetrate the tenant.
An Unknown User Connects
If we look at the Office 365 audit UserLoggedIn event for an Unknown user with the Office 365 audit log search (Figure 1), we see some information that helps us understand what happened, but not everything that Azure Active Directory captured.
Knowing that someone tried to do something from a certain IP address is not altogether helpful. Other pieces of information (not shown here) are the record type (15), meaning that it comes from the Secure Token Service (STS) within Azure Active Directory, the user type is 5, meaning an application, and an ActorContextId is present. This is a GUID pointing to an Office 365 tenant. Collectively, we can guess that the audit event comes from an attempt to access the tenant by a user belonging to another tenant. Azure Active Directory recognizes the security tokens created by that tenant, so the login attempt succeeds.
ISV Office 365 Audit Reporting Products
ISV reporting and analysis products available for Office 365 usually help administrators do a better job of figuring out what happens in their tenant through a mixture of stronger query capabilities, analysis tools, and visualization, plus storing more than the default 90 days of audit data kept by Office 365. Scrolling through lines and lines of audit data in the Security and Compliance Center is no one’s idea of a fun afternoon.
Office 365 Cloud App Security
However, ISV products all depend on the Office 365 audit log, and they can’t create missing data. If we want to go deeper into audit data, we must use Office 365 Cloud App Security (CAS), a version of the Microsoft Cloud App Security product built specially for Office 365. If you have Office 365 E5 licenses, you have access to CAS (the feature used to be called Advanced Security Management). Otherwise, you can buy a CAS add-on for Office 365 (roughly $36 per user/year depending on your country), or the full-blown CAS product.
CAS captures more data from the same sources than the Office 365 audit log does. In addition, CAS can ingest audit information from other sources (like firewalls and proxies) and stores the audit data for 180 days instead of 90. But the biggest advantage offered by CAS is the range of analytics available to help administrators understand the flood of audit data that a busy tenant generates.
The CAS record for the same event has more detail than the Office 365 audit log, including the user principal name of the user who tried to login (Figure 2). The record also says that SharePoint Online is the application involved in the event. Given that it obviously exists, you wonder why Microsoft doesn’t send this data to the Office 365 audit log.
Some trial and error shows that events like this appear when users try to access OneDrive for Business or SharePoint Online sites or documents that are not shared with them. And sometimes you see spikes in events caused when users try to use a link to a document multiple times (Figure 3).
Events like this might happen for innocent reasons. The owner of the document might change its sharing settings or remove the document. Someone might share a link that others cannot use. These things happen in the real world. In any case, it’s good to understand when something strange happens and why it might be happening.
The Need for Cloud App Security
Microsoft would like it if every Office 365 tenant had E5 licenses or bought the add-on for Cloud App Security, and if everyone had CAS, they would have lots more data at their fingertips when the need arose to investigate potential security breaches. Whether or not you use the data is another matter. CAS does fill in gaps in knowledge, but you must know how to use the tool.
As evident in Gartner reviews, CAS is valuable. The question therefore comes down to cash. Can you afford to pay an extra $36 annually for every user in the tenant to use Office 365 CAS and would you get value from this investment? Would a cheaper ISV product make Office 365 audit data more accessible and useful? Only you can answer!
Follow Tony on Twitter @12Knocksinna.
Want to know more about how to manage Office 365? Find what you need to know in “Office 365 for IT Pros”, the most comprehensive eBook covering all aspects of Office 365. Available in PDF and EPUB formats (suitable for iBooks) or for Amazon Kindle.