Not Able to Connect to Microsoft Azure Arc and Azure Stack HCI Endpoints

Since mid-June 2024, some Microsoft customers have been reporting Azure Arc and Azure Stack HCI connectivity issues from on-premises devices to the cloud service endpoints. Devices were not able to connect to Microsoft Azure services like Azure Arc Resource Bridge and Azure VMware Solution.

The issues have mostly been reported by customers with restrictive firewall or proxy rules, using IP address ranges or restrictive URL filtering. In this article, I’ll look at what you can do to resolve the issues.

Not able to connect to Microsoft Azure services

The issues and connectivity limitations are caused by changes Microsoft made to Azure Arc on June 17, 2024. Microsoft moved the Azure Arc services to Azure Front Door instead of simple load balancing and gateways.

Azure Front Door makes sense for a solution like Azure Arc with global inbound connections and  an endless volume of clients and systems. It allows for better performance, scalability, and reliability over regular load balancers and single endpoints.

What is Azure Front Door?

Azure Front Door is a content delivery network based on 192 and more Microsoft edge locations with Points of Presence in 109 metropolitan areas. It also adds additional features like DDoS prevention and additional Web Application Firewall features. Azure Front Door also makes use of Microsoft’s own threat intelligence, which is used to protect cloud services like Microsoft 365, Xbox, and other services.

As all Microsoft’s services are automated with nearly zero human intervention, Azure Front Door also improves the automation options for Azure Arc and Azure Stack HCI deployment and scalability. Azure Front Door offers a solid Automation SDK and automated certificate rotation, which is required for any Azure Arc connection.

Services impacted

For now, only the following services are impacted by the recent changes, but new services, or additional Azure Arc services, will also make use of Azure Front Door in the future:

• Azure Arc enabled Servers
• Azure Arc enabled Kubernetes
• Azure Stack HCI
• Azure Arc enabled VMware vSphere
• Azure Arc enabled System Center Virtual Machine Manager

The changes are targeted to improve overall performance and security, in addition the goal was to simplify endpoint management and reduce the overall endpoints for Microsoft Azure Arc Services.

The wider network documentation for Azure Arc endpoints was updated recently and should now reflect the required changes for customers. A full JSON list of the of the Microsoft Azure Service Tags and IP address ranges, including Azure Arc, is available from Microsoft, which can be used for firewall and proxy configuration.

According to Microsoft, only customers with the environmental and infrastructure setup below should be impacted by the recent changes. To be honest, most enterprises and larger IT environments should review the changes:

• Customers using firewall and proxy filter for outgoing traffic based on the destination IP address
• Customers using the public endpoints, Internet facing endpoints for Microsoft Azure Arc
• Customers who do not use Azure Private Link, as it bypasses Azure Front Door
• Customers with IP address filters that do not allow traffic to all the address ranges defined in the Azure Arc Infrastructure service tag document.

Microsoft advises you to review and implement the changes as soon as possible but only customers that fulfil all the requirements should be impacted. In any case, you should review and implement the changes, even if you only meet part of the requirements.

Why set network policies via DNS and services?

In general, to avoid being impacted by similar changes in the future, it is best practice to implement firewall and proxy rule sets without using IP address ranges.IP based firewall and proxy policies are things of the past and are not able to keep up with our ever evolving and changing technologies.

Based on those learnings and in cooperation with the major cloud and service providers, firewall solution providers have developed other methodologies to manage policies, routing, and security.

There are also firewall solutions from Palo Alto, CheckPoint, zScaler, Barracuda, and FortiNet that provide a feature often referred to as service-based routing or service-based policies. Those policies often include cloud services from providers like Microsoft, Google, and AWS and are automatically updated by the vendors. Service-based routing makes life easier with dynamic cloud services and hosted solutions.

Managing IP address ranges for cloud services, with always evolving and changing endpoints, makes it nearly impossible to stay up to date with old school IP management and policies. The IP approach also makes it hard to adopt a zero-trust security approach.

Cloud solutions and services require fast and constant change to keep up with security and resiliency demand from customers and other external environmental factors.  

Before you leave

The recent changes were made to unify endpoints and increase performance, reliability, and security of Microsoft Azure Arc services but with some impact on customer environments. As I personally learned over the last weeks, only customers with an out of date and IP-based security policy were impacted.

Those policies are no longer a valid security solution and you should consider a more modern, agile, and reliable security approach for your environment.