Choosing the circuit types under which to run an SD-WAN deployment is a critical design step. Each circuit type has different strengths and weaknesses that can affect SD-WAN operation.
This section describes design decisions and best practices to help operators make the best circuit choice for Aruba’s two SD-WAN solutions.
Table of contents
- Aruba EdgeConnect SD-WAN & SD-Branch WAN Transport Design
- Zero Touch Provisioning
- Hub Designs
- Branch Designs
- Internet Egress
- Overlay Components
- Cloud Integration
Commodity Internet is the most common transport type for SD-WAN deployments due to its low cost and high bandwidth.
The main benefit of choosing commodity Internet is full IP reachability to any other location using the Internet as a transport type. The downside is the limit or lack of guaranteed performance from the transports, especially when traffic must cross numerous backbone Internet providers.
MPLS circuits provide private connectivity between locations across a service provider network and ensure more predictable SLA-backed performance. Although MPLS can serve as a viable transport for SD-WAN, MPLS is generally the most expensive form of transport. In most SD-WAN solutions, MPLS is phased out of the network and replaced with commodity Internet, relying on the SD-WAN technology to provide the SLA.
If the MPLS network is used to host services such as provider SIP circuits, firewalls, or cloud ingress points, running SD-WAN on the circuit may present routing challenges for accessing the services. Carefully consider the routing design to accommodate service requirements and access flows. Route-leaking between the underlay and overlay or hairpinning traffic through a hub may be required.
As SD-WAN is adopted, it is typical to phase out MPLS circuits in favor of the higher speeds and lower costs Internet circuits can offer.
Mobile carrier data is traditionally used as a path of last resort at many locations to provide last-mile resiliency if the physical circuits at a location are unavailable.
Because mobile circuits have lower throughput and data caps, they generally are not recommended for use as primary circuits. However, some modern 5G providers now offer plans with no data caps and high speeds that may be appropriate to consider for use as a primary circuit.
Private Layer 2 circuit types include VPLS, Metro-Ethernet, and Pseudo Wire. These circuit types are commonly found in DCI environments or in interconnections in large campus environments. Though they are less common for large WAN transports they are still often deployed.
Both Aruba EdgeConnect SD-Branch and EdgeConnect SD-WAN support layer 2 WAN transports, but these designs require additional consideration. Consult with your local Aruba Solution Architect if deploying SD-WAN over this transport type.
Low Orbit WAN circuits, such as SpaceX Starlink, provide a new RF-based WAN solution. They are generally high-speed, low-latency broadband Internet circuits used in remote or rural locations.
For network operators in these areas, it can be a challenge obtaining other WAN circuit types with adequate bandwidth and SLA performance required for their business needs. The emerging low orbit WAN circuit is a worthy consideration.
Many SD-WAN implementations aim to remove private circuits from their environment completely. Commodity Internet circuits provide large amounts of bandwidth cost effectively and SD-WAN technology now enables networks to steer traffic to the correct circuit, improving the performance of commodity Internet to better meet business requirements.
A combination of multiple high speed Internet circuits with a 4G/5G backup is the most desirable end state for circuit pairings. It is common to see dual, or even triple, Internet circuits at the data center, with dual Internet or single Internet with mobile backup at the branches.
Businesses with stricter transport SLAs or complex established MPLS networks may prefer keeping provider MPLS in their infastructure. In this case, paring MPLS, commodity Internet, and mobile backup makes sense. The business can choose to reduce the bandwidth of their MPLS circuits gradually as more traffic is shifted to overlays built on commodity Internet.
When pairing WAN circuits for a site, the need to provide last-mile resiliency cannot be overlooked. Avoid the common practice of bundling circuits together for the last mile so ingress occurs at the same physical location. The common ingress point can be susceptible to a layer 1 failure event, or backhoe event, taking out all WAN circuits at the location.
Some customers use 5G circuits as the primary WAN transport, along with commodity Internet. Service providers now offer unlimited 5G plans that work well with a wired commodity Internet ciruit to provide last mile resiliency.
SD-WAN solutions enable the deployment of devices “out of the box” with automated connection to a single cloud platform for management. This process is known as Zero Touch Provisioning (ZTP).
To benefit from ZTP, branch networks must be standardized so templates can be applied for consistent, universal configuration across multiple locations.
ZTP can be beneficial even if some sites have unique topologies that cannot be configured remotely. However, the true value lies in the ability to use templates to deploy all topologies universally.
SD-WAN fabrics generally have hub locations, typically data centers or other sites where applications reside. The hub location is responsible for:
- WAN site reachability to applications residing in the hub
- Aggregation of IPsec tunnels, for hub-and-spoke topologies
- Serving as a regional routing hub for large deployments
- Centralized security services, as needed
- Cloud onramp instead of brick-and-mortar hubs.
This section reviews key considerations when determining the set-up of hub locations for SD-WAN topologies.
When installing SD-WAN appliances in the data center, two deployment methods are commonly used:
- Inline deployment
- Out-of-path deployment.
For inline deployments, the SD-WAN router serves as the edge router, terminating WAN circuits directly. This is the preferred deployment method with two gateways acting as an active/standby pair. This method works well in simple environments with one or two SD-WAN appliances that do not require horizontal scale. It also works well in greenfield environments when support for migrating a legacy WAN is not required.
The diagram below illustrates an in-path deployment.
For out-of-path deployments, the SD-WAN appliance does not physically terminate the WAN circuits. Instead, traffic is redirected to the SD-WAN appliances, generally using routing protocols or a redirect protocol. This method is not the recommended starting design, and should be used only if either of two specific requirements are needed:
The edge router must be kept in place and cannot be replaced with a gateway. This may be desirable when the edge router is doing other complex functions, such as third party IPsec tunnel termination.
More WAN bandwidth is needed than a simple two-box, in-path deployment can provide. The gateways can scale past two, providing horizontal scale, if deployed out-of-path.
The diagram below illustrates an out-of-path deployment.
A branch is defined as a small remote location that does not host applications. Restaurant chains and retail outlets are examples of businesses with hundreds of branches spread over large geographical areas. For remote locations with larger footprints, refer to the Aruba small campus reference architecture.
This section presents key considerations for designing branches, with a focus on SD-WAN appliance integration.
SD-WAN appliances at branch locations are almost always deployed as edge routers, in-path, replacing legacy routers and firewalls. They physically terminate the WAN circuits and handle all route peering with the providers. These branch routers may be deployed as the default gateway for the VLANs at the branch or they may participate in routing with the branch LAN.
LAN-side design for switching and wireless infrastructure is covered in the Campus Design section of the VSG. For specific branch designs, refer to the reference architectures for EdgeConnect SD-WAN and EdgeConnect SD-Branch.
As more business resources are migrated to the Internet, designing Internet egress becomes more important. Many corporate resources are migrating to SaaS solutions, such Office 365, accessed over the Internet. The change in application location requires network architects to rework traditional Internet egress designs.
The conversion of all traffic to SD-WAN enables more flexible and direct Internet egress designs, without compromising security. With application recognition and SD-LAN’s enhanced policy application, different Internet egress policies can be applied to different traffic.
Trusted business applications can be sent directly to the Internet or sent through on-box firewall features of the SD-WAN appliance. Traffic also can be sent to a cloud firewall service for more detailed inspection and centralized firewalling. Finally, select traffic can still be backhauled to a data center for more specific in-house inspection as needed.
Regardless of the location in the network, it is critical for clients to use the geographical-appropriate DNS. Many Internet resources have geolocated onramps for their services. For example, Microsoft has numerous onramps to the core software services based on geographic locations. If you are located in Seattle and request Office 365 DNS entries, but your DNS server is in New York, you receive IP addresses for the New York on-ramp, which creates suboptimal traffic flows.
The specific interworkings and features of an overlay are highly dependent on the SD-WAN solution. This section outlines how overlays work generically. More solution-specific detail is provided in the EdgeConnect SD-WAN and EdgeConnect SD-Branch sections.
Overlays, which are typically IPsec tunnels, normalize all transports within a WAN by building logical connections between the SD-WAN appliances. After the tunnels are established, the SD-WAN solution begins to monitor the tunnels to obtain performance metrics and transport health.
The goal of an overlay is to ensure application stability and reliability, using the following mechanisms:
Traffic Identification - SD-WAN appliances are configured to differentiate traffic by type. The identification can be simple matching of DSCP markings or use of L3/L4 ACLs. More advanced application recognition engines can be employed, based on the capability of the SD-WAN solution.
Traffic Policy - The operator creates policy for different types or groups of traffic and assigns a specific SLA. The SD-WAN solution then steers the traffic over specific transports to ensure the best path is selected based on the assigned policy and SLA.
Combat WAN Transport Instability - SD-WAN appliances can apply is forward error correction (FEC) to traffic, which enables the network to recover from packet loss caused by a variety of network layer conditions, such as queue overflows, constrained bandwidth links, or long equipment delays in the carrier network. Packet-level FEC adds a parity packet for every ‘N’ number of packets in each block, with ‘N’ being 2, 4, or 8. The parity packet enables the far-end SD-WAN appliance to reconstitute lost packets before the packets are delivered to TCP, UDP, or other transport layers. This avoids the loss of information in the case of UDP traffic, as well as the delays associated with multiple round-trip retransmissions or the performance impacts of congestion avoidance mechanisms in the case of TCP.
Topology Choice - Topology dictates where IPsec tunnels are built and can affect the scalability of the SD-WAN solution. There are two main topology types: hub-and-spoke and full mesh. In hub-and-spoke, IPsec tunnels are built only between the spokes and the hubs. This minimizes the number of IPsec tunnels, allowing for higher scale, but also creates suboptimal traffic flows between the spokes, requiring them to transit the hub. With a full mesh topology, the spokes can all communicate directly, but this creates a very high number of IPsec tunnels which limits the scalability of the topology.
The best practice is to use hub-and-spoke topologies for overlays that do not require optimized spoke-to-spoke routing. The main traffic type that requires a full mesh topology is ‘real-time’ to allow latency-sensitive applications to route directly between spokes.
SD-WAN has changed the way many architects connect their networks to cloud providers such as AWS, Azure, or GPC. EdgeConnect SD-WAN and EdgeConnect SD-Branch support numerous deployment models to connect to cloud resources. EdgeConnect deployed directly into the IaaS provider is most common and generally recommended, which is further discussed below. For more information on how to integrate SD-WAN with a multi-cloud solution such as Megaport or Alkira, contact your Aruba account team.
Aruba SD-WAN solutions help to automate much of the deployment and reduce complexity, as described in the deployment guides. An outline of the underlying solution architecture is illustrated below.
To integrate with Microsoft Azure Aruba recommends customers deploy a vWAN solution to interconnect their workload Virtual Networks (VNET). An SD-WAN VNET is then used to provide connectivity between the Azure VNETs and the SD-WAN fabric. More information on this method can be found in Microsoft’s documentation.
Within Azure a pair of redundant EdgeConnect appliances are deployed via either Aruba Orchestrator for SD-WAN or Aruba Central for SD-Branch. On the LAN side EdgeConnect appliances will peer BGP to the VNET hub. SD-WAN fabric routes will be advertised to the hub, and workload VNET routes will be learned. On the WAN side the interfaces will connect to the Azure Internet Gateway to obtain internet reachability. This will allow the devices to register with Orchestrator or Central and establish SD-WAN IPSEC tunnels and routing adjacencies.
Note: While the EdgeConnect can be deployed directly into vWAN, it is not the reccommended approach due to current caveats, including lack of support for Express Route connections.
To integrate with Amazon Web Services (AWS) Aruba recommends customers deploy a transit gateway solution to interconnect their workload Virtual Private Cloud (VPC). An SD-WAN VPC is then used to provide connectivity between the AWS VPCs and the SD-WAN fabric. More information on this method can be found in Amazon’s documentation.
Within AWS a pair of redundant EdgeConnect appliances are deployed via either Aruba Orchestrator for SD-WAN or Aruba Central for SD-Branch. On the LAN side EdgeConnect appliances will peer BGP to the Transit Gateway (TGW). SD-WAN fabric routes will be advertised to the TGW, and workload VPC routes will be learned. On the WAN side the interfaces will connect to the AWS Internet Gateway to obtain internet reachability. This will allow the devices to register with Orchestrator or Central and establish SD-WAN IPSEC tunnels and routing adjacencies.