HPE Aruba Data Center Reference Architectures

The HPE Aruba ESP (Edge Services Platform) data center reference architectures support high-availability computing racks using redundant top-of-rack (ToR) switches in EVPN-VXLAN or traditional Layer 2 Two-Tier topologies.

Table of contents

• HPE Aruba Data Center Reference Architectures
  • Performance and Redundancy
    • EVPN-VXLAN Spine and Leaf
    • Layer 2 Two-Tier
  • Reference Architecture Components Selection
    • Aruba CX Data Center Switch Overview
    • EVPN-VXLAN Solution Switches
      • Spine Switches
      • Leaf Switches
      • Server Access Switches
      • EVPN-VXLAN Architecture Capacity Planning
        • Bandwidth Calculations
        • Network and Compute Scaling
    • Layer 2 Two-Tier Solution Switches
      • Core Switches
      • Access Switches
    • Out-of-Band Management Switches
    • Aruba Fabric Composer
    • Pensando Policy and Services Manager
    • NetEdit
  • Reference Architecture Physical Layer Planning
    • Cables and Transceivers
    • Port Speed Groups
    • Split Ports
    • Media Access Control Security

Performance and Redundancy

EVPN-VXLAN Spine and Leaf

The Aruba ESP EVPN-VXLAN solution is built on a physical spine-and-leaf topology, which optimizes performance and provides a horizontally scalable design that can accommodate data center growth. The Layer 3 links between spine and leaf switches enable adding spine capacity without disrupting existing network components. A data center can start with as few as two spine switches. When additional capacity is required, up to six spine switches can be deployed in a single fabric for CX 8xxx and CX 10000 leaf switches. High-throughput designs using CX 9300 leaf switches allow for extreme spine scaling of up to 15 CX 9300 spines supporting dual ToR switches. The figure below shows the reference architecture with three spine switches and dual-ToR switches.

Certain application environments do not require high availability at the individual computing host. In this case, a single ToR switch per rack provides a more cost-effective data center network. In this type of implementation, host positioning and non-switch redundancy mechanisms much be considered, because a ToR switch under maintenance impacts connectivity to all computing hosts in the rack. Spine and leaf deployments can comprise a mix of both single and dual ToR racks. The following topology shows a single ToR design with two spine switches.

Layer 2 Two-Tier

The Layer 2 two-tier topology physically resembles a two-spine spine-and-leaf design. Fault tolerance is achieved using multi-chassis Layer 2 link aggregation between the core and access layers in contrast to the Layer 3 links used in a spine-and-leaf solution. The VSX feature enables upgrading and removing individual switches without disrupting other network components. The core size is fixed at two switches, which makes upgrading physical links and aggregation bundles the primary methods of increasing bandwidth capacity between access and core switches.

Reference Architecture Components Selection

The following section provides guidance for hardware selection based on computing host, availability, and bandwidth requirements.

Aruba CX Data Center Switch Overview

The HPE Aruba CX portfolio offers five 1U fixed configuration data center switch models. The CX 8325 model offers high ToR port density. The CX 10000 adds enhanced features along with the same ToR port density. The CX 8100 offers high ToR port density for small and medium data centers consisting of 1 and 10 Gbps connected hosts. The CX 9300 offers the highest throughput capacity and the most flexibility in a 1U form-factor. The CX 8360 model offers a variety of port configurations for small and medium sized topologies.

The Aruba CX 10000 distributed services switch (DSS) supports non-switching features to consider when selecting a ToR switch. In addition to inline stateful firewall enforcement and enhanced traffic visibility, it includes IPsec encryption services, DDoS protection, and NAT.

All models offer the following data center switching capabilities:

• High-speed, fully distributed architecture with line-rate forwarding
• High availability and in-service ToR upgrades with VSX
• Cloud-native and fully programmable modern operating system built on a microservices architecture
• Error-free network configuration with software-defined orchestration tools
• Distributed analytics and guided troubleshooting to provide full visibility and rapid issue resolution
• Hot-swappable and redundant load-sharing fans and power supplies
• Power-to-port and port-to-power cooling options for different data center designs
• Jumbo frame support for 9198 byte frames
• Advanced Layer 2 and Layer 3 features to support an EVPN-VXLAN overlay
• Distributed active gateways to support host mobility.

The HPE Aruba CX 6300 model offers an economical Layer 2 ToR for racks with a high number of 1 Gbps connected hosts in an EVPN-VXLAN solution.

EVPN-VXLAN Solution Switches

The Aruba ESP reference architecture for an EVPN-VXLAN data center comprises switches in two roles: spine and leaf.

Spine Switches

The EVPN-VXLAN architecture is built around spine switches with high-density, high-speed ports. The primary function of spine switches is to provide high-speed routed capacity between tunnel endpoints for VXLAN encapsulated traffic. The primary design considerations when choosing a spine switch are:

• Port density
• Ports speeds
• Maximum routes in BGP RIB

Aruba 1U switches support a range of data center fabric sizes, offering 400 Gbps, 100 Gbps, and 40 Gbps connections to leaf switches.

The CX 9300 offers the greatest spine capacity and flexibility in the 1U switch lineup.

• When using the Aruba CX 9300 as both spine and leaf switches, it supports up to 32 leaf racks in a single ToR switch topology or up to 16 leaf racks in a dual ToR switch topology using 400 Gbps links over single-mode or multimode fiber optic cable.
• When using the Aruba CX 9300 as both spine and leaf switches, it supports extreme horizontal spine scaling. A single ToR topology supports up to 16 spines, and a dual ToR topology supports up to 15 spines, delivering a respective non-oversubscribed fabric capacity of 6.4 Tbps or 6.0 Tbps to each leaf rack.
• The CX 9300 spine can double (64 single ToR/32 dualToR) or quadruple (128 single ToR/64 dual ToR) the number of leaf racks supported over its physical port count when using breakout cabling combined with 100 Gbps connections to CX 8xxx and CX 10000 leaf switches. Single-mode transceivers and fiber are required to support four leaf switches per spine port. Two leaf switches per spine port is supported over multimode fiber or when using AOCs.
• The CX 9300 can support a mix of 400 Gbps links to service leaf racks and 100 Gbps links to standard computing racks to alleviate centralized service congestion points. A CX 9300-based spine also provides an upgrade path from 100 Gbps to 400 Gbps for up to 32 leaf switches by replacing a CX 8xxx leaf with a CX 9300.

The CX 8325 and CX 8360 offer cost-effective, high-speed spine capacity using 40/100 Gbps links.

• The Aruba CX 8325 can support up to 32 leaf racks in a single ToR switch topology or up to 16 computing racks in a dual ToR switch topology.
• The Aruba CX 8360 can support up to 12 leaf racks in a single ToR switch topology or up to six computing racks in a dual ToR switch topology.

The table below summarizes the spine SKUs available and their corresponding leaf rack capacity.

The table below lists the optics supporting CX 9300 spine connectivity over structured cabling:

The table below lists the available AOC breakout cables for connecting CX 9300 spines to CX 8xxx/10000 leaf switches:

Leaf Switches

The Aruba ESP Data Center reference architecture primarily uses five models as 1U data center ToR switches. The CX 8325 series and CX 10000 switches support high-density host racks using 1 GbE / 10 GbE / 25 GbE ports. The CX 8100 offers high ToR port density for small and medium data centers with 1 GbE and 10 GbE host ports. The Aruba CX 8360 series offers a variety of models that support 1GbE / 10 GbE RJ45 ports, and flexible variations of 1 GbE, 10 GbE, 25 GbE, and 50 GbE modular transceiver ports. The CX 9300 in a leaf role is intended to connect 100 GbE high-throughput hosts with a CX 9300 spine layer using 400 Gbps links.

The Aruba CX 10000 distributed services switch (DSS) adds inline firewall features typically provided by service leaf firewalls or VM hypervisors attached to leaf switches or dedicated firewall appliances attached to a services leaf. The Aruba CX 10000 also offers IPsec to encrypt traffic between data centers, DDoS, NAT, and enhanced telemetry services. The Aruba CX 10000 switch should be selected when these features are required by downstream hosts or to meet other data center goals. DSS features are not available on other CX switch models. A mix of DSS and non-DSS ToR leaf switch models can connect to a common spine.

For redundant ToR designs, the high- and medium-density SKUs provide the minimum of four uplink ports required for a two-spine switch topology. For non-redundant ToR design, medium- and low-density SKUs provide the minimum of two uplink ports required for a two-spine switch topology.

The table below summarizes the leaf SKUs available and their corresponding supported designs.

Server Access Switches

The HPE Aruba CX 6300 and CX 8100 switches can be used to extend VLANs from a leaf switch to adjacent racks. This strategy provides an economical solution for connecting a rack with a high number of low-speed connected hosts. CX 6300 models support both built-in and modular power supplies.

EVPN-VXLAN Architecture Capacity Planning

The following section provides capacity planning guidance for the Aruba ESP data center spine-and-leaf reference architecture.

Bandwidth Calculations

A spine-and-leaf network design provides maximum flexibility and throughput for Aruba ESP data center implementation. To achieve the greatest level of performance, a spine-and-leaf topology can be designed for zero oversubscription of bandwidth. This results in a data center network that will never be congested because the bandwidth available to hosts is equal to the bandwidth between leaf-and-spine switches.

A significant advantage of a spine-and-leaf design is the ability to add capacity as needed simply by adding additional spine switches and/or increasing the speed of the uplinks between leaf-and-spine switches. A rack with 40 dual-homed servers with 10 GbE NICs could theoretically generate a total load of 800G of traffic. For that server density configuration, a 1:1 (non-oversubscribed) fabric could be built with four spine switches using 4x100 GbE links on each. In practice, most spine-and-leaf topologies are built with server-to-fabric oversubscription ratios between 2.4:1 and 6:1.

Network and Compute Scaling

The Aruba ESP data center reference architecture provides capacity for most deployments. Distributed gateways and symmetric IRB forwarding optimize fabric capacity consumption. Total fabric capacity can be increased incrementally by adding spines to accommodate growing host compute requirements over time.

The border leaf is typically the node with the highest control plane load since it handles both internal and external connections. Route summarization is a good practice to reduce the redistribution of IP prefixes among domains.

The Aruba ESP data center reference architecture was tested thoroughly in an end-to-end solution environment that incorporates best-practice deployment recommendations, applications, and load profiles that represent production environments.

Refer to the product data sheets on Aruba Campus Core and Aggregation Switches for detailed specifications not included in this guide.

Layer 2 Two-Tier Solution Switches

The Aruba ESP reference architecture for a Layer 2 two-tier data center comprises switches in two roles: core and access.

Core Switches

The Layer 2 two-tier architecture is built around a pair of core switches with high-density, high-speed ports. The core switches provide fast Layer 2 switching between data center computing racks and all Layer 3 functions for the data center, including IP gateway services, routing between subnets, routed connectivity outside of the data center, and multicast services. The primary design considerations when choosing a spine switch are:

• Port density
• Ports speeds
• MAC address table size
• ARP table size
• IPv4/IPv6 route table size

Aruba 1U switch models support a full range of small to large data center core options.

The CX 9300 offers the most capacity and flexibility in the core role in the 1U switch lineup.

• When using the Aruba CX 9300 in both core and access roles, it supports up to 28 computing racks in a single ToR switch topology or up to 14 computing racks in a dual ToR switch topology using 400 Gbps links over single-mode or multimode fiber optic cable.
• A CX 9300 core can double (56 single ToR/28 dualToR) or quadruple (112 single ToR/56 dual ToR) the number of supported access racks when using breakout cabling combined with 100 Gbps connections to CX 8xxx and CX 10000 access switches. Single-mode transceivers and fiber are required to support four leaf switches per spine port. Two leaf switches per spine port are supported over multimode fiber or when using AOCs.

The CX 8325 and CX 8360 offer cost-effective, high-speed core capacity using 40/100 Gbps links.

• The Aruba CX 8325 can support up to 28 access racks in a single ToR switch topology or up to 14 access racks in a dual ToR switch topology.
• The Aruba CX 8360 can support up to 8 access racks in a single ToR switch topology or up to four access racks in a dual ToR switch topology.

The table below summarizes the core switch SKUs available and their corresponding access rack capacity, assuming two core ports are consumed per core switch for redundant external connectivity in addition to the two VSX ISL ports.

The table below lists the optics supporting CX 9300 core connectivity over structured cabling:

The table below lists the available AOC breakout cables for connecting a CX 9300 core to CX 8xxx/10000 access switches:

Access Switches

There are four access switch models in the Aruba ESP Data Center reference architecture. All models are 1U ToR switches. The CX 8325 series and CX 10000 switches support high-density racks using 1 GbE / 10 GbE / 25 GbE host ports. The Aruba CX 8360 series offers a variety of models supporting 1GbE / 10 GbE RJ45 ports, and flexible variations of 1 GbE, 10 GbE, 25 GbE, and 50 GbE modular transceiver ports. The CX 9300 in an access role is intended to connect 100 GbE and 200 GbE high-throughput hosts to a CX 9300 core layer using 400 Gbps links.

The Aruba CX 10000 distributed services switch (DSS) adds inline firewall features typically provided by VM hypervisors attached to access switches or dedicated firewall appliances attached to the core. The Aruba CX 10000 switch should be selected when these features are required by downstream hosts or to meet other data center goals. DSS features are not available on other CX switch models. A mix of DSS and non-DSS switches connected to a common core is supported.

The table below summarizes the access switch SKUs available.

Out-of-Band Management Switches

The Aruba ESP data center reference architecture uses a management LAN built on dedicated switching infrastructure to ensure reliable connectivity to data center infrastructure for automation, orchestration, and traditional management access. The table below lists the recommended switch models.

Aruba Fabric Composer

Aruba Fabric Composer (AFC) is offered as a self-contained ISO or virtual machine OVA and can be installed in both virtual and physical host environments as a single instance or as a high-availability, three-node cluster. AFC can manage EVPN-VXLAN spine-and-leaf fabric and Layer 2 two-tier topologies. AFC is available as an annual per-switch software subscription.

The AFC solutions overview provides additional information.

Pensando Policy and Services Manager

The Pensando Policy and Services Manager (PSM) runs as a virtual machine OVA on a host. PSM requires vCenter for installation. It is deployed as a high-availability, quorum-based cluster of three VMs.

PSM supports Aruba CX 10000 series switches. Management of PSM is integrated into AFC.

PSM can be downloaded from the Aruba Support Portal. Entitlement to PSM is included by adding the following required SKU when purchasing an Aruba CX 10000 switch.

NetEdit

NetEdit runs as a VM OVA on a host. Aruba NetEdit is available from the Aruba Service Portal. Customers must visit the Aruba Airheads Community and create an Airheads account in order to download the NetEdit software.

Ordering information for Aruba NetEdit is provided at the end of this data sheet.

Reference Architecture Physical Layer Planning

The following section provides guidance for planning the physical layer of data center switches.

Cables and Transceivers

Refer to the following documents to ensure that supported cables and transceivers are selected when planning physical connectivity inside the data center:

HPE Server Networking Transceiver and Cable Compatibility Matrix

ArubaOS-Switch and ArubaOS-CX Transceiver Guide

Port Speed Groups

For ToR configurations that require server connectivity at multiple speeds, it is important to note that setting the speed of a port might require adjacent ports to operate at that same speed.

Aruba CX 8325 and Aruba CX 10000 switches have a default speed of 25GbE. Changing the speed to 10GbE will impact groups of 12 ports on the Aruba CX 8325 and groups of four ports on the Aruba CX 10000. Aruba CX 8360 and 9300 switches allow individual ports to operate at different speeds without impacting adjacent ports unless Media Access Control security (MACsec) is in use on an 8360. Ports configured to use MACsec must all be configured to operate at the same speed.

Split Ports

A CX 9300 can use breakout cables to split any single physical port into either two or four 100 Gbps ports. Other platforms can use breakout cables to split a 40 Gb/s or 100 Gb/s port into four lower-speed connections (4x10 Gb/s and 4x25 Gb/s). Refer to the ArubaOS-Switch and ArubaOS-CX Transceiver Guide when selecting supported breakout cables and switch support for split ports.

Media Access Control Security

MACsec is a standard defined in IEEE 802.1AE that extends standard Ethernet to provide frame-level encryption on point-to-point links. This feature is typically used in environments where additional layers of data confidentiality are required or where it is impossible to physically secure the network links between systems. Refer to the table below for details of MACsec support in the Aruba switching portfolio:

---