HPE Aruba Networking Data Center Reference Architectures

HPE Aruba Networking data center reference architectures support high-availability computing racks using redundant top-of-rack (ToR) switches in EVPN-VXLAN overlay and traditional topologies.

Table of contents

• HPE Aruba Networking Data Center Reference Architectures
  • EVPN-VXLAN Spine and Leaf
  • Two-Tier
  • Reference Architecture Components Selection
    • HPE Aruba Networking CX Data Center Switch Overview
    • EVPN-VXLAN Solution Switches
      • Spine Switches
      • Leaf Switches
      • Server Access Switches
      • EVPN-VXLAN Architecture Capacity Planning
        • Bandwidth Calculations
        • Network and Compute Scaling
    • Two-Tier Solution Switches
      • Core Switches
      • Access Switches
    • Out-of-Band Management Switches
    • Aruba Fabric Composer
    • Pensando Policy and Services Manager
    • NetEdit
  • Reference Architecture Physical Layer Planning
    • Cables and Transceivers
    • Interface Groups
    • Split Ports
    • Media Access Control Security (MACsec)
  • Scale Validation
    • Spine and Leaf with EVPN-VXLAN Overlay
      • Hardware and Firmware
      • Switch Scale Configuration
      • Multidimensional Dynamic Table Values
    • Two-Tier Architecture
      • Hardware and Firmware
      • Configured Test Scale
      • Multidimensional Dynamic Table Values

EVPN-VXLAN Spine and Leaf

The HPE Aruba Networking EVPN-VXLAN solution is built on a physical spine-and-leaf topology, which optimizes performance and provides a horizontally scalable design that accommodates data center growth. The Layer 3 links between spine and leaf switches enable adding spine capacity without disrupting existing network components. A data center can start with two spine switches, and then add spine switches in the future when additional capacity is required. The figure below shows the reference architecture with two spine switches and dual-ToR switches.

Certain application environments do not require high availability at the individual computing host. In this case, a single ToR switch per rack provides a more cost-effective data center network. In this type of implementation, host positioning and non-switch redundancy mechanisms must be considered, because a ToR switch under maintenance affects connectivity to all computing hosts in the rack. Spine and leaf deployments can include a mix of both single and dual ToR racks.

Two-Tier

The Two-Tier topology physically resembles a spine-and-leaf design with two spines. Fault tolerance is achieved using multi-chassis Layer 2 link aggregation between the core and access layers, in contrast to the Layer 3 links used in a spine-and-leaf solution. The VSX feature enables upgrading and removing individual switches without disrupting other network components. The core size is fixed at two switches, which makes upgrading physical links and aggregation bundles the primary methods of increasing bandwidth capacity between access and core switches.

Reference Architecture Components Selection

The following section provides guidance for hardware selection based on computing host, availability, and bandwidth requirements.

HPE Aruba Networking CX Data Center Switch Overview

The HPE Aruba Networking CX portfolio offers five 1U fixed configuration data center switch models.

• The CX 8325 model offers high ToR port density for 10 and 25 Gbps connected hosts.
• The CX 10000 adds enhanced features along with the same ToR port density.
• The CX 8100 offers high ToR port density for small and medium data centers consisting of 1 and 10 Gbps connected hosts.
• The CX 9300 offers the highest throughput capacity and the most flexibility in a 1U form-factor.
• The CX 9300S offers high throughput ToR capacity for 100 and 200 Gbps connected hosts.
• The CX 8360 model offers a variety of port configurations for small and medium sized topologies.

The CX 10000 distributed services switch (DSS) supports non-switching features to consider when selecting a ToR switch. In addition to inline stateful firewall enforcement and enhanced traffic visibility, it includes IPsec encryption services, DDoS protection, and NAT.

All models offer the following data center switching capabilities:

• High-speed, fully distributed architecture with line-rate forwarding
• High availability and in-service ToR upgrades with VSX
• Cloud-native and fully programmable modern operating system built on a microservices architecture
• Error-free network configuration with software-defined orchestration tools
• Distributed analytics and guided troubleshooting to provide full visibility and rapid issue resolution
• Hot-swappable and redundant load-sharing fans and power supplies
• Front-to-back and back-to-front cooling options for different data center designs
• Jumbo frame support for 9198 byte frames
• Advanced Layer 2 and Layer 3 features to support an EVPN-VXLAN overlay
• Distributed active gateways to support host mobility.

The HPE Aruba CX 6300 model offers an economical Layer 2 ToR for racks with a high number of 1 Gbps connected hosts.

EVPN-VXLAN Solution Switches

The HPE Aruba Networking reference architecture for an EVPN-VXLAN data center includes switches in two roles: spine and leaf.

Spine Switches

The EVPN-VXLAN architecture is built around spine switches with high-density, high-speed ports. The primary function of spine switches is to provide high-speed routed capacity between tunnel endpoints for VXLAN encapsulated traffic. When choosing a spine switch, primary design considerations are:

• Port density
• Ports speeds
• Maximum routes in BGP RIB.

HPE Aruba Networking 1U switches support a range of data center fabric sizes, offering 400 Gbps, 100 Gbps, and 40 Gbps connections to leaf switches.

The CX 9300-32D offers the greatest spine capacity and flexibility in the 1U switch lineup.

• When using a CX 9300S-32C8D leaf switch, a maximum of eight CX 9300-32D spines can connect up to 32 leaf racks in a single ToR switch topology or 16 leaf racks in a dual ToR switch topology using 400 Gbps links. This configuration targets high-speed compute and AI applications using 100 and 200 Gbps connected hosts.
• When using the CX 9300-32D as both spine and leaf switches, it supports up to 32 leaf racks in a single ToR switch topology or up to 16 leaf racks in a dual ToR switch topology using 400 Gbps links over single-mode or multimode fiber optic cable. This configuration supports 400/200/100-Gbps leaf connected compute and AI applications.
• Using the CX 9300-32D as both spine and leaf switches supports extreme horizontal spine scaling. A single ToR topology supports up to 16 spines, and a dual ToR topology supports up to 15 spines, delivering a respective non-oversubscribed fabric capacity of 6.4 Tbps or 6.0 Tbps to each leaf rack.
• The CX 9300-32D spine can double (64 single ToR/32 dualToR) or quadruple (128 single ToR/64 dual ToR) the number of leaf racks supported over its physical port count when using breakout cabling combined with 100 Gbps connections to CX 8xxx and CX 10000 leaf switches. Single-mode transceivers and fiber are required to support four leaf switches per spine port. Two leaf switches per spine port are supported over multimode fiber or when using AOCs.
• The CX 9300-32D spine can support a mix of 400 Gbps links to service leaf racks and 100 Gbps links to standard computing racks to alleviate centralized service congestion points. A CX 9300-32D based spine also provides an upgrade path from 100 Gbps to 400 Gbps for up to 32 leaf switches by replacing a CX 8xxx leaf with a CX 9300 or 9300S switches.

The CX 8325 and CX 8360 offer cost-effective, high-speed spine capacity using 40/100 Gbps links.

• The CX 8325 can support up to 32 leaf racks in a single ToR switch topology or up to 16 computing racks in a dual ToR switch topology.
• The CX 8360 can support up to 12 leaf racks in a single ToR switch topology or up to six computing racks in a dual ToR switch topology.

The table below summarizes the spine SKUs available and their corresponding leaf rack capacity.

The table below lists the optics that support CX 9300 spine connectivity over structured cabling:

The table below lists the available AOC breakout cables for connecting CX 9300 spines to CX 8xxx/10000 leaf switches:

Leaf Switches

The HPE Aruba Networking data center reference architecture primarily uses six models as 1U data center ToR switches.

• The CX 8325 series and CX 10000 switches support high-density host racks using 1 GbE / 10 GbE / 25 GbE ports.
• The CX 9300-32D in a leaf role is intended to connect 100 GbE, 200 GbE, and 400 GbE high-throughput hosts to a CX 9300-32D spine using 400 Gbps links.
• The CX 9300S supports 100 GbE and 200 GbE high-throughput hosts to a CX 9300-32D spine. It also can be optimized for 25 GbE connected hosts. Additionally, the 9300S provides secure border leaf options using high-speed MACsec interfaces.
• The CX 8100 offers high ToR port density for small and medium data centers with 1 GbE and 10 GbE host ports.
• The CX 8360 series offers a variety of models that support 1GbE / 10 GbE RJ45 ports, and flexible variations of 1 GbE, 10 GbE, 25 GbE, and 50 GbE modular transceiver ports.

The CX 10000 distributed services switch (DSS) adds inline firewall features typically provided by dedicated firewall appliances attached to a services leaf or VM hypervisors attached to leaf switches. The CX 10000 also offers IPsec encryption between data centers, NAT, DDoS, and enhanced telemetry services. The CX 10000 switch should be selected when these features are required by downstream hosts or to meet other data center goals. DSS features are not available on other CX switch models. A mix of DSS and non-DSS ToR leaf switch models can connect to a common spine.

Redundant ToR designs require at least four uplink ports for a two-spine switch topology. A minimum of two ports connect to spine switches and two additional ports are members of a high-speed VSX ISL. The CX 9300S is an exception that can connect all eight 400 Gbps uplink ports to spine switches, when using 200 Gbps ports for the VSX ISL. A non-redundant ToR design requires at least two high-speed uplink ports for a two-spine topology.

The table below summarizes the leaf SKUs available and their corresponding supported designs.

Note: Three CX 9300S-32C8D bundles are Trade Agreement Act (TAA) compliant with the same capabilities listed in the table above.
S0F81A (front-to-back air flow)
S0F83A (back-to-front air flow)
S0F87A (back-to-front air flow and DC power supplies)

Server Access Switches

CX 6300 and CX 8100 switches can be used to extend VLANs from a leaf switch to adjacent racks. This strategy provides an economical solution for connecting a rack with a high number of low-speed connected hosts. CX 6300 server access switches are typically connected to CX 8325 or 10000 leaf switches. CX 6300 models support both built-in and modular power supplies

EVPN-VXLAN Architecture Capacity Planning

The following section provides capacity planning guidance for the HPE Aruba Networking data center spine-and-leaf reference architecture.

Bandwidth Calculations

A spine-and-leaf network design provides maximum flexibility and throughput in a data center implementation. To achieve the greatest level of performance, a spine-and-leaf topology can be designed for zero oversubscription of bandwidth. This results in a data center network that will never be congested because the bandwidth available to hosts is equal to the bandwidth between leaf-and-spine switches.

A significant advantage of a spine-and-leaf design is the ability to add capacity as needed simply by adding additional spine switches and/or increasing the speed of the uplinks between leaf-and-spine switches. A rack with 40 dual-homed servers with 10 GbE NICs could theoretically generate a total load of 800G of traffic. For that server density configuration, a 1:1 (non-oversubscribed) fabric could be built with four spine switches using 4x100 GbE links on each. In practice, most spine-and-leaf topologies are built with server-to-fabric oversubscription ratios between 2:1 and 6:1.

Network and Compute Scaling

The HPE Aruba Networking data center reference architecture provides capacity for most deployments. Distributed gateways and symmetric IRB forwarding optimize fabric capacity consumption. Total fabric capacity can be increased incrementally by adding spines to accommodate growing host compute requirements over time. The CX 10000 DSS switch enables policy enforcement without changing spine-and-leaf traffic optimizations.

The border leaf is typically the node with the highest control plane load since it handles both internal and external connections. Route summarization is a good practice to reduce the redistribution of IP prefixes among domains. Both CX 10000 and 9300S switches support secure border leaf capabilities to external networks and between fabrics.

The HPE Aruba Networking data center reference architecture was tested thoroughly in an end-to-end solution environment that incorporates best-practice deployment recommendations, applications, and load profiles that represent production environments.

Refer to the product data sheets on HPE Aruba Networking Campus Core and Aggregation Switches for detailed specifications not included in this guide.

Two-Tier Solution Switches

The HPE Aruba Networking reference architecture for a Two-Tier data center includes switches in two roles: core and access.

Core Switches

The Two-Tier architecture is built around a pair of core switches with high-density, high-speed ports. The core switches provide fast Layer 2 switching between data center computing racks and all Layer 3 functions for the data center, including IP gateway services, routing between subnets, routed connectivity outside of the data center, and multicast services. The primary design considerations when choosing a spine switch are:

• Port density
• Ports speeds
• MAC address table size
• ARP table size
• IPv4/IPv6 route table size

HPE Aruba Networking 1U switch models support a full range of small to large data center core options.

The CX 9300-32D offers the most capacity and flexibility in the core role of the 1U switch lineup.

• When using the CX 9300-32D in both core and access roles, it supports up to 28 computing racks in a single ToR switch topology or up to 14 computing racks in a dual ToR switch topology using 400 Gbps links over single-mode or multimode fiber optic cable.
• A CX 9300-32D core can double (56 single ToR/28 dualToR) or quadruple (112 single ToR/56 dual ToR) the number of supported access racks when using breakout cabling combined with 100 Gbps connections to CX 8xxx and CX 10000 access switches. Single-mode transceivers and fiber are required to support four leaf switches per spine port. Two leaf switches per spine port are supported over multimode fiber or when using AOCs.

CX 8325 and CX 8360 offer cost-effective, high-speed core capacity using 40/100 Gbps links.

• The CX 8325 can support up to 28 access racks in a single ToR switch topology or up to 14 access racks in a dual ToR switch topology.
• The CX 8360 can support up to 8 access racks in a single ToR switch topology or up to four access racks in a dual ToR switch topology.

The table below summarizes the core switch SKUs available and their corresponding access rack capacity, assuming two core ports are consumed per core switch for redundant external connectivity in addition to the two VSX ISL ports.

The table below lists the optics that support CX 9300 core connectivity over structured cabling:

The table below lists the available AOC breakout cables for connecting a CX 9300-32D core to CX 8xxx/10000 access switches:

Access Switches

The HPE Aruba Networking data center reference architecture includes six access switch models. All models are 1U ToR switches.

• The CX 8325 series and CX 10000 switches support high-density racks using 1 GbE / 10 GbE / 25 GbE host ports.
• The CX 8360 series offers a variety of models supporting 1GbE / 10 GbE RJ45 ports, and flexible variations of 1 GbE, 10 GbE, 25 GbE, and 50 GbE modular transceiver ports.
• The CX 8100 series offers a cost effective model for 1 GbE / 10 GbE connected hosts.
• The CX 9300-32D in an access role is intended to connect 100 GbE and 200 GbE high-throughput hosts to a CX 9300-32D core layer using 400 Gbps links.
• The CX 9300S supports 100 GbE and 200 GbE high-throughput hosts to a CX 9300-32D core, but it also can be optimized for 25 GbE connected hosts.

The CX 10000 distributed services switch (DSS) adds inline firewall features typically provided by dedicated firewall appliances attached to the core or VM hypervisors attached to access switches. The CX 10000 switch should be selected when these features are required by downstream hosts, or to meet other data center goals. DSS features are not available on other CX switch models. A mix of DSS and non-DSS switches connected to a common core is supported.

The table below summarizes the access switch SKUs available.

Note: Three CX 9300S-32C8D bundles are Trade Agreement Act (TAA) compliant with the same capabilities listed in the above table.
S0F81A (front-to-back air flow)
S0F83A (back-to-front air flow)
S0F87A (back-to-front air flow and DC power supplies)

Out-of-Band Management Switches

The HPE Aruba Networking data center reference architecture uses a management LAN built on dedicated switching infrastructure to ensure reliable connectivity to data center infrastructure for automation, orchestration, and traditional management access. The table below lists the recommended switch models.

Aruba Fabric Composer

HPE Aruba Networking’s Aruba Fabric Composer (AFC) is offered as a self-contained ISO or virtual machine OVA and can be installed in both virtual and physical host environments as a single instance or as a high-availability, three-node cluster. AFC can manage EVPN-VXLAN spine-and-leaf fabric and Two-Tier topologies. AFC is available as an annual per-switch software subscription.

The AFC solutions overview provides additional information.

Pensando Policy and Services Manager

The Pensando Policy and Services Manager (PSM) runs as a virtual machine OVA on a host. PSM requires vCenter for installation. It is deployed as a high-availability, quorum-based cluster of three VMs.

PSM supports CX 10000 series switches. Management of PSM is integrated into AFC.

PSM can be downloaded from the HPE Networking Support Portal. Entitlement to PSM is included by adding the following required SKU when purchasing a CX 10000 switch.

NetEdit

HPE Aruba Networking’s NetEdit software runs as a VM OVA on a host. NetEdit is available from the HPE Networking Support Portal.

Ordering information for NetEdit is provided at the end of this data sheet.

Reference Architecture Physical Layer Planning

The following section provides guidance for planning the physical layer of data center switches.

Cables and Transceivers

Refer to the following documents to ensure that supported cables and transceivers are selected when planning physical connectivity inside the data center:

HPE Server Networking Transceiver and Cable Compatibility Matrix

HPE Aruba Networking ArubaOS-Switch and ArubaOS-CX Transceiver Guide

Interface Groups

For ToR configurations that require server connectivity at multiple speeds, it is important to note that setting the speed of a port might require adjacent ports to operate at that same speed.

CX 8325 and CX 10000 host facing ports have a default speed of 25GbE. Changing the speed to 10GbE will impact groups of 12 ports on the CX 8325 and groups of four ports on the CX 10000. Some CX 8360 switches use interface groups and others support individual port speed settings without impacting adjacent ports. CX 9300-32D switches allow individual ports to operate at different speeds. The CX 9300S 400 Gbps ports support individual speed settings, while the remaining 100G and 200G ports can be assigned two speed modes in interface groups of four.

The following diagram illustrates 9300S port groups:

Split Ports

Split ports enable an individual high-speed interface to establish multiple lower speed links using active optical breakout cables or optical transceivers.

The CX 9300-32D can split an individual 400 Gbps port into 4 x 100 Gbps, 2 x 100 Gbps or 2 x 200 Gbps links.

The CX 9300S supports two split interface profile modes that optimize split port capabilities for 100 Gbps or 25 Gbps operational requirements. The default profile (profile 1) optimizes 100 Gbps operation. In this mode, the eight 400 Gbps ports can be split into 4 x 100 Gbps, 2 x 100 Gbps, or 2 x 200 Gbps links, and eight 200 Gbps ports can be split into 2 x 100 Gbps links.

The following diagram illustrates split port operation on the CX 9300s using split interface profile 1 with interface-groups 3 and 6 set to 200 Gbps operation:

Note: Currently shipping HPE Aruba Networking 200G to 2 x 100G AOC split cables support only Q-DD interfaces. These are supported in the CX 9300S 400G interfaces, but not in the 200G QSFP28/56 interfaces. Future cabling options will support 200G to 2 x 100G split operation on CX9300S 200G ports.

When a CX 9300S 200G port group is set to 40 Gbps operation in split interface profile 1, the ports are capable of 2 x 10 Gbps split. Split interface profile 2 is recommended when optimizing the 9300S for 25 Gbps or 10 Gbps operation.

The CX 9300S split interface profile 2 optimizes 25 Gbps operation, where six 200 / 100 / 40 Gbps ports can be split into four 25 Gbps links. The number of 400 Gbps ports supporting split operation is reduced to four, when using split interface profile 2.

The following diagram illustrates split port operation on the CX 9300s using split interface profile 2 with interface-groups 4 and 5 set to 200 Gbps operation:

Note: When the CX 9300S 200G ports in interface-group 4 or 5 are set to 40 Gbps operation (depicted in green in the diagram above), ports within that group only support 4 x 10 Gbps or 2 x 10 Gbps split operation.

The CX 9300S requires a reboot to switch between split interface port profiles.

The QSA28 network adapter (845970-B21) supports 25 Gbps and 10 Gbps optics in QSFP28 ports and 10 Gbps optics in QSFP+ ports. The QSA28 can be used with the CX 9300S to enable lower port speed operation on ports that do not support split operation or have split operation disabled due to the port profile selection.

Most other platforms can split a 40/100 Gbps port into four lower-speed connections (4x10 Gb/s or 4x25 Gb/s).

Refer to the HPE Aruba Networking ArubaOS-Switch and ArubaOS-CX Transceiver Guide when selecting supported breakout cables, adapters, and transceivers.

Media Access Control Security (MACsec)

MACsec is a standard defined in IEEE 802.1AE that extends standard Ethernet to provide frame-level encryption on point-to-point links. This feature is typically used in environments where additional layers of data confidentiality are required or where it is impossible to physically secure the network links between systems.

MACsec can be used to encrypt communication between switches within a data center, between two physically separate data center locations over a data center interconnect (DCI), or between switches and attached hosts.

The table below details MACsec support in the HPE Aruba Networking switch portfolio:

Scale Validation

HPE Aruba Networking’s test lab performs multidimensional scale validation of data center architectures. A comprehensive, solution-level test case for each architecture is implemented using recommended best practices.

The validated scale values below represent specific test cases and are not intended to indicate the maximum achievable scale for a specific architecture. The test case is intended to provide a sample reference of achievable scale across multiple switch resources, in contrast to unidimensional data sheet values that specify maximum values for a feature in isolation. Each customer environment is unique and may require optimizing resources in a different manner.

Topology architectures are connected to a high performance testing platform that generates large-scale client traffic.

Spine and Leaf with EVPN-VXLAN Overlay

The spine-and-leaf/EVPN-VXLAN data center was validated using CX 8325-32C spine switches and CX 10000-48Y6C leaf switches.

The following diagram illustrates the HPE Aruba Networking test lab’s topology (simulated racks not depicted).

The underlay uses IPv4 routed-only ports between spine and leaf switches and a single OSPF area to share loopback and VTEP reachability. The testing environment consists of three physical racks with redundant leaf switches and 13 simulated racks to support a total of 16 overlay VTEPs. The testing platform simulates non-redundant leaf switches, resulting in a lower number of underlay OSPF adjacencies than when using a purely physical setup, which does not affect EVPN-VXLAN overlay scale testing parameters.

Layer 2 and Layer 3 overlay scalability were tested. Sixty-four VRFs were defined, each with five VLANs [three standard VLANs, an isolated private VLAN (PVLAN), and a primary PVLAN]. Dual-stacked VLAN SVIs were defined on standard VLANS and primary PVLANs. HPE Aruba Networking’s Active Gateway feature provided a dual-stacked, distributed Layer 3 gateway on each leaf switch. Both ARP and ND suppression were enabled.

Two VLAN SVIs per VRF were defined on each border leaf to connect to a pair of external firewalls. Bidirectional Forwarding Detection (BFD) was enabled on external BGP peerings for fast routing failure detection.

Hardware and Firmware

The following switch models and firmware versions were tested in the designated roles:

Note: The internal switch architecture of the 10000-48Y6C is based on the 8325-48Y8C. Validated values for the 10000-48Y6C also apply to the 8325-48Y8C.

Switch Scale Configuration

The following per-switch configuration values established Layer 3 and Layer 2 scale for the testing environment.

Multidimensional Dynamic Table Values

The following table values were populated during the solution test.

Note: *The AOS-CX “show ip route” and “show ipv6 route” command outputs include /32 and /128 EVPN host routes, which do not consume a route table entry. In the table above, the first value represents the number of displayed routes when using a show route command. The second number represents the number of actual route entries consumed in the route table during the test.

Two-Tier Architecture

The Two-Tier data center was validated using CX 8360-12C core switches and two types of server access switches: 8360-48XT4Cv2 and 8100-40XT8XF4C. A total of four server access racks were connected to the VSX-redundant core.

The following diagram illustrates the HPE Aruba Networking test lab’s topology (simulated access racks not depicted).

Four VRFs were defined, with 128 VLANs assigned per VRF (127 server facing VLANs and one transit VLAN). HPE Aruba Networking’s Active Gateway feature provided host gateway redundancy on the core switches.

OSPFv2 and OSPFv3 were used for IPv4 and IPv6 routing on a transit VLAN between core switches and external firewalls. BFD was enabled for fast OSPF neighbor failure detection.

PIM-SM, IGMP, and MLD were enabled on core routed interfaces. IGMP and MLD snooping were enabled on server access switches.

MSTP was enabled with a single instance.

Hardware and Firmware

The following switch models and firmware versions were tested in the designated roles:

Configured Test Scale

The following per-switch configuration values established Layer 3 and Layer 2 scale for the testing environment.

Multidimensional Dynamic Table Values

The following table values were populated during the solution test.

---