Secure an Aruba EVPN Fabric

AFC orchestrated security policy is applied to east-west traffic using CX 10000 switches and AMD Pensando’s Policy and Services Manager in this example fabric. ACL policy also can be applied to east-west and north-south traffic.

Table of contents

• Secure an Aruba EVPN Fabric
  • Overview
  • Configure PSM Integration with Aruba Fabric Composer
  • Configure Macro Firewall Policy
    • Configure PSM Networks
    • Review Dynamic Endpoint Groups
    • Configure Static Endpoint Groups
    • Configure Firewall Rules
    • Configure Ingress Policy
  • VM Microsegmentation
    • Configure Policy Applications
    • Configure Microsegmentation Rules
    • ESXi Microsegmentation Using AFC
    • Add the Primary PVLAN to the EVPN
    • Externally Advertise Microsegmented Networks
  • Verify Policy in PSM
    • Verify Networks

Overview

This chapter describes enforcing policy in the Aruba ESP data center network using the Aruba CX 10000 switch platform’s stateful firewall. Policy is implemented using the AMD Pensando Policy and Services Manager (PSM) orchestrated by Aruba Fabric Composer (AFC). Stateful firewall policy filters traffic between VLANs, between hosts in the same VLAN, and between VM guests assigned to the same hypervisor (microsegmentation).

The Aruba CX 10000 series switch is a data-center-class Distributed Services Switch (DSS). It includes hardware dedicated to performing stateful firewall functions on data center host traffic. Configuring and managing these capabilities require deployment of PSM. AFC is recommended to manage PSM. Refer to the Aruba ESP Data Center Design VSG and the Pensando Policy and Services Manager for Aruba CX 10000: User Guide for additional details.

Note: Aruba CX 10000 switches running AOS-CX 10.13 and above require a feature pack license to enable firewall features. Information on the installation and purchase of feature packs can be found in the Feature Pack Ordering Guide and Deployment Guide.

Configure PSM Integration with Aruba Fabric Composer

Use this procedure to associate a PSM cluster with a fabric in AFC to enable centralized management of firewall policy.

Step 1 On the menu bar at the top right of the AFC user interface, select Guided Setup.

Step 2 At the top of the Guided Setup window, click the Distributed Services tab, then click PENSANDO PSM.

Step 3 On the Host page, enter the following values and click VALIDATE.

• Name: RSVDC-FB1-PSM
• Description: PSM cluster for RSVDC fabric 1
• Host: 172.16.104.51
• Username: admin
• Password: < Password for the PSM admin user >

Note: The Host field requires only a single DNS name or IP address of one PSM VM. The IP addresses of the remaining cluster members are discovered automatically.

Step 4 After the validation success message appears, click NEXT.

Step 5 On the Settings page, select the fabric. Leave other settings at their default and click NEXT.

Step 6 On the Summary page, verify that the PSM cluster settings are correct and click APPLY.

Note: Aruba CX 10000 switches that are members of the same fabric make a join request to the PSM cluster at the completion of this step. AFC instructs PSM to admit CX 10000 switches, when auto-admission to the PSM cluster has been set to false.

Step 7 On the AFC top right menu bar, click the CLI Commands icon.

Step 8 On the CLI Command Processor page, enter the following values, then click RUN.

• Switches: < Select Aruba CX 10000 fabric leaf switches >
• Commands: show psm

Step 9 Verify that each CX 10000 has the following values in the command output.

• Operational Status: admitted
• Host Addresses: < IP address list of PSM cluster members >
• VRF: mgmt
• Operational Addresses: < IP address list of PSM cluster members >

Configure Macro Firewall Policy

A firewall policy is a collection of rules applied to a PSM Network object or Virtual Routing and Forwarding (VRF) objects. When PSM policy is applied to a DSS switch, a Network object corresponds to a VLAN on the switch, and a VRF object corresponds to a VRF on the switch.

A policy contains a set of rules that specify the traffic allowed or denied between endpoint groups. Each rule contains service qualifiers or applications (sets of service qualifiers) that identify traffic type by port number. An implicit “deny all” rule is applied at the end of every policy.

The following firewall policy example restricts traffic allowed to the database server VLAN. It is applied to a PSM Network. For information on choosing between applying a policy to a Network or a VRF, refer to the Network Design section of the Data Center Design VSG.

The primary purpose of the sample policy is to protect database servers. At the completion of this process, web application servers are allowed to send MSSQL, ICMP, and traceroute traffic to the database servers. Other traffic is denied.

The process defines the following:

• PSM Network objects to inform CX 10000 switches which VLANs to redirect to the firewall engine
• Endpoint group objects to specify IP ranges applied to firewall rules
• Firewall rules to specify allowed or denied traffic
• Firewall policy that stitches the above components together

The web servers are distributed across both CX 10000 switches and non-DSS switch models in this example. The database servers are connected only to a CX 10000 leaf pairs.

Configure PSM Networks

In this procedure, PSM Network objects are created to specify which VLANs redirect traffic for firewall policy enforcement. Creating a PSM Network in AFC will not create a VLAN on the switch. A corresponding VLAN and SVI must be defined, if they do not already exist. For complete configuration steps for prerequisite VRFs, VLANs, and SVIs, refer to Deploying the Fabric earlier in this guide.

Step 1 On the Distributed Services tab of Guided Setup, choose RSVDC-FB1/PROD-DC-VRF in the Selected VRF dropdown. Click CONFIGURE NETWORKS.

Step 2 To add a PSM Network for the web server VLAN, click the right ACTIONS menu and select Add to start the Network workflow.

Step 3 On the Name page, enter a Name and Description, then click NEXT.

Step 4 On the Settings page, enter the database server VLAN ID in the VLAN field to associate it with the Network object, then click NEXT.

Step 5 On the Summary page, verify that the information is correct and click APPLY.

Step 6 Repeat the procedure to define a Network object for every VLAN in PROD-DC-VRF.

Caution: Communication failures may result between two hosts in different VLANs, if a Network object is not associated to both VLANs. It is best practice to define a Network for every VLAN in a VRF.

Review Dynamic Endpoint Groups

Endpoint group objects represent source and destination IP addresses referenced in firewall rules. Dynamic endpoint groups are autopopulated based on the assignment of vSphere tags to VMs identified through AFC’s vCenter integration.

All VMs assigned the same tag are autopopulated as a member of the dynamic group. When member IP addresses change or tag assignments are changed in vCenter, dynamic group objects are updated automatically.

Step 1 On the Configuration menu, select Policy > Endpoint Groups.

Step 2 Verify that the VMs assigned tags in vSphere are populated in the dynamic endpoint groups.

Note: If no dynamic endpoint groups are present, verify that Automated Endpoint Group Provisioning is enabled in Configuration > Integration > VMware vSphere. Verify that VMs in vSphere have been assigned tags and are currently running. Refer to the Deploying the Fabric section for more information on integration.

Configure Static Endpoint Groups

Additional endpoint groups are created to represent IP addresses not automatically populated in a dynamic group.

Step 1 To create an endpoint group for the network services subnet, in the Configuration > Policy > Endpoint Groups context, click the right ACTIONS menu and select Add.

Step 2 On the Name page, enter a Name and Description, then click NEXT.

Step 3 On the Type page, leave Type set to the default Layer 3 value and click NEXT.

Step 4 On the Endpoints page, click the Manual Endpoint entry radio button, enter the web server subnet value in the IPv4 Network Address field, and click ADD.

Note: A single endpoint group can be a collection of IP ranges and individual addresses.

Static endpoint groups can be made using AFC’s vSphere integration. When making static assignments in the Endpoint Group wizard based on a VM Name or VM Tag, the IP addresses added to the endpoint group are established at the time of the endpoint group’s creation. Future changes to VM IP addresses or tag assignments are not updated automatically in the address object. Use dynamic endpoint groups to automatically update VM IP assignments.

Step 5 Verify that the subnet was added to the list at the bottom of the Endpoint Group window and click NEXT.

Step 6 On the Summary page, verify that all information is correct and click APPLY.

Step 7 Repeat the procedure to create additional endpoint groups to be used in firewall rules using the following values:

Configure Firewall Rules

In the following procedure, firewall rules are created to define the inbound traffic allowed to the backend database server VLAN.

In addition to allowing communication from frontend web servers, rules that enable network service functions and troubleshooting protocols are defined. This is required because any traffic not explicitly allowed in firewall policy is blocked by an implicit deny rule.

It is important to consider the policy direction, when defining rules. In this example, an ingress policy is used to filter both DSS and non-DSS sourced traffic to the database servers. Policy direction and design vary based on requirements and administrator preferences.

An individual rule identifies source and destination endpoint groups, a service qualifier or application to specify the traffic type, and an action of “allow” or “deny”. A rule does not allow or deny traffic until it is added to a policy.

Step 1 In the AFC left navigation pane, click Rules.

Step 2 To create a rule that allows web servers to reach database servers using the MSSQL protocol, click the right ACTIONS menu and select Add.

Step 3 On the Name page, enter a Name and Description, then click NEXT.

Note: A helpful rule naming convention is to include the action of allow or deny, endpoint group, and service in the rule name. This simplifies policy construction later.

Step 4 On the Settings page, leave the Type and Action fields at their default values and click NEXT.

Step 5 On the Endpoint Groups page, select the endpoint groups dynamically created using VM tags as follows, then click NEXT.

• Source Endpoint Groups: rsvdc-vcenter.example.local_prod:app1_webui
• Destination Endpoint Groups: rsvdc-vcenter.example.local_prod:app1_db

Note: AFC automatically updates IP addresses in a firewall rule that uses dynamically created endpoint groups when a VM administrator changes IP assignments or VM tags in vCenter.

Step 6 On the Applications and Service Qualifiers page, begin typing the predefined mssql_server name in the Service Qualifier field, click mssql_server when it appears in the list, then click NEXT.

Step 7 On the Summary page, verify that all information is correct and click APPLY.

Step 8 Repeat the procedure to create rules with the following values for use in the the database server Network ingress policy.

Note: Reduce the total number of rules created by defining rules that can be applied to more than one policy.

A new rule can be cloned from an existing rule by selecting ACTIONS > Clone, when minor modifications are required for similar rules.

Configure Ingress Policy

When using an EVPN-VXLAN overlay, a PSM ingress policy filters all traffic arriving from other switches inside the fabric (DSS or non-DSS). This includes both Layer 2 and Layer 3 VXLAN forwarded traffic, and traffic originating from outside the data center fabric. An ingress policy also applies to routed traffic between hosts attached to the same CX 10000 switch, when using AOS-CX 10.10.1000 or later.

By default, an ingress policy does not apply to traffic between hosts in the same subnet attached to the same CX 10000 switch. This type of policy application can be achieved when using a PVLAN-based microsegmentation strategy, as described in ESXi VM Microsegmentation. Egress policy also can be applied to filter traffic between hosts attached to the same CX 10000 that are in the same VLAN and subnet, as long as the traffic traverses the switch.

The diagram below illustrates the components that protect the database servers from non-MSSQL traffic sourced by web servers:

In the following procedure, the previously created rules are assigned to a policy to filter traffic destined for the database server VLAN.

Step 1 In Configuration / Policy / Policies, click the right ACTIONS menu and select Add.

Step 2 On the Name page, enter a Name and Description, then click NEXT.

Step 3 On the Settings page, leave the default value of Distributed Firewall for Type selected and click NEXT.

Step 4 On the Policy Rules page, click the right ACTIONS menu and select Add > Existing.

Step 5 In the Select Rules window, click the checkboxes for rules allowing traffic destined to the database servers in the following order, then click APPLY.

• Allow-PROD-WEB-to-DB-MSSQL
• Allow-Admins-to-PROD-DC-VRF
• Allow-All-ICMP
• Allow-All-UDP-Traceroute

Note: After applying an ingress policy for the database server VLAN, all traffic destined for a host in the VLAN must be explicitly allowed by a firewall rule in the policy or it is blocked. The only exception is traffic originated in the database server VLAN where the source and destination are attached to the same CX 10000 switch.

Step 6 Verify that the rule set is in the desired order and click NEXT.

Step 7 On the Enforcers page, select the following values. Click the ADD button near the bottom left of the window (below the dropdowns).

• Fabric: RSVDC-FB1
• Direction: Ingress
• VRF: PROD-DC-VRF
• Networks: DB-NET-PROD - VLAN: 102

Step 8 Verify that the enforcer information was added correctly and click NEXT.

Step 9 On the Summary page, verify that the information is correct and click APPLY.

The frontend web application server to backend database server policy configuration is now complete.

VM Microsegmentation

Microsegmentation enables the application of firewall policy between VM guests on the same hypervisor.

A PVLAN-based configuration forces all traffic from microsegmented VMs to the upstream CX 10000 switch for firewall policy inspection. Without the PVLAN setup, a hypervisor forwards traffic directly between hosts in the same VLAN. Egress and ingress policy can be applied.

The following diagram illustrates the networking components used to support a microsegmentation policy:

In this procedure, egress firewall policy is applied between two VMs on the same ESXi host. Egress policy brings all traffic initiated by hosts in the VLAN into scope for policy rule creation.

AFC is used to create a virtual distributed switch (vDS) and microsegmentation PVLAN in vSphere. A corresponding PVLAN structure is created on CX 10000 switches attached to the ESXi host. An SVI is created with proxy ARP on the primary PVLAN to allow proxied communication between VMs placed in an isolated PVLAN, and a firewall policy is applied to the primary PVLAN Network to specify allowed traffic between hosts in the isolated PVLAN. In this example implementation, the microsegmentation is created across multiple CX 10000 switches.

The AFC microsegmentation workflow uses a series of wizards to complete microsegmentation set-up in a single fluid process. In addition to creating some of the vCenter configuration, the following related config components can be modified within the same workflow:

• PSM Network definition for the primary PVLAN
• VLAN SVI for the primary PVLAN
• CX 10000 MC-LAG configuration
• Policy endpoint groups
• Policy applications and service qualifiers
• Policy rules
• Firewall policy
• vCenter vDS and virtual port groups
• vCenter LAG/LACP configuration
• Assignment of distributed port groups to vCenter VMs

Configure Policy Applications

Applications in AFC provide an administrator the flexibility to bundle multiple service qualifiers together into a set of protocols. The definition can represent a broad set of requirements for an application or logically group any collective set of protocols together for firewall or ACL policy purposes.

The following procedure defines an application that represents a set of network services that must be reachable from the microsegmented data center hosts.

Step 1 In the AFC left navigation pane, click Applications.

Step 2 Click the right ACTIONS menu and select Add.

Step 3 On the Name page, enter a Name and Description, then click NEXT.

Step 4 On the Service Qualifiers page, begin to enter the name of a service qualifier. A list of matching qualifiers is displayed. Press TAB or ENTER when a selection is highlighted to autocomplete the entry, or click the qualifier name in the list. Continue selecting service qualifiers until the following selections appear in the Qualifiers field, then click NEXT.

• syslog
• dns
• ntp
• radius_auth_udp
• radius_acct_udp
• ldaps

Note: If a predefined service qualifier does not exist for the desired protocol and port combination, click ADD to create a new service qualifier.

Step 5 On the ALG Settings page, leave Type set to the default value of None and click NEXT.

Step 6 On the Summary page, verify that all information is correct and click APPLY.

Configure Microsegmentation Rules

Using the procedure in the Configure Firewall Rules section, create the firewall rules below. The AFC Microsegmentation wizard supports rule creation as a part of its workflow. Predefining rules simplifies the microsegmentation process.

Note: Rules between microsegmented endpoints can be granular to an individual level. In the Allow-MSEG1 examples above, the same rule is applied to allow communication between the collective set of VMs assigned to the microsegmentation. It is best practice to apply policy to sets of hosts when possible to minimize the number of rules.

In this example implementation, dynamic endpoint groups are used in microsegmentation firewall rules.

ESXi Microsegmentation Using AFC

Microsegmentation uses PSM firewall policy to filter traffic between VMs installed on the same hypervisor, when attached to a CX 10000 switch.

The AFC Create Microsegmentation workflow automates microsegmentation in a vSphere environment.

The workflow below can be used for a new ESXi host deployment or on an existing host. The ESXi host must have at least one available VNIC that is not currently assigned to a virtual switch. Only unassigned VNICs are displayed and available in the wizard. If unused VNICs are not available on the target ESXi host, microsegmentation can be created by running appropriate AFC wizards individually after configuring PVLANs in vCenter.

Step 1 On the DISTRIBUTED SERVICES tab of Guided Setup. Select RSVDC-FB1/PROD-DC-VRF in the Selected VRF field and click CONFIGURE MICROSEGMENTATION.

Step 2 On the Settings page, enter the following values to select an ESXi host, then click ADD.

• Name: MSEG1
• Host Addresses: < ESXi host target for microsegmentation >
• NICs: vmnic3   vmnic4

Step 3 Repeat the step above to assign additional ESXi hosts to the microsegmentation instance. All assigned ESXi hosts must be attached to a CX 10000 switch.

Step 4 Verify that all ESXi hosts were added to the list at the bottom of the dialogue box and click NEXT.

Step 5 On the PVLAN page, enter the following values. Next to Isolated VLAN VNICs, click SELECT VNICS.

• Portgroup Name Prefix: DPG
• Primary VLAN: 50
• Isolated VLAN: 51

Step 6 Check the VM virtual network adapters to be assigned to the PVLAN distributed port group, then click APPLY.

Note: Each column supports filtering and sorting to manage the displayed VNICs in large VM deployments.

Step 7 Verify that the number of Isolated VLAN NICs is correct, then click NEXT.

Step 8 On the Fabric page, select the data center fabric and click NEXT.

Step 9 On the LAGs page, enter the following VM values. Next to Switch LAG, click ADD.

• vSphere LAG Name: MSEG1
• Host: < ESXi host >
• Host NICs: < VM NICs connected to switch MC-LAG >

Step 10 LAG wizard: On the Settings page, enter a Name, Description, and LAG Number. Click NEXT.

Step 11 LAG wizard: On the Ports page, select the switch corresponding to the ESXi host, verify that ports are preselected correctly, and click NEXT.

Step 12 LAG wizard: On the LACP Settings page, leave settings at their defaults and click NEXT.

Step 13 LAG wizard: On the VLANs page, enter both primary and isolated VLAN IDs in the VLANs field, then click NEXT.

Step 14 LAG wizard: On the Summary page, verify that all information is correct and click APPLY. The wizard closes and returns to the main Distributed Virtual Switch workflow.

Step 15 On the LAGs page, in the lower left, click ADD.

Step 16 Repeat steps 9–15 to create additional MC-LAGs for each VM included in the microsegmentation, then click NEXT.

Step 17 On the VRF page, select the VRF, then click ADD on the Network field to launch the Network wizard.

Step 18 Network wizard: On the Name page, enter a Name and Description, then click NEXT.

Step 19 Network wizard: On the Settings page, review the auto-selected primary PVLAN value and click NEXT.

Note: When creating a PSM Network within the microsegmentation workflow, the VLAN is automatically set to the primary PVLAN ID. This value cannot be modified. If a different VLAN ID is present, click CANCEL to exit the Network wizard, and click BACK to the PVLANs page for review.

Step 20 Network wizard: On the Summary page, verify that the settings are correct and click APPLY. The wizard closes and returns to the main Distributed Virtual Switch workflow.

Step 21 Verify that the new PSM Network is populated in the Network Field, then click ADD on the SVI field to launch the IP Interface wizard.

Step 22 IP Interface wizard: On the Interface Type page, review the auto-populated values for Type, VLAN, and Enable Local Proxy ARP fields. Enter the following values, then click NEXT.

• Switches: < CX 10000 switches attached to the ESXi microsegmentation >
• IPv4 Subnetwork Address: 10.5.50.0/24
• IPv4 Addresses: 10.5.50.1
• Active Gateway IP Address: 10.5.50.1
• Active Gateway IP MAC Address: 02:00:0A:05:00:01

Note: Enable Local Proxy ARP is checked by the wizard and cannot be modified. Proxy ARP is required to enable communication between VMs assigned to the same isolated PVLAN.

Step 23 IP Interface wizard: On the Name page, enter a Name and Description, then click NEXT.

Step 24 IP Interface wizard: On the Summary page, verify that all information is correct, then click APPLY. The wizard closes and returns to the main Distributed Virtual Switch workflow.

Step 25 Verify that the new SVI was populated in the SVI field and click NEXT to proceed.

Step 26 On the Policy page, click ADD in the Policies field to start the Policy wizard.

Step 27 Policy wizard: On the Name page, enter a Name and Description, then click NEXT.

Step 28 Policy wizard: On the Settings page, verify that the prepopulated Type field value is set to Distributed Firewall and click NEXT.

Step 29 Policy wizard: Click ACTIONS, then select Add > Existing.

Step 30 Policy wizard. In the Select Rules window, select the checkbox for existing rules in the following order, and click APPLY.

• Allow-MSEG1-HTTPS
• Allow-MSEG1-SSH
• Allow-Base-Net-Services
• Allow-All-ICMP
• Allow-All-UDP-Traceroute

Step 31 Policy wizard: Verify the rule order matches requirements, then click NEXT.

Step 32 Policy wizard: On the Enforcers page, verify the pre-populated egress enforcer is listed, then click NEXT.

Note: Policy direction and design vary based on requirements and administrator preferences. When all hosts are attached to CX 10000 switches, egress policy is recommended to filter east-west traffic. If an ingress policy is preferred, the pre-populated egress enforcer can be removed by clicking the trash can icon.

Step 33 Policy wizard: On the Summary page, verify that all information is correct, then click APPLY. The Policy wizard closes, returning to the main Distributed Virtual Switch workflow.

Step 34 On the Policy page, verify that the newly created policy is populated in the Policies field, then click NEXT.

Step 35 On the Summary page, verify that all information is correct, then click APPLY.

At the completion of this step, AFC creates the PVLAN and LAG/LACP configuration on all included ESXi hosts and the MC-LAGs on uplink ToR switches.

Add the Primary PVLAN to the EVPN

The primary PVLAN is added to the EVPN config to advertise reachability of the microsegmented hosts in the fabric. A microsegmentation can exist in a single rack or multiple racks in the fabric.

Step 1 On the Configuration menu, select Routing > EVPN.

Step 2 To modify the EVPN configuration, click the right ACTIONS menu and select Add.

Step 3 On the Introduction page, read the configuration note and click NEXT.

Step 4 On the Switches page, unselect the Create EVPN instance across the entire Fabric and all Switches contained within it checkbox. Specify all DSS switches connected to the PVLAN microsegmentation, then click NEXT.

Step 5 On the Name page, enter the Name Prefix previously used to create an EVPN map and click NEXT.

Note: It is best practice to use the same prefix value for all EVPN configuration within a single fabric.

Step 6 On the VNI Mapping page, the previously used value for Base L2VNI is autopopulated. Enter the microsegmentation primary PVLAN in the VLANs field, then click NEXT.

Step 7 On the Settings page, the previously used MAC resource pool is autopopulated in the MAC Address Resource Pool field. Specify AUTO as the Route Target Type and click NEXT.

Step 8 On the Summary page, verify that all values are correct and click APPLY.

Externally Advertise Microsegmented Networks

During the AFC EVPN-VXLAN creation, BGP was enabled for the PROD-DC VRF. AFC configures redistribution of connected interfaces for all leaf switches where the VRF is configured, when BGP is enabled. This configuration installs the primary PVLAN SVI prefix into BGP, which facilities the advertisement of the primary PVLAN network to campus and other fabrics without the need for additional steps.

Verify Policy in PSM

AFC policy is pushed to AMD Pensando’s PSM, which installs the policy on CX 10000 DPUs.

Verify Networks

Step 1 On PSM, expand Tenants in the left pane and click Networks.

Step 2 Verify that the PSM Networks displayed have the correct ingress and egress policies applied.

Note: Each defined Network should display “Propagation Complete” in the Propagation Status column, indicating that all CX 10000 switches have installed the displayed policy.

Step 3 To view a complete list of policies, on the left navigation menu, click Security Policies.

Step 4 Review the list of policies. To view the rules included in a policy, click a policy name.

Step 5 To view the full text of a rule in the displayed rule set, mouse-over a rule in the list.

---