In this article i will go over one of the workaround instructions to address CVE-2021-44228 and CVE-2021-45046 in vRealize Operations 7.x. I have tested the workaround on vROPS 7.5 as its still shipped with VCF 3.x and i haven’t yet seen documentation on a workaround for this version. If you are looking for instructions for version 8.x consult kb article 87076. This has been tested on December 21 2021. Please check the official documentation or open a ticket for production usage.
Create a snapshot of the vROPS components to make sure we have something to revert to in case anything were to go wrong.
Log into the vROPS instance admin UI typically https://ip_address/admin and take the cluster offline. This applies to all nodes including but not limited to Analytic, Primary, Replica, Data, Remote Collectors and Witness nodes.
Give a reason and press ok
Verify the cluster is offline before continuing
Log in via ssh to a temporary path ex /tmp. Because vROPS 7.5 doesn`t come with the newer OpenSSL modules we need to find other means to get the files to the server without using a direct download method like wget.
In my case in put the code below in a file called vrops-log4j-fix.sh in my /tmp directory
#!/bin/bash
file=/tmp/impacted_jars.txt
echo "Searching for impacted .jar files. Please wait..."
find /usr/lib -type f -name "*.jar" -exec sh -c "zipinfo -1 '{}' | grep "org/apache/logging/log4j/core/lookup/JndiLookup.class" && echo {}" \; | grep "/usr/lib" > $file
line_qty=$(wc -l < $file)
if [ $line_qty -ne 0 ]; then
echo "Found $line_qty impacted .jar files"
else
echo "No impacted .jar files found"
exit 0
fi
echo "Starting to patch impacted .jar files"
while IFS= read -r line;
do
echo "patching -> $line"
own_user=$(stat -c '%U' "$line")
own_group=$(stat -c '%G' "$line")
zip -q -d "$line" org/apache/logging/log4j/core/lookup/JndiLookup.class
if [ $? -ne 0 ]; then echo "ERROR: Fail to Patch $line"; fi
chown $own_user:$own_group "$line"
done < $file
rm -f $file
Make the file executable by running chmod +x vrops-log4j-fix.sh
And then run the script by running ./vrops-log4j-fix.sh
The system will go through and find impacted .jar files and try to patch them. If successful we should end up with something like this
Next we will do the same with cp-log4j-fix.sh file
#!/bin/bash
#set -x
FAILURE="0"
WRAPPER_FILES="/usr/lib/vmware-vcops/user/conf/collector/wrapper.conf"
for f in $WRAPPER_FILES
do
last_idx=""
if [[ -f $f ]]; then
echo "********************************"
echo "Updating file: $f"
let last_idx=$(egrep "^wrapper.java.additional.[[:digit:]]+=" $f | cut -d= -f1 | awk -F '.' '{print $4}' | sort -n | tail -1)
if [[ -z $last_idx ]]; then
echo -e "ERROR: Failed to get JVM additional index"
let FAILURE="1"
continue
fi
((last_idx++))
echo -e "\
#Fixing Apache Log4j2 Remote Code Execution Vulnerability\
wrapper.java.additional.$last_idx=-Dlog4j2.formatMsgNoLookups=true" >> $f
if [[ $? != 0 ]]; then
echo -e "ERROR: Failed to update file: $f\
"
let FAILURE="1"
else
echo -e "Sucessfully updated file: $f\
"
fi
else
echo -e "ERROR: file is not found: $f\
"
let FAILURE="1"
fi
done
CASA_JVM="/usr/lib/vmware-casa/casa-webapp/bin/setenv.sh"
echo "********************************"
echo "Updating file: $CASA_JVM"
echo 'JAVA_OPTS="$JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true"' >> $CASA_JVM
if [[ $? != 0 ]]; then
echo -e "ERROR: Failed to update file: $CASA_JVM\
"
let FAILURE="1"
else
echo -e "Sucessfully updated file: $CASA_JVM\
"
fi
if [[ "X$FAILURE" == "X1" ]]; then
exit 1
fi
exit 0
and lastly the data-rc-witness-log4j-fix.sh file
#!/bin/bash
#set -x
FAILURE="0"
WRAPPER_FILES="/usr/lib/vmware-vcops/user/conf/analytics/wrapper.conf
/usr/lib/vmware-vcops/user/conf/collector/wrapper.conf
/usr/lib/vmware-vcops/user/conf/gemfire/wrapper.conf
/usr/lib/vmware-vcops/user/conf/tomcat-enterprise/wrapper.conf"
for f in $WRAPPER_FILES
do
last_idx=""
if [[ -f $f ]]; then
echo "********************************"
echo "Updating file: $f"
let last_idx=$(egrep "^wrapper.java.additional.[[:digit:]]+=" $f | cut -d= -f1 | awk -F '.' '{print $4}' | sort -n | tail -1)
if [[ -z $last_idx ]]; then
echo -e "ERROR: Failed to get JVM additional index"
let FAILURE="1"
continue
fi
((last_idx++))
echo -e "\
#Fixing Apache Log4j2 Remote Code Execution Vulnerability\
wrapper.java.additional.$last_idx=-Dlog4j2.formatMsgNoLookups=true" >> $f
if [[ $? != 0 ]]; then
echo -e "ERROR: Failed to update file: $f\
"
let FAILURE="1"
else
echo -e "Sucessfully updated file: $f\
"
fi
else
echo -e "ERROR: file is not found: $f\
"
let FAILURE="1"
fi
done
CATALINA_FILES="/usr/lib/vmware-casa/casa-webapp/bin/setenv.sh
/usr/lib/vmware-vcops/tomcat-web-app/bin/setenv.sh"
for f in $CATALINA_FILES
do
if [[ -f $f ]]; then
echo "********************************"
echo "Updating file: $f"
echo 'JAVA_OPTS="$JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true"' >> $f
if [[ $? != 0 ]]; then
echo -e "ERROR: Failed to update file: $f\
"
let FAILURE="1"
else
echo -e "Sucessfully updated file: $f\
"
fi
else
echo -e "ERROR: file is not found: $f\
"
let FAILURE="1"
fi
done
if [[ "X$FAILURE" == "X1" ]]; then
exit 1
fi
exit 0
It would look similar to this in the end
To verify that CVE-2021-44228 was applied run the following
ps axf | grep --color log4j2.formatMsgNoLookups | grep -v grep
Running ./vrops-log4j-fix.sh will also verify that there are no .jar files that need to be patched
Next bring the instance back online in the admin console
