VCF compatibility checks are safety controls, not generic obstacles to an upgrade or bring-up. Broadcom documents temporary bypasses for two narrowly defined missing-compatibility-data failures. The files and properties differ by release and workflow, so the procedures must not be mixed.
This article covers:
- VCF 5.x and VCF on VxRail 5.x upgrades that match Broadcom KB 323368.
- VCF 9.0.x bring-up failures on the VCF Installer appliance that match Broadcom KB 403075.
Do Not Use This as a General Compatibility Bypass
Use the applicable workaround only after confirming that the error matches the linked KB and that the compatibility data is current.
Do not use these procedures to approve an actually unsupported Bill of Materials. In particular, KB 323368 explicitly says not to bypass an NSX incompatibility warning during an upgrade to VCF 9.0.0.0. Some NSX versions are genuinely incompatible, and that upgrade must not be started.
Before changing anything:
- Confirm the exact VCF version, workflow, and error in the relevant log.
- Update or synchronize the compatibility data first.
- Open a Broadcom Support case if the error does not precisely match one of the documented scenarios.
- Schedule a maintenance window and ensure that the appliance has a current supported backup and recovery path.
- Plan to re-enable the checks immediately after the single blocked workflow completes.
Which Scenario Applies?
| Scenario | Appliance | Properties changed | Services restarted |
|---|---|---|---|
vcf5-upgrade | SDDC Manager, VCF 5.x or VCF on VxRail 5.x | compatibility.flag: vcf.compatibility.controllers.compatibilityCheckEnabled; LCM application-prod.properties: lcm.enable.vvs.compatibility.upgrade.precheck | lcm.service |
vcf9-bringup | VCF Installer, VCF 9.0.x bring-up | LCM application-prod.properties: vcf.compatibility.controllers.compatibilityCheckEnabled; Domain Manager application-prod.properties: vcf.domainmanager.validation.enableVvsInteropValidation | lcm.service, domainmanager.service |
The VCF 5.x upgrade property belongs in compatibility.flag. The VCF 9.0.x bring-up property belongs in the LCM application-prod.properties file. Choosing the wrong scenario edits the wrong configuration.
Reversible, Idempotent Script
The following Bash script supports status, disable, and enable. A first disable captures the exact pre-change files in a root-only state directory. The enable action reads those saved files only to restore each managed property’s original effective state—present with its original value, or absent—without overwriting unrelated settings that an upgrade may have changed.
Before installing anything, the script renders and validates every target file. It also takes per-run rollback copies. On failure, it stages and verifies same-directory rollback files before atomically installing them, then restarts and checks the affected services. If a complete rollback cannot be verified, it keeps the recovery copies, does not restart services against an uncertain mixed configuration, and exits with an explicit manual-recovery error. It also refuses duplicate entries, unexpected values, and a mixed original/disabled state.
#!/usr/bin/env bash
set -euo pipefail
umask 077
SCENARIO="${1:-}"
ACTION="${2:-status}"
STATE_ROOT="/root/.vcf-validation-controls"
STAGING=""
ROLLBACK_DIR=""
ARCHIVE_DIR=""
INSTALL_STARTED=0
declare -a TEMPS
usage() {
echo "Usage: $0 {vcf5-upgrade|vcf9-bringup} {status|disable|enable}" >&2
exit 2
}
[[ "$SCENARIO" == "vcf5-upgrade" || "$SCENARIO" == "vcf9-bringup" ]] || usage
[[ "$ACTION" == "status" || "$ACTION" == "disable" || "$ACTION" == "enable" ]] || usage
if [[ "$EUID" -ne 0 ]]; then
echo "Run this script as root on the appliance specified for the selected scenario." >&2
exit 1
fi
case "$SCENARIO" in
vcf5-upgrade)
FILES=(
"/opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag"
"/opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties"
)
KEYS=(
"vcf.compatibility.controllers.compatibilityCheckEnabled"
"lcm.enable.vvs.compatibility.upgrade.precheck"
)
SERVICES=("lcm.service")
;;
vcf9-bringup)
FILES=(
"/opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties"
"/etc/vmware/vcf/domainmanager/application-prod.properties"
)
KEYS=(
"vcf.compatibility.controllers.compatibilityCheckEnabled"
"vcf.domainmanager.validation.enableVvsInteropValidation"
)
SERVICES=("lcm.service" "domainmanager.service")
;;
esac
STATE_DIR="${STATE_ROOT}/${SCENARIO}.active"
for file in "${FILES[@]}"; do
if [[ ! -f "$file" || -L "$file" ]]; then
echo "Required file is missing or is not a regular, non-symbolic-link file: $file" >&2
exit 1
fi
done
secure_object() {
local path="$1"
local expected_type="$2"
local permissions
if [[ "$expected_type" == "directory" ]]; then
[[ -d "$path" && ! -L "$path" ]] || return 1
else
[[ -f "$path" && ! -L "$path" ]] || return 1
fi
[[ "$(stat -c '%u' "$path")" == "0" ]] || return 1
permissions="$(stat -c '%A' "$path")"
[[ "${permissions:5:1}" != "w" && "${permissions:8:1}" != "w" ]]
}
property_count() {
local file="$1"
local key="$2"
awk -v wanted="$key" '
/^[[:space:]]*#/ { next }
{
equals = index($0, "=")
if (equals == 0) next
property = substr($0, 1, equals - 1)
gsub(/^[[:space:]]+|[[:space:]]+$/, "", property)
if (property == wanted) count++
}
END { print count + 0 }
' "$file"
}
property_value() {
local file="$1"
local key="$2"
awk -v wanted="$key" '
/^[[:space:]]*#/ { next }
{
equals = index($0, "=")
if (equals == 0) next
property = substr($0, 1, equals - 1)
gsub(/^[[:space:]]+|[[:space:]]+$/, "", property)
if (property == wanted) {
value = substr($0, equals + 1)
gsub(/^[[:space:]]+|[[:space:]]+$/, "", value)
found = 1
}
}
END { if (found) print value }
' "$file"
}
show_status() {
local index count value service
for index in "${!FILES[@]}"; do
count="$(property_count "${FILES[$index]}" "${KEYS[$index]}")"
if [[ "$count" -eq 0 ]]; then
value="<not set>"
else
value="$(property_value "${FILES[$index]}" "${KEYS[$index]}")"
fi
printf '%s: %s=%s (active entries: %s)\n' \
"${FILES[$index]}" "${KEYS[$index]}" "$value" "$count"
done
if [[ -d "$STATE_DIR" ]]; then
echo "Saved pre-disable state: $STATE_DIR"
else
echo "Saved pre-disable state: none"
fi
for service in "${SERVICES[@]}"; do
printf '%s: ' "$service"
systemctl is-active "$service" || true
done
}
validate_saved_state() {
local manifest="${STATE_DIR}/manifest.tsv"
local index saved count line_count
local manifest_index manifest_path manifest_key manifest_hash actual_hash
if ! secure_object "$STATE_DIR" directory ||
! secure_object "$manifest" file; then
echo "Saved state is missing or has unsafe ownership, permissions, or object type: $STATE_DIR" >&2
return 1
fi
for index in "${!FILES[@]}"; do
saved="${STATE_DIR}/${index}.original"
if ! secure_object "$saved" file; then
echo "Saved state file is missing or unsafe: $saved" >&2
return 1
fi
line_count="$(awk -F '\t' -v wanted="$index" '$1 == wanted { count++ } END { print count + 0 }' "$manifest")"
if [[ "$line_count" -ne 1 ]]; then
echo "Saved-state manifest does not contain exactly one entry for index $index." >&2
return 1
fi
IFS=$'\t' read -r manifest_index manifest_path manifest_key manifest_hash < <(
awk -F '\t' -v wanted="$index" '$1 == wanted { print; exit }' "$manifest"
)
if [[ "$manifest_index" != "$index" ||
"$manifest_path" != "${FILES[$index]}" ||
"$manifest_key" != "${KEYS[$index]}" ]]; then
echo "Saved-state manifest mapping mismatch for index $index." >&2
return 1
fi
actual_hash="$(sha256sum "$saved" | awk '{print $1}')"
if [[ "$actual_hash" != "$manifest_hash" ]]; then
echo "Saved-state checksum mismatch: $saved" >&2
return 1
fi
count="$(property_count "$saved" "${KEYS[$index]}")"
if [[ "$count" -gt 1 ]]; then
echo "Saved state has duplicate active entries for ${KEYS[$index]}." >&2
return 1
fi
done
}
render_property() {
local source="$1"
local key="$2"
local mode="$3"
local value="$4"
local destination="$5"
awk -v wanted="$key" -v mode="$mode" -v replacement="$value" '
{
equals = index($0, "=")
property = ""
if ($0 !~ /^[[:space:]]*#/ && equals > 0) {
property = substr($0, 1, equals - 1)
gsub(/^[[:space:]]+|[[:space:]]+$/, "", property)
}
if (property == wanted) {
if (mode == "set" && !written) print wanted "=" replacement
written = 1
next
}
print
}
END {
if (mode == "set" && !written) print wanted "=" replacement
}
' "$source" > "$destination"
chmod --reference="$source" "$destination"
chown --reference="$source" "$destination"
}
mkdir -p -- "$STATE_ROOT"
chmod 700 "$STATE_ROOT"
LOCK_FILE="${STATE_ROOT}/operations.lock"
exec 9>"$LOCK_FILE"
chmod 600 "$LOCK_FILE"
if ! flock -n 9; then
echo "Another vcf-validation-controls process is running; no changes were made." >&2
exit 1
fi
cleanup() {
local index temp
for temp in "${TEMPS[@]-}"; do
if [[ -n "$temp" && -f "$temp" ]]; then
rm -f -- "$temp"
fi
done
if [[ -n "$STAGING" && -d "$STAGING" ]]; then
for index in "${!FILES[@]}"; do
rm -f -- "${STAGING}/${index}.original"
done
rm -f -- "${STAGING}/manifest.tsv"
rmdir -- "$STAGING" 2>/dev/null || true
fi
if [[ "$INSTALL_STARTED" -eq 0 && -n "$ROLLBACK_DIR" && -d "$ROLLBACK_DIR" ]]; then
for index in "${!FILES[@]}"; do
rm -f -- "${ROLLBACK_DIR}/${index}.current"
done
rmdir -- "$ROLLBACK_DIR" 2>/dev/null || true
fi
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
if [[ -e "$STATE_DIR" || -L "$STATE_DIR" ]]; then
validate_saved_state
elif [[ "$ACTION" == "enable" ]]; then
echo "No saved pre-disable state exists for $SCENARIO; refusing to guess the original values." >&2
exit 1
fi
echo "Current state:"
show_status
if [[ "$ACTION" == "status" ]]; then
exit 0
fi
for index in "${!FILES[@]}"; do
count="$(property_count "${FILES[$index]}" "${KEYS[$index]}")"
if [[ "$count" -gt 1 ]]; then
echo "Refusing to continue: duplicate active entries exist for ${KEYS[$index]}." >&2
exit 1
fi
done
if [[ -d "$STATE_DIR" ]]; then
ORIGINAL_STATE_COUNT=0
DISABLED_STATE_COUNT=0
for index in "${!FILES[@]}"; do
current_count="$(property_count "${FILES[$index]}" "${KEYS[$index]}")"
original="${STATE_DIR}/${index}.original"
original_count="$(property_count "$original" "${KEYS[$index]}")"
current_matches_original=0
if [[ "$current_count" -eq "$original_count" ]]; then
if [[ "$current_count" -eq 0 ]]; then
current_matches_original=1
else
current_value="$(property_value "${FILES[$index]}" "${KEYS[$index]}")"
original_value="$(property_value "$original" "${KEYS[$index]}")"
if [[ "$current_value" == "$original_value" ]]; then
current_matches_original=1
fi
fi
fi
current_is_disabled=0
if [[ "$current_count" -eq 1 ]] &&
[[ "$(property_value "${FILES[$index]}" "${KEYS[$index]}")" == "false" ]]; then
current_is_disabled=1
fi
if [[ "$current_matches_original" -eq 1 ]]; then
ORIGINAL_STATE_COUNT=$((ORIGINAL_STATE_COUNT + 1))
elif [[ "$current_is_disabled" -eq 1 ]]; then
DISABLED_STATE_COUNT=$((DISABLED_STATE_COUNT + 1))
else
echo "Refusing to continue: ${KEYS[$index]} matches neither the saved original state nor the managed disabled state." >&2
exit 1
fi
done
if [[ "$ORIGINAL_STATE_COUNT" -ne 0 && "$DISABLED_STATE_COUNT" -ne 0 ]]; then
echo "Refusing to continue: managed properties are in a mixed original/disabled state. Recover them under an approved change before retrying." >&2
exit 1
fi
fi
if [[ "$ACTION" == "disable" && ! -d "$STATE_DIR" ]]; then
for index in "${!FILES[@]}"; do
count="$(property_count "${FILES[$index]}" "${KEYS[$index]}")"
if [[ "$count" -eq 1 ]] &&
[[ "$(property_value "${FILES[$index]}" "${KEYS[$index]}")" != "true" ]]; then
echo "Refusing to layer this bypass over a property that is already not true: ${KEYS[$index]}." >&2
exit 1
fi
done
STAGING="$(mktemp -d "${STATE_ROOT}/${SCENARIO}.new.XXXXXX")"
manifest="${STAGING}/manifest.tsv"
: > "$manifest"
chmod 600 "$manifest"
for index in "${!FILES[@]}"; do
saved="${STAGING}/${index}.original"
cp -p -- "${FILES[$index]}" "$saved"
chown root:root "$saved"
chmod 600 "$saved"
hash="$(sha256sum "$saved" | awk '{print $1}')"
printf '%s\t%s\t%s\t%s\n' \
"$index" "${FILES[$index]}" "${KEYS[$index]}" "$hash" >> "$manifest"
done
mv -- "$STAGING" "$STATE_DIR"
STAGING=""
echo "Saved the exact pre-disable files in $STATE_DIR"
validate_saved_state
fi
if [[ "$ACTION" == "enable" ]]; then
validate_saved_state
fi
ROLLBACK_DIR="$(mktemp -d "${STATE_ROOT}/${SCENARIO}.rollback.XXXXXX")"
for index in "${!FILES[@]}"; do
cp -p -- "${FILES[$index]}" "${ROLLBACK_DIR}/${index}.current"
temporary="$(mktemp "${FILES[$index]}.vcf-validation.XXXXXX")"
TEMPS[$index]="$temporary"
if [[ "$ACTION" == "disable" ]]; then
mode="set"
value="false"
else
original="${STATE_DIR}/${index}.original"
original_count="$(property_count "$original" "${KEYS[$index]}")"
if [[ "$original_count" -eq 0 ]]; then
mode="remove"
value=""
elif [[ "$original_count" -eq 1 ]]; then
mode="set"
value="$(property_value "$original" "${KEYS[$index]}")"
else
echo "Saved state has duplicate active entries for ${KEYS[$index]}; refusing restoration." >&2
exit 1
fi
fi
render_property \
"${FILES[$index]}" "${KEYS[$index]}" "$mode" "$value" "$temporary"
rendered_count="$(property_count "$temporary" "${KEYS[$index]}")"
if [[ "$mode" == "remove" ]]; then
[[ "$rendered_count" -eq 0 ]]
else
[[ "$rendered_count" -eq 1 ]]
[[ "$(property_value "$temporary" "${KEYS[$index]}")" == "$value" ]]
fi
if cmp -s -- "${FILES[$index]}" "$temporary"; then
rm -f -- "$temporary"
TEMPS[$index]=""
fi
done
rollback() {
local result="$1"
local index restore_temp
local restore_failed=0
local state_failed=0
local service_failed=0
local -a restore_temps
trap - ERR INT TERM
set +e
echo "Change failed; staging an atomic restoration from $ROLLBACK_DIR" >&2
for index in "${!FILES[@]}"; do
restore_temp="$(mktemp "${FILES[$index]}.vcf-rollback.XXXXXX")"
if [[ -z "$restore_temp" ]]; then
echo "Could not create a rollback staging file beside ${FILES[$index]}." >&2
restore_failed=1
continue
fi
restore_temps[$index]="$restore_temp"
if ! cp -p -- "${ROLLBACK_DIR}/${index}.current" "$restore_temp" ||
! cmp -s -- "${ROLLBACK_DIR}/${index}.current" "$restore_temp"; then
echo "Could not stage and verify the rollback copy for ${FILES[$index]}." >&2
restore_failed=1
fi
done
if [[ "$restore_failed" -eq 0 ]]; then
for index in "${!FILES[@]}"; do
if ! mv -- "${restore_temps[$index]}" "${FILES[$index]}"; then
echo "Atomic rollback installation failed for ${FILES[$index]}." >&2
restore_failed=1
else
restore_temps[$index]=""
fi
done
fi
for restore_temp in "${restore_temps[@]-}"; do
if [[ -n "$restore_temp" && -f "$restore_temp" ]]; then
rm -f -- "$restore_temp"
fi
done
if [[ "$ACTION" == "enable" &&
-n "$ARCHIVE_DIR" && -d "$ARCHIVE_DIR" &&
! -e "$STATE_DIR" && ! -L "$STATE_DIR" ]]; then
if ! mv -- "$ARCHIVE_DIR" "$STATE_DIR"; then
echo "Could not return archived state to $STATE_DIR; it remains at $ARCHIVE_DIR." >&2
state_failed=1
fi
fi
if [[ "$restore_failed" -eq 0 ]]; then
for service in "${SERVICES[@]}"; do
if ! systemctl restart "$service" ||
! systemctl is-active --quiet "$service"; then
echo "Rollback restored the files, but $service did not return active." >&2
service_failed=1
fi
done
else
echo "ROLLBACK INCOMPLETE: services were not restarted. Preserve $ROLLBACK_DIR and recover the listed files before proceeding." >&2
fi
if [[ "$restore_failed" -ne 0 || "$state_failed" -ne 0 || "$service_failed" -ne 0 ]]; then
echo "Manual recovery is required; preserve $ROLLBACK_DIR and the messages above." >&2
exit 70
fi
echo "Rollback completed successfully; the original command still failed." >&2
exit "$result"
}
trap 'rollback $?' ERR
trap 'rollback 130' INT
trap 'rollback 143' TERM
CHANGED=0
INSTALL_STARTED=1
for index in "${!FILES[@]}"; do
if [[ -n "${TEMPS[$index]}" ]]; then
mv -- "${TEMPS[$index]}" "${FILES[$index]}"
CHANGED=1
fi
done
if [[ "$CHANGED" -eq 1 ]]; then
for service in "${SERVICES[@]}"; do
systemctl restart "$service"
done
else
echo "The requested property state was already present; no service restart was needed."
fi
for service in "${SERVICES[@]}"; do
systemctl is-active --quiet "$service"
done
if [[ "$ACTION" == "enable" ]]; then
ARCHIVE_DIR="${STATE_DIR}.restored.$(date +%Y%m%dT%H%M%S).$$"
mv -- "$STATE_DIR" "$ARCHIVE_DIR"
echo "Archived the pre-disable state at $ARCHIVE_DIR"
fi
trap - ERR INT TERM
INSTALL_STARTED=0
for index in "${!FILES[@]}"; do
rm -f -- "${ROLLBACK_DIR}/${index}.current"
done
rmdir -- "$ROLLBACK_DIR"
ROLLBACK_DIR=""
echo "Resulting state:"
show_status
Save it as vcf-validation-controls.sh, make it executable, and run only the scenario that matches the verified KB:
chmod 700 vcf-validation-controls.sh
# Inspect without changing either property.
./vcf-validation-controls.sh vcf5-upgrade status
# VCF 5.x upgrade failure matching KB 323368.
./vcf-validation-controls.sh vcf5-upgrade disable
# VCF 9.0.x bring-up failure matching KB 403075.
./vcf-validation-controls.sh vcf9-bringup disable
Do not run both disable commands on the same appliance. The scenarios apply to different workflows and use different property locations.
The active pre-disable state is stored under /root/.vcf-validation-controls. Keep that directory root-only. If the script reports duplicate properties, a pre-existing false value, incomplete saved state, or a rollback, stop and resolve that condition before retrying the blocked VCF workflow.
Validate, Complete One Workflow, and Restore
After disable:
- Confirm every displayed property is
false, every property has exactly one active entry, and every listed service isactive. - Retry only the workflow that failed with the matching compatibility-data error.
- Do not use the bypass to continue past a new or different compatibility warning.
- As soon as the workflow completes, restore the exact saved property state with the corresponding
enablecommand:
# After the VCF 5.x upgrade workaround:
./vcf-validation-controls.sh vcf5-upgrade enable
# Or, after the VCF 9.0.x bring-up workaround:
./vcf-validation-controls.sh vcf9-bringup enable
The result must show each managed property restored to its original effective state—normally true or <not set>—with no duplicate active entries, and every affected service must be active. A restored property must not remain false. Run the matching status command again and retain the output with the change record.
After successful restoration, the script archives the saved pre-disable state with a timestamp instead of deleting it. It never copies an old whole-file backup over a post-upgrade configuration. Review and remove archived state and any rollback directory only under the approved change record; if restoration is unclear, engage Broadcom Support.
These bypasses are temporary workarounds for documented missing-data failures. They are not a supported method for creating an incompatible VCF deployment.