Cosmin.us

Upgrading VCF Log Management to 9.1.1 Step by Step

Cosmin Trif — Tue, 08 Sep 2026 16:30:00 +0000

With VCF 9.1.1 released on September 3, 2026, I wanted to continue patching the management services in my lab. This post is a follow-up to my VCF 9.1 management services upgrade walkthrough and focuses on Log Management through VCF Operations.

VCF 9.1.1 is a maintenance release focused mostly on supportability improvements. The main VCF 9.1.1 release notes and the VCF Operations 9.1.1 release notes are the references I used for this upgrade.

In VCF 9.1, Log Management is a VCF management service hosted on the VCF services runtime. The supported lifecycle workflow is therefore in VCF Operations under Build -> Lifecycle -> VCF Management -> Upgrade. This walkthrough covers a 9.1.0.x maintenance upgrade; an environment coming from an older VCF or Aria release should follow the matching Log Management upgrade procedure first.

Versions in This Upgrade

These are the versions reported by the VCF Operations UI in my lab:

Component	Current Version	Target Version
Log Management	9.1.0.0400.25544947	9.1.1.0.25679624

The target build is also listed in the VCF Operations 9.1.1 release notes.

Before You Begin

For an existing 9.1.0.x environment, patch Fleet Lifecycle to 9.1.1.0 before any other VCF component. After Fleet Lifecycle is updated, the release notes allow the remaining components to be patched according to the documented dependencies. I ran this Log Management operation by itself so that the task and validation were easy to follow.

Before starting:

Verify that the external SFTP backup location is configured and reachable under Build -> Lifecycle -> VCF Management -> Backup & Restore. Keep the backup encryption passphrase available for a restore.
Take an on-demand component backup and verify that it completed successfully before patching Log Management.
Confirm that the lifecycle metadata is current and that the required 9.1.1 binaries are available in the configured software depot. If the depot is offline, prepare the binaries with the VCF Download Tool.
Make sure no other upgrade or patch operation is running. The individual-component procedure says another component cannot be upgraded or patched while an operation is in progress.
Plan a maintenance window and record the Log Management integrations and log sources you need to test afterward.

There is also a 9.1.1 known issue when Log Management and VCF Operations for networks are installed in parallel: the platform can be disrupted, causing Real-time Metrics to fail and the Operations for networks task to remain stuck for more than three hours. Install those components sequentially instead.

Selecting Log Management

The target in my lab is already 9.1.1.*. If the target is still on an older release, click Select Version, choose 9.1.1.*, and apply the target before selecting the component. If the page shows a warning that VCF Operations must be patched independently, finish that patch and wait for it to succeed before continuing with the other management components.

Filter the component list for log, select Log management, and make sure the intended instance is selected. In my lab the instance is instance-a. Expand Check Required Binaries and resolve any missing binary before continuing.

The upgrade path should read 9.1.0.0400.25544947 -> 9.1.1.0.25679624. The screenshot below shows the target version, the selected Log Management instance, and the Ready for upgrade state.

Running the Prechecks

With only Log Management selected, click Run Prechecks (1). Follow the task by opening Precheck details.

The precheck workflow in my lab completed successfully in about 14 minutes. It passed the package staging, vCenter, database upgrade, and single-component backup checks.

Do not start the upgrade until the precheck status is Passed. If a check fails, open its details, remediate the reported issue, and run the precheck again.

Starting the Upgrade

Return to the component list and click Upgrade on the Log Management row. The row changes to Upgrade in progress. Click Upgrade details to follow the workflow.

The workflow runs through these subtasks:

Set Log Management upgrade context for the workflow.
Stage the Log Management plugin in VCF services runtime.
Run the Log Management prechecks.
Stage the Log Management package in VCF services runtime.
Prepare Log Management for upgrade.
Perform the Log Management upgrade.
Run the post-upgrade inventory sync.

The workflow started at 10:53 AM in my lab. The task took about 1 hour and 11 minutes, completing at 12:04 PM. That is the total workflow time shown in my task, not a promise about service downtime in another environment. Wait for the final inventory sync and confirm that the overall task is Completed.

Verifying the New Version

Go to Build -> Lifecycle -> VCF Management -> Components and open the Log Management component for the instance you upgraded.

Under Summary, verify that the status is Running, the version is 9.1.1.0.25679624, and the component is managed by the expected VCF services runtime. In my lab the deployment size is Small with one replica.

After the component reports Running, validate the parts of the service that matter in your environment:

Open Operate -> Logs and run a known query.
Confirm that new events are arriving from representative vSphere, ESXi, and application sources.
Check dashboards, alerts, notification targets, and any custom integrations.
Verify that any custom log forwarding configuration still points to the correct Log Management instance.

9.1.1 Changes to Keep in Mind

The release notes call out a few Log Management changes that are easy to miss during validation:

The Additional Storage Required metric and its related symptoms, alerts, and dashboards were removed because the old value could report a misleading storage shortfall when no retention period was defined.
The Server Side Encryption field was removed from the Add New Storage screen. AWS S3 uses its default AES-256 behavior; for non-AWS S3-compatible storage, configure bucket-level encryption on the storage server itself.
If a Log Management Disaster Recovery restore leaves the Log Analysis page showing Unable to access logs due to user permissions, the documented workaround is to restart the Log Management service through Fleet Lifecycle using the component Stop / Start action.

Once the component and log sources have been validated, keep or expire the backup artifacts according to your retention policy. I also recommend clearing the browser cache before using the updated VCF Operations interface.

Log Management in my lab is now on 9.1.1. If this is part of a larger VCF 9.1.1 maintenance run, continue with the remaining components in the order and dependency boundaries documented in the VCF 9.1.1 release notes.

Upgrading VCF SDDC Lifecycle to 9.1.1 Step by Step

Cosmin Trif — Tue, 08 Sep 2026 12:00:00 +0000

With VCF Services Runtime upgraded to 9.1.1, the next component I wanted to upgrade in my lab was SDDC Lifecycle. In this post I will go over upgrading it from 9.1.0.0400 to 9.1.1.0 through VCF Operations.

VCF 9.1.1 was released on September 3, 2026. This is a maintenance release focused mostly on supportability improvements. The release notes can be found here.

Before You Begin

For an existing 9.1.0.x environment, upgrade Fleet Lifecycle to 9.1.1 first, before any other VCF component. In my lab it is already on 9.1.1.0.25713934. Continue to follow the release-specific dependencies in the 9.1.1 release notes when planning the remaining components.

Before starting:

Verify that the external SFTP backup location is configured and reachable under Build -> Lifecycle -> VCF Management -> Backup & Restore. Keep the encryption passphrase available for a restore. The configuration steps are here.
Take an on-demand backup. Go to Backup & Restore -> Backup -> Backup Now, select the VCF instance and SDDC Lifecycle, and verify that the backup completes successfully. Review the supported backup methods.
Make sure the required upgrade binaries are available through the assigned depot. For offline or disconnected environments, prepare them with the VCF Download Tool using the binary management documentation.

I ran this as a single component operation and waited for it to complete before starting another management-service operation. That keeps the workflow easy to follow and avoids competing lifecycle tasks during the maintenance window.

Here are the versions from my lab:

Component	Before	After
SDDC Lifecycle	9.1.0.0400.25570103	9.1.1.0.25713940

The target version and build match the VCF 9.1.1 Bill of Materials.

Selecting SDDC Lifecycle

To get started, log in to VCF Operations with an Administrator account and go to Build -> Lifecycle -> VCF Management -> Upgrade.

The Target VCF version in my lab is already set to 9.1.1.* from the previous upgrade. If yours is still on 9.1.0.*, click on Select Version and set the target to 9.1.1.*, as covered in the runtime walkthrough.

Clear any existing component selections, filter the list for sddc, and select SDDC lifecycle for the instance you want to upgrade. In my lab this is instance-a. Make sure only the intended component is selected; the buttons should show Run Prechecks (1) and Upgrade (1).

Expand Check Required Binaries and resolve any missing binaries before proceeding. Check that the upgrade path is 9.1.0.0400.25570103 -> 9.1.1.0.25713940.

Running the Prechecks

Click on Run Prechecks (1). The row changes to Prechecks in progress.

Click on Precheck details to follow the workflow. The detail view shows the precheck context being set and the SDDC Lifecycle precheck binaries being staged.

In my lab the precheck started at 8:56 AM and completed at 9:11 AM on September 8. The final result was Passed, including the component package, vCenter, database upgrade, and single-component backup checks.

Once the prechecks pass, return to the upgrade list. The row should show Ready for upgrade.

Resolve any failed check before continuing, following Broadcom’s individual component upgrade procedure.

Starting the Upgrade

Click on Upgrade on the SDDC Lifecycle row.

The status changes to Upgrade in progress. Click on Upgrade details to follow the workflow.

The workflow sets the upgrade context, stages the SDDC Lifecycle plugin in VCF Services Runtime, runs the component prechecks, stages the package, prepares and performs the upgrade, and finishes with an inventory sync.

Wait for the workflow to finish. If a task fails, open its details, resolve the reported issue, and retry it before moving on.

Verifying the Upgrade

Once the upgrade is complete, the workflow shows Completed. In my lab all seven subtasks completed, including Inventory sync post SDDC lifecycle upgrade. The upgrade started at 9:15 AM and finished at 10:04 AM on September 8, about 49 minutes for the full workflow.

We can verify the installed version by going to Build -> Lifecycle -> VCF Management -> Components and opening SDDC Lifecycle for the instance.

Under Summary, check that the status is Running and the version is 9.1.1.0.25713940. Also confirm that it is managed by the expected VCF Services Runtime instance.

One known issue to keep in mind: if you plan to scale VCF Services Runtime from Small to Small (High Availability), upgrade SDDC Lifecycle to 9.1.1 before doing that. The VCF Operations 9.1.1 release notes document a failure when Fleet Lifecycle is on 9.1.1 while SDDC Lifecycle is still on 9.1.0.

SDDC Lifecycle in my lab is now on 9.1.1. We can continue with the remaining management components, following the release-specific dependencies above.

Upgrading VCF Services Runtime to 9.1.1 Step by Step

Cosmin Trif — Mon, 07 Sep 2026 18:30:00 +0000

With Fleet Lifecycle upgraded to 9.1.1, the next component I wanted to upgrade in my lab was VCF Services Runtime. In this post I will go over upgrading it from 9.1.0.0200 to 9.1.1.0 through VCF Operations.

VCF 9.1.1 was released on September 3, 2026. The release notes can be found here. The runtime fixes listed in the VCF Operations release notes include improvements for mixed-case FQDNs, services recovering after a cluster power-on, and log buffers filling the logging disk.

Before You Begin

For an existing 9.1.0.x environment, upgrade Fleet Lifecycle to 9.1.1 first, before any other VCF component. Broadcom also requires the hosting runtime to be upgraded before the dependent Identity Broker and Salt RaaS upgrades to 9.1.1. The release-specific order is documented here.

Before starting:

Verify that an external SFTP backup location is configured and reachable under Build -> Lifecycle -> VCF Management -> Backup & Restore. This is a prerequisite for patching VCF management services. Check the configuration status and keep the encryption passphrase available for a restore. The configuration steps are here.
Take an on-demand backup before the upgrade. Go to Backup & Restore -> Backup -> Backup Now, select the VCF instance and component, and verify that the backup completes. Include the runtime and the hosted services that support backup in your recovery plan. Broadcom lists the runtime’s file-based backup as covering service accounts only; it does not replace the individual service backups. Review the supported backup methods.
Check the runtime and hosted-service health before starting, including node CPU, memory, and disk utilization. Plan a maintenance window: hosted services can restart during maintenance. Broadcom describes these checks in the VCF Management Services health documentation.
Make sure the required binaries are available through the assigned software depot. For offline or disconnected environments, prepare them with the VCF Download Tool using the binary management documentation.

For this walkthrough we will upgrade one runtime instance and let it finish before starting another management-service operation. If VCF Operations itself is being patched, wait for that patch to complete first, as the banner in the screenshot below instructs. Also wait for any Software Depot patch to finish so the upgrade binaries are available.

Here are the versions from my lab:

Component	Before	After
VCF Services Runtime	9.1.0.0200.25555874	9.1.1.0.25714471

Fleet Lifecycle was already on 9.1.1.0.25713934. Both 9.1.1 build numbers match the VCF 9.1.1 Bill of Materials.

Selecting the Target Version

To get started, log in to VCF Operations with an Administrator account and go to Build -> Lifecycle -> VCF Management -> Upgrade.

We can see that Fleet Lifecycle is already upgraded, but the management components still have 9.1.0.* selected as the target. If the new release is not available, use Sync to refresh the lifecycle metadata and wait for the task to complete.

Under Management Components, click on Select Version.

In the Set Target Version dialog, select 9.1.1.* from the VCF global version drop-down and click on Set Version.

Please note that this target applies to management components across all VCF instances. Setting the target does not start the upgrades; we still need to select the component to upgrade.

Running the Runtime Prechecks

Clear any existing component selections, then filter the list for runtime and select the VCF services runtime row for the instance you want to upgrade. In my lab this is instance-a.

Make sure only the intended runtime is selected. The buttons should show Run Prechecks (1) and Upgrade (1). Check that the upgrade path is 9.1.0.0200.25555874 -> 9.1.1.0.25714471.

Expand Check Required Binaries and confirm that the required binaries are available. Resolve any missing binaries before proceeding.

Click on Run Prechecks (1). Once the task completes, review Precheck details and resolve any errors before starting the upgrade, following the individual component upgrade procedure.

Here we can see the runtime row showing Ready for upgrade, with the Precheck details link available.

Starting the Upgrade

Once the prechecks have completed successfully, click on Upgrade on the runtime row.

The status changes to Upgrade in progress. We can follow the progress by clicking on Upgrade details.

The workflow starts by setting the runtime upgrade context and staging the runtime plugin. In this capture we can see Staging binaries followed by Running Component Stage Prechecks.

We now have to wait for the workflow to finish. It continues through runtime prechecks, package staging, upgrade preparation, the upgrade itself, and the final inventory sync.

If a task fails, open its details and resolve the reported issue before retrying. The 9.1.1 known issues also explain that a completed workflow can retain failed subtask attempts after successful retries. Review the final task status and component health when checking the result.

Verifying the Upgrade

Once the upgrade is complete, the workflow shows Completed. The final Inventory sync post VCF services runtime upgrade subtask also shows Completed.

In my lab the workflow started at 9:01 AM and completed at 2:46 PM on September 7, about 5 hours and 45 minutes. This is the total workflow time shown in my task, not a measurement of service downtime. Allow for your environment to take a different amount of time.

We can verify the installed version by going to Build -> Lifecycle -> VCF Management -> Components and opening VCF Services Runtime for the instance.

Under Summary, check that the status is Running and the version is 9.1.1.0.25714471.

Before moving on, open Services runtime health and check the nodes and hosted services for outstanding health issues. Verify the services you use, such as sign-in, log collection, and lifecycle operations, and confirm that backups are still working.

The runtime in my lab is now on 9.1.1. We can continue with the remaining management components, following the release-specific dependencies above. Upgrading the runtime does not by itself confirm that every hosted component is on 9.1.1; check each component’s version separately.

Upgrading VCF Operations to 9.1.1 Using a .PAK File

Cosmin Trif — Thu, 03 Sep 2026 06:50:00 +0000

With the release of VMware Cloud Foundation 9.1.1 on September 3, 2026, I wanted to get my VCF Operations lab upgraded. This is a maintenance release focused mostly on supportability improvements. The VCF 9.1.1 release notes can be found here, and the VCF Operations release notes are here.

In this post I will go over upgrading VCF Operations from 9.1.0.0400 to 9.1.1 using the upgrade PAK file. If you are still on an older release, I also covered upgrading VCF Operations to 9.0.1 using a PAK file. That is a separate walkthrough, not a required intermediate upgrade.

Important: Broadcom does not recommend using the PAK method for maintenance upgrades in a fleet-managed VCF 9.1.0.x environment. In that case, upgrade Fleet Lifecycle to 9.1.1 first and use Build -> Lifecycle -> VCF Management -> Upgrade. I used the manual PAK workflow in my lab, and this post only covers the VCF Operations appliance. The screenshots document that lab result; they are not a replacement for the official component order or evidence that Fleet Lifecycle can be skipped. The PAK method remains the documented path when upgrading from Aria Operations 8.x or VCF Operations 9.0.x to 9.1.

What Is New in VCF Operations 9.1.1

A few changes in this release stood out to me:

SDDC Manager log collection can now be managed from VCF Operations.
A network proxy can be configured when deploying a new cloud proxy from the OVA.
The VMware Salt for VCF Components API is now available for managing configuration across VCF components.
Active Directory lookup, password management, and certificate management now cover more users, accounts, and VCF components.

The release also fixes problems with mixed-case FQDNs during upgrade operations, the UI not loading when NTP is out of sync, services not recovering after a cluster power-on, and log buffers filling the logging disk. Broadcom also published a separate VCF Operations 9.1.1 fixed-CVE workbook.

Versions Used in This Upgrade

The PAK file and the VCF Operations node use different build numbers. This is expected.

Item	Before	After
Installed update package	9.1.0.0400.25541550	9.1.1.0.25679887
VCF Operations node	9.1.0.0400.25541561	9.1.1.0.25679751

In my case the PAK file was Operations-Upgrade-9.1.1.0.25679887.pak, and the download was 7.03 GB.

Before You Begin

Before starting the upgrade:

Review the release notes and verify the supported upgrade path.
Confirm that every VCF Operations node is online and healthy and that DNS and NTP are working correctly.
Run the pre-upgrade assessment and review the results.
Back up any customized content that you need to preserve.
Take the VCF Operations cluster offline and snapshot every cluster node without virtual machine memory and without quiescing the guest file system. Broadcom documents the full process here. Bring the cluster online again before starting the upgrade.
Plan for downtime. The VCF Operations cluster restarts and remains unavailable during the update.

There is a known 9.1.1 issue where the Management Pack compatibility precheck can show raw resource keys and JSON. If that happens, download the APUAT support-bundle HTML report from the same precheck run to see the readable results.

Downloading the Upgrade PAK

To get started, log in to the Broadcom Support Portal and go to My Downloads -> VMware Cloud Foundation -> VMware Cloud Foundation 9 -> 9.1.1.0. Under VCF Operations, click View Group.

Read both agreements and accept the terms to enable the downloads.

Download Operations-Upgrade-9.1.1.0.25679887.pak. The portal also provides the checksums that we can use to validate the file after the download.

Checking the Cluster Before the Upgrade

Open the VCF Operations administration interface at https://<primary-node-fqdn>/admin and log in with the local admin account.

Go to System Status and make sure the cluster is Online and every node is Running. In my lab the starting node version was 9.1.0.0400.25541561.

Go to Software Update. The previously installed update in my lab was 9.1.0.0400.25541550. Click Install a Software Update.

Uploading the PAK File

Click Browse, select Operations-Upgrade-9.1.1.0.25679887.pak, and click Upload. Leave Install the PAK file even if it is already installed unchecked.

The upload can take some time because the PAK is just over 7 GB. Once it finishes, confirm that the signature is valid and that the version is 9.1.1.0.25679887, then click Next.

Review the End User License Agreement, select I accept the terms of this agreement, and click Next.

Review the update information and the software update best practices. This screen reminds us to have a backup or snapshots, run the pre-upgrade assessment, and remove the snapshots after the upgrade. Click Next.

On the last page, click Install. The installer restarts the cluster before copying the files, and the administration interface will become unavailable for a while.

Monitoring the Upgrade

After the administration interface comes back, log in again and go to Software Update to follow the progress. The workflow moves through 14 steps. In the capture below it is at 7 of 14 - Preapply Validated and is preparing for certificate renewal.

Once all 14 steps complete, the status changes to Applied and Cleaned. In my single-node lab the install took about 40 minutes after the PAK finished uploading.

Verifying the Upgrade

Go back to System Status and verify that the cluster is Online and every node is Running. The final node version in my lab is 9.1.1.0.25679751.

In the standard lifecycle workflow, cloud proxy and license server lifecycle is bundled with VCF Operations. Because this lab used the maintenance PAK route, verify both explicitly instead of assuming they were updated. The 9.1.1 release notes list cloud proxy build 25679891 and license server build 25679819. I also recommend checking that collections have resumed, integrations are healthy, dashboards load, and fresh metrics are coming in.

Don’t forget to remove the snapshots once everything has been validated. Since we are doing an upgrade, I also strongly recommend clearing the browser cache before using the new VCF Operations version.

And with that we are finished upgrading the VCF Operations cluster to 9.1.1. If this is part of a full VCF 9.1.1 upgrade, don’t forget to continue with the remaining components in the order listed in the release notes.

Updating ESXi with the Authenticated Broadcom Depot or an Offline Bundle

Cosmin Trif — Tue, 25 Aug 2026 23:05:00 +0000

This is the corrected follow-up to my earlier ESXi command-line upgrade articles. The shared public VMware host-update depot used in those examples is no longer valid. Broadcom now uses customer-specific authenticated download URLs, and its current standalone command-line procedure documents an offline depot ZIP with esxcli software profile update.

The procedure and release information below were reviewed on August 25, 2026. Reopen the linked Broadcom articles during every change: depot authentication, entitlements, available profiles, release builds and support status can change.

Scope: This article is for a standalone, profile-based ESXi update when esxcli is an approved lifecycle method. For a VCF-managed host, use the supported VCF lifecycle workflow. For a cluster managed with a vSphere Lifecycle Manager desired image, remediate through vLCM so the host does not drift from that image. NSX-prepared hosts have additional sequencing and kernel-module requirements.

Do Not Select a Build Merely Because It Is Called Latest

An image-profile name is an exact change target, not a moving alias. Choose the release required by the approved bill of materials, upgrade path, hardware support and product interoperability. Then copy the exact profile name reported by the selected depot.

The three old examples were pinned to releases that Broadcom’s build-number table now shows as superseded:

Historical article target	What the current table shows
ESXi 7.0 Update 3f, build 20036589	Later ESXi 7 entries exist, through Update 3w, build 24784741. ESXi 7 reached End of General Support on October 2, 2025.
ESXi 8.0 Update 3b, build 24280767	Later releases exist in the 8.0 Update 3 train, including Update 3k, build 25595708.
ESX 9.0 GA, build 24755229	Later 9.0 releases and a separate 9.1 train exist. The table lists 9.0.2.0100 build 25595025 and 9.1.0.0200 build 25557999.

Those values describe the table on the review date; they are not recommendations and must not be copied as an automatic target. In particular, a numerically higher build in another maintenance or major-version train is not proof that it is the correct upgrade for an environment.

Confirm the Lifecycle Method and Exact Target

Before downloading or applying an image:

Confirm the supported upgrade path in the Broadcom Product Interoperability Matrix, including vCenter Server, NSX, vSAN and every integrated VMware product.
Verify the server model, CPU, boot device, NICs, storage controllers, drivers, firmware and BIOS in the Broadcom Compatibility Guide. Use the current vendor-customized image when the hardware vendor requires it.
Read the target release notes and identify whether the required artifact is under Products or Solutions in the Broadcom Support Portal. Broadcom explains that base releases and patch releases can appear in different areas.
For an NSX-prepared host, follow Broadcom KB 312013 for the matching NSX kernel-module bundle and upgrade sequence. Do not apply a generic image and assume the NSX packages will remain correct.
If the host is VCF-managed or governed by a vLCM desired image, stop using this standalone procedure and use that product’s lifecycle workflow.

ESXi 7 is now a legacy, end-of-general-support release. Do not treat a newer ESXi 7 patch as a substitute for an approved migration plan. Use post-EoGS content or a downgrade only under the applicable support policy and Broadcom Support guidance.

Prepare the Host

Before changing the host:

Take a current ESXi configuration backup and verify the recovery path.
Migrate or shut down all user VMs. Do not manually move or power off vCLS VMs unless Broadcom Support directs you to do so.
Confirm the remaining cluster can satisfy the intended vSphere HA failover policy while this host is unavailable.
For vSAN, confirm cluster health and object compliance, wait for active resynchronizations to finish, and select the maintenance-mode data-migration option appropriate to the storage policies and available capacity.
Keep out-of-band console access available and patch one host at a time unless the validated design explicitly permits parallel remediation.

Enter maintenance mode from the vSphere Client and confirm it from the ESXi shell:

vim-cmd hostsvc/hostsummary | grep inMaintenanceMode

Continue only when the result is true.

Preferred Method: Use an Offline Depot Bundle

Broadcom’s current standalone esxcli procedure uses an offline depot ZIP. Download the artifact for the exact approved release from the Broadcom Support Portal or the hardware vendor, then upload it to a datastore accessible by the host. Confirm the upload completed and retain the portal filename and release information with the change record.

Set the path to the uploaded file. Replace the uppercase example components with the actual datastore, directory and filename:

OFFLINE_BUNDLE='/vmfs/volumes/DATASTORE/updates/ESXI_DEPOT_BUNDLE.zip'

List every image profile in that bundle:

esxcli software sources profile list -d "$OFFLINE_BUNDLE"

Read the target release notes to distinguish standard, no-tools, security-only and vendor-customized profiles. Copy one exact profile name from the command output only after confirming it matches the approved target:

TARGET_PROFILE='COPY_EXACT_PROFILE_NAME_FROM_LIST_OUTPUT'

Inspect the current host profile and perform a dry run:

esxcli software profile get
esxcli software profile update --dry-run -d "$OFFLINE_BUNDLE" -p "$TARGET_PROFILE"

Treat every dependency, acceptance-level, hardware, space or package-removal message as a failed precheck until it is understood. A clean dry run is still not proof that the host will boot or rejoin the cluster successfully. Do not add --force, --no-sig-check, --ok-to-remove or compatibility-bypass flags to make the command pass.

Broadcom documents a false dry-run failure on ESXi 7 releases earlier than 7.0 Update 3i when a target profile contains the newer ESXio content. Do not simply ignore that failure. Move to the fixed baseline through an approved path, use the supported lifecycle workflow, or obtain Broadcom Support guidance for the exact source and target.

When the dry-run result and change target have been reviewed, apply the same profile without --dry-run:

esxcli software profile update -d "$OFFLINE_BUNDLE" -p "$TARGET_PROFILE"

profile update normally preserves vendor/OEM components unless they are superseded. Do not replace it with profile install unless the exact upgrade documentation or Broadcom Support requires installation semantics and the planned package removals have been reviewed. Starting with ESXi 8.0 Update 2, esxcli software vib update and esxcli software vib install are not supported for changing the ESXi version.

Alternative: Use the Authenticated Broadcom Online Depot

Use the online method only when direct host access to dl.broadcom.com is permitted and the change workflow accepts a customer-specific token in the depot URL. Broadcom says only a Product Administrator can generate a download token for an entitled Site ID. A newly generated token can also take a short time to become valid.

Treat the token like a credential: do not publish it, add it to source control, include it in screenshots, or paste it into tickets and chat transcripts. Prefer the offline-bundle method when shell history, process arguments or logs containing the authenticated URL would violate credential-handling policy.

Temporarily enable the outbound HTTP client ruleset:

esxcli network firewall ruleset set -e true -r httpClient

Set the authenticated main-depot URL, replacing REPLACE_WITH_DOWNLOAD_TOKEN with the token for the entitled Site ID:

DEPOT_URL='https://dl.broadcom.com/REPLACE_WITH_DOWNLOAD_TOKEN/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml'

List the depot profiles and copy the exact approved profile name from the output:

esxcli software sources profile list -d "$DEPOT_URL"
TARGET_PROFILE='COPY_EXACT_PROFILE_NAME_FROM_LIST_OUTPUT'

Run only the precheck first:

esxcli software profile update --dry-run -d "$DEPOT_URL" -p "$TARGET_PROFILE"

When the dry run succeeds and its output has been reviewed, apply the update as a separate action:

esxcli software profile update -d "$DEPOT_URL" -p "$TARGET_PROFILE"

Disable the temporary firewall exception and remove the shell variables when the depot is no longer needed:

esxcli network firewall ruleset set -e false -r httpClient
unset TARGET_PROFILE DEPOT_URL

An HTTP 401 or 403 response can indicate a wrong Site ID, insufficient entitlement, a token that has not propagated yet or a token-access problem. Validate it with Broadcom’s authenticated-download troubleshooting guide instead of reverting to the retired public depot. Fix proxy and certificate trust problems; do not disable TLS certificate verification.

Reboot and Validate Before Exiting Maintenance Mode

If the profile update reports that a reboot is required, keep the host in maintenance mode and reboot it:

esxcli system shutdown reboot -r 'apply ESXi profile update'

After the host finishes booting, verify the version and active profile:

vmware -vl
esxcli software profile get

Before returning the host to service, also validate:

Management networking, physical NICs, DNS and host connectivity to vCenter Server.
Storage adapters, paths, datastores and expected multipathing policy.
vSphere HA, DRS and cluster alarms.
vSAN health, object compliance and any post-reboot resynchronization.
NSX transport-node, tunnel, datapath and kernel-module status when applicable.
Compliance with the intended vLCM image or VCF bill of materials.

Exit maintenance mode only after these checks succeed:

vim-cmd /hostsvc/maintenance_mode_exit

Disable SSH or ESXi Shell again if it was enabled only for this change.

Official References

Safely Removing an SDDC Manager-Deployed NSX Edge Cluster from VCF

Cosmin Trif — Tue, 25 Aug 2026 12:05:00 +0000

This is the corrected follow-up to my older article about simplifying NSX Edge removal in VCF. The most important correction is that the Edge Cluster Deployment Removal Tool does not preserve everything attached to an Edge cluster. Broadcom requires added dependencies to be removed or disconnected before the cleaner runs.

The procedure below is based on Broadcom KB 316110, reviewed on August 25, 2026. Always reopen the KB before a change because the attachment, checksum, prerequisites, and supported releases can change.

Destructive operation: The cleaner removes the selected deployment’s NSX gateways, Edge cluster and node VMs, network objects, associated vCenter objects, and SDDC Manager inventory records. A successful dry run is necessary, but it is not a rollback mechanism and does not replace dependency analysis or backups.

Confirm That the Deployment Is Supported

Use this cleaner only for an NSX Edge cluster deployment created through one of these SDDC Manager workflows:

An Application Virtual Network (AVN) bring-up workflow.
The SDDC Manager Edge Cluster Creation workflow, including the corresponding UI workflow.

KB 316110 describes the cleaner as intended for freshly deployed Edge clusters. Stop and open a Broadcom Support case when the deployment’s origin or current inventory association is unclear.

The current cleaner also exposes a separate --db-removal-only mode for a specific VCF 9.x imported-Edge inventory scenario. That mode does not perform the deployment cleanup described here and is outside this article’s scope; use it only under the current KB instructions or Broadcom Support guidance.

Do not use this workflow when:

The Edge cluster is being used by Workload Control Plane (WCP), vSphere with Tanzu, or dependent Kubernetes services.
The Edge cluster was created outside the supported SDDC Manager workflows.
Edge VMs, gateways, or inventory records were already deleted manually and SDDC Manager no longer matches NSX or vCenter.
Fabric or logical objects created by SDDC Manager were renamed and their original names are unknown.
The requested action is to shrink, migrate, or repair an active Edge cluster rather than remove the deployment.

Manual deletion can leave SDDC Manager inventory out of sync with NSX and vCenter. Broadcom directs administrators who reach that state to open a support case rather than continue deleting objects; see KB 437372.

Remove Dependencies Before Running the Cleaner

Inventory everything using the Edge cluster and migrate or remove it before continuing. This includes, as applicable:

Added Tier-0 or Tier-1 gateway configuration.
Segments linked to a Tier-1 gateway.
Interfaces, services, static routes, NAT, VPN, load-balancing, DHCP, or other configuration attached above the SDDC Manager-created deployment.
External Connections and Transit Gateway dependencies in applicable VCF 9.x deployments.
WCP and related components, which must be removed through the supported vCenter workflow before the Edge cluster can be reconsidered for cleanup.

Revert renamed fabric or logical entities to the names assigned by the SDDC Manager deployment workflow. The cleaner has only limited ability to disconnect linked segments; do not rely on the dry run to discover or safely migrate every application dependency.

Obtain approval from the network and application owners after confirming that removing the identified gateways and Edge nodes will not interrupt a service that should remain available.

Prepare Backups and Recovery Access

Before running a destructive cleanup:

Verify a current, successful NSX file-based backup and confirm access to its passphrase and remote backup files.
Follow the current VCF backup guidance for SDDC Manager. Take an SDDC Manager VM snapshot only when the Broadcom procedure or Support guidance for the specific scenario instructs you to do so, and remove it promptly after successful validation.
Do not snapshot NSX Manager, Global Manager, or Edge appliances. Broadcom documents that VM snapshots of NSX appliances are unsupported and can cause database corruption or cluster instability; use the supported NSX file-based backup instead. See KB 433822.
Schedule a maintenance window and keep console access to SDDC Manager, NSX, and vCenter available.

Download and Verify the Current Tool

Open KB 316110 and download its current Edge cleaner attachment. Do not reuse an older tarball or a bookmarked attachment URL.

The KB publishes the filename and hashes for its current attachment and warns that the hashes change when the file changes. Copy the tarball to /home/vcf on the SDDC Manager appliance, not /tmp, then calculate its SHA-256 hash:

cd /home/vcf
sha256sum 'edge_cluster_cleaner_<version>.tar.gz'

Replace <version> with the exact version in the downloaded filename. Compare the entire result with the current SHA-256 value shown in KB 316110. Stop if the filename or hash does not match.

Extract the verified attachment and inspect the bundled help:

tar -xzvf 'edge_cluster_cleaner_<version>.tar.gz'
cd cleanup
./remove_edge_cluster.sh --help

Use the flags reported by the downloaded tool. If its syntax differs from the KB or this article, stop and reconcile the version before continuing.

Run a Dry Run First

Use the exact Edge cluster name and, when supplied, the exact workload-domain name. Use a vCenter SSO administrator for the workload domain. Do not place its password on the command line; when neither a password flag nor password environment variable is supplied, the tool prompts for it without storing it in shell history.

./remove_edge_cluster.sh \
 --cluster '<EDGE_CLUSTER_NAME>' \
 --workload '<WORKLOAD_DOMAIN_NAME>' \
 --user '<SSO_ADMIN_USER>' \
 --dryrun \
 --verbose

The --workload argument is optional in the documented tool, but specifying the intended domain makes the target explicit. Review the dry-run output and generated log from beginning to end. Confirm that:

The tool authenticated to the intended environment.
The selected Edge cluster and workload domain are correct.
Every object proposed for deletion belongs to the intended deployment.
No unexpected dependency, renamed object, validation failure, or inventory mismatch appears.

If any check fails, do not add force or warning-suppression flags. Correct the prerequisite or open a Broadcom Support case, then repeat the dry run.

Run the Live Cleanup

After the dry run succeeds and the change is approved, run the same command without --dryrun:

./remove_edge_cluster.sh \
 --cluster '<EDGE_CLUSTER_NAME>' \
 --workload '<WORKLOAD_DOMAIN_NAME>' \
 --user '<SSO_ADMIN_USER>' \
 --verbose

Read the warning and confirm the target interactively. Do not use --skip-warning merely to make the example noninteractive. Keep the terminal connected until the cleaner finishes and retain its log with the change record.

KB 316110 notes that a transient timeout can occasionally stop cleanup partway through. In that documented timeout condition, inspect the log and rerun the same cleaner invocation. If the Edge cluster was dissolved but its Edge node VMs remain, follow the KB’s specific direction to remove those residual Edge Transport Node VMs through the NSX UI. Do not improvise manual deletion for other partial states.

Validate All Three Inventories

After the cleaner reports success, verify the result in:

SDDC Manager: the Edge deployment and its workload-domain inventory association are removed.
NSX Manager: the intended gateways, Edge cluster, Edge nodes, uplink segments, and SDDC Manager-created transport-zone objects are gone, with no new alarms or stale dependencies.
vCenter Server: the intended Edge node VMs, port groups, and deployment resource pool are gone, while unrelated infrastructure remains present.

Also test the management and workload paths that were intentionally migrated away from the removed Edge cluster. If the three inventories disagree, stop. Do not manually delete more objects to make the screens match; collect the cleaner log and support bundles and open a Broadcom Support case.

What the Cleaner Normally Removes

Depending on whether the deployment came from AVN bring-up or direct SDDC Manager Edge creation, KB 316110 says the cleaner normally removes some or all of the following:

The deployment’s Tier-0 and Tier-1 gateways.
The Edge cluster and Edge node VMs.
Edge uplink segments and deployment-created VLAN transport zones.
Deployment-created vCenter port groups and resource pool.
The corresponding SDDC Manager inventory records.

That list is the reason dependency removal and a reviewed dry run come first. The cleaner is a deployment-removal tool, not an application-migration or dependency-preservation tool.

Official References

VMSA-2025-0004: Current Remediation and Fixed Versions

Cosmin Trif — Tue, 25 Aug 2026 12:00:00 +0000

Broadcom published VMSA-2025-0004 on March 4, 2025, for three vulnerabilities affecting VMware ESXi and related desktop and cloud products. Fixed versions are available, and the advisory lists no workarounds for any of the three vulnerabilities.

The practical response is therefore straightforward: identify every affected product, move it to a fixed or later supported build using its supported update method, and verify the resulting build. Access restrictions and monitoring remain useful defense-in-depth controls, but they do not replace the update.

Fixed Versions

The following table reflects Broadcom’s VMSA response matrix. Where a newer supported update is available and compatible with your environment, use that newer update rather than treating the minimum fixed build as a preferred long-term target.

Product	Affected branch	Minimum remediation listed by Broadcom
VMware ESXi	8.0	`ESXi80U3d-24585383` or `ESXi80U2d-24585300`, depending on the maintained branch
VMware ESXi	7.0 (EOGS)	`ESXi70U3s-24585291`
VMware Workstation	17.x	17.6.3
VMware Fusion	13.x	13.6.3
VMware Cloud Foundation	5.x	Async patch to `ESXi80U3d-24585383`
VMware Cloud Foundation	4.5.x (EOGS)	Async patch to `ESXi70U3s-24585291`
VMware Telco Cloud Platform	5.x, 4.x, 3.x, 2.x	Follow Broadcom KB 389385
VMware Telco Cloud Infrastructure	3.x, 2.x	Follow Broadcom KB 389385

The ESXi 7.0 and VCF 4.5.x entries record the advisory’s March 2025 minimum fixes; they are not current supported target releases. vSphere 7.0 reached End of General Support on October 2, 2025, and Broadcom’s current VCF release matrix places VCF 4.x under End of General Support. Unless a specific Extended Support agreement or Broadcom Support direction applies, plan migration to a supported release rather than stopping at those legacy builds.

Cloud Foundation administrators should use the VCF async-patching procedure linked from the advisory rather than independently applying an ESXi image that has not been accounted for in the VCF bill of materials. Also confirm hardware, firmware, driver, add-on, backup, replication, and management-product interoperability before beginning a production rollout.

The advisory is the source of truth if its matrix changes after this article is published.

Understand the Three Vulnerabilities Correctly

These vulnerabilities are related, but they do not all provide the same capability:

CVE-2025-22224 — VMCI heap overflow: A malicious actor with local administrative privileges inside a VM may execute code as that VM’s VMX process on the host. Broadcom rates it Critical with a maximum CVSSv3 score of 9.3.
CVE-2025-22225 — ESXi arbitrary write: An actor who already has privileges within the VMX process may trigger an arbitrary kernel write and escape the VMX sandbox. Broadcom rates it Important with a maximum CVSSv3 score of 8.2.
CVE-2025-22226 — HGFS information disclosure: An actor with administrative privileges inside a VM may exploit an out-of-bounds read to disclose memory from the VMX process. Broadcom rates it Important with a maximum CVSSv3 score of 7.1.

This distinction matters. CVE-2025-22224 provides code execution in the VMX process; it should not be described by itself as complete host takeover. CVE-2025-22225 is the sandbox-escape step described by the advisory. CVE-2025-22226 is an information-disclosure issue, not a code-execution issue.

Broadcom’s current advisory states that it has information suggesting exploitation of all three CVEs has occurred in the wild.

Recommended Remediation Workflow

Inventory the affected products and exact builds. Include standalone ESXi, VCF-managed hosts, Workstation, Fusion, and applicable Telco products.
Compare each build with the current advisory. Do not rely only on a scanner’s advisory title; confirm the installed build against the response matrix.
Plan the supported update path. Check compatibility and product-specific prerequisites. For VCF, use the documented async-patching workflow referenced by Broadcom.
Protect recoverability. Confirm that current configuration backups and workload recovery plans are usable before changing production infrastructure.
Evacuate and maintain hosts correctly. For clustered ESXi hosts, migrate or shut down workloads as appropriate and place each host into maintenance mode before applying its supported lifecycle operation.
Apply the fixed or a later supported build. There is no Broadcom workaround that substitutes for this step.
Verify the result. Confirm the reported build, cluster health, management connectivity, storage and network health, and workload availability before returning the host to service.
Continue monitoring. Review security telemetry and incident-response data for activity that predates remediation; applying the fixes blocks these vulnerable code paths but does not prove that earlier exploitation did not occur.

No Workaround Means No Equivalent Configuration Fix

Broadcom lists None in the Workarounds column for CVE-2025-22224, CVE-2025-22225, and CVE-2025-22226. Restricting administrative access inside VMs, segmenting management networks, and monitoring hosts are sensible controls, but they do not remove the vulnerable code and should not be presented as equivalent remediation.

Historical Context

My original March 2025 article described the initial urgency correctly but incorrectly stated that patches were not yet available. Broadcom’s initial advisory already included fixed versions. I have corrected the original article and retained it as historical context, while this page provides the current remediation-first guidance.

Official References

Safely Removing Stale NSX VIBs from an ESXi Host

Cosmin Trif — Tue, 25 Aug 2026 09:00:00 +0000

This post replaces the unsafe package-removal command that appeared in my older article about removing stale NSX packages from an ESXi host. Broadcom now documents a clear distinction between removing NSX through its management workflow and cleaning up VIBs that remain afterward.

Warning: Do not manually remove individual NSX VIBs with esxcli software vib remove. Broadcom identifies that method as unsupported. It can break distributed-switch state, disconnect host networking, and in some failure cases leave reimaging as the recovery path.

When This Procedure Applies

Use this workflow for the documented stale-package condition: NSX has already been removed from the host through NSX Manager, or the host has been removed from NSX Manager, but NSX VIBs remain on ESXi. One example is the dependency error covered by Broadcom KB 409544, where vsipfwlib remains dependent on nsx-proto2-libs.

This is not a shortcut for dismantling an active transport node. If NSX Manager still shows the host as configured, if workloads or VMkernel adapters still depend on NSX-backed networking, or if the supported removal command reports attached resources, resolve those dependencies before continuing. Do not bypass the command’s safety checks by deleting individual VIBs.

Step 1: Evacuate the Host and Enter Maintenance Mode

Before attempting either the NSX Manager removal or the stale-VIB cleanup:

Evacuate powered-on workloads from the host.
Put the host into maintenance mode.
Confirm that the cluster and its storage can tolerate the host being unavailable.
Keep console or out-of-band management access available in case normal host networking is interrupted.

Do not perform this cleanup on an in-service host.

Step 2: Remove or Unprepare the Host in NSX Manager

Use NSX Manager as the first removal method. Select the applicable Remove NSX or unprepare action for the host transport node. The exact navigation label varies by NSX release. Wait for NSX Manager to report the host as Not Configured before treating the remaining packages as stale.

For one currently documented reinstall scenario, Broadcom directs administrators to go to System > Fabric > Hosts, move the host out of the prepared cluster, and use Remove NSX if it does not transition to Not Configured. See KB 401684 and confirm the matching procedure for your installed NSX version.

Step 3: Confirm Which NSX VIBs Remain

esxcli software vib list | grep -E 'nsx|vsipfwlib'

Record the output for troubleshooting. If the command returns no matching packages, there are no NSX VIBs for this procedure to remove. Reconcile the host’s state in NSX Manager instead of continuing blindly.

Step 4: Use the Supported NSX Removal Command

For the stale-VIB condition documented in KB 409544, run:

nsxcli -c del nsx

Allow the command to complete. If it fails, stop and address the reported dependency or attached-resource condition. Do not replace it with individual esxcli software vib remove commands. Broadcom’s KB 426464 explicitly says to correct the condition causing del nsx to fail rather than using the unsupported ESXCLI method.

Step 5: Reboot and Verify the Host

KB 409544 requires a reboot after the removal. Reboot the host while it remains in maintenance mode. After it has returned, log in and repeat the package check:

esxcli software vib list | grep -E 'nsx|vsipfwlib'

For the KB 409544 condition, the expected result is no matching NSX VIBs. Also verify that:

The management interface is reachable.
NSX Manager reports the host as Not Configured if it is meant to remain unprepared.
No stale NSX-backed distributed-switch resources remain associated with the host.

If you intend to prepare the host again, initiate that operation from NSX Manager. Wait for the transport node to report a healthy state, then validate management connectivity, workload networking, and tunnel status before exiting maintenance mode.

If ESXCLI Removal Was Already Attempted

If individual NSX VIBs were already removed and the host lost network connectivity, do not continue removing packages. Use console or out-of-band access and follow the recovery article that matches the symptom:

KB 426464: ESX host loses network connectivity after running ESXCLI removal on NSX VIBs warns that affected hosts can require reimaging.
KB 394809: NSX VIBs were uninstalled using ESXCLI, causing network outages documents a console-based recovery path and highly recommends reimaging afterward to avoid unexpected effects.

When the host’s current state does not match either documented scenario, collect diagnostics and open a Broadcom Support case rather than combining cleanup commands from unrelated KB articles.

Official References

Safely Canceling and Deleting Alerts with the VCF Operations API

Cosmin Trif — Tue, 25 Aug 2026 00:00:00 +0000

The VCF Operations API uses different payloads for finding, canceling, and deleting alerts. Treating those operations as interchangeable can either fail validation or broaden a cleanup beyond the alerts you intended to remove.

This follow-up corrects the API section in Bulk Delete Alerts in VCF Operations. The safe sequence is:

Query candidate alerts without changing them.
Review every returned alert ID and account for pagination.
Cancel only the approved IDs with a uuid-values body.
Verify those IDs are now CANCELED.
Delete the same IDs with a direct AlertQuery body.
Query again to confirm that they are gone.

Before You Begin

Fix or tune the alert definition first. Deleting alert history does not stop the condition from generating new alerts.
Use a VCF Operations account whose application role and object scope permit the required alert operations.
Take any database backup required by your change policy before a large cleanup.
Start with one small batch. The API documents a default query page size of 1,000, so a single response is not proof that you found every match.
Use a trusted certificate. The examples intentionally do not bypass TLS verification.

Set the appliance name, create a private working directory, and read an existing session token without placing it in shell history. Store the authorization header in a mode-600 file so the token is not expanded into each curl process argument, then remove it as soon as the cleanup is verified:

export OPS_FQDN='ops.example.com'
umask 077
OPS_WORK_DIR="$(mktemp -d /tmp/vcf-ops-alert-cleanup.XXXXXX)"
read -r -s -p 'VCF Operations token: ' OPS_TOKEN
printf '\n'
printf 'Authorization: OpsToken %s\n' "$OPS_TOKEN" \
 > "${OPS_WORK_DIR}/authorization.header"
chmod 600 "${OPS_WORK_DIR}/authorization.header"
unset OPS_TOKEN

Current VCF Operations API requests use this header:

Authorization: OpsToken TOKEN_VALUE

You can obtain the session token from POST /suite-api/api/auth/token/acquire. Do not save the token, account password, or response in a shared script or repository.

Step 1: Query a Narrow Candidate Set

Use POST /suite-api/api/alerts/query with an AlertQuery. Prefer an exact alertDefinitionId, resource ID, or an already reviewed list of alert IDs. The alertName field uses contains logic and is therefore less precise.

Create a query for one exact alert definition:

{
 "compositeOperator": "AND",
 "activeOnly": true,
 "alertDefinitionId": [
 "11111111-2222-3333-4444-555555555555"
 ]
}

Save that JSON as ${OPS_WORK_DIR}/alert-candidates.json, replace the example UUID, and query the first page:

curl --fail-with-body \
 --request POST \
 "https://${OPS_FQDN}/suite-api/api/alerts/query?page=0&pageSize=1000" \
 --header @"${OPS_WORK_DIR}/authorization.header" \
 --header 'Content-Type: application/json' \
 --data-binary @"${OPS_WORK_DIR}/alert-candidates.json" \
 --output "${OPS_WORK_DIR}/alert-page-0.json"

Review the count and identifying fields before doing anything destructive:

jq '.pageInfo, [.alerts[] | {alertId, alertDefinitionId, alertDefinitionName, resourceId, status}]' \
 "${OPS_WORK_DIR}/alert-page-0.json"

If pageInfo.totalCount exceeds the number returned, query every remaining page and review it. Keep batches small enough to audit. Never replace the query with {} merely to avoid pagination.

Step 2: Build and Review the Exact Cancel List

The modify-alerts endpoint does not accept an AlertQuery. It requires a uuid-values body whose only required field is the uuids array.

After reviewing the first batch, build the body from its returned IDs:

jq '{uuids: [.alerts[].alertId]}' \
 "${OPS_WORK_DIR}/alert-page-0.json" \
 > "${OPS_WORK_DIR}/alerts-to-cancel.json"

jq . "${OPS_WORK_DIR}/alerts-to-cancel.json"

Record the approved count and retain the reviewed ID list with the change record.

Step 3: Cancel Only Those IDs

Cancel the exact list with POST /suite-api/api/alerts?action=cancel:

curl --fail-with-body \
 --request POST \
 "https://${OPS_FQDN}/suite-api/api/alerts?action=cancel" \
 --header @"${OPS_WORK_DIR}/authorization.header" \
 --header 'Content-Type: application/json' \
 --data-binary @"${OPS_WORK_DIR}/alerts-to-cancel.json" \
 --output "${OPS_WORK_DIR}/alert-cancel-response.json"

jq '[.alerts[] | {alertId, status, cancelTimeUTC}]' \
 "${OPS_WORK_DIR}/alert-cancel-response.json"

Canceling changes alert state; it does not delete the alert record.

Step 4: Build the Targeted Delete Query

The delete endpoint accepts a direct AlertQuery, not an alert-query wrapper. Reuse the reviewed IDs and require CANCELED status:

jq '{
 compositeOperator: "AND",
 alertStatus: ["CANCELED"],
 alertId: .uuids
 }' \
 "${OPS_WORK_DIR}/alerts-to-cancel.json" \
 > "${OPS_WORK_DIR}/alerts-to-delete.json"

jq . "${OPS_WORK_DIR}/alerts-to-delete.json"

Run the same body through the read-only query endpoint first:

curl --fail-with-body \
 --request POST \
 "https://${OPS_FQDN}/suite-api/api/alerts/query?page=0&pageSize=1000" \
 --header @"${OPS_WORK_DIR}/authorization.header" \
 --header 'Content-Type: application/json' \
 --data-binary @"${OPS_WORK_DIR}/alerts-to-delete.json" \
 --output "${OPS_WORK_DIR}/alert-delete-preview.json"

jq '.pageInfo, [.alerts[] | {alertId, alertDefinitionName, resourceId, status}]' \
 "${OPS_WORK_DIR}/alert-delete-preview.json"

Stop if the preview contains an unexpected ID, definition, resource, or status. The preview count should match the reviewed cancel list for that batch.

Step 5: Delete and Verify

Delete the targeted canceled alerts with DELETE /suite-api/api/alerts/bulk:

curl --fail-with-body \
 --request DELETE \
 "https://${OPS_FQDN}/suite-api/api/alerts/bulk" \
 --header @"${OPS_WORK_DIR}/authorization.header" \
 --header 'Content-Type: application/json' \
 --data-binary @"${OPS_WORK_DIR}/alerts-to-delete.json"

The documented success response is HTTP 204 and indicates that the deletion was started. It does not return a deleted-object list. Query the exact IDs again until the API returns no matching alerts, and confirm the expected result in the UI and audit history before closing the change.

Delete the temporary authorization header immediately. Retain or securely remove the reviewed JSON files according to the change-record policy, then remove the directory when it is empty:

rm -f -- "${OPS_WORK_DIR}/authorization.header"
unset OPS_TOKEN
# After retaining or deleting the reviewed JSON files:
rmdir -- "$OPS_WORK_DIR"
unset OPS_WORK_DIR

Requests to Avoid

Do not send an empty {} query to the bulk-delete endpoint.
Do not use only alertStatus: ["CANCELED"] unless the approved scope is explicitly every canceled alert in the environment.
Do not put an AlertQuery inside an alert-query wrapper; the endpoint expects the query fields at the JSON root.
Do not send alert IDs directly to DELETE /suite-api/api/alerts; the documented bulk deletion route is /suite-api/api/alerts/bulk.
Do not assume the first 1,000 results are the complete set.
Do not schedule destructive cleanup until the query, count limits, audit logging, and failure handling have been tested with a small batch.

References

Using a 90-Day Planning Horizon Correctly in VCF Operations

Cosmin Trif — Tue, 25 Aug 2026 00:00:00 +0000

A 90-day capacity goal can describe three different things in VCF Operations: past demand used by the projection, future warning lead time, or the future period considered by VM rightsizing. They are controlled separately.

This article replaces the earlier 90-day capacity procedure, which incorrectly treated Time Remaining thresholds as a historical lookback setting.

The Three Meanings of 90 Days

Planning question	Relevant control	What it does not control
Should the forecast consider up to 90 days of past demand?	Historical Data window, if that value is available in the installed build	Alert severity or procurement lead time
Should an alert become critical when exhaustion is projected within 90 days?	Time Remaining criticality thresholds	Historical lookback
Should Recommended Size cover demand projected across a 90-day future horizon?	The applicable Time Remaining warning/green threshold plus the documented 30-day extension	Past-data retention

Changing one row does not configure the other two.

What the Capacity Engine Actually Does

VCF Operations forecasts demand rather than simply averaging a fixed block of historical utilization. Current Broadcom guidance describes these behaviors:

Recent samples receive more weight through exponential decay.
Conservative risk uses the upper bound of the projection range.
Aggressive risk uses the mean of the upper and lower projection bounds.
Peak-focused mode incorporates peaks detected in historical demand and can be combined with either risk posture.
Business Hours limits the demand samples used for projection to the configured hours; it does not merely give those hours extra weight.
Capacity Remaining is based on a short forward projection, while Time Remaining estimates when demand will intersect usable capacity.
vSphere HA admission-control reservations reduce usable cluster capacity and can therefore change the result even when current utilization appears low.

These settings change the model’s behavior. None of them, by itself, means “use exactly 90 days of history.”

Before Changing a Policy

Identify the active policy on the exact cluster, VM group, or object you are evaluating.
Record the existing Capacity settings and take screenshots for the change record.
Confirm whether the object is using the Demand model, the optional Allocation model, or both. Do not add arbitrary allocation ratios to hide a Demand-based alert.
Check vSphere HA admission control before interpreting usable capacity as missing physical capacity.
Decide which 90-day outcome you actually need: historical evidence, alert lead time, VM sizing horizon, or a documented combination.

Clone an inherited policy when you need a separate production, non-production, cluster, or VM posture. Broadcom recommends separate VM policies when rightsizing settings should differ from higher-level capacity policies.

Goal 1: Use a 90-Day Future Alert Lead Time

If procurement must begin when capacity is projected to run out within a quarter, configure Time Remaining severity thresholds for that operational lead time.

For example, an organization might choose warning at 120 days and critical at 90 days so the earlier warning leaves a 30-day intake buffer. Those numbers are governance choices, not Broadcom defaults. Keep the thresholds in a logical order, document who owns each severity, and verify how the installed version labels warning, immediate, and critical.

This configuration means “alert us based on a future exhaustion date.” It does not mean the forecast reads only the last 90 days.

Goal 2: Use a 90-Day VM Recommended Size Horizon

Broadcom’s current rightsizing explanation states that Recommended Size considers the peak projected demand from now through 30 days beyond the Time Remaining threshold used by the policy. Its example uses a 30-day warning threshold and therefore produces a 60-day sizing horizon.

Under that documented rule, a 90-day Recommended Size horizon corresponds to a 60-day applicable threshold plus the 30-day extension:

60-day policy threshold + 30-day extension = 90-day sizing horizon

Do not set the threshold to 90 days and call the sizing horizon 90 days; that would extend the evaluated period to 120 days under the same rule. Because UI wording can differ by release, verify the resulting projection on a VM’s Capacity tab before applying the policy broadly.

Use a VM-specific policy if clusters need different Time Remaining severities. Select Conservative, Aggressive, and Peak Focused according to workload risk and peak behavior, not merely to force a preferred recommendation.

Goal 3: Consider 90 Days of Historical Demand

The Historical Data setting is the control associated with the model’s past-data window. Broadcom KB 437777 explicitly treats it separately from Time Remaining risk and gives an example of temporarily reducing a 30-day window to one day when an obsolete spike is distorting a forecast.

To request a 90-day historical window:

Go to Infrastructure Operations > Configurations > Policy Definition.
Edit or clone the policy that is active on the intended objects.
Open Capacity and unlock the inherited Historical Data setting if necessary.
Select 90 days only if the installed VCF Operations build exposes 90 days as a supported value.
Save the policy, confirm its object assignment and precedence, and record the effective value.

If the installed build does not offer a 90-day value, do not substitute a 90-day Time Remaining threshold and describe it as a lookback. Use 90-day historical charts or reports as supporting evidence, and confirm the supported forecasting-window choices for that release with Broadcom Support.

Historical-data retention is also separate. Retaining a metric for reporting does not guarantee that every retained sample is used by the current projection.

Correcting Anomalous History

Use the Capacity view’s Reset control when a known change or anomaly makes the current projection unrepresentative. Broadcom documents two relevant behaviors:

Selecting a new projection start date causes forecasting to use data from that point forward.
Selecting a start and end time for exclusion omits that interval from capacity planning and forecasting.

A reset does not delete the underlying historical metric data. Charts, reports, and retained time-series data remain available.

Do not assume an object maintenance schedule is a substitute for this exclusion workflow. Maintenance behavior is designed around monitoring and alert handling; use the capacity engine’s own reset or exclusion controls when the goal is to change forecasting input.

Recalculate and Validate

Capacity forecasting runs periodically, so a policy change may not appear after one five-minute collection cycle. Broadcom KB 437777 describes two supported validation paths:

For a global recalculation, go to Administration > Control Panel > Dynamic Thresholds and start recalculation during an approved window.
For a specific cluster, open its Capacity tab and use Reset only when changing the projection baseline is intended.

Then validate all of the following:

The expected policy is active on the target object.
The effective Historical Data value matches the approved setting.
Time Remaining severity changes at the intended future thresholds.
The Capacity chart shows the expected projection start point and any excluded interval.
VM Recommended Size uses the intended future horizon.
HA reservation and capacity buffers match the actual resilience design.
Demand and Allocation models are interpreted separately, with the more constrained model understood.

Test on a small object group before broad assignment. Keep the before-and-after values with the change record so a surprising forecast can be traced to a specific policy decision.

References

Deploying a VCF 9.1 Fleet with the VCF Installer

Cosmin Trif — Fri, 07 Aug 2026 09:00:00 +0000

Last year, when VCF 9.0 shipped, I covered the VCF Installer in three posts: deploying the appliance, configuring it, and deploying a VCF 9 instance. With VCF 9.1 generally available since May 12, 2026, it was time to run the whole flow again in the lab, and enough has changed that it earns a fresh walkthrough. This post covers the complete path in one go: downloading and deploying the 9.1 installer appliance, connecting the depot, downloading binaries, and deploying a new VCF fleet on top of an existing vCenter. I used the 9.1.0.0400 installer from the July 2026 refresh.

What changed since the 9.0 posts

The overall shape is the same (deploy the OVA, connect a depot, download binaries, run the deployment wizard) but 9.1 moves a few important pieces, per the official What’s New for the VCF Installer and the depot KBs:

The download token is gone. Depot authentication now uses a Download Service ID + Activation Code obtained from the VCF Business Services console. The token flow from my 9.0 configuration post no longer applies. See KB 443647.
VCF Management Services are deployed by default. The installer now provisions the VCF services runtime, fleet lifecycle, identity broker, software depot, Salt RaaS, telemetry, and the centralized License Server as part of a new deployment.
Integrated planning workflow. The wizard generates CPU, memory, storage, VLAN, and FQDN requirements up front and validates your infrastructure against them, including a Pre-fill generated FQDNs button.
Auto-generated passwords. The installer can generate the component passwords for you, retrievable after deployment.
Dual-stack support. The appliance OVA now asks for an IP Address Version (IPv4 only, or IPv4 and IPv6), and existing dual-stack environments are supported, though wizard-driven deployments on an existing vCenter remain IPv4-only; dual-stack goes through a JSON spec.
Lowercase FQDNs are now formalized. The 9.1 planning docs state do not use capital letters in the FQDN, and uppercase component FQDNs are a documented deployment-failure known issue.

Download the VCF Installer

The appliance OVA comes from the Broadcom Support Portal at support.broadcom.com: select the VMware Cloud Foundations division, go to My Downloads, and pick VMware Cloud Foundation 9 with your target release. If the portal navigation fights you, KB 401497 walks through it.

Select the release. I grabbed 9.1.0.0400 from the July 2026 patch wave
Agree to the Terms and Conditions
Download VCF-SDDC-Manager-Appliance-9.1.0.0400.25570100.ova (2.28 GB) and verify the checksum

Deploy the VCF Installer appliance

Sizing is unchanged from 9.0: 4 vCPUs, 16 GB RAM, and 914 GB storage (thin provisioning keeps that honest), per the official deployment page. The docs describe two placement options: on one of the ESX hosts that will form the management domain (the appliance is converted into SDDC Manager during deployment), or outside the management infrastructure entirely, where one appliance can deploy multiple platforms. As in the 9.0 post, I am deploying the OVA through an existing vCenter.

Log in to vCenter with a user that can deploy an OVF
Right click the cluster and click Deploy OVF Template

Select Local file and upload the installer OVA
Click Next

Give the appliance a name
Select a deployment location
Click Next

Select a compute resource and wait for the compatibility checks
Click Next

Review the deployment details, note the 914 GB thick footprint vs 4.2 GB thin
Click Next

Read and accept the license agreements
Click Next

Select a datastore for the deployment
Click Next

Select the destination network: the appliance needs to reach the ESX hosts, the VM management network, DNS, and NTP
Click Next

Fill in the template customization: root and local user passwords (minimum 15 characters with the usual complexity rules; the local password is what you will use for admin@local in the UI and vcf over SSH), hostname, networking, DNS, and NTP
Click Next

This is also where 9.1 differs from 9.0: the networking section now starts with an IP Address Version selector, with IPv6 address, prefix length, and gateway fields if you go dual-stack.

Review the summary
Click Finish and wait for the deployment to complete

Log in and connect the depot

Once the appliance is up, navigate to https://ip_address/ui or https://hostname/ui. Same as 9.0, the services take a few minutes to start after first boot.

For username use admin@local
Enter the local user password from the OVA deployment
Click Log in

The Get Started page opens with the Download Binaries card flagging No Depot Connection.

Click Depot Settings and Binary Management

Both depot types are available: online (direct or via proxy) and offline, fed by the VCF Download Tool, only one can be connected at a time. For air-gapped environments the official offline depot procedure covers the Download Tool flow; William Lam also documented the new HTTP offline depot support in 9.1.

Click Configure under the Online Depot tile

Here is the big change from 9.0: instead of pasting a download token, the dialog shows the appliance’s Download Service ID (the docs and KB call it the Software Depot ID). Take that ID to the BSC portal at vcf.broadcom.com, register it, and you get an Activation Code back; the full flow is in the official depot procedure and KB 443647. The download token from KB 390098 that we used in the 9.0 post is not used here anymore.

Copy the Download Service ID and register it in the BSC portal to obtain the Activation Code
Enter the Activation Code (enable the proxy toggle if you need one)
Click Authenticate

Download the binaries

With the depot connected, the Binary Management section lists everything available for the release.

Confirm the Online Depot tile shows Depot connection active
In Binary Management, select the product and version, VMware Cloud Foundation 9.1.0.0 in my case
Select the binaries and click Download

The component list is a good illustration of how 9.1 patch waves work: the installer pulls the latest patch of each component rather than a single uniform build: SDDC Manager 9.1.0.0400, NSX 9.1.0.0200, VCF Automation 9.1.0.0200, VCF Operations 9.1.0.0400, and so on. William Lam has a good write-up of this default behavior if you need to pin earlier versions. Also note the sizes, VCF services runtime alone is 17.42 GB and VCF Automation 14.95 GB, so give the downloads some time.

Start the Deployment Wizard

Click Return Home: the Deploy card sits next to Download Binaries. Progress in the wizard is saved locally in the browser after each step, so you can safely step away

Click Deployment Wizard and select VMware Cloud Foundation

The dropdown also offers VMware vSphere Foundation: the 9.1 installer deploys VVF through the same wizard framework, with a reduced flow (no NSX Manager or SDDC Manager steps). That is a post for another day; everything below follows the VCF path.

Deploy VMware Cloud Foundation

The wizard runs through four phases: Introduction, Plan, Prepare, Deploy; the official procedure is Start a New VCF Fleet or a New VCF Instance Deployment.

Introduction

The About page is worth an actual read: it diagrams the VCF fleet concept and calls out that VCF 9.1 introduces VCF management services inside the management domain, next to vCenter, NSX Manager, and SDDC Manager. The installer deploys the management domain; workload domains come later from VCF Operations.

Three deployment paths are offered: a new VCF fleet, a new VCF instance joining an existing fleet’s VCF Operations, or deploying deferred components. Same as in my 9.0 fleet post, this lab is a fresh start.

Select Deploy a new VCF fleet
Click Next

Plan

Check I have an existing vCenter instance: the wizard will collect and validate the details later
Click Next

You can also flag an existing VCF Operations 9.1 instance or an NSX-registered vCenter here. VCF Automation is no longer a hard requirement at deployment time; it can be connected or deployed later as a day-N operation.

Select the Deployment model. I went with Simple, which locks the size to Small
Review the component sizing table and click Next

The sizing table is part of the new integrated planning workflow, and it is worth pausing on: VCF management services alone ask for 40 vCPUs, 82 GB RAM, and 3 TB of disk in this smallest configuration.

Review the Network Options: by default VCF Operations, VCF Automation, and the management services land on the VM management network, and a distributed Transit Gateway is configured to make the domain VPC-ready
Click Next (or Customize if you need separate networks)

One caveat straight from the wizard: with an existing vCenter the wizard supports IPv4-only; dual-stack deployments have to go through a JSON spec.

The Review Prerequisites step is the payoff of the planning workflow: capacity totals (2 hosts, 84 vCPUs, 254 GB RAM, 5,244 GB disk for my configuration), VLAN requirements (none for this path), and the exact FQDN and IP requirements: 12 IPs for the management services pool, 5 for VCF Automation, plus FQDNs for Cloud proxy, NSX Manager, VCF Operations, and the License Server. The Pre-fill generated FQDNs in wizard button carries these into the later steps. Remember: lowercase only, forward and reverse resolvable.

Create the DNS records and reserve the IP ranges
Click Next

Prepare

Select the Version: 9.1.0.0, with the latest downloaded patch of each component applied automatically
Name the VCF instance and the management domain
Click Next

The VCF Management step collects the FQDNs for VCF Operations, the License Server, and the fleet, instance, and services-runtime components; this is where the pre-filled FQDNs from the Plan phase save the typing.

Enter the details of the existing vCenter: FQDN, administrator, and root credentials
Click Connect: the cluster running the vCenter appliance is selected as the deployment destination
Click Next

Provide the IP pools, a single IPv4 range works for both: minimum 12 addresses for the VCF Management Services pool and 5 for the VCF Automation pool
Click Next

Enter the NSX Manager cluster and appliance FQDNs
Decide on the overlay networking mode
Click Next

The Configure overlay using ESX Management VMkernel Networking toggle is one of my favorite quality-of-life items in 9.x: NSX 9 can run overlay over the ESX management VMkernel network without a VLAN-backed transport configuration. Unchecked, NSX is configured in VLAN transport zone mode instead.

Enter the SDDC Manager FQDN and the installer appliance password
Click Next

The info banner explains why the appliance password is requested: when the installer appliance runs on one of the management domain hosts, it is converted into the SDDC Manager appliance during deployment: the same appliance transition the official docs describe, and the reason the prerequisites page noted the SDDC Manager FQDN is optional in that placement.

Deploy

Review the summary, every section from the wizard is expandable here
Optionally click Download JSON Spec
Click Next

The JSON preview tab and the downloadable spec make the whole deployment repeatable: the same file drives API-based deployments, and it is the vehicle for the options the wizard does not expose, like dual-stack networking or an alternative internal cluster CIDR.

Wait for the 11 validation checks to complete: deployment specification, security configuration, DNS resolution, versions and bundles, existing SDDC Manager configuration, password policies, network configuration, existing components, and the capacity calculations
Resolve any errors, acknowledge warnings if appropriate, and click Deploy

From here the installer takes over: management services, VCF Operations, NSX, SDDC Manager conversion, and the rest of the management domain roll out in sequence. Depending on hardware, expect this to run for a few hours.

If validation or deployment fails

A few 9.1-specific failure modes worth knowing before you hit Deploy:

“Deploy and configure VCF Management Platform” fails: the management services use an internal cluster CIDR of 198.18.0.0/15 by default; if that overlaps your network, KB 440541 covers switching it to 240.0.0.0/15 or 250.0.0.0/15 via the JSON spec.
“VCF Management Services Deployment Task” fails on NTP: the NTP server must be reachable over UDP 123 from the deployed appliances, not just from the installer: KB 439240.
vMotion / vSAN network connectivity validations fail: the installer does not recognize standby uplinks; set the vmnics active/active or disable the check per KB 437248.
Uppercase FQDNs: deployment fails for fleet components, identity broker, services runtime, and VCF Automation; lowercase everything (known issues).

For nested labs, William Lam maintains a comprehensive set of installer workarounds for 9.1 (pNIC speed, vSAN HCL, and MTU prechecks included. And a neat trick from the same source: the installer doubles as a standalone pre-check tool) you can run the full validation suite against your infrastructure without deploying anything.

References

Official documentation

Broadcom KBs

My VCF 9.0 installer series

Upgrading VCF 9.1 Management Services to 9.1.0.0400 Step by Step

Cosmin Trif — Mon, 03 Aug 2026 10:00:00 +0000

On July 13, 2026 Broadcom shipped the next VCF 9.1 patch wave, and this post walks through the part of it that lives under Build -> Lifecycle -> VCF Management -> Upgrade: the VCF Management services. The headline releases on this surface are VCF Operations 9.1.0.0400 and VCF Automation 9.1.0.0200, mostly security enhancements plus a batch of fixes. This post is the follow-up to my 9.1.0.0300 walkthrough.

To be clear on scope: this post covers only the management services. SDDC Manager 9.1.0.0400 and the core components (vCenter, NSX, ESX, vSAN) go through the VCF Instances flow and are not covered here; see the short section at the end.

One thing to understand up front: patch releases are not synchronized among all VCF components, and you can pick and choose which components to apply. That is why my target list below is a mix of 0400, 0200, and 0100 builds. It looks odd the first time. It is expected.

Versions in This Update

Here is the full From/To picture from my lab, exactly as the VCF Operations UI reports it.

Component	Current Version	Target Version
Fleet Lifecycle	9.1.0.0300.25536195	9.1.0.0400.25570104
VCF Operations	9.1.0.0300.25531073	9.1.0.0400.25541561
VCF Operations for Networks	9.1.0.0100.25427421	9.1.0.0200.25517220
Identity Broker	9.1.0.0.25368698	9.1.0.0100.25522734
VCF Automation	9.1.0.0100.25429499	9.1.0.0200.25556825
Log Management	9.1.0.0100.25435121	9.1.0.0400.25544947
SDDC Lifecycle	9.1.0.0300.25536194	9.1.0.0400.25570103
Real-Time Metrics	9.1.0.0100.25435036	9.1.0.0400.25544944
Real-Time Metrics Store	9.1.0.0.25370367	9.1.0.0200.25555874
Salt Master	9.1.0.0100.25434834	9.1.0.0400.25544946
Salt RaaS	9.1.0.0100.25434834	9.1.0.0400.25544946
Software Depot	9.1.0.0.25371105	9.1.0.0400.25570105
Migration Service Engine	9.1.0.0.25370929	9.1.0.0200.25556825
VCF Services Runtime	9.1.0.0.25370367	9.1.0.0200.25555874

SDDC Manager 9.1.0.0400 (build 25570100) ships in the same wave but lives outside this component list. See the scope section at the end of the post.

Before You Start

A few rules before clicking anything. Back up your components before patching, and remember only one upgrade or patch operation can run at a time. That single-operation rule matters more than it sounds with fourteen components in the queue, plan for a long session or spread it over a maintenance window. If your depot is online, the binaries pull directly. If you run offline, update the VCF Download Tool to 9.1.0.0400 (build 25550178) first and download the binaries with it.

Two known issues shape how I ran this:

SDDC Lifecycle 9.1.0.0400: the batch precheck can fail with VCF_LCM_FLEET_UPGRADE_400_COMPONENT_NOT_NEEDED_USER_INPUT_SPEC_PROVIDED. Do not use Run Prechecks (ALL). Precheck each component individually.
Software Depot 9.1.0.0400: patching the depot takes it temporarily offline. Do not run any other component patch at the same time, and retry anything that failed afterwards.

Upgrade Fleet Lifecycle

Broadcom’s ordering guidance for maintenance releases starts with Fleet Lifecycle. For patch releases the order is flexible, but I still go Fleet Lifecycle first; it drives patch deployment for the rest of the fleet components.

In VCF Operations, go to Build -> Lifecycle -> VCF Management -> Upgrade. Open the “…” menu on the Fleet Lifecycle card and click Select version.

The Fleet Lifecycle Component Target Version dialog opens. Set the target to 9.1.0.0400.25570104.

Back on the card, the upgrade path now reads 9.1.0.0300.25536195 to 9.1.0.0400.25570104. Click Upgrade.

Kick off the upgrade. The card flips to Upgrade in progress.

Open Upgrade details. The first two subtasks are “Set Fleet lifecycle upgrade context” and “Stage Fleet lifecycle plugin in VCF services runtime”.

The status messages move through Staging binaries and Running Component Stage Prechecks. Nothing to do here but watch.

All 7 subtasks land on Completed. In this run the whole thing took about 45 minutes, 11:08 to 11:54 AM.

Set the VCF Management Target Version

With Fleet Lifecycle done, click CHANGE TARGET VERSION.

The Set Target Version dialog offers the VCF global version 9.1.0.*.

The component list now shows everything Ready for upgrade, with the mixed 0400/0200/0100 targets from the table above. This is the per-component patch numbering at work, not a depot problem.

Because of the SDDC Lifecycle known issue, I did not touch RUN PRECHECKS (ALL). I ran prechecks per component instead, right before each upgrade. It costs a few extra clicks per component, but it avoids the VCF_LCM_FLEET_UPGRADE_400_COMPONENT_NOT_NEEDED_USER_INPUT_SPEC_PROVIDED failure entirely.

Upgrade VCF Operations

First up, VCF Operations itself to 9.1.0.0400.25541561. Note that the cloud proxy and the license server are patched automatically as part of this one, no separate steps needed for either.

The workflow runs through its subtasks; I moved on once it was In Progress.

Upgrade Identity Broker

Next, Identity Broker to 9.1.0.0100.25522734.

This one is a 6-subtask workflow.

Upgrade VCF Automation

VCF Automation goes to 9.1.0.0200.25556825.

The precheck workflow runs first, then the upgrade takes over.

Upgrade VCF Operations for Networks

VCF Operations for Networks moves to 9.1.0.0200.25517220.

Upgrade Log Management

Log Management targets 9.1.0.0400.25544947.

Same pattern: precheck, then the upgrade workflow.

Upgrade SDDC Lifecycle

SDDC Lifecycle goes to 9.1.0.0400.25570103: the component behind the batch precheck known issue, upgraded here with its own individual precheck.

Upgrade Real-Time Metrics and Metrics Store

Real-Time Metrics targets 9.1.0.0400.25544944.

The Real-Time Metrics row switches to Upgrade in progress.

Then the upgrade workflow itself.

The Real-Time Metrics Store is a separate component; it targets 9.1.0.0200.25555874.

The Real-Time Metrics Store row switches to Upgrade in progress.

And the upgrade workflow follows.

Upgrade Salt Master

Salt Master targets 9.1.0.0400.25544946.

The Salt Master row switches to Upgrade in progress.

The upgrade workflow takes over and works through its subtasks.

Upgrade Software Depot

Software Depot goes to 9.1.0.0400.25570105. Per the known issue in the release notes, the depot goes briefly offline during its own patch, so avoid launching other component patches at the same time and retry anything that failed afterwards.

The Software Depot row switches to Upgrade in progress.

Then the upgrade workflow, while the depot is briefly unavailable.

Upgrade Migration Service Engine

Migration Service Engine targets 9.1.0.0200.25556825.

The Migration Service Engine row switches to Upgrade in progress.

Fun detail in the workflow panel: the staging task exposes the internal component name VCD_MIGRATOR: “Stage VCD_MIGRATOR plugin in VCF services runtime”.

Upgrade VCF Services Runtime

VCF Services Runtime goes to 9.1.0.0200.25555874.

The VCF Services Runtime row switches to Upgrade in progress.

Then the upgrade workflow with its subtasks.

Upgrade Salt RaaS

Salt RaaS ran last in my lab, targeting 9.1.0.0400.25544946.

At the time of this capture the precheck workflow was still running, “Run Salt RaaS upgrade prechecks” shows In Progress.

What About SDDC Manager and the Core Components?

They are out of scope for this post. SDDC Manager 9.1.0.0400 (build 25570100) and the core components from this wave (ESX 9.1.0.0200, vSAN 9.1.0.0200, NSX 9.1.0.0200, and vCenter 9.1.0.0200, later bumped to 9.1.0.0300 on July 29, 2026) are applied through Build -> Lifecycle -> VCF Instances. That is the same flow I documented in Upgrading VCF 9.1 to 9.1.0.0300 Step by Step.

Validate the Update

Before calling it done:

Check the Component Versions page, every patched management component should show On Target.
Check the task list, no failed workflows left behind.
Confirm the VCF Operations UI is reachable and healthy.
Retry anything that failed while the Software Depot was being patched; the depot was offline during that window.

For SDDC Manager and the core components, see the previous section, the flow in Upgrading VCF 9.1 to 9.1.0.0300 Step by Step still applies one-to-one.

Fixing Swagger’s localhost:8600 URL in VCF Automation 9.1

Cosmin Trif — Fri, 31 Jul 2026 00:00:00 +0000

While putting together the VCF Automation 9.1 API getting started post, I spent some time in the in-product API Help Center. Log into the VM Apps tenant, click your username, API Help Center > Explore Automation APIs. The Swagger pages render fine, but the moment you try an actual call, it fails. The Servers box for the VCF Automation (Blueprint) API shows:

http://localhost:8600

Swagger UI builds Try It Out requests from servers[0].url, so the browser dutifully sends every request to port 8600 on your own workstation. Connection refused, every time. The documentation is correct; the advertised host is not.

This post is about finding where that value actually comes from (which took three wrong turns through the appliance’s Kubernetes internals) and the override that finally fixed it without touching a container image. The short version:

The localhost:8600 value is not in any configuration on the appliance. A grep across a full support-bundle cluster dump finds no ConfigMap, Helm value, gateway route, or forwarded-headers setting that produces it.
The spec is a static JSON file generated by springdoc at build time (the giveaway is its "description": "Generated server url") and it ships as a classpath resource inside the Spring Boot application in the tango-blueprint-service-app pod. Whoever built it had the service listening on port 8600, and the URL froze there.
The api-docs page itself is served by a different pod than the spec, which is why the obvious nginx-level fixes go nowhere.
The fix: mount a corrected copy of the spec from a ConfigMap and prepend file:/override/ to spring.web.resources.static-locations via JAVA_OPTS, so Spring serves the fixed file and never falls through to the copy shipped in the image. Survives pod restarts with nothing to reconcile it away; a product upgrade will wipe it.

Everything below was done on a lab appliance running build 9.1.0.0100.25429499. This is an unsupported modification. More on that at the end.

The Symptom

In the tenant API Help Center, open the VCF Automation API (the Blueprint one, version 2019-09-12). The spec renders, the paths are all correct, but the servers dropdown offers exactly one entry: http://localhost:8600. There is no way to type a different URL: the dropdown only lists what the spec declares, and this spec declares one hardcoded server with no variables.

Browser DevTools (Network tab) shows what the page actually loads. First an index of every API the Help Center knows about:

/tenant/<your-tenant>/api-docs/assets/content/soft/automation-apis.json

That index maps each tile to a spec URL: about sixteen specs across a dozen backend services. The Blueprint entry points at:

/blueprint/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json

And that document contains the problem, verbatim:

"servers": [
 {
 "url": "http://localhost:8600",
 "description": "Generated server url"
 }
]

Two details in that response are worth pausing on, because they shaped the whole investigation:

"Generated server url" is the literal default description springdoc attaches when it auto-computes the server URL from the incoming request. So springdoc generated this, but if it were computed per request, it would show the appliance FQDN, not localhost.
The response carries a last-modified date from months before I ever opened the page, plus accept-ranges: bytes. That is a static file being read off disk (or classpath), not a document rendered per request.

Put together: someone ran the Blueprint service on a build machine, where it listened on localhost:8600, exported the generated spec to a file, and shipped that file in the product. The URL froze at build time.

Ruling Out Configuration

Before touching anything, I wanted to know whether some appliance setting produces or rewrites that value, because if it did, that would be the supported place to fix it.

It does not. I had a full support bundle from this appliance handy (that is a story for another post), and the extracted cluster object dump is conclusive:

The string 8600 appears nowhere in the cluster as a port, URL, or config value. No Service exposes it, no container declares it, no ConfigMap mentions it. The only matches are UID substrings.
There is no springdoc, server.forward-headers-strategy, or server-URL configuration in any ConfigMap in the prelude or vmsp-platform namespaces.
The Envoy gateway routes (Gateway API HTTPRoute objects) that publish the api-docs paths perform path rewrites only: nothing rewrites hosts or response bodies.

So there is no knob. The value lives in content, not configuration, and only Broadcom can fix it properly. Everything from here on is about making the appliance serve a corrected copy.

Wrong Turn #1: The Pod That Serves the Page Doesn’t Serve the Spec

The api-docs page shell (the Angular app with the tiles) is served by the cloud-automation-ui-app pod in the prelude namespace. Its nginx config (ConfigMap cloud-automation-ui) rewrites /tenant/<org>/api-docs and /provider/api-docs into static content under /usr/share/nginx/html/dist. My first instinct was an nginx sub_filter there, rewriting localhost:8600 in responses.

Dead end, for a structural reason: the spec never passes through that pod. The gateway routes the two URLs to different backends entirely:

Path	HTTPRoute	Backend
`/tenant/<org>/api-docs/...` (the page, the index JSON)	`tenant-in-path-automation-ui`	`cloud-automation-ui`
`/blueprint/api-docs/...` (the spec)	`blueprintapi-docs`	`tango-blueprint:8080`, prefix rewritten to `/api-docs/`

A sub_filter in the UI pod would rewrite a response it never sees. The actual owner of the spec is the Blueprint service itself: Deployment/tango-blueprint-service-app, image blueprint-webapp:9.1.0.0100.25429499: a Spring Boot app, which also explains the Spring Security headers on the spec response.

Wrong Turn #2: There Is No File to Edit

Fine, exec into the right pod and sed the file in place:

kubectl -n prelude exec deploy/tango-blueprint-service-app -- \
 sh -c 'find / -name "api-doc-public-classic-vra-*.json" -not -path "/proc/*" 2>/dev/null'

Nothing. And looking for the application itself:

kubectl -n prelude exec deploy/tango-blueprint-service-app -- \
 sh -c 'find / -maxdepth 3 -name "*.jar" -not -path "/proc/*" 2>/dev/null | head'

/opt/bc-fips/bcutil-fips-2.1.5.jar
/opt/bc-fips/bctls-fips-2.1.22.jar
/opt/bc-fips/bcpkix-fips-2.1.10.jar
/opt/bc-fips/bc-fips-2.1.2.jar
/opt/bc-fips/bcmail-fips-2.1.6.jar
/jdk/lib/jrt-fs.jar

Only the Bouncy Castle FIPS libraries and the JDK, the shallow search doesn’t even turn up the application archive itself. Wherever the application lives, the spec is packaged inside it as a classpath resource: find sees nothing, and sed has nothing to reach. Even if you dug out the archive, patched it, and re-packed it in the running container, the fix would be pointless: the JVM holds it open, so a restart is needed to pick up the change, and the restart is exactly what rebuilds the container filesystem from the pristine image. The fix would erase itself at the moment of application.

Wrong Turn #3: kubectl edit Ate My JAVA_OPTS

The mechanism that does work is Spring’s static resource resolution. Spring Boot serves static content from an ordered list of locations, and the first location that contains a matching file wins:

spring.web.resources.static-locations =
 classpath:/META-INF/resources/, classpath:/resources/, classpath:/static/, classpath:/public/

Setting that property replaces the default list. So if I prepend a filesystem directory and re-append the defaults, a file I control shadows the identically-pathed resource on the application classpath, and everything else keeps being served from the classpath exactly as before.

The path arithmetic works out neatly. After the gateway’s prefix rewrite, the backend sees the request as /api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json. With file:/override/ first in the list, Spring checks /override/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json before falling through to the classpath. That is exactly where a ConfigMap mount can put a corrected copy.

The wrong-turn part: my first attempt applied all of this with kubectl edit deploy. The volume and volumeMount saved; the JAVA_OPTS append silently did not. That env value is one enormous line that already contains $(SERVICE_ACCOUNT_SECRET) (literal Kubernetes variable-expansion syntax) and somewhere in the editor round-trip the appended text was lost; I never pinned down exactly where. (The same string is also a shell command-substitution landmine if you ever round-trip the value through a shell, which is why the fix below patches programmatically.) The pod rolled, the override directory appeared, and the spec still said localhost:8600 because the JVM never got the property. Verify the property reached the Deployment before concluding the approach failed.

The Fix

All commands run as root on the appliance, where kubectl is already configured. Replace vcfa.domain.com with your appliance FQDN.

1. Produce the corrected spec. No spelunking inside the image needed, pull it through the gateway and fix the one string:

curl -sk https://vcfa.domain.com/blueprint/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json \
 | sed 's|http://localhost:8600|https://vcfa.domain.com|g' \
 > /root/api-doc-public-classic-vra-2019-09-12.json

Sanity-check it: zero remaining hits, and still valid JSON:

grep -c localhost:8600 /root/api-doc-public-classic-vra-2019-09-12.json
python3 -m json.tool /root/api-doc-public-classic-vra-2019-09-12.json > /dev/null && echo "valid json"

2. Create the ConfigMap and back up the Deployment:

kubectl -n prelude create configmap blueprint-apidocs-override \
 --from-file=/root/api-doc-public-classic-vra-2019-09-12.json

kubectl -n prelude get deploy tango-blueprint-service-app -o yaml \
 > /root/tango-blueprint-deploy.backup.yaml

3. Add the volume and mount. A strategic merge patch merges named list entries, so this adds without disturbing the existing volumes and mounts:

kubectl -n prelude patch deploy tango-blueprint-service-app --type=strategic -p '
spec:
 template:
 spec:
 volumes:
 - name: apidocs-override
 configMap:
 name: blueprint-apidocs-override
 containers:
 - name: tango-blueprint-service-app
 volumeMounts:
 - name: apidocs-override
 mountPath: /override/api-docs/classic-apis
'

4. Append the properties to JAVA_OPTS (programmatically. This reads the current value, appends both the current (spring.web.resources.static-locations) and the pre-2.4 (spring.resources.static-locations) property names) whichever one this Spring Boot generation ignores does no harm, and writes a JSON patch. It refuses to double-apply:

kubectl -n prelude get deploy tango-blueprint-service-app -o json > /root/d.json

python3 - <<'EOF'
import json
d = json.load(open("/root/d.json"))
env = d["spec"]["template"]["spec"]["containers"][0]["env"]
i = [n for n, e in enumerate(env) if e["name"] == "JAVA_OPTS"][0]
loc = "file:/override/,classpath:/META-INF/resources/,classpath:/resources/,classpath:/static/,classpath:/public/"
add = " -Dspring.web.resources.static-locations=" + loc + " -Dspring.resources.static-locations=" + loc
if "static-locations" in env[i]["value"]:
 json.dump([], open("/root/patch.json", "w"))
 print("already present - nothing to do"); raise SystemExit
json.dump([{"op": "replace",
 "path": "/spec/template/spec/containers/0/env/%d/value" % i,
 "value": env[i]["value"] + add}],
 open("/root/patch.json", "w"))
print("patch written for env index", i)
EOF

kubectl -n prelude patch deploy tango-blueprint-service-app --type=json --patch-file=/root/patch.json

5. Wait for the rollout. The Deployment uses the Recreate strategy with a single replica, so the Blueprint service is briefly down while the pod is replaced:

kubectl -n prelude rollout status deploy/tango-blueprint-service-app --timeout=180s

Verification

Learned from wrong turn #3: verify each layer, not just the end result.

The property is on the Deployment:

kubectl -n prelude get deploy tango-blueprint-service-app \
 -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="JAVA_OPTS")].value}' \
 | tr ' ' '\n' | grep -i static

-Dspring.web.resources.static-locations=file:/override/,classpath:/META-INF/resources/,...
-Dspring.resources.static-locations=file:/override/,classpath:/META-INF/resources/,...

The override file is mounted in the new pod:

kubectl -n prelude exec deploy/tango-blueprint-service-app -c tango-blueprint-service-app -- \
 ls -la /override/api-docs/classic-apis/

And the actual test, the spec as served through the gateway:

curl -sk https://vcfa.domain.com/blueprint/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json \
 | grep -A2 '"servers"'

 "servers": [
 {
 "url": "https://vcfa.domain.com",

Hard-reload the API Help Center page (the SPA caches aggressively) and the Servers box shows the appliance FQDN. Try It Out now sends requests where they belong.

What This Does and Does Not Fix

Worth being precise here, because the Help Center is bigger than one spec:

Fixed: the 2019-09-12 Blueprint spec shown above. Its 2019-01-15 sibling from the same /blueprint/api-docs/classic-apis/ directory - if it shows the same URL - is the same fix: repeat step 1 for it, then rebuild the ConfigMap with both files: kubectl -n prelude create configmap blueprint-apidocs-override --from-file=/root/api-doc-public-classic-vra-2019-09-12.json --from-file=/root/api-doc-public-classic-vra-2019-01-15.json --dry-run=client -o yaml | kubectl -n prelude replace -f -. The mounted volume picks up the change on its own within a minute or so; no restart needed.
Same pattern, different pod: the catalog, deployment, and policies tiles point at api-doc-public-classic-vra-*.yaml files under /catalog/api-docs/classic-apis/ - the same baked-file shape, owned by the catalog service. If those show the same URL, it is the same fix against that Deployment.
Possibly not broken at all: several tiles point at live springdoc endpoints (/identity/doc/v3/api-docs, /iaas-api/swagger/v3/api-docs/..., /project-service/api/swagger/v3/api-docs/...). Those generate their spec per request, so check before assuming. A quick curl ... | jq .servers against each URL from the index tells you which ones need attention.

Caveats

This is an unsupported modification to a Broadcom appliance, and it behaves accordingly:

It survives pod restarts, and nothing reconciles it away. The fix lives in the Deployment spec, so every pod the Deployment creates from now on gets the mount and the property - the verified state above is itself a freshly recreated pod. The tango-blueprint-service Helm release is deployed once at install/upgrade time with nothing reconciling it continuously, so it should come back intact after an appliance reboot as well; re-check the spec once after your next reboot to confirm. (Do not assume this generalizes: several other components in the same namespace are Flux-managed and would revert an edit within minutes.)
A product upgrade will wipe it. The LCM re-renders the Deployment from the new chart. Expect to re-apply, and re-check after any lifecycle operation.
Roll back at any time with the backup from step 2. Not with kubectl apply, though - the backup carries the pre-patch resourceVersion (so the apply is rejected as a conflict), and client-side apply against a Helm-created object has no last-applied-configuration to prune from, so the added volume would survive the merge and the next pod would wedge on the then-deleted ConfigMap. kubectl replace does a full PUT and removes the volume, mount, and JAVA_OPTS additions in one shot; strip the resourceVersion so the PUT is unconditional, and delete the ConfigMap only after the Deployment no longer references it:

sed '/^ resourceVersion:/d' /root/tango-blueprint-deploy.backup.yaml \
 | kubectl -n prelude replace -f -
kubectl -n prelude rollout status deploy/tango-blueprint-service-app --timeout=180s
kubectl -n prelude delete configmap blueprint-apidocs-override

File an SR anyway. As of this writing there is no KB and nothing in the 9.1 release notes for this - the proper fix has to ship in the product, and that only happens if it gets reported. Reference the build number and the "Generated server url" value in the spec.

If you only need a working Try It Out for yourself and would rather not touch the appliance at all, there is a zero-footprint alternative: download the spec, fix servers[0].url in the copy, and import it into Postman; it seeds the collection’s baseUrl from the spec, and auth works as described in the getting started post.

Final Notes

The frustrating part of this bug is how little is actually wrong: every path in the spec is correct, authentication is correct, the API itself works. One string, frozen at build time on some build machine where the Blueprint service happened to listen on port 8600, breaks the one feature the page exists to provide.

The investigation pattern is the reusable part. When a value appears in a rendered page and nothing in the configuration produces it, stop grepping ConfigMaps and follow the bytes instead: DevTools tells you the exact URL, the response headers tell you whether it is static or generated, and the gateway routes tell you which pod actually owns it. In this case each of those three steps invalidated an otherwise-reasonable fix (the nginx rewrite targeted the wrong pod, the in-pod sed targeted a file that does not exist, and the editor-based patch failed silently) before the resource-shadowing approach landed on the one layer where the platform genuinely offers an override.

References

VCF Automation 9.1 API Getting Started: authentication against the same appliance, and where the API docs moved in 9.1
Broadcom TechDocs - What Are the Automation APIs and How Do I Use Them (VCF 9.1)
VM Apps Org APIs on the Broadcom Developer Portal, the hosted alternative to the in-product Help Center
Spring Boot Reference - Serving Static Content, the static-locations resolution order this fix relies on
springdoc-openapi FAQ; how the “Generated server url” value is computed, and why it goes stale behind a proxy

VCF Automation 9.1 API Getting Started

Cosmin Trif — Thu, 30 Jul 2026 07:00:00 +0000

Back in 2022 I wrote vRA 8 API getting started, two curl calls and you had a bearer token. Since then the product has been renamed twice (vRealize Automation → VMware Aria Automation → VCF Automation) and, more importantly for anyone with scripts in production, the authentication flow changed in VCF 9.x. Neither of the two calls from that post works against a 9.1 appliance.

This is the 9.1 version of the same post: the minimum you need to do before you can actually query the API, with curl.

What Changed Since vRA 8

The short version, before the steps:

	vRA 8.x	VCF Automation 9.1 (VM Apps org)
Refresh token	`POST /csp/gateway/am/api/login?access_token` with username/password	Generated in the UI (My Account > API Tokens), no programmatic call
Bearer token	`POST /iaas/api/login` with the refresh token	`POST /tm/oauth/tenant/{tenant}/token`, standard OAuth `grant_type=refresh_token`
Bearer lifetime	~8 hours	1 hour
Refresh token lifetime	90 days	90 days by default (configurable, at creation time only)
API docs	`https://vra_url/automation-ui/api-docs/`	In-product API Help Center, plus developer.broadcom.com

The two things that break existing scripts:

You cannot get a refresh token with username and password anymore. The docs are explicit: “As of VCF 9.0, the Identity Service API functionality has changed and is no longer used to obtain a refresh token programmatically.” The refresh token is now an API token you generate once in the UI and treat like a credential.
There is a tenant in the URL now. VCF Automation 9 has a provider/tenant model. Everything that used to be vRA lives inside a VM Apps organization, and the token endpoint is scoped to that org name, so step zero is finding out what yours is called.

Step 0: Find Your VM Apps Tenant Name

Log into the Provider Management Portal (https://vcfa.domain.com, the admin account) and go to Organizations. Note two things:

The name of your VM Apps organization, if you upgraded from Aria Automation 8.x, this is the classic tenant that was created during migration (the docs use vcf-e2e-vidm as an example).
The FQDN of the appliance itself.

You need both for every call that follows.

Step 1: Generate an API Token (the New Refresh Token)

This is the part that moved from curl to the UI:

Open the VCF Automation UI at https://vcfa.domain.com and switch the organization to your VM Apps tenant.
Log in with a tenant account (it needs to be an org member with at least the user service role, an org_owner works fine).
Click your name in the top-right corner and select My Account.
Go to the API Tokens tab, click New, give the token a name, and click Create.
Copy the token; it is shown once.

By default the token lives for 129,600 minutes (90 days). If you want a different lifespan, set it before creating the token under General Settings > Timeouts in the Provider Management Portal (0 means unlimited); the lifetime is baked in at creation and cannot be changed retroactively. To kill a token early, revoke it from the same UI or via DELETE /cloudapi/1.0.0/tokens/urn:vcloud:token:<token_id>.

Step 2: Exchange the API Token for an Access Token

This replaces the old POST /iaas/api/login. The endpoint is a standard OAuth token endpoint, scoped to your tenant:

curl -k --location --request POST 'https://vcfa.domain.com/tm/oauth/tenant/your-vm-apps-tenant/token' \
 --header 'Content-Type: application/x-www-form-urlencoded' \
 --header 'Accept: application/json' \
 --data-urlencode 'grant_type=refresh_token' \
 --data-urlencode 'refresh_token=YOUR_API_TOKEN'

The response contains the bearer token:

{
 "access_token": "eyJhbGciOi...",
 "token_type": "Bearer",
 "expires_in": 3600,
 "refresh_token": "YOUR_API_TOKEN"
}

The access token expires after one hour, noticeably shorter than the ~8 hours you got from /iaas/api/login in 8.x, so long-running scripts should be prepared to re-run the exchange.

For scripting, the whole thing collapses into one line with jq:

ACCESS_TOKEN=$(curl -sk --request POST "https://$VCFA_HOST/tm/oauth/tenant/$VCFA_TENANT/token" \
 -H 'Content-Type: application/x-www-form-urlencoded' \
 -H 'Accept: application/json' \
 --data-urlencode 'grant_type=refresh_token' \
 --data-urlencode "refresh_token=$VCFA_API_TOKEN" | jq -r .access_token)

Step 3: Use It

The good news: once you have the access token, the API surface you know from vRA 8 is still there. The 9.1 programming guide tutorials use the same service endpoints (/iaas/api, /catalog/api, /deployment/api, /blueprint/api, /abx, the Orchestrator gateway) with a plain bearer header:

curl -sk "https://$VCFA_HOST/iaas/api/projects" \
 -H "Authorization: Bearer $ACCESS_TOKEN" | jq .

If a call comes back 403, verify what the account actually is inside the org. The docs still expose the CSP gateway for this (note the csp-auth-token header instead of Authorization):

# your org id
curl -sk "https://$VCFA_HOST/csp/gateway/am/api/loggedin/user/orgs" \
 -H "csp-auth-token: $ACCESS_TOKEN" | jq .

# org role: must be org_owner or org_member
curl -sk "https://$VCFA_HOST/csp/gateway/am/api/loggedin/user/orgs/$ORG_ID/roles" \
 -H "csp-auth-token: $ACCESS_TOKEN" | jq .

# service roles: must include at least "user"
curl -sk "https://$VCFA_HOST/csp/gateway/am/api/loggedin/user/orgs/$ORG_ID/service-roles" \
 -H "csp-auth-token: $ACCESS_TOKEN" | jq .

One caveat from Broadcom: scripts that authenticate against new VM Apps organizations (created in 9.x, not upgraded from 8.x) should use the VCF Automation Provider Management Access Control APIs rather than the CSP APIs, treat the CSP gateway as a compatibility surface, not a foundation for new tooling.

Where the API Docs Moved

The old in-product Swagger at https://vra_url/automation-ui/api-docs/ and the developer.vmware.com portal are both gone. In 9.1:

In-product: log into the VM Apps tenant, click your username in the top-right corner, and open API Help Center > Explore Automation APIs.
Online: the VM Apps Org APIs on the Broadcom Developer Portal.

Also worth knowing before you port scripts from 8.x: the Pipelines/Code Stream APIs (/codestream/api/*), the CMX Kubernetes APIs, and the Migration Assessment APIs are removed in 9.1.

A Note on All Apps Organizations

Everything above covers the VM Apps organization, the classic vRA-shaped experience. The new All Apps side of VCF Automation 9 (the Cloud Director-heritage Tenant Manager) authenticates differently again: provider tokens come from POST /oauth/provider/token, and there is a session endpoint at /cloudapi/1.0.0/sessions/provider that does accept basic auth and returns the bearer in the x-vmware-vcloud-access-token response header. That API family deserves its own post; Christian Ferber has a good writeup linked below in the meantime.

References

VCF Operations 9.1 Patch Stuck at “Installing PAK File”

Cosmin Trif — Wed, 29 Jul 2026 17:30:00 +0000

The 9.1.0.0300 patch went through this lab without drama at the beginning of July. Three weeks later, the 9.1.0.0400 express patch for VCF Operations hit the same single-node appliance and stopped dead at Installing PAK file: no progress, no error, and a reboot later, still nothing. What followed was a proper archaeology dig through every layer of state the upgrade machinery keeps, and almost every layer turned out to be holding a different version of the truth.

The short version:

The upgrade died two seconds into the OS update phase because the glob over /etc/yum.repos.d/* matched nothing (the directory had been emptied (or removed; the error is identical either way)) and sed exiting 2 on it is fatal to updateCoordinator.py.
The failure was recorded in pakManager’s results file, and from then on a retry guard rejected every new attempt in milliseconds, while CaSA kept retrying roughly every 11–12 seconds, over 1,100 times in the retained logs alone.
The UI said Installing the whole time because CaSA’s own pak history had the APPLY_OS_UPDATE operation with a null completed_time: stale state, not activity.
Rebooting did not retry anything; it just rebuilt the same wedge, with the cluster still offline because offline is persisted state.
The recovery attempts then failed in two brand-new ways: a 409 pak_installation_active raised when the precheck’s state check collided with the assessment install the precheck itself had just launched, and a NullPointerException when LCM asked CaSA for an upgrade task that no longer existed anywhere.

Versions in play: VCF Operations 9.1.0.0300.25531073 upgrading to 9.1.0.0400.25541550, on a single-node deployment, driven by the express patch workflow in fleet management. All timestamps below are UTC, straight from the logs.

The Symptom: Installing PAK File, Forever

The express patch staged cleanly and started applying, then sat at Installing PAK file for hours. A reboot later it still showed the same thing. Time to stop watching the progress bar and go look at what the appliance was actually doing.

The places that matter for any PAK-based upgrade on a VCF Operations node:

What	Where
pakManager action logs	`/storage/log/vcops/log/pakManager/vcopsPakManager.root.<action>.log`
Per-PAK script stdout/stderr	`/storage/log/vcops/log/pakManager/<pak-id>/`
PAK working state + results	`/storage/db/pakRepoLocal/<pak-id>/<pak-id>.results`
pakManager action history	`/storage/db/pakRepoLocal/vcopsPakManagerCommonHistory.json`
CaSA logs + per-PAK history	`/storage/log/vcops/log/casa/`
Staged PAK binaries	`/storage/db/casa/pak/dist_pak_files/VA_LINUX/`

The PAK for this patch is vRealizeOperationsManagerEnterprise-910040025541550: the ID is the version string with the dots removed, 9.1.0.0400.25541550.

Finding the Loop

The first thing that stood out was not an error message but a rotation pattern. vcopsPakManager.root.apply_os_update.log keeps twenty rotated files of 1 MB each, and every one of them was less than four hours old; the log was burning a full rotation roughly every 11 minutes. Something was running apply_os_update constantly.

Each attempt ended the same way, in milliseconds (log prefixes trimmed for width here and throughout):

[VerifyActionNotRunningOrRun] - apply_os_update result: "Failed"
[VerifyActionNotRunningOrRun] - apply_os_update exit code: "1"
[HandleError] - Exiting with exit code: 1, message: The action: apply_os_update
on the pak file: "vRealizeOperationsManagerEnterprise-910040025541550" previously
failed, cleanup the system and remove_history before attempting again--exiting

That is pakManager’s retry guard: once an action has Failed in the PAK’s .results file, every subsequent invocation is rejected before any script runs. And CaSA (the admin/cluster service that orchestrates PAK operations) was invoking it again roughly every 11–12 seconds, each time via:

/usr/bin/sudo -n /usr/lib/vmware-python-3/bin/python
 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py
 --action apply_os_update --pak vRealizeOperationsManagerEnterprise-910040025541550
 --json --force_content_update true --roles ADMIN,DATA,UI

The retained logs held 1,125 rejections spanning just under four hours, and CaSA’s logs show the loop reaching back to the previous evening. An install loop where every iteration is refused on arrival: the upgrade could not move forward and would never stop trying. So the interesting question became: what failed the first time?

The Root Cause: One sed Against an Empty Directory

CaSA’s operation history gives the setup, and the per-PAK stderr log (/storage/log/vcops/log/pakManager/vRealizeOperationsManagerEnterprise-910040025541550/apply_os_update_stderr.log) gives the kill. The timeline on July 28:

17:42:27 → 17:42:33: validation runs and passes
17:42:45 → 17:44:29: BRING_CLUSTER_OFFLINE completes
17:44:37: APPLY_OS_UPDATE starts
17:44:39: the very first command of the VA upgrade phase runs, and dies:

[b2b - updateCoordinator - os_upgrade] - Entering VA upgrade phase
[b2b - updateCoordinator - os_upgrade] - Disable existing repositories
[b2b - updateCoordinator - runScript] - Script command:
 "/usr/bin/sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/*"
[b2b - updateCoordinator - runScript] - stderr:
 "/usr/bin/sed: can't read /etc/yum.repos.d/*: No such file or directory"
[b2b - updateCoordinator - runScript] - exit code: 2
[b2b - updateCoordinator - programExit] - Unable to disable existing repositories for updates.

The glob over /etc/yum.repos.d/* matched nothing: the shell passed the literal string through, sed could not open it, and exited 2. That error is byte-identical whether the directory is empty or missing outright; by the time anyone could look, the files were back, so all the log proves is that there was nothing there to match. updateCoordinator.py treats a non-zero exit here as fatal, reasonably, since it is about to drive tdnf package operations and wants external repositories disabled first.

Two details make this sting:

The upgrade does not need working external repos at all. The very next steps in a healthy run copy the PAK’s own local-pak-iso.repo into that directory and install everything from the ISO embedded in the PAK. The sed only needs the glob to match at least one file. A directory with nothing in it is the one state it cannot survive.

This exact command succeeded three weeks earlier. The 0300 patch on July 5 logged the identical sed with an empty stderr and exit 0, then dropped local-pak-iso.repo in place and carried on. So the directory was populated on July 5 and held nothing the glob could match by July 28, something removed those files in between. More on the suspects at the end.

One more log-reading trap: the failure looks like it took ten minutes, because after the fatal error updateCoordinator.py spends from 17:44:39 to 17:54:47 generating a support bundle before it exits. The Failed result lands in the state files at 17:54:47, ten minutes after the two-second failure. If you correlate timestamps across logs, that gap is the tool being helpful, not the upgrade doing work.

Why It Looked Stuck Instead of Failed

Here is the part that turned a two-second failure into a day of confusion. The upgrade machinery keeps (at least) four separate stores of state, and they disagreed:

Store	Path	Owner	What it said
PAK results file	`/storage/db/pakRepoLocal/<pak-id>/<pak-id>.results`	pakManager	`apply_os_update_result: Failed`
Action history	`/storage/db/pakRepoLocal/vcopsPakManagerCommonHistory.json`	pakManager	`apply_os_update: Failed`
Per-PAK operation history	`/storage/log/vcops/log/casa/casa_pak_history_<pak-id>.json`	CaSA	`APPLY_OS_UPDATE` started, never completed
Upgrade orchestrator documents	CaSA runtime	CaSA	operation active, keep retrying

The UI renders from CaSA’s view. Its per-PAK history file ended like this:

{
 "operation": "APPLY_OS_UPDATE",
 "start_time": 1785260677162,
 "completed_time": null
}

An operation with a start time and a null completion is, as far as the UI is concerned, still installing. pakManager knew the upgrade had failed; CaSA never wrote a terminal result for the operation. Both were internally consistent. Neither was lying. They just were not the same state machine.

Why Rebooting Did Not Help

BRING_CLUSTER_OFFLINE had completed before the failure, and offline is persisted admin state. After the reboot, the platform came up and refused to start its services on purpose, from the support bundle’s sysenv/status.txt, captured while it was down:

vmware-vcops.service - LSB: vRealize Operations Platform
 Active: failed (Result: exit-code)
 vpostgres Cannot be started as the slice is offline
 analytics Cannot be started as the slice is offline
 collector Cannot be started as the slice is offline
 api Cannot be started as the slice is offline

So the reboot traded a stuck upgrade on a running cluster for a stuck upgrade on a down cluster. One genuinely useful side effect: the retry loop did not survive the restart. CaSA’s orchestrator documents came back empty (current_pak_activity all nulls from the first post-reboot poll onward) so the hammering stopped; the last apply_os_update attempt in the logs is from just before the reboot. But nothing resumed, either. The upgrade was now fully inert: no process running, no retries, UI still saying Installing, and monitoring down with it.

Bringing the cluster back online restored the data-collection side of the house (services active, slice ONLINE, still on 9.1.0.0300) while the upgrade state stayed exactly as wedged as before. Stale state does not age out. Waiting was never going to fix this.

Clearing the PAK State: CLI Archaeology

The guard message says what it wants: “cleanup the system and remove_history before attempting again.” Turning that sentence into working commands took several tries, and the failed attempts are worth documenting because the CLI’s own error messages actively mislead.

First lesson: run it with python3. The script does not behave as a direct executable: invoked by path, the shell tries to interpret Python as shell (abridged, full path shortened):

vcopsPakManager.py: line 6: from: command not found
vcopsPakManager.py: line 10: import: command not found
vcopsPakManager.py: line 31: syntax error near unexpected token `('

python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py

Run bare like that, it prints its full usage, which is where the trap sits. The guard message names remove_history as if it were an action. It is not. It is a parameter of the cleanup action:

cleanup - remove temp and working files
 params:
 --remove_history (ACTION|all) - remove the history for the action. The parameter
 is the action, or "all" to remove all history
 --remove_pak - removes the pak file and all history
 --skip_history_update - removes the pak without updating history

The tour of error messages on the way there:

--action remove_history ... → Action: "remove_history" is not valid
--action cleanup --remove_history --pak <pak> → Action: --pak is not supported for remove_history
--action cleanup --remove_history → option --remove_history requires argument
--action cleanup --remove_pak <pak> → No PAK file specified

The second one is the nasty one: --remove_history requires an argument, so it silently consumed --pak as its value and then complained that --pak is not a valid action name. And the fourth shows --remove_pak is a bare flag, not an option that takes the PAK name, the PAK always goes in --pak. The forms that worked:

python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py \
 --action cleanup --pak vRealizeOperationsManagerEnterprise-910040025541550

python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py \
 --action cleanup --remove_history all --pak vRealizeOperationsManagerEnterprise-910040025541550

And one final surprise from the results file afterwards: --remove_history all flipped a dozen keys to "history removed", but not the ones that mattered:

apply_adapter_result: history removed
apply_system_update_result: history removed
new_validate_result: history removed
...
validate_result: Failed ← survived
apply_os_update_result: Failed ← survived
apply_os_update_exit_code: 1 ← survived

Several statuses outlived the removal, including, of all things, the apply_os_update Failed/exit-1 pair that is exactly what the retry guard checks. Keep that in mind for the endgame below.

While in there, this is also the moment to fix the actual root cause, put repo definitions back in /etc/yum.repos.d/. A snapshot, a sibling appliance, or the Photon base image all work as a source; strictly speaking the upgrade only needs the glob to match one file, but restoring the stock set is the right fix. Healthy looks like this:

local-pak-iso.repo photon-extras.repo photon-release.repo photon-srpms.repo
photon-debuginfo.repo photon-iso.repo photon.repo photon-updates.repo

The Precheck 409: Colliding With Your Own Assessment

With pakManager’s state cleared and the repo directory restored, the next upgrade precheck failed after about a minute with:

An unexpected error occurred in step ops_precheck. Reference Code: EEF54002.
409: upgrade.verification.pak_installation_active
PreConditionFailedCasaException: key=upgrade.verification.pak_installation_active

My first theory was the obvious one, that null-completed_time entry still sitting in CaSA’s history. The CaSA log killed that theory and replaced it with something stranger. The precheck works by staging and installing an assessment PAK (APUAT-for-814x) on the appliance, and the sequence around the failure reads:

19:03:20 verifyConsistentState PAK_INSTALL → Cluster is in consistent state
19:03:21 APUAT-for-814x staged; validation, then apply_system_update begins
19:04:01 OrchestratorStatusDocument: current_pak_activity =
 {"pak_id":"APUAT-for-814x-910025541563","operation_name":"APPLY_SYSTEM_UPDATE","step":12,...}
19:04:01 verifyConsistentState PAK_INSTALL, again
19:04:02 Cluster state check failed: PreConditionFailedCasaException:
 key=upgrade.verification.pak_installation_active

The same consistency check passed at 19:03:20 with no PAK active, and failed 41 seconds later because a PAK installation was genuinely active, the assessment install the precheck itself had just launched. The stale enterprise-PAK entry was nowhere in it; CaSA was telling the truth about a collision between two halves of the same workflow. Meanwhile the assessment PAK, indifferent to the LCM step failing above it, ran to a clean finish in under twelve minutes, one more case of the layers not sharing a reality.

Two useful tools on the CaSA side regardless. Its deployment state is queryable directly, and note it reported "DONE" while the 409 was still reproducible, so treat it as one more opinion, not an oracle:

curl -k -u admin https://localhost/casa/deployment/cluster/installationstate

And the non-destructive reset is restarting the service:

systemctl restart vmware-casa

I restarted CaSA and re-ran the precheck; this run went clean end to end: the assessment PAK staged, validated, installed, and cleaned up in thirteen minutes, and the step reported success. Whether the restart mattered or the retry simply avoided the earlier collision, I cannot prove from the logs; what the evidence does show is that the null-completion history entry was still in place, untouched, during the passing run. It was never the thing the 409 keyed on.

The NullPointerException: Asking an Empty Orchestrator for a Task

Precheck green, one more attempt at the actual upgrade, and a new reference code:

An unexpected error occurred in step ops_apply_upgrade. Reference Code: 90AD4093.
Detail: Cannot invoke "com.broadcom.vcf.lcm.sddc.upgrade.sdk.client.appliance.model.Task.getId()"
because "task" is null

An NPE in the LCM client is a product bug in the sense that it hides the real condition, but the underlying state is easy to read. CaSA’s orchestrator, logged on every status poll, was completely empty:

UpdateOrchestratorDocument={"casa_webapp_pak_id":null,"casa_webapp_force_content_update":false,
 "casa_pak_status":{}}
OrchestratorStatusDocument: {"current_pak_activity":{"pak_id":null,"operation_name":null,
 "step":0,"total_steps":0,...},...,"queued_pak_ids":[],"is_upgrade_orchestrator_active":false,...}

And the PAK’s working directory was down to two files:

/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-910040025541550/
├── manifest.txt
└── vRealizeOperationsManagerEnterprise-910040025541550.results ← still says Failed

Two separate subtractions added up to this. The reboot had emptied CaSA’s runtime documents and queue, that happened back at 12:43, and it is what stopped the retry loop. The cleanup action had then removed the extracted PAK content from pakRepoLocal: which is its job. Between the two, nothing on the appliance still described an in-flight upgrade, so when LCM told CaSA to proceed with the apply step, there was no registered PAK, no queued operation, and nothing to build a task from. CaSA returned no task; the LCM client dereferenced it; NPE. The pendulum had swung all the way from state says failed, refuse to run to no state at all, nothing to run.

The original binary survives, though, cleanup did not touch the staged PAK repository:

locate .pak | grep 0400
/storage/db/casa/pak/dist_pak_files/VA_LINUX/vRealizeOperationsManagerEnterprise-910040025541550.pak
(plus the 0400 license-server and cloud-proxy PAKs, trimmed)

The Clean Path Out

⚠️ What follows is the planned remediation from this point, not a procedure I have verified end to end, which is exactly why step 1 exists. It is a lab; on a production deployment, this is the moment to take the reference codes (EEF54002, 90AD4093) and the support bundle to an SR instead of editing state by hand.

At this point the environment is: cluster online and collecting, on 9.1.0.0300; repo directory restored; precheck passing; PAK binary staged; and two pieces of stale state left, the Failed survivor keys in the results file and the null-completion entry in CaSA’s per-PAK history. The way out is to stop half-clearing and reset the PAK completely, then let the workflow rebuild everything from the binary:

Snapshot the appliance. The cluster is healthy right now; protect that before another attempt.
Remove the PAK and all of its history. Per the usage text, --remove_pak “removes the pak file and all history.” Note this is the one cleanup variant I have not exercised, and given that --remove_history all left the Failed keys standing, do not take the usage text on faith:
```
python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py \
 --action cleanup --remove_pak --pak vRealizeOperationsManagerEnterprise-910040025541550
```
Afterwards, verify the pakRepoLocal directory (with its .results file) is actually gone, and that the staged binary in /storage/db/casa/pak/dist_pak_files/VA_LINUX/ is still there, if --remove_pak reaches into the staged repository too, the workflow will need to re-download the PAK, which is inconvenient rather than fatal.
Clear CaSA’s per-PAK history so the UI stops rendering the ghost Installing operation and nothing can ever mistake it for an active install again, with the service stopped, move the stale file aside:
```
systemctl stop vmware-casa
mv /storage/log/vcops/log/casa/casa_pak_history_vRealizeOperationsManagerEnterprise-910040025541550.json /root/
systemctl start vmware-casa
```
Re-run the express patch from the beginning, not as a resume. With the staged .pak in place there is nothing to re-download; the workflow re-stages, re-validates, and (with /etc/yum.repos.d/ populated) should get past the two-second mark it never survived before. Expect the cluster to go offline again for the apply; that is the designed behavior (bring_cluster_offline: true in the manifest), so schedule it.

Verification

What success looks like once the re-run goes through:

The appliance reports 9.1.0.0400.25541550 and the express patch workflow completes without a reference code.
curl -k -u admin https://localhost/casa/deployment/cluster/installationstate returns "DONE", with no failed step behind it this time.
The new casa_pak_history_vRealizeOperationsManagerEnterprise-910040025541550.json shows every operation, APPLY_OS_UPDATE included, with a non-null completed_time.
The cluster slice is ONLINE and collecting, and no apply_os_update invocations appear in the pakManager logs after completion, the loop stays dead.
/etc/yum.repos.d/ is still populated. If it is empty again, find the culprit before the next patch does this all over.

What Actually Emptied /etc/yum.repos.d?

The unresolved thread. The directory was demonstrably populated on July 5 (the 0300 patch’s sed succeeded and copied local-pak-iso.repo into it) and demonstrably held nothing the glob could match on July 28. Neither observed upgrade run removed those files, and the PAK cleanup actions do not reach into /etc. Something else did, in the three-week window between patches.

The usual suspects are STIG/hardening remediation, a compliance scan “fixing” findings, or an overzealous cleanup script, and the hardening theory is less hypothetical than it sounds: the appliance ships a STIG-hardening Ansible role that the upgrade itself runs during apply_system_update, and its task list includes a literal “Find yum repos” step. A framework that polices exactly this directory demonstrably lives on the box; the July runs used it non-destructively, but whatever emptied the directory operated on the same territory. I have not caught the culprit in the act, so the case stays open.

If that something is scheduled, it will do it again, and the next express patch will die the same two-second death. Before every PAK-based upgrade on a VCF Operations appliance, it now costs me one command to check:

ls /etc/yum.repos.d/

Empty directory, no upgrade. It is that direct.

Takeaways

A stuck progress bar is a claim, not a fact. The UI renders CaSA’s operation history; a null completed_time displays as Installing forever, even when the failure happened in two seconds and nothing has run since. The pakManager logs are where the truth lives.
Know the four state stores: per-PAK results file, vcopsPakManagerCommonHistory.json, CaSA’s casa_pak_history_<pak>.json, and CaSA’s runtime orchestrator documents. Every confusing symptom in this saga was two of them disagreeing, including the 409, where the precheck disagreed with the assessment install it had launched itself.
The retry guard message is honest but the CLI is not literal about it: remove_history is a parameter of --action cleanup, not an action, and a missing argument makes it eat the next flag with a misleading error.
Verify what state-clearing commands actually cleared. --remove_history all reported nothing wrong while leaving the apply_os_update: Failed keys (the exact ones the guard checks) untouched. After any cleanup, read the .results file; do not assume.
cleanup without the root cause fixed is a trap. The guard fires before the script runs; clear the state while /etc/yum.repos.d/ is still empty and the next attempt fails in two seconds and re-arms it, from an offline cluster.
Reboots do not clear persisted upgrade state. Offline stays offline, Failed stays Failed, and the null completion stays null. What a reboot did do here is wipe CaSA’s runtime orchestrator: which killed the retry loop, made the wedge quieter and easier to misread as idle, and set up the eventual task-is-null NPE.
Fatal-error support bundles skew timelines. Nearly all of the ten minutes this “upgrade” ran were generateSupportBundle.py executing after the fatal error. Read the stderr log inward from the first ERROR, not the timestamps outward.
Check /etc/yum.repos.d/ before any PAK upgrade, and find out what is emptying it, whatever removed those files will break the next patch identically if it runs again.

Final Notes

Every stage of this one was the same lesson wearing different clothes: the upgrade machinery is several loosely-coupled state machines, and each error message was one machine reporting its own local truth. pakManager said previously failed; CaSA said still installing; the precheck said installation active, correctly, about the assessment install it had just started itself; LCM said task is null. Every statement was locally accurate, none of them described the system, and the actual root cause (a directory that a three-week-old patch run proves used to be populated) never appeared in any of them.

The two-second failure was trivial. The day was spent on state.

VCF Operations for Networks 9.1 Search Queries: The vRNI Query Language Lives On

Cosmin Trif — Mon, 27 Jul 2026 07:00:00 +0000

If you spent years typing flows where firewall action = deny into vRealize Network Insight, then watched the product get renamed to Aria Operations for Networks and then absorbed into VCF Operations as VCF Operations for Networks, you probably have one practical question: did the search language survive?

It did, completely. I compared the VCF Operations for Networks 9.1 search documentation against the legacy Aria Operations for Networks 6.13 docs page by page, and the query grammar is identical: same entity types, same where operators, same group by / order by / limit clauses, same planning and path queries. Even the REST API still answers under the old /api/ni/ (Network Insight) namespace. Your saved searches and your muscle memory carry over as-is.

The short version:

VCF Operations 9.1 has one console but at least four separate query languages: the networks entity search (vRNI heritage), the Enhanced Search metric queries (vROps heritage), log search under Operate > Logs, and the new-in-9.1 PromQL Custom Query for real-time metrics. Knowing which bar you are typing into matters more than any syntax detail.
The networks search DSL is unchanged from vRNI/AON: <projection> of <entity> where <condition> group by <property> order by <field> limit N, plus top N, in last N days, path queries (VM 'a' to VM 'b'), and plan queries for micro-segmentation.
The 9.1 differences are cosmetic or peripheral: NSX Advanced Load Balancer is renamed Avi Load Balancer in the docs, “VMware Cloud (VMC)” prose became “VMware Cloud on AWS”, the VMware SD-WAN (VeloCloud) search-queries page is gone from the 9.1 doc tree, and AWS/Azure data sources are deprecated in 9.1.
The classic community query lists (the ones we all bookmarked in the vRNI days) still apply. I have collected the best of them below, together with the official examples, into one cookbook.

Everything here is based on the 9.1 documentation set and verified against the legacy 6.13 docs; sources are linked at the end.

One Console, Four Query Languages

The single most confusing thing about search in VCF Operations 9.1 is that “search” is not one feature. The console consolidates products that each brought their own query engine, and none of them were merged:

Query surface	Where it lives in 9.1	Grammar	Lineage
Networks entity search	Inside the VCF Operations for Networks section (Operate pillar)	`entity where ... group by ...`	vRNI / Aria Operations for Networks
Enhanced Search (metric queries)	The global search bar in the top menu, on every page	`Metric of Object Type where ... childOf ...`	vROps / Aria Operations
Log search	Operate > Logs	Log filtering, alert queries, field extraction	Aria Operations for Logs
Custom Query (new in 9.1)	Operate > Real-Time Metrics / Troubleshooting Workbench	PromQL	New real-time metrics pipeline

This post is mainly about the first one, because that is where the network troubleshooting value is. The second gets its own section further down, since the two are easy to conflate: the same word where appears in both grammars but the operator sets are different. The 9.1 release notes only call out one genuinely new query mechanism, and it is the fourth: “Use PromQL-based search queries to derive operational insights.” Neither the networks DSL nor the Enhanced Search grammar changed in 9.1.

A note on architecture, since the rebrand hides it: VCF Operations for Networks is still a separate, pre-hardened appliance, deployed as a Day-2 operation from VCF Operations and lifecycle-managed by Fleet Management, not code merged into the core VCF Operations binary. Only one instance can be integrated per fleet. Licensing is automatic starting with VCF 9: the component is licensed through the vCenter instance it is connected to, and there is nothing to license manually anymore.

Anatomy of a Networks Search Query

The documentation splits queries into four categories: structured queries, entity queries, planning queries, and path queries. In practice you will spend most of your time on structured queries, which read like a relaxed SQL:

sum(bytes) of flows where Flow Type = 'Routed' group by Source Subnet Network order by sum(bytes)

The pieces, in the order the parser expects them:

Entity Type (Mandatory)

Every structured query is anchored on an entity type: vms, hosts, flows, firewall rules, switch ports, routes, nsx managers, kubernetes pods, and many more. Singular and plural both work, and synonyms are accepted (virtual machine = vm). Queries are not case-sensitive.

Typing just the entity type is already a valid query: vms lists every VM. The search bar validates as you type: a check mark means the query is valid, a cross means it is not. An old vRNI trick from the community lists that is still worth trying: type help into the search field to bring up the search help.

Filters: `where` Plus Operators

Conditions are property, operator, value, combined with and / or and parentheses:

Operator	Example
`=` / `!=`	`flows where source ip address = '10.16.240.0/24'`
`>` `<` `>=` `<=`	`vms where memory >= 4096 mb`, `vms where cpu usage rate < 70%`
`like` / `not like`	`vms where name like 'app'`
`in` / `not in`	`flows where port in (22, 23, 80, 443)`
`is set` / `is not set`	`vms where firewall rule is set`
`matches` / `not matches`	`vm where name matches '[a-z]vm-delta[0-9]'` (regex)
grouping `()`	`flows where (src tier = 'App' and destination tier = 'DB') OR (destination tier = 'App' and source tier = 'DB')`

The underrated one is the nested in: a sub-query as the right-hand side:

vm where host in (host where Blade like 'sys/chassis-1')
vm where vlan in (vlan of vm where os in ('Microsoft Windows Server 2003', 'Microsoft Windows Server 2008'))

The second example is a trimmed version of the documented “total attack surface” query (the full one lists five vulnerable operating systems and ends with group by Vlan): every VM that shares a VLAN with a VM running a vulnerable OS.

Projections: What to Show

By default a query returns the entity’s standard field set. A projection in front changes that:

Property list: os of vms, cpu cores of vms, or several at once: CPU Usage Rate, Network Rate, mac address, ip, host of vm 'Quality control-VM26'
Count: count of vms, count of vm where operating system like 'Microsoft Windows Server 2008'
List(): fetch one entity while filtering on another: List(host) of vms where memory <= 2gb
Aggregates (sum, max, min, avg): sum(memory) of hosts, sum(bytes) of flows
Series for time-series graphs: series(avg(cpu usage)) of vms where cpu cores = 4

Grouping, Ordering, Limiting

vms group by host
sum(bytes) of flows group by dest vm order by sum(bytes)
vms order by memory limit 5

Three rules worth memorizing, because they are the usual cause of the red X:

limit must be preceded by order by.
order by accepts exactly one field; descending is the default (asc flips it).
When a group by query has a projection or an order by, the projection must contain an aggregation function. sum(bytes) of flows group by dest vm is valid; bytes of flows group by dest vm is explicitly called out in the docs as invalid.

Time: `top N` and `in last N days`

The examples in the official advanced-queries page lean heavily on two inline constructs:

top 7 vm group by name, Vlan order by sum(Total Network Traffic) in last 7 days
Kubernetes alerts where alert code = 'ImagePullBackOff' in last 24 hours
New firewall rules in last 24 hours

There is also a UI time control next to the search bar with presets (“last 24 hours”, “last 3 days”), an At option for a point in time, and a Between option for a range, useful for re-running the same query against last Tuesday’s state, which remains one of this product’s superpowers.

Path and Planning Queries

Path queries draw the hop-by-hop topology between two endpoints, including the underlay:

VMware VM 'VM01' to VMware VM 'VM02'
vm 'app1-web' to Internet

Planning queries feed the micro-segmentation donut, the flow analysis view you use before writing distributed-firewall rules. The documented forms take a bare argument:

plan security in last 7 days
plan securitygroup1
plan host1

The community lists extend these with filters: plan vm where folder = 'Production' and plan flows where vm = 'app1-web' and traffic type = 'INTERNET_TRAFFIC' both come from mtellin’s gist rather than the official grammar reference, but they follow the same where rules as everything else.

Entity and Free-Text Searches

Plain strings work too: 'prod-68-1' (exact name, quoted), prod (partial name), vm app1 (entity type plus partial name). And a community favorite from the gist: paste a bare MAC address like 00:50:56:a6:c9:6a into the search bar to find the machine that owns it, a great party trick when someone shows up with nothing but a switch log.

vCenter Tags

vm where tag = 'Environment:Production'

The syntax is '{category}:{tagname}'. Tags are also available as a Group-by criterion in flow analysis (up to three at once).

The bookmark icon next to the search bar saves any valid query; saved searches are per-user, and six defaults ship with the product (All Flows, Applications, Azure, Kubernetes Dashboard, Top Trends, NSX). Two features worth more attention than they get:

Help > Useful Searches is a curated, tagged library of pre-built queries, parameterized ones open pre-filled with sample values you swap out.
View Search Query on dashboard widgets (VM Details, Host Details, NSX Manager, Flow Insights, and a few others) reveals the query powering the widget, which you can copy into the search bar and modify. This is the fastest way I know to learn the DSL: find a widget that almost shows what you want, steal its query, adjust.

And almost any search can become an alert: run the query, then create a notification from the results page. Threshold queries (flows where firewall action = deny, packet-drop queries) make surprisingly good custom alerts.

The Query Cookbook

The queries below are collected from the official 9.1 search-queries reference and the two classic community lists (Pat’s vRNI search queries post and mtellin’s gist) deduplicated and grouped by what I actually use them for. Since the grammar is unchanged from vRNI, the community-era queries remain valid in 9.1 unless they depend on a data source that is gone (more on that below).

Flows and Traffic Patterns

What it answers	Query
Denied flows, environment-wide	`flows where firewall action = deny`
Internet traffic by source VM	`sum(bytes) of flows where Flow Type = 'Internet' group by src vm`
Top talkers, limit 10	`sum(bytes) of flows group by src vm order by sum(bytes) limit 10`
Top ports by volume	`sum(bytes) of flow group by port order by sum(bytes)`
Sum and average per port	`sum(bytes), avg(bytes) of flows group by port`
Top routed subnet pairs (last 7 days)	`top 7 flow where Flow Type = 'Routed' group by Source Subnet Network, Destination Subnet Network order by sum(Bytes) in last 7 days`
Traffic that never leaves its subnet, micro-segmentation candidates	`top 7 flow where Flow Type = 'Switched' group by Subnet Network order by sum(Bytes) in last 7 days`
VMs talking outside the country	`top 7 flow where Destination Country != 'United States' group by Source VM, Destination Country order by sum(Bytes) in last 7 days`
Traffic between two application tiers, either direction	`Flows where (src tier = 'App' and destination Tier = 'DB') OR (destination tier = 'App' and source tier = 'DB')`
Data center interconnect traffic between two sites	`sum(bytes) of flows where (Dst Manager = 'site2' AND src manager = 'site1') OR (Dst Manager = 'site1' AND src manager = 'site2')`

NSX Firewall and Security

What it answers	Query
Effective rules between two VMs	`firewall rules from 'Prod-Midtier-1' to 'Prod-Db-1'`
Any/any allow rules, segmentation debt	`firewall rule where action = allow and service any = true`
Masked (shadowed) rules	`Firewall Rule Masked Alert`
Rules created recently	`New firewall rules in last 24 hours`
Rules no flow has ever hit	`Firewall rule where flow is not set`
Flow count per rule	`Flow group by firewall rule`
Flows hitting specific rule IDs	`Flow where rule id in (1011, 1012, 1013)`
VMs excluded from the DFW	`vm where firewall status = 'Excluded'`
VMs covered by a given rule	`vm where Firewall Rule = 'Prod MidTier to Prod DB - DBService'`
Security groups referenced by no rule at all	`Security group where Indirect Incoming Rules is not set and Indirect Outgoing Rules is not set and Direct Incoming Rules is not set and Direct Outgoing Rules is not set`
Rule and group membership churn	`firewall rule membership change`, `security group membership change`

VM and Host Inventory

What it answers	Query
VM count per OS	`vms group by operating system`
Segments with the most VMs	`vm group by l2 network`
Datastores with the most VMs	`vm group by datastore`
Hosts per vSphere version / build	`host group by version`, `host group by OS`
Network identity card for every VM	`l2 network, vlan, ip address, default gateway of vms`
VMs still running vulnerable operating systems	`count of vm where Operating System like 'Microsoft Windows Server 2003' or Operating System like 'Microsoft Windows Server 2008'`
Everything about one VM in a single line	`CPU Usage Rate, Network Rate, Memory Usage Rate, mac address, ip, vxlan, host of vm 'Quality control-VM26'`

Physical Network and Routing

What it answers	Query
All switches from one vendor	`switches where vendor = Cisco`
Switch ports dropping packets, worst first	`switchport where vendor = 'Cisco' order by total packet drops`
Routes in a VRF	`routes where vrf = 'Provider Edge 3'`
Routes via a given next hop	`routes where NextHop Router = 'California-Edge'`
BGP sessions that are not up	`BGP Connection where BGP Status != 'ACTIVE'`
BGP neighbors of a router	`list(BGP Neighbor) of NSX-T Router 'Router Name'`
ECMP-enabled routers on a manager	`NSX-T Router where ECMP = 'Enabled' and manager = 'NSX-T Manager Name'`

Kubernetes (VKS / Antrea)

What it answers	Query
Pod distribution per node	`kubernetes Pods group by Kubernetes Node`
Nodes not in Ready state	`kubernetes nodes where Ready != 'True'`
Pods created in the last hour	`new kubernetes pod in last 1 hour`
Dropped flows per service	`flows where firewall action = 'DROP' group by Kubernetes Service`
Cross-namespace compliance check	`flows from Kubernetes Namespace 'PCI' to Kubernetes Namespace 'Non-PCI'`

This category got more relevant in 9.1: container IPFIX flows from VKS clusters running Antrea now export to the VCF Operations for Networks collector, with pod-to-pod and pod-to-external traffic dashboards.

Dashboard Shortcuts

Not structured queries, but single keywords from the gist that jump straight to built-in views: problem, change (recent changes in the environment), topn, storage, and pci compliance of VC Manager '192.168.10.10' (the PCI dashboard: which, per the gist, required an Enterprise license back in the vRNI days).

A Caveat on Community Shorthand

The community lists include a few compact forms you will not find in the official grammar reference: vm by VLAN, show VLAN, vm by network rate, top 10 vms by max packet drops, flows where flow type = 'VM-Physical' by bytes. They come from vRNI-era posts that predate the current documented grammar, which only uses group by / order by for these jobs. I have kept them out of the tables above for that reason. If you rely on one of them, the in-product validation check mark will tell you immediately whether your build still accepts it.

The Other Search Bar: Enhanced Search Metric Queries

The global search bar at the top of the VCF Operations console (the one that is on every page) is a different animal with its own documented query language, under the section Broadcom titles “Enhanced Search Capability”. It searches the vROps-side world: objects, metrics, properties, dashboards, reports, views, alert definitions, symptom definitions, recommendations, notifications, IP addresses, super metrics, and (since recent releases) features, type li and select Feature to jump straight to Licenses, navigation path included.

Its metric query grammar looks like this:

CPU|Usage % of Virtual Machine
CPU|Usage % of Virtual Machine where CPU|Usage % > 15 and Memory|Usage % > 15
CPU|Demand MHz of Virtual Machine where Configuration|Name contains 'Centos'
CPU|Usage % of Virtual Machine where CPU|Usage % > 15 or Memory|Usage % > 15 childOf vSphere World

The differences from the networks DSL are exactly where you would trip:

The pattern is Metric of Object Type, with the pipe character separating metric group and metric name (CPU|Usage %). If you start with a metric name, naming the object type is mandatory.
String operators are camelCase keywords: equals, notEquals, contains, notContains, startsWith, notStartsWith: there is no like, no in, no matches here. Numeric operators (>, >=, <, <=, =, !=) plus arithmetic (+ - * / with parentheses) and and/or round out the set. String values are case-insensitive.
childOf at the end of the query scopes results to descendants of an ancestor object; the closest thing this grammar has to the networks DSL’s nested in.
Hard limits: 300 characters per query, at most five different metric names. Break either and a red banner explains the issue; for syntax errors specifically it reads “The query contains syntax error(s). Please modify it to get suggestions.”
There is no group by or order by in the query itself; slicing happens on the Metric Search Results page instead, via Transformation (Min/Max/Average/Sum/First/Last/Current), Roll Up (hour through year), Parents, value-distribution buckets, and time settings. The Transformation filter only activates when the query has a where condition and the object type has at most 200 objects over at most one month. From any result row, the vertical-ellipsis menu jumps into the Troubleshooting Workbench.

One honest note: none of this is new in 9.1. I compared the 9.0 and 9.1 pages (the content is identical) and the same grammar exists in Aria Operations 8.x. What is new in 9.1 on the metrics side is the PromQL-based Custom Query in Real-Time Metrics, which is a separate topic (and a separate language) altogether.

What Actually Changed in 9.1 for Operations for Networks

Since search itself did not move, here is what did: the 9.1 items relevant to a networks operator, from the release notes and product support notes:

Network Assessment and Value report: evaluates your current VDS-based networking and produces a CAPEX/OPEX analysis for VCF networking capabilities. It replaces the vCenter Assessment report, which was removed.
VPC Planning: guided vSphere-to-VPC transition planning, with subnet tables and a CSV export of the generated VPC design.
Expanded IPFIX for VKS clusters: Antrea container flows land in the collector; pod-to-pod and pod-to-external dashboards.
Expanded migration planning: automatic migration wave generation, clustering workloads by flow, latency, and bandwidth data.
Network health dashboards: NSX Edge appliance and ESX host networking capacity monitoring surfaced through VCF Operations Health and Diagnostics.
Deprecations: AWS and Azure data sources are deprecated in 9.1, and the networks API had 22 operations marked deprecated, the largest deprecation set in the Operations component. The VMware SD-WAN (VeloCloud) search-queries reference page is also absent from the 9.1 documentation tree.

That deprecation list is worth acting on if you have search-based alerting or scripts built on AWS/Azure entity queries (AWS EC2 where Security Group = ..., Azure NSG Rule where Action = 'ALLOW'), the syntax still exists in the 9.1 reference today, but the data sources feeding those entities are on the way out.

Getting There from Standalone 6.x

If you are still on a standalone appliance (I have covered upgrading Aria Operations for Networks to 6.14 via the upgrade bundle and via Aria Suite Lifecycle before) the path into 9.x is: get to 6.14.0 Patch 6 or 6.14.1+, import the instance into VCF Operations Fleet Management, upgrade it in place to VCF Operations for Networks 9.0.x, and from there directly to 9.1. Only one instance can be imported per fleet. Every search query in this post works the same on both sides of that migration, which is precisely the point.

The API Corner

For automation, the search subsystem is exposed over REST, still under the Network Insight namespace:

POST /api/ni/search/ql: takes the same query-language strings you type in the search bar
POST /api/ni/search: entity search
POST /api/ni/search/aggregation: aggregations
POST /api/ni/search/groupby: group-by queries

The schemas are on the VCF Operations for Networks API reference. The ql endpoint is the interesting one: anything you prototype in the search bar can be lifted verbatim into a script.

References

VCF 9.1 NSX Rotation Error: PASSWORD_MANAGER_NSX_USER_API_FAILED

Cosmin Trif — Wed, 15 Jul 2026 17:30:00 +0000

Continuing the credential cleanup in the same 9.1 environment from the expired password sync and VCFOPSPWID03 posts, one credential refused to settle: the service account SDDC Manager uses to talk to the NSX Manager API, svc-sddcmanager-a-nsx-mgmt--8199 in my lab. It showed up as disconnected, every remediation attempt failed, and the environment kept trying to rotate it in a loop.

This one took several wrong turns before the actual root cause surfaced, and almost every wrong turn taught me something worth writing down. The short version:

SDDC Manager generated a 128-character password for the service account, but its own remediate path enforces a 127-character maximum, an off-by-one between the generator and the validator that keeps the rotation failing.
The service account is an OS-level PAM user on the NSX appliance: invisible to the NSX API, UI, and CLI, manageable only as root with standard Linux tools.
Failed rotation attempts had locked the account at the NSX API level.
And the final boss: SDDC Manager’s rotation machinery was re-changing the password within a couple of seconds of any manual reset, so every fix I tested was already stale by the time I tested it.

The Symptom: A Service Account Stuck Disconnected

The credential in question, as SDDC Manager tracks it:

Field	Value
Username	`svc-sddcmanager-a-nsx-mgmt--8199`
Resource type	`NSXT_MANAGER`
Credential type	`API`
Account type	`SERVICE`

The rotation failures are visible on the SDDC Manager appliance in the operations manager log:

tail -f /var/log/vmware/vcf/operationsmanager/operationsmanager.log |
 grep -Ei 'svc-sddcmanager|ROTATE|REMEDIATE|diagnosticMessage'

The recurring entry that matters (flattened here):

errorCode: PASSWORD_MANAGER_NSX_USER_API_FAILED
diagnosticMessage: Response from NSX <nsx-manager-fqdn>: 403 -
The credentials were incorrect or the account specified has been locked.

A 403 for a service account that only SDDC Manager itself manages. Nobody had touched this password by hand: SDDC Manager rotated it, and then could no longer authenticate with the value it had just set.

Note: The examples in this post use curl -k, which skips TLS certificate verification, fine in a lab where the appliances still run self-signed certificates. In production, drop -k and trust the appliance CA instead. If any password you type contains !, disable history expansion first (set +H in bash, unsetopt BANG_HIST in zsh) so the shell does not mangle it. And the read -rp / read -rsp prompts use bash syntax: on the appliances that is the shell you get, but if you run any of this locally from zsh, start bash first or use the zsh form (read -rs 'VAR?prompt: '); in zsh, read -rsp silently leaves the variable empty, and an empty password produces confusing downstream errors instead of a clear failure.

The Off-by-One: 128 Generated vs 127 Accepted

Digging through the failed rotation tasks showed the first real clue: the password SDDC Manager had generated for the service account was 128 characters long, while the remediate path rejects anything over 127 characters.

The generator takes its cue from the NSX node authentication policy. NSX advertises a maximum_password_length of 128 by default, SDDC Manager generates a password at that maximum, and then its own validation refuses to remediate with it. The account ends up in the worst possible state: the rotation half-succeeds often enough to keep changing the password, but remediation can never complete, so the credential is marked disconnected and the platform keeps retrying.

Dead Ends Worth Knowing About

Before the fix, the detours, each one reasonable, each one wrong.

Remediating from the UI

The obvious first move. The remediation workflow failed with the same 403, and the UI adds nothing to the diagnosis, the useful error text only exists in operationsmanager.log.

Guessing the Credential Tuple

My first API attempts failed validation before doing anything, because a PATCH /v1/credentials call must reference the exact tuple SDDC Manager has on file: resourceName, resourceType, credentialType, and username all have to match. Do not guess any of them. Read them back first.

Get a token (I ran these directly on the SDDC Manager appliance against localhost, but the FQDN works the same):

SDDC="https://localhost"
read -rp "SSO username: " SSO_USER
read -rsp "SSO password: " SSO_PASS; echo

TOKEN=$(jq -nc --arg u "$SSO_USER" --arg p "$SSO_PASS" \
 '{username:$u,password:$p}' |
 curl -sk -H 'Content-Type: application/json' -d @- \
 "$SDDC/v1/tokens" | jq -r .accessToken)

unset SSO_PASS
[ -n "$TOKEN" ] && [ "$TOKEN" != "null" ] || echo "Token acquisition failed" >&2

The block prints nothing on success, the token lands silently in $TOKEN. Test it with a cheap authenticated call before building anything on top of it:

curl -sk -o /dev/null -w "%{http_code}\n" \
 -H "Authorization: Bearer $TOKEN" "$SDDC/v1/domains"

A 200 means the token works. Anything else, start over at the token block.

List the NSX credentials and pull the exact tuple:

curl -sk -H "Authorization: Bearer $TOKEN" \
 "$SDDC/v1/credentials?resourceType=NSXT_MANAGER" |
 jq -r '.elements[] | select(.accountType=="SERVICE") |
 [.resource.resourceName, .resource.resourceType, .credentialType, .username] | @tsv'

That returns the values used in the remediation calls later in this post.

If jq fails here with “Cannot iterate over null”, the response was not a credentials list; it is almost always an authentication error object instead. Check the HTTP status and the raw body:

curl -sk -o /tmp/creds.json -w "%{http_code}\n" \
 -H "Authorization: Bearer $TOKEN" \
 "$SDDC/v1/credentials?resourceType=NSXT_MANAGER"

jq . /tmp/creds.json

A 401 means the token is empty, expired (they are short-lived), or the literal string null from a failed acquisition, re-run the token block above and mind its failure check. A 401 body complaining that “jwt strings must contain exactly 2 period characters” is the same problem stated more directly: whatever is in $TOKEN is not a JWT at all, and the usual cause is a rejected /v1/tokens login quietly turning into null. Check that the username is a full SSO identity such as administrator@vsphere.local, not a bare username.

Capture the service account username while you are here, the remediation and rotation calls later run in this same SDDC Manager session and reuse it:

SVC=$(curl -sk -H "Authorization: Bearer $TOKEN" \
 "$SDDC/v1/credentials?resourceType=NSXT_MANAGER" |
 jq -r '.elements[] | select(.accountType=="SERVICE") | .username')

echo "$SVC"

If you spend time in this API, the /v1/credentials overview covers the basic operations.

Changing the VCF Operations Password Policies

VCF Operations 9.1 exposes password policy settings, and lowering a maximum length there looked like the clean fix for the 128-character problem. It did nothing. Those policies do not govern what SDDC Manager generates for service accounts, the generator follows the NSX node authentication policy, not VCF Operations. Worth knowing before you spend an afternoon proving it.

The First Fix: Lower the NSX Maximum Password Length

Since the generator follows the NSX policy maximum and the validator caps at 127, the workaround is to lower the NSX maximum_password_length so generated passwords land under the limit. I went to 100, comfortably below 127, still far longer than any human-managed password.

One trap first: the setting lives in the auth-policy endpoint. There is no /api/v1/node/aaa/password-complexity endpoint, that name belongs to the NSX CLI command (get/set password-complexity), and the API returns 404 for it. The REST path you want is /api/v1/node/aaa/auth-policy, the same endpoint from the password reuse post.

Back up the current policy before changing anything:

read -rsp "NSX admin password: " NSXPASS; echo

curl -sk -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy |
 tee /tmp/auth-policy-backup.json | jq .

{
 "api_failed_auth_lockout_period": 900,
 "api_max_auth_failures": 5,
 "minimum_password_length": 12,
 "maximum_password_length": 128,
 ...
}

Lower the maximum (maximum_password_length accepts 8 to 128):

curl -sk -u admin:"$NSXPASS" -X PUT \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy \
 -H 'Content-Type: application/json' \
 -d '{"maximum_password_length": 100}'

Policy changes take a short while to apply and synchronize across the manager cluster nodes in 9.x. Read the value back before moving on:

curl -sk -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy | jq .maximum_password_length

New rotations now generate passwords of at most 100 characters, comfortably under the 127-character validation. Problem solved, except the account was still disconnected, and remediation still failed with the same 403.

Plot Twist: TEST_BEFORE_REMEDIATE

The remediation task logs showed the failure happening in a TEST_BEFORE_REMEDIATE stage. Before SDDC Manager remediates a credential, it first authenticates against NSX as the service account with the password it believes is current. If that test fails (wrong password or locked account) the whole remediation aborts with PASSWORD_MANAGER_NSX_USER_API_FAILED, and nothing is remediated.

That explains the loop: the earlier failed rotations had burned through NSX’s API authentication failure allowance (api_max_auth_failures: 5 in the policy above), the account got locked, and from then on even a correct password could not pass the pre-remediation test.

So before anything else could work, I had to look at the account itself on the NSX side, which is where this got interesting.

The Hidden User: An API-Only PAM Account

The service account does not exist as far as the NSX management plane is concerned:

GET /api/v1/node/users: not listed.
NSX UI under user management, not there.
NSX CLI get user / set user: unknown user.

But on the NSX Manager appliance as root, it is a perfectly ordinary Linux account:

grep svc- /etc/passwd

svc-sddcmanager-a-nsx-mgmt--8199:x:...:...:/home/svc-sddcmanager-a-nsx-mgmt--8199:...

The account authenticates to the NSX API through PAM, has password history in /etc/security/opasswd, and is managed by exactly nothing except root and the standard Linux toolbox. No SSH access, no UI, API-only. That means the diagnostics you would normally run through the NSX API happen with faillock and chage instead:

SVC="svc-sddcmanager-a-nsx-mgmt--8199"

faillock --user "$SVC"
chage -l "$SVC"

faillock confirmed the lockout, a stack of recent authentication failures, courtesy of the rotation loop. Clear it:

faillock --user "$SVC" --reset

While in there, two more root-level tools are worth knowing for this account type:

passwd "$SVC" # set a known password, as root
echo "" > /etc/security/opasswd # wipe PAM password history. See warning

⚠️ /etc/security/opasswd holds the PAM password history for every user on the appliance, not just the service account. Emptying it is a blunt lab move to get past “password has been already used” rejections during repeated testing. Do not do this on a production appliance without a support ticket telling you to.

Keeping the Account from Locking Again

The lockout pattern here matches Broadcom KB 420736: SDDC Manager trips the NSX API lockout with its own failed authentication attempts, and remediation can never win against a locked account. The durable fix is to add SDDC Manager’s IP to the NSX lockout_immune_addresses list, so its authentication failures never lock anything.

Run this on the SDDC Manager appliance, so hostname -i resolves to the right IP; the jq filter appends it to the existing list without clobbering other entries:

Note: PUT /api/v1/cluster/api-service replaces the entire cluster-wide API service configuration document, and NSX restarts the API service to apply it, expect a brief API interruption, so run this when nothing else is mid-call. The pipeline below reads the full current document and keeps a copy in /tmp/api-service-backup.json before anything is overwritten.

curl -sk -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/cluster/api-service |
 tee /tmp/api-service-backup.json |
 jq --arg ip "$(hostname -i)" \
 '.lockout_immune_addresses += [$ip] | .lockout_immune_addresses |= unique' |
 curl -sk -u admin:"$NSXPASS" -X PUT \
 -H 'Content-Type: application/json' -d @- \
 https://<nsx-manager-fqdn>/api/v1/cluster/api-service

With the lockout cleared and immunity in place, I reset the service account password as root, confirmed it worked with a quick authentication test…

read -rsp "Service account password you just set: " SVCPASS; echo

curl -sk -o /dev/null -w "%{http_code}\n" \
 -u "$SVC:$SVCPASS" https://<nsx-manager-fqdn>/api/v1/cluster/status

unset SVCPASS

…and got 403. Seconds after setting it. With a password I had just set moments earlier.

The Real Root Cause: Racing the Rotation Engine

This was the point where the investigation stopped being about policies and started being about timing. The sequence kept repeating:

As root on NSX: passwd the service account to a known value.
Test authentication with that value: 200: for a moment.
Test again a few seconds later: 403.

Nothing was “wrong” with any password. SDDC Manager’s rotation machinery was re-changing the service account password within a couple of seconds of any reset. The disconnected credential kept the retry loop hot, every retry rotated the password on the appliance, and every manual test raced the engine and lost. The vault was not stale, the policy was not blocking anything anymore, my known-good password simply stopped being the current password almost immediately after I set it.

Watching the log in one terminal makes the race visible:

tail -f /var/log/vmware/vcf/operationsmanager/operationsmanager.log |
 grep -Ei 'svc-sddcmanager|ROTATE|REMEDIATE'

The fix is to stop fighting the engine and synchronize with it instead: set the password and hand it to SDDC Manager inside that couple-second window, so the value on the appliance and the value being remediated converge before the next rotation fires.

⚠️ What follows is a lab maneuver. Manually resetting a platform-managed service account and racing the rotation engine is recoverable here; on a production NSX Manager, capture operationsmanager.log, the failed task IDs, and the faillock output and open a support request first. See the last bullet in Hardening Takeaways.

In practice, that means staging everything in advance:

Terminal 1: NSX Manager as root, with a non-interactive password set ready to run:

read -rsp "New service account password: " SVCPASS; echo
echo "$SVC:$SVCPASS" | chpasswd
unset SVCPASS

Terminal 2 (the same SDDC Manager session from earlier, so $SDDC, $TOKEN, and $SVC are still set, with the remediation payload pre-built) the only thing left is to press Enter:

read -rsp "Same password as set on NSX: " SVCPASS; echo

TASK_ID=$(jq -nc \
 --arg fqdn "<nsx-manager-fqdn>" \
 --arg user "$SVC" \
 --arg pw "$SVCPASS" \
 '{operationType:"REMEDIATE",
 elements:[{resourceName:$fqdn, resourceType:"NSXT_MANAGER",
 credentials:[{credentialType:"API", username:$user, password:$pw}]}]}' |
 curl -sk -X PATCH \
 -H "Authorization: Bearer $TOKEN" \
 -H 'Content-Type: application/json' -d @- \
 "$SDDC/v1/credentials" | jq -r .id)

unset SVCPASS

Run terminal 1, then terminal 2 immediately, within the window, before the engine rotates again. Poll the task:

curl -sk -H "Authorization: Bearer $TOKEN" \
 "$SDDC/v1/credentials/tasks/$TASK_ID" | jq '{name, status}'

This time TEST_BEFORE_REMEDIATE authenticated successfully (the password it tested was, at that moment, actually the password on the appliance) and the remediation completed. The credential flipped back to healthy, and the rotation loop stopped, because there was nothing left to retry.

To finish cleanly, trigger one supervised rotation so the account ends up on a fresh SDDC Manager-generated password (now capped at 100 characters by the lowered policy) instead of a hand-typed one:

jq -nc \
 --arg fqdn "<nsx-manager-fqdn>" \
 --arg user "$SVC" \
 '{operationType:"ROTATE",
 elements:[{resourceName:$fqdn, resourceType:"NSXT_MANAGER",
 credentials:[{credentialType:"API", username:$user}]}]}' |
curl -sk -X PATCH \
 -H "Authorization: Bearer $TOKEN" \
 -H 'Content-Type: application/json' -d @- \
 "$SDDC/v1/credentials" | jq .

Poll the task the same way as before; with the loop broken and the policy cap in place, the rotation completes cleanly.

Verification

The rotation task in SDDC Manager completes successfully, and no new ROTATE/REMEDIATE retries appear in operationsmanager.log for the service account.
The credential status is back to healthy (GET /v1/credentials shows no error state for the tuple).
faillock --user "$SVC" on the NSX appliance stays empty.
lockout_immune_addresses in GET /api/v1/cluster/api-service contains the SDDC Manager IP.
The NSX auth-policy holds the values you intend to keep, in my case maximum_password_length: 100 stays, since the 128/127 mismatch would otherwise come straight back on the next rotation.

Clean up the working files, since the backups describe your security posture and the lockout-immune IP list:

rm -f /tmp/auth-policy-backup.json /tmp/api-service-backup.json
unset NSXPASS

Hardening Takeaways

A few things I am keeping from this one, beyond the fix itself:

Add SDDC Manager to lockout_immune_addresses proactively. Any credential workflow that can fail more times than api_max_auth_failures allows can lock its own account and wedge remediation. KB 420736 describes the pattern.
Back up auth-policy before any PUT: tee to a file costs nothing, and the endpoint carries every knob (lockout periods, complexity, history) in one document, so an overzealous edit has a wide blast radius.
Check NTP while you are in the logs. This environment’s operationsmanager.log also showed intermittent JWT clock-skew errors between components, harmless-looking noise that can turn into real authentication failures. Time sync issues and credential troubleshooting mix badly; rule them out early.
Know when to stop and open an SR. Manually resetting a service account password as root and racing the rotation engine is recoverable in a lab. In production, the moment a platform-managed service account is in a rotation loop, capture operationsmanager.log, the failed task IDs, and the faillock output, and open a support request: KB 420736 describes the lockout pattern, and KB 423038 is a related reference worth attaching alongside it.

Final Notes

Every layer of this problem produced the same useless symptom (a 403) for four different reasons: a password too long to remediate, a locked account, a pre-remediation test that authenticates as the account it is about to fix, and finally a rotation engine overwriting the password faster than I could test it. The 128-versus-127 mismatch set the loop in motion, but the thing that made it unfixable-looking was the timing: every diagnostic I ran was measuring a password that had already been replaced.

Two habits would have shortened this significantly. First, tail -f the operations manager log before forming a theory: the rotation attempts were right there, timestamped, showing the engine acting seconds after every manual change. Second, when a platform manages an account, assume it will keep managing it while you troubleshoot: either pause the machinery or synchronize with it. Fighting an automated rotation engine with manual password resets is a race you lose every time, until you make the race the plan.

Fixing VCFOPSPWID03: NSX Password Update Failed as Call from NSX Received Invalid Response (VCF 9.1)

Cosmin Trif — Sat, 11 Jul 2026 17:30:00 +0000

While rotating credentials in the same 9.1 environment I covered in the expired password sync post, an Update of the NSX Manager admin password (credential type API) from VCF Operations kept failing after about a minute with error code VCFOPSPWID03 and a spectacularly unhelpful message: “call from NSX received invalid response.”

The actual cause turned out to be simple, NSX rejected the new password because it had already been used on that manager. This being a lab, I was deliberately recycling a previous password. That goes against best practices, and NSX enforces it. But the way the error surfaces (and the way the reuse policy behaves in this build) is worth writing down.

The Error: “Call from NSX Received Invalid Response”

The password update banner in VCF Operations:

The failed request under Management Tasks, with the task failing in Stage 1 after roughly a minute:

The full error text:

Error Code: VCFOPSPWID03
Password task UPDATE for <nsx-manager-fqdn> has failed. Error message:
NSX password update failed at <nsx-manager-fqdn>: Failed to update password
for NSX user admin for <nsx-manager-fqdn>, as call from NSX received
invalid response.

The same failure is visible on the VCF Operations node in the component-manager-plugin-*.log (in a support bundle it is under logs/; on the appliance under /storage/log/vcops/log/):

[...UpdatePasswordTask.execute] - Executing Update password task...
[...PasswordTaskUtil.monitorPasswordTask] - Password Task : <task-id> Status: IN_PROGRESS
[...PasswordTaskUtil.monitorPasswordTask] - Password Task : <task-id> Status: FAILED
[...Task.injectFailedEventWithLocale] - Injecting task failure event. Error Code : 'VCFOPSPWID03', Retry : 'false'
java.lang.RuntimeException: Password task UPDATE for <nsx-manager-fqdn> has failed. [...]
as call from NSX received invalid response.

“Invalid response” suggests connectivity or certificate problems, and that is exactly the wrong place to go digging. NSX was up, reachable, and answering; it just did not like the request.

Finding the Real Error

Drill into the failed password task and expand the details. The Cause field carries the message that the top-level error flattened away:

Description: Password update for resource : <nsx-manager-fqdn>,
 user : admin and credential type : API
Progress Messages: Failed to get NSX user details.
Cause: Specified password invalid: Password has been already
 used. Choose another.

There it is. Not a connectivity issue, not a certificate issue, password reuse. NSX checked the new password against its password history and refused it.

The NSX Password Reuse Policy

Password history on NSX is not the classic PAM /etc/security/opasswd mechanism (mine was empty; that was my first dead end). It is governed by the password_remembrance setting in the node authentication policy, alongside the complexity settings.

Prompt for the admin password once instead of putting it inline: passwords containing ! get mangled by history expansion in interactive shells (set +H disables that in bash; in zsh use unsetopt BANG_HIST), and inline credentials linger in shell history:

set +H
read -rsp "NSX admin password: " NSXPASS; echo

curl -k -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy

{
 "api_failed_auth_lockout_period": 900,
 "api_max_auth_failures": 5,
 "minimum_password_length": 12,
 "password_remembrance": 5,
 ...
}

Or on the NSX CLI: get password-complexity.

Per the VCF 9.1 NSX documentation, password_remembrance is the number of previous passwords a user cannot reuse. Three details matter here:

NSX supports one password policy (you cannot exempt a single user like admin from history checks) and the manager password-policy settings synchronize across the management cluster nodes.
In VCF 9.1 the default value is 5 and the documented valid range is 1 to 12 for new configurations. Out of the box, the last five passwords are blocked, which is why my recycled lab password was refused without anyone ever hardening this environment. (The same doc notes that policy settings customized before an upgrade are retained afterward, so upgraded environments may be carrying older values.)
Older NSX 4.x documentation describes 0 as “check disabled”. This 9.1 build rejects 0 outright, as I found out next.

Note: These examples use curl -k, which skips TLS certificate verification, fine in a lab where the manager still has a self-signed certificate. In production, drop -k and trust the NSX CA instead. When you are done, unset NSXPASS.

Why You Cannot Just Disable Password History

My first instinct was to turn the history check off entirely, since 0 was the documented disabled/default value in NSX 4.x. Both the CLI (set password-complexity) and the API refuse it in this build:

curl -k -u admin:"$NSXPASS" -X PUT \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy \
 -H "Content-Type: application/json" \
 -d '{"password_remembrance": 0}'

{
 "error_code": 255,
 "error_message": "Request body not valid - PasswordComplexityProperties.password_remembrance: value '0' must be greater than or equal to '1'",
 "module_name": "common-services"
}

That matches the 9.1 documentation, which lists a valid range of 1 to 12 for new password_remembrance configurations. In this build there is no accepted way to set 0: the lowest you can configure is 1, which blocks only the single most recent password.

Recommended Fix: Use a New Compliant Password

In production (and honestly in most labs too) the right response to “Password has been already used” is the boring one: leave the history policy alone, pick a new password that satisfies the policy (minimum length, character classes, no dictionary-based words, not among the remembered previous passwords), and re-run the Update action in VCF Operations. The workflow completes, and the appliance password, the stored credential, and the expiration metadata all stay in sync.

The next section covers the lab case, where re-using one specific password was the whole point.

Lab-Only Workaround: Temporarily Lower Password History to 1

Treat this as a reversible change, not a new setting:

Confirm the cause first. The failed task’s Cause must actually say “Password has been already used”; this workaround fixes nothing else.
Record the current value, so you can restore it exactly:

curl -sk -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy | jq .password_remembrance

Lower it to 1 (the minimum of the valid range) so NSX only blocks reuse of the immediately previous password; anything older becomes acceptable again:

curl -k -u admin:"$NSXPASS" -X PUT \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy \
 -H "Content-Type: application/json" \
 -d '{"password_remembrance": 1}'

Read the value back before retrying. API policy changes take about 20 seconds to apply and synchronize across the manager cluster nodes, rerun the GET from step 2 and confirm it returns 1.
Re-run the Update action in VCF Operations. In my case the task completed with the recycled password, and because the change went through the supported workflow, the appliance password, the stored credential, and the expiration metadata all ended up in sync, no manual vault surgery required.
Restore the exact value you recorded in step 2: not “whatever the default is”:

curl -k -u admin:"$NSXPASS" -X PUT \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy \
 -H "Content-Type: application/json" \
 -d '{"password_remembrance": <original-value>}'

Two follow-ups worth knowing:

Even with password_remembrance: 1, you can never “change” to the password that is currently set (the workflow requires current and new to differ) and the most recent entry in the history may still be blocked. If you absolutely must land back on a very recently used password (labs again), rotate through one or two throwaway passwords first.
Keep an eye on api_max_auth_failures: enough failed update attempts with a wrong current password will lock the account on top of everything else.

If Reuse Is Not Your Problem

The same flattened “call from NSX received invalid response” error can hide any password policy rejection, not just reuse. Start with the failed task’s Cause field: in this environment it carried NSX’s real error text, and reading it costs nothing.

If the Cause is still too vague, you can provoke NSX into stating its verbatim reason by attempting the change directly against the node API, with one important caveat:

⚠️ This is not a validation call. PUT /api/v1/node/users/<userid> is the real password-change API; there is no dry-run. If the request succeeds, the NSX admin password has just changed outside VCF Operations: the Fleet vault and SDDC Manager still hold the old credential, and every workflow that authenticates to NSX inherits that drift until you rotate once more through the VCF Operations Update action (entering the just-set password as the current one). Only run this with a password you actually intend to keep.

First find the user ID (the built-in admin is typically 10000):

curl -k -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/users

Then attempt the change, letting jq build the payload so special characters survive the shell:

read -rsp "New password: " NEWPASS; echo

jq -nc --arg old "$NSXPASS" --arg new "$NEWPASS" \
 '{password: $new, old_password: $old}' |
curl -k -u admin:"$NSXPASS" -X PUT \
 -H "Content-Type: application/json" -d @- \
 https://<nsx-manager-fqdn>/api/v1/node/users/<userid>

unset NEWPASS

Whatever check is failing comes back in plain text: reuse, minimum length, character classes, or the Linux PAM dictionary baseline. That last one deserves a mention: the docs state the default password complexity rules “as enforced by the Linux PAM module” always apply on top of the configurable settings, and there is no auth-policy knob to turn them off. Dictionary-based passwords (including the <VendorName>1!-style classics every lab inherits) get rejected on API-driven changes even when they satisfy length and character-class rules. If the error says “dictionary word” or “too simplistic,” no password_remembrance setting will save you; pick a password without recognizable words.

If the direct call succeeds but the VCF Operations workflow still fails, then you genuinely have a workflow-side problem, and the component-manager-plugin-*.log on the VCF Operations node is the place to look, but remember the vault is now behind, so plan one more rotation through Update once the workflow is fixed.

Verification

After the update task completes:

The Update Password request in Management Tasks completes successfully.
The new password authenticates against the NSX API: a quick policy read doubles as a credential check (prompt again so the variable holds the new password, and unset NSXPASS when done):

read -rsp "New NSX admin password: " NSXPASS; echo
curl -k -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy

UI login to NSX Manager with the new password works.
password_remembrance is restored to the exact value you recorded before the workaround, not whatever the troubleshooting session left behind.
The password entry in Fleet shows a healthy status and refreshed expiration metadata, the account query from the expired password sync post works for this.

Final Notes

The error message is the villain of this story. “Call from NSX received invalid response” sent me looking at connectivity and certificates, when the task details had the real answer one click deeper: “Password has been already used. Choose another.” When a VCF Operations password task fails with VCFOPSPWID03, expand the task and read the Cause before touching anything else.

The second lesson is that the NSX build in VCF 9.1 rejects disabling password history, new password_remembrance values are limited to 1 through 12, with a default of 5. In a lab, temporarily dropping it to 1 is a reasonable escape hatch for recycling a password, as long as you restore the original value afterward. In production, reusing passwords is the actual problem, use a new compliant password and keep the history policy strict.

Fixing Expired Password Sync in VCF 9.1 Fleet Management

Cosmin Trif — Tue, 07 Jul 2026 17:30:00 +0000

In VCF 9.1, Fleet Password Management accounts can get stuck in EXPIRED status, and the password workflows changed compared to 9.0. In 9.0, you may have seen UI actions such as Disconnected or Remediate. In the 9.1 environment I was working on, those actions were not available. The UI exposed Update, and the Fleet Password Management API provides the same kind of workflow: query password accounts, then update or rotate the password.

Symptoms

The issue in this case was simple to describe but easy to chase in the wrong place:

Some password accounts showed as EXPIRED in Fleet password management.
The VCF Operations root password had already been changed outside Fleet.
The 9.1 UI did not expose the older remediation-style options, so the useful paths were the Update action for single accounts and the API for repeatable remediation.

This post walks through the recovery path I used for VCF 9.1.

What Changed in VCF 9.1 Password Management

The important realization is that VCF 9.1 Fleet Password Management does not expose a documented “store this already changed password in the vault” workflow.

The UI exposes Update for a password account. For automation or for working through a larger list of expired entries, the official API exposes:

POST /suite-api/api/fleet-management/password-management/accounts/query
PUT /suite-api/api/fleet-management/password-management/accounts/{passwordAccountKey}/password

The update workflow requires both the current password and the new password. In the API these fields are currentPassword and newPassword; in the UI they are entered through the Update action. Either way, this is a rotation workflow. If a password was already changed directly on the appliance, you need to provide that current real password and rotate once more through Fleet so the appliance password, the vault, and the expiration metadata are back in sync.

References:

Get a VCF Operations API Token

The VCF Operations API supports OpsToken authorization. Set the VCF Operations API host first:

OPS="https://<vcf-operations-fqdn>"

Note: These examples use curl -k, which skips TLS certificate verification, common in labs where the appliance still has its self-signed certificate. With -k you are sending credentials to whatever answers on that FQDN. In production, drop -k and trust the VCF Operations CA instead, for example curl --cacert /path/to/vcf-ops-ca.pem ..., or add the CA to the system trust store.

Use an account that is authorized to call the Fleet password management APIs. Avoid echoing the password back to the terminal. The read -rp / read -rsp prompts in this post use bash syntax; if your interactive shell is zsh, either start bash first or use the zsh form, for example read -rs 'VCFOPS_PASSWORD?VCF Operations password: '.

Acquire a token:

read -rp "VCF Operations username: " VCFOPS_USERNAME
read -rsp "VCF Operations password: " VCFOPS_PASSWORD; echo

TOKEN=$(jq -nc \
 --arg u "$VCFOPS_USERNAME" \
 --arg p "$VCFOPS_PASSWORD" \
 '{username:$u,password:$p}' |
 curl -sk -H 'Content-Type: application/json' -d @- \
 "$OPS/suite-api/api/auth/token/acquire" | jq -r .token)

unset VCFOPS_PASSWORD
[ -n "$TOKEN" ] && [ "$TOKEN" != "null" ] || echo "Token acquisition failed" >&2

Test the token:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 "$OPS/suite-api/api/versions/current" | jq .

Query Expired Password Accounts

Now query the expired password entries:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{"status":"EXPIRED"}' \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=200" \
 | tee /tmp/expired-password-accounts.json | jq .

For a cleaner table:

jq -r '.vcfPasswordAccounts[] |
 [.appliance, .applianceFqdn, .userName, .credentialType, .expiryDate, .passwordAccountKey] |
 @tsv' /tmp/expired-password-accounts.json

I usually sort the output into a tracking table like this:

Appliance	FQDN	User	Credential Type	Action
VCF_OPERATIONS	`<vcf-operations-fqdn>`	root	SSH	Update in UI or rotate through API
SDDC_MANAGER	`<sddc-manager-fqdn>`	root	SSH	Confirm current password, then rotate
VCF_SERVICES_RUNTIME	`<vcf-services-runtime-fqdn>`	vmware-system-user	SSH	Confirm ownership before rotating

Do not blindly rotate everything at once. Some accounts may be service users, some may belong to appliances that are temporarily unreachable, and some may be stale entries for components that no longer exist. Also note that domain-level credentials such as ESXi, vCenter, and NSX are owned by SDDC Manager rather than the Fleet vault; those rotate through the SDDC Manager /v1/credentials API instead.

Update an Expired Password Account

For a single account, you can do this from the VCF Operations UI by selecting the expired password account and using Update. Enter the current working password and the new final password.

For repeatable remediation, use the API equivalent below.

One important CLI note: do not paste passwords with special characters directly into an inline curl or shell variable assignment. Passwords that contain an exclamation mark can be changed or rejected by the shell before they ever reach the API because ! may trigger history expansion in interactive shells.

If you are using Bash, disable history expansion for the session:

set +H

If you are using Zsh, disable bang history expansion:

unsetopt BANG_HIST

The safer pattern is to prompt for the password and let jq build the JSON payload.

Pick one expired entry and export the key:

KEY="<passwordAccountKey>"

Run the update call. The currentPassword must be the password that works on the target right now. The newPassword is the new final password you want Fleet to set and store.

read -rsp "Current password: " CUR; echo
read -rsp "New password: " NEW; echo

jq -nc --arg c "$CUR" --arg n "$NEW" \
 '{currentPassword:$c,newPassword:$n}' |
curl -sk -X PUT \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d @- \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/${KEY}/password" \
 | tee /tmp/pwupdate-task.json | jq .

unset CUR NEW

The response should create a password update workflow request. Track the task in VCF Operations, or use the returned request ID if your environment exposes the workflow request endpoint.

After the task completes, query the account again. Filter by the FQDN of the appliance you just updated (it is in your tracking table or /tmp/expired-password-accounts.json) so the result does not depend on how many password accounts the fleet has:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{"applianceFqdn":"<appliance-fqdn>"}' \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=200" \
 | jq --arg KEY "$KEY" '.vcfPasswordAccounts[] | select(.passwordAccountKey==$KEY)'

You want to see the account move out of EXPIRED and show a new expiryDate.

If the Password Was Already Changed Outside Fleet

This was the awkward case in my environment.

If the password has already been changed manually on the appliance, the Fleet vault may still have the old value. Since 9.1 does not provide a documented vault-only update, use the actual current appliance password as the current password and rotate once more to a new final password through the UI Update action or the API.

That gives Fleet a successful workflow and lets it update:

The appliance password
The stored credential
The expiration metadata
The password account status

If you want to keep the exact same password that was already set manually, that is where the product gap shows up. The documented workflow wants a change from current to new.

If the Current Password Is Unknown

If nobody knows the current password, Fleet cannot update it because the workflow requires the current password.

Use the native recovery method for that appliance first. For example:

Open the VM console.
Reset the account to a temporary known password.
Confirm SSH or console login works.
Immediately rotate from temporary to final through the UI Update action or the Fleet API.

Do not leave the temporary password as the final state. The point is to re-enter the supported Fleet workflow as quickly as possible.

If an Entry Looks Stale

Before rotating, verify that the appliance still exists and belongs to the environment:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{"applianceFqdn":"<appliance-fqdn>"}' \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=50" \
 | jq .

Also check DNS and reachability from the shell where you are running the remediation:

getent hosts <appliance-fqdn>
curl -k -I https://<appliance-fqdn>

If the component has been removed but the password entry remains, do not delete database rows manually. Open a support case or use the supported inventory cleanup path for that component.

Verification Checklist

After the expired entries are addressed, verify the following:

The expired account query returns fewer entries, ideally zero for the scope you remediated.
Updated accounts show a refreshed expiryDate.
The password update workflow completed successfully.
Direct login to the target appliance works with the new password.

Query all remaining expired entries:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{"status":"EXPIRED"}' \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=200" \
 | jq '.vcfPasswordAccounts'

Finally, clean up the working files, since they contain fleet inventory details such as appliance FQDNs, usernames, and password account keys:

rm -f /tmp/expired-password-accounts.json /tmp/pwupdate-task.json

Final Notes

The main lesson is that VCF 9.1 password management needs to be treated as an update or rotation workflow. The password account state is exposed through VCF Operations Fleet Password Management, and expired entries need to be handled through the UI Update action or the documented account query and password update API calls. This is the same rotate-only philosophy the platform has had since the vRSLCM Locker password management days; the tooling changed, the workflow shape did not.

For this kind of issue, I would avoid two shortcuts:

Do not use VCF 9.0 UI guidance for VCF 9.1.
Do not update Fleet LCM database rows by hand.

The reliable path is to update expired accounts through the UI or API and verify the resulting password account state.

Upgrading VCF 9.1 to 9.1.0.0300 Step by Step

Cosmin Trif — Sun, 05 Jul 2026 17:30:00 +0000

The 9.1.0.0300 patch train for VMware Cloud Foundation 9.1 is applied through a few different lifecycle views. In my lab, the update covered the Fleet Lifecycle component, VCF Operations, SDDC Lifecycle, and SDDC Manager.

Before starting, review the official Broadcom release notes:

Broadcom describes these patch releases as time-sensitive fixes that can be applied between major, minor, and maintenance releases. That also means every component may not move at the same time, so check what is actually offered in your environment instead of assuming the whole VCF bill of materials changes together.

Versions in This Upgrade

Here is what was upgraded in this run.

Component	From	To
Fleet Lifecycle	9.1.0.0.25371109	9.1.0.0300.25536195
VCF Operations	9.1.0.0200.25500350	9.1.0.0300.25531073
SDDC Lifecycle	9.1.0.0200.25495903	9.1.0.0300.25536194
SDDC Manager	9.1.0.0200 build 25496021	9.1.0.0300 build 25536191

As always, take backups or snapshots that match your normal rollback procedure, confirm that the Broadcom depot is reachable, and make sure no other lifecycle operations are running before you start.

Start in VCF Management

Log in to VCF Operations and go to Build -> Lifecycle -> VCF Management -> Upgrade. If the lifecycle metadata has not synced recently, use Sync before checking for available target versions.

The Fleet Lifecycle card is the first item I updated.

Click the target version selector and choose the 9.1.0.0300 target.

After selecting the target, the Fleet Lifecycle card should show the upgrade path to 9.1.0.0300.25536195. Click Upgrade.

Upgrade Fleet Lifecycle

Once the task starts, the Fleet Lifecycle card changes to Upgrade in progress.

Open Upgrade details to monitor the subtask list. In my run, the task staged binaries, ran component stage prechecks, and initialized the VCF component stage.

When this finishes, the Fleet Lifecycle card should show the new current version.

Set the VCF Management Target Version

Next, set the management component target version. In the Target Version 9.1.0.* section, click Change Target Version.

In my lab I used the wildcard 9.1.0.* global version. Select the patch release for VCF Operations and any other management components that might need to be upgraded.

Click Set Version and then run the prechecks. The page should show the components that are ready for upgrade.

Upgrade VCF Operations

Start with VCF Operations. The row in my environment moved from 9.1.0.0200.25500350 to 9.1.0.0300.25531073.

Click Upgrade for the VCF Operations row.

Use Upgrade details to monitor the precheck and update process. In this run, the task started an appliance precheck, initiated the VCF Ops precheck, and then began applying the system update.

When the VCF Operations update completes, the Tasks tab should show the refresh and upgrade tasks as completed.

Upgrade SDDC Lifecycle

After VCF Operations is done, return to the upgrade list. SDDC Lifecycle should still show as ready if you did not upgrade everything at once.

In my environment, SDDC Lifecycle moved from 9.1.0.0200.25495903 to 9.1.0.0300.25536194.

Click Upgrade for SDDC Lifecycle and monitor the upgrade details.

The details panel should show the SDDC Lifecycle plugin being staged in the VCF services runtime.

Wait for the task to complete before moving on to the SDDC Manager upgrade plan.

Download the SDDC Manager Patch Binary

Now go to Build -> Lifecycle -> VCF Instances -> instance-a -> Binary Management.

Set VCF Version to 9.1, select Patch Binaries, and confirm the SDDC Manager 9.1.0.0300 binary is downloaded. In this run the binary was released on Jun 29, 2026 and was 2.41 GB.

If it is not downloaded yet, select the binary and click Download.

Edit the Workload Domain Upgrade Plan

Go to the management domain under the VCF instance, then open the Upgrades tab. In this example the management domain is mgmt-a.

Click Edit Plan.

In Plan Component Upgrade, select SDDC Manager and choose target version 9.1.0.0300. In my run, ESX, NSX, and vCenter were already on 9.1.0.0100 and were not selected for this 0300 SDDC Manager patch plan.

Click Next and review the plan. The wizard notes that SDDC Manager is upgraded first so it can continue orchestrating the remaining component lifecycle work.

Click Submit Plan.

Start or Schedule the SDDC Manager Upgrade

The upgrade sequence should now show the SDDC Manager 9.1.0.0300 upgrade as the available step. You can run it immediately with Upgrade Now or schedule it for a maintenance window.

After starting the upgrade, monitor the VMware Cloud Foundation Upgrade Status page. You can also click View Upgrade Activity for deeper task progress.

In my run, the visible steps included:

Run VCF Prerequisite
Setup Common Appliance Platform
Validate Services Before Upgrade
Remove Packages Pre Upgrade

Do not refresh services or reboot appliances manually while the upgrade is running. Let the lifecycle workflow complete.

Validate the Upgrade

After the workflow completes, return to the management domain and open Component Versions.

The SDDC Manager row should show the same current and target version, with status On Target.

I also recommend checking:

VCF Management -> Tasks for failed or partially completed tasks
VCF Operations health and adapters
SDDC Manager service health
Fleet password management for accounts showing EXPIRED. I covered the remediation in Fixing Expired Password Sync in VCF 9.1 Fleet Management
Workload domain upgrade status
Depot/binary cleanup after you are comfortable with the rollback window

If you keep older bundles around for too long, they can consume a lot of space. I covered bundle cleanup separately in Cleaning up VCF Bundles.

Exporting and Restoring vCenter 8.x Permissions Before Breaking Enhanced Linked Mode

Cosmin Trif — Tue, 30 Jun 2026 15:45:00 +0000

Breaking Enhanced Linked Mode is one of those changes where the technical command is not the hardest part. The hard part is making sure administrators, service accounts, and external products can still log in and do their jobs after the vCenters no longer share the same SSO domain.

In this post I will walk through a repeatable way to export and restore vCenter 8.x RBAC data before splitting Enhanced Linked Mode. The focus is on three permission layers:

Custom roles
Explicit object or inventory permissions
Global permissions

The short version is this: object permissions usually survive the split because they live in the vCenter database, but global permissions and SSO-domain objects need special handling. If you depend on global permissions, do not start the split until you have exported them and have a plan to recreate or import them afterward.

What Changes When You Split ELM

Enhanced Linked Mode lets multiple vCenter Server instances share one vCenter Single Sign-On domain. When you break that relationship, the vCenters stop sharing the same SSO namespace.

That matters because not all permissions are stored the same way.

Item	What to expect
Custom roles	Stored in the vCenter database, so they usually survive the split. Export them anyway and re-import if anything drifted.
Object permissions	Usually remain on the vCenter inventory objects, but the principals must still resolve after the split.
Global permissions	Must be exported and recreated/imported after the split.
Local SSO users/groups	Must be recreated if they are still needed.
AD/LDAP identity sources	Must be recreated or verified on the standalone vCenter.
Solution users/plugins	Usually need product-specific re-registration or repair.

If the environment is managed by VMware Cloud Foundation / SDDC Manager, stop and validate supportability first. Cross-domain repointing is not something I would treat as a casual vCenter-only operation in a VCF managed environment.

Download the Helper Scripts

I use three helper scripts for this workflow:

Script	Purpose
Export-VCenterRbac.ps1	Exports custom roles and explicit object permissions from a vCenter.
Convert-AuthzDoctorPermissions.ps1	Converts `authz-doctor` output into a clean CSV for global permissions.
Import-VCenterRbac.ps1	Imports roles, optional global permissions, and object permissions after the split.

The scripts assume PowerShell 7+ and PowerCLI:

Install-Module VMware.PowerCLI -Scope CurrentUser
Set-PowerCLIConfiguration -InvalidCertificateAction Warn -Confirm:$false

Run the import with -WhatIf first. This is especially important when restoring global permissions.

Before You Export Anything

Do the boring checks first. This is where most bad outcomes are avoided.

Confirm which vCenters are staying linked and which vCenters are being split.
Confirm you have working SSO administrator access.
Confirm you have root access to each VCSA so you can run authz-doctor.
Take a file-based backup of each vCenter.
Take powered-off snapshots of all ELM nodes at the same point in time if that is your rollback method.
If vCenter HA is enabled, remove it before the split.
Identify any local SSO users/groups that need to exist after the split.
Identify all AD/LDAP identity sources that need to be recreated.
Identify solution accounts for NSX, SRM, Aria, backup products, monitoring tools, and plugins.

I also like to create at least one explicit vCenter-root permission for a known AD admin group on each vCenter before the change. Do not rely only on global permissions for your break-glass path.

Step 1: Export Roles and Object Permissions

Create one export folder per vCenter. Do not mix them. The object paths and managed object references are specific to the vCenter they came from.

cd C:\temp\vcenter-elm-rbac

.\Export-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -OutDir .\exports\vcsa01

.\Export-VCenterRbac.ps1 `
 -Server vcsa02.example.com `
 -OutDir .\exports\vcsa02

.\Export-VCenterRbac.ps1 `
 -Server vcsa03.example.com `
 -OutDir .\exports\vcsa03

Each export folder should contain:

manifest.json
roles.json
object-permissions.json
object-permissions.csv
global-permissions.template.csv

roles.json is used for import. object-permissions.json is also used for import. The CSV is there so you can review the permissions quickly in Excel or another editor.

Open manifest.json and make sure the count looks reasonable:

Get-Content .\exports\vcsa01\manifest.json | ConvertFrom-Json

Step 2: Export Global Permissions with authz-doctor

Global permissions are the part I care about most in an ELM split. They are easy to forget because they live under Administration > Access Control > Global Permissions, not on the normal inventory tree.

The authz-doctor tool is preinstalled on recent vCenter builds, but not on all of them. If the path below does not exist on your appliance, download the RPM attached to Broadcom KB 369938 and install it with rpm -i --force first, which is the install command the KB uses.

On each vCenter appliance, SSH in as root and run:

/usr/lib/vmware-vpx/scripts/authz-doctor/authz-doctor.py permission_check > /tmp/authz-doctor-permissions.txt

Copy the file back to the matching export folder:

scp root@vcsa01.example.com:/tmp/authz-doctor-permissions.txt ./exports/vcsa01/

The output is a bordered ASCII table. The permission rows look like this:

+-----------------------------------+-------+---------+-----------+-----------+--------+
| Principal | Group | Role Id | Role Name | Propagate | Entity |
+-----------------------------------+-------+---------+-----------+-----------+--------+
| VSPHERE.LOCAL\vpxd-extension-xxxx | False | -1 | Admin | True | Global |
| DOMAIN\vSphere-Admins | True | -1 | Admin | True | Global |
+-----------------------------------+-------+---------+-----------+-----------+--------+

The converter detects permission rows by their six-column layout and skips the border and header lines, so you can feed it the raw permission_check output without any cleanup.

Then convert the global permission rows into CSV:

.\Convert-AuthzDoctorPermissions.ps1 `
 -InputFile .\exports\vcsa01\authz-doctor-permissions.txt `
 -OutputFile .\exports\vcsa01\global-permissions.csv `
 -GlobalOnly

Repeat this for each vCenter.

The resulting file should have this shape:

"Principal","IsGroup","RoleId","RoleName","Propagate","Entity"
"DOMAIN\vSphere-Admins","True","-1","Admin","True","Global"

Step 3: Review Principals Before the Split

Do not blindly import the CSV later. Review it first.

Pay special attention to:

administrator@vsphere.local or users from the old SSO domain
Local SSO groups that will not exist after the split
Solution users such as vpxd-*, vsphere-ui-*, NSX, SRM, backup, or monitoring accounts
AD groups whose domain name or identity source alias may change
Custom roles that exist on one vCenter but not another

If you are moving from a shared SSO domain to standalone SSO domains, a principal name that made sense before the split may not resolve afterward. The permission entry can exist, but it is useless if the identity source cannot resolve the user or group.

Step 4: Add Temporary Object-Level Admin Access

Before breaking ELM, add a direct permission at the root of each vCenter inventory for the admin group you will use after the split.

In the vSphere Client:

Select the vCenter object at the top of the inventory.
Go to Permissions.
Add your AD admin group.
Assign the appropriate role, usually Administrator for the migration window.
Enable propagation.

This gives you a local object permission that is independent of Global Permissions. It is your practical safety net.

Step 5: Break Enhanced Linked Mode

Follow the supported vCenter 8.x ELM split procedure for your environment. At a high level, the flow is:

Shut down the vCenter being split from the SSO domain.
From one of the remaining linked vCenters, unregister the powered-off node.
Power on the vCenter being split.
Repoint it to a standalone SSO domain.
Validate that the vCenter no longer shows the other linked vCenters.

Be aware that the unregister step (cmsso-util unregister) restarts all vCenter services on the node where you run it, so plan for a short vSphere Client outage on that remaining vCenter, not just on the one being split.

One extra step if your ELM group has only two nodes: after the repoint, Broadcom KB 370062 says to also run cmsso-util unregister on the remaining node, so the stale partner information for the node that left is released.

The key point for this post is timing: do not import permissions until the standalone vCenter has its identity source and SSO domain in the final state.

Step 6: Recreate Identity Sources and Local SSO Objects

After the vCenter is standalone, log in with the new SSO administrator and recreate or verify:

AD/LDAP identity sources
Local SSO users
Local SSO groups
Any SSO group nesting used by the exported permissions
Certificates or trust needed for LDAPS

Also check Global Permissions for stale entries that still reference the previous SSO domain and clean them up. The split KB links a dedicated procedure for removing stale global permissions after a domain repoint, and the import is much easier to reason about when the old domain’s dead entries are gone first.

Before importing permissions, test that the principal resolves. For example, search for the AD group in the vSphere Client permission picker. If the picker cannot resolve it, the import will not give you meaningful access.

Step 7: Dry-Run the Import

Start with a dry run:

.\Import-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -InDir .\exports\vcsa01 `
 -GlobalPermissionsCsv .\exports\vcsa01\global-permissions.csv `
 -UsePrivateMobForGlobalPermissions `
 -WhatIf

The import order is:

Create or update custom roles.
Import global permissions if a CSV is provided.
Import object permissions.

The global permission import uses the vCenter MOB endpoint for AuthorizationService.AddGlobalAccessControlList. That is why the switch is intentionally named -UsePrivateMobForGlobalPermissions. I want it to be obvious that this is different from the normal object permission path through AuthorizationManager.

The MOB route is specifically the vCenter 8.x answer. vSphere 9.0 added a supported REST API for global permissions (POST /api/vcenter/authorization/permissions with a GlobalAcl DynamicID), so if you are doing this on 9.x, use that instead.

If you only want to restore roles and global permissions first, skip object permissions:

.\Import-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -InDir .\exports\vcsa01 `
 -GlobalPermissionsCsv .\exports\vcsa01\global-permissions.csv `
 -UsePrivateMobForGlobalPermissions `
 -SkipObjectPermissions `
 -WhatIf

Step 8: Run the Import

Once the dry run looks right, run it without -WhatIf:

.\Import-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -InDir .\exports\vcsa01 `
 -GlobalPermissionsCsv .\exports\vcsa01\global-permissions.csv `
 -UsePrivateMobForGlobalPermissions

If you changed principal names during the split, edit global-permissions.csv before running the import. For example, change:

OLDSSO\vSphere Admins

to:

CORP\vSphere Admins

or whatever the correct identity source now exposes.

Step 9: Verify Access

Do not stop after the script completes. Verify the permission model from both the API side and the user side.

Run authz-doctor again:

/usr/lib/vmware-vpx/scripts/authz-doctor/authz-doctor.py permission_check > /tmp/authz-doctor-permissions-after.txt

Check Global Permissions in the vSphere Client:

Administration > Access Control > Global Permissions

Then test real logins:

Log out of the SSO administrator account.
Log in as a user from the restored AD admin group.
Confirm you can see the expected inventory.
Confirm you can perform a low-risk administrative action.
Test at least one non-admin role if you have delegated access.

For object permissions, you can also re-export and compare the count:

.\Export-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -OutDir .\exports\vcsa01-after

Compare-Object `
 (Import-Csv .\exports\vcsa01\object-permissions.csv) `
 (Import-Csv .\exports\vcsa01-after\object-permissions.csv) `
 -Property Principal,RoleName,Propagate,EntityPath

Some differences are expected if you intentionally changed principals or cleaned up stale entries. Unexpected missing admin groups are not expected.

Common Gotchas

The import cannot fix an identity source that does not exist. Recreate AD/LDAP first.

Custom role IDs can change. The import script matches role names first because role IDs are not always stable across systems.

Object permissions can point to objects that no longer exist. The script tries the original managed object reference first and then the inventory path. If neither resolves, review the CSV and decide whether the permission is still needed.

Solution users are not normal human permissions. If NSX, SRM, Aria, a backup product, or a monitoring tool owns a registration, use that product’s supported reconnect or repair procedure instead of blindly restoring stale solution-user permissions.

Global permissions are broad. Review them carefully before bringing them back into a standalone vCenter.

References

The goal is not just to preserve a list of permissions. The goal is to preserve access that still resolves, still matches the right role, and still works after each vCenter stands on its own.

Clearing a Full vSAN Trace Ramdisk Across ESXi Hosts in Parallel

Cosmin Trif — Wed, 24 Jun 2026 10:30:00 +0000

While reviewing an SOS support bundle from a VMware Cloud Foundation environment, I noticed every ESXi host in the management cluster was logging the same warning, over and over, right up to the moment the bundle was collected:

[vob.visorfs.ramdisk.full] Cannot extend visorfs file
/vsantraces/vsantracesLSOMVerbose--...zst because its ramdisk
(vsantraceFailover) is full.

These are -INFO level VOB events, not errors, and they do not touch your data or VMs. But when the same symptom appears on all hosts at once, fires continuously, and never clears on its own, it is worth understanding what is actually happening and fixing it cluster-wide rather than logging into each host by hand. This post walks through the diagnosis and a small Bash script that queries, fixes, and reclaims space on every host in parallel.

Understanding the vSAN Trace Ramdisk

vSAN writes diagnostic traces (DOM, LSOM, CLOM, PLOG, and others) to /vsantraces, an in-memory location backed by a ramdisk. Under normal operation these traces rotate into compressed .zst archives and the ramdisk stays well under capacity. ESXi also keeps a secondary vsantraceFailover ramdisk that catches trace writes when the primary cannot be extended.

The warning above means the failover ramdisk has hit 100% and the trace daemon can no longer write new trace data. Because tracing is purely diagnostic, the cluster keeps running normally, but you lose trace history, the logs fill with noise, and a genuinely useful troubleshooting tool is effectively offline.

A few things are easy to get wrong here, so it is worth being precise:

This is not caused by verbose tracing being left on. On a healthy vSAN ESA cluster, LSOMVerbose is enabled by default. Confirm the configured level before assuming someone changed it.
A vsantraced restart re-initializes the daemon but does not purge files already sitting on the failover ramdisk. If the ramdisk is full, restarting alone will often leave it full.
The real reclaim comes from removing the old rotated .zst archives, which are the bulk of the consumed space.

Diagnosing Before You Touch Anything

The first job is to confirm the state on every host with read-only commands. Three pieces of information tell you almost everything:

# What trace level is actually configured?
esxcli vsan trace get

# Is the failover ramdisk full?
vdf -h | grep -i vsantrace

# What is consuming the space?
ls -lhS /vsantraces/ | head

If vdf shows a line like vsantraceFailover 300M 300M 0B 100%, that is your smoking gun. The ls output will typically show several large vsantraces--*.zst archives (often the configured max file size each) as the dominant consumers.

A note on SSH: ESXi SSH is disabled by default, and a recent SOS bundle will reflect that. Enable it per host first (vCenter > Host > Configure > Services > SSH > Start) before running anything below, and disable it again when you are done. If you would rather keep SSH off entirely, the same commands can be issued through vCenter with PowerCLI Get-EsxCli.

Querying Every Host at Once

Logging into hosts one at a time does not scale, and the whole point is that this condition tends to hit the entire cluster together. The script below fans out over SSH: it launches one background job per host, waits for all of them, and writes each host’s output to its own file. A per-connection timeout keeps a single unreachable host from hanging the whole run.

#!/usr/bin/env bash
#
# query-vsan-traces.sh
# Query (and optionally fix) all ESXi hosts in parallel for the
# vSAN "vsantraceFailover ramdisk full" condition.
#
# Usage:
# ./query-vsan-traces.sh # QUERY only (read-only, default)
# ./query-vsan-traces.sh --fix # QUERY, then restart vsantraced
# ./query-vsan-traces.sh --clean # QUERY, then delete OLD .zst archives
# ./query-vsan-traces.sh --clean host1 # restrict the action to specific hosts
# ./query-vsan-traces.sh --clean --yes # skip the confirmation prompt
# SSH_USER=root ./query-vsan-traces.sh # override the SSH user (default: root)

set -u

# --- config -----------------------------------------------------------------
HOSTS_DEFAULT=(
 esx-01.example.lab
 esx-02.example.lab
 esx-03.example.lab
 esx-04.example.lab
)
SSH_USER="${SSH_USER:-root}"
# Per-connection timeouts so one unreachable host can't hang the whole run.
SSH_OPTS=(-o ConnectTimeout=10 -o BatchMode=yes -o StrictHostKeyChecking=accept-new)
OUTDIR="vsan-trace-report-$(date +%Y%m%d-%H%M%S)"
# ---------------------------------------------------------------------------

# --- arg parsing ------------------------------------------------------------
DO_FIX=0
DO_CLEAN=0
ASSUME_YES=0
HOSTS=()
for arg in "$@"; do
 case "$arg" in
 --fix) DO_FIX=1 ;;
 --clean) DO_CLEAN=1 ;;
 --yes|-y) ASSUME_YES=1 ;;
 -*) echo "Unknown option: $arg" >&2; exit 2 ;;
 *) HOSTS+=("$arg") ;;
 esac
done
[ ${#HOSTS[@]} -eq 0 ] && HOSTS=("${HOSTS_DEFAULT[@]}")

mkdir -p "$OUTDIR"

# The diagnostic commands run on each host. Keep them read-only.
read -r -d '' QUERY_CMDS <<'EOF'
echo "===== $(hostname) ====="
echo "--- esxcli vsan trace get ---"
esxcli vsan trace get 2>&1
echo "--- ramdisk usage (vsantrace) ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
echo "--- top trace files by size ---"
ls -lhS /vsantraces/ 2>/dev/null | head -n 15 || echo "(/vsantraces not present)"
echo "--- recent ramdisk-full VOBs (last 20) ---"
grep "vob.visorfs.ramdisk.full" /var/log/vobd.log 2>/dev/null | tail -n 20 || echo "(none)"
EOF

# Remediation A: restart vsantraced and show before/after ramdisk usage.
read -r -d '' FIX_CMDS <<'EOF'
echo "===== $(hostname) ====="
echo "--- BEFORE: ramdisk usage ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
echo "--- restarting vsantraced ---"
/etc/init.d/vsantraced restart 2>&1
sleep 3
echo "--- AFTER: ramdisk usage ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
EOF

# Remediation B: delete OLD .zst archives, keeping the newest 3 per host.
# Active (non-.zst) trace files are never touched.
read -r -d '' CLEAN_CMDS <<'EOF'
echo "===== $(hostname) ====="
echo "--- BEFORE: ramdisk usage ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
echo "--- deleting all but the newest 3 .zst archives ---"
ls -t /vsantraces/*.zst 2>/dev/null | tail -n +4 | while read -r f; do
 echo "rm $f"
 rm -f "$f"
done
echo "--- AFTER: ramdisk usage ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
EOF

run_on_host() {
 local host="$1" cmds="$2" suffix="$3"
 local out="$OUTDIR/${host}${suffix}.txt"
 if ssh "${SSH_OPTS[@]}" "${SSH_USER}@${host}" "$cmds" >"$out" 2>&1; then
 echo "OK $host -> $out"
 else
 echo "FAIL $host (see $out for error)"
 fi
}

fan_out() {
 local cmds="$1" suffix="$2"
 local pids=()
 for h in "${HOSTS[@]}"; do
 run_on_host "$h" "$cmds" "$suffix" &
 pids+=("$!")
 done
 wait "${pids[@]}"
}

confirm() {
 [ "$ASSUME_YES" -eq 1 ] && return 0
 printf "Proceed on all %d hosts? [y/N] " "${#HOSTS[@]}"
 read -r reply
 case "$reply" in y|Y|yes|YES) return 0 ;; *) echo "Aborted."; exit 0 ;; esac
}

# --- 1. QUERY (always) ------------------------------------------------------
echo "Querying ${#HOSTS[@]} hosts in parallel as user '$SSH_USER'..."
echo "Output dir: $OUTDIR"
echo
fan_out "$QUERY_CMDS" ""
echo
echo "Hosts with the failover ramdisk at 100%:"
grep -l "vsantraceFailover.*100%" "$OUTDIR"/*.txt 2>/dev/null \
 || echo " (none; ramdisks have headroom)"

# --- 2. FIX (only with --fix) -----------------------------------------------
if [ "$DO_FIX" -eq 1 ]; then
 echo; echo "FIX MODE: restart 'vsantraced' on ${#HOSTS[@]} host(s)."
 echo "Only affects diagnostic tracing: no vSAN data, VMs, or I/O touched."
 confirm
 fan_out "$FIX_CMDS" "-fix"
fi

# --- 3. CLEAN (only with --clean) -------------------------------------------
if [ "$DO_CLEAN" -eq 1 ]; then
 echo; echo "CLEAN MODE: delete old .zst archives on ${#HOSTS[@]} host(s)."
 echo "Keeps the newest 3 archives per host; active trace files untouched."
 confirm
 fan_out "$CLEAN_CMDS" "-clean"
fi

Step 1: Run the Query

With your hostnames filled into HOSTS_DEFAULT (or passed on the command line), run it with no arguments first. This is read-only and safe:

./query-vsan-traces.sh

You get one output file per host plus a summary line listing exactly which hosts are sitting at 100%. Confirm from the esxcli vsan trace get output that the trace level is the default before going further, if it is, you know this is a stuck-ramdisk problem, not a misconfiguration.

Step 2: Try the Restart

The lightest-touch remediation is to restart the trace daemon, which re-initializes the trace directories:

./query-vsan-traces.sh --fix

The script prints before/after vdf output for each host. In my case the restart completed cleanly on all hosts but the failover ramdisk stayed at 100%; the daemon came back, but the files already on the ramdisk were not purged. That is the expected outcome when the ramdisk is already full, and it is exactly why the script does not stop here.

Step 3: Reclaim the Space

The reclaim step deletes the old rotated .zst archives, which are what actually fills the ramdisk. The script keeps the newest three archives per host so you do not lose all recent history, and it never touches the active (non-.zst) trace files:

./query-vsan-traces.sh --clean

Each remediation mode prompts for confirmation once, listing the hosts it is about to act on, and you can skip the prompt with --yes in an automation context. After the cleanup, the AFTER vdf line should show the failover ramdisk back under capacity, and the ramdisk.full VOBs should stop appearing within a minute or two.

Why This Order Matters

It is tempting to jump straight to deleting files, but running the query first buys you two things. You confirm the trace configuration is actually default (ruling out a real misconfiguration), and you capture a record of the pre-change state in the per-host output files. The restart is offered before the cleanup simply because it is the lower-impact action; when it does not free space, the cleanup is the definitive fix.

If a host is still at 100% after a cleanup, that points to something actively re-filling the ramdisk faster than rotation can drain it: a stuck writer, or a persistent trace target that is unwritable. At that point you are past the mechanical fixes and it is worth opening a support request rather than looping on the same commands.

Conclusion

The “vsantraceFailover ramdisk full” warning looks alarming because it repeats endlessly across every host, but it is a contained, diagnostic-only condition with a safe, repeatable fix. The key is to treat the whole cluster as a unit: query every host in parallel, confirm the configuration is default, attempt the cheap restart, and reclaim space by clearing old trace archives when the restart is not enough. A small Bash wrapper turns what would be a tedious host-by-host chore into three commands you can run from your workstation, with a written record of each host’s state along the way.

Fixing "Error attempting Upgrade Database Schema" When Upgrading vCenter 8.0 U3x to 9.1.0.0

Cosmin Trif — Tue, 16 Jun 2026 12:45:00 +0000

If you’re moving vCenter Server from 8.0 U3x to 9.1.0.0 (typically as part of a VCF 9.1 deployment or a VVF 9.1 upgrade) the run can fail partway through with:

Error attempting Upgrade Database Schema
Please check vcintegrity migration logs for details.

This is one of those failures that looks scary in the UI but has a clean, well-understood root cause: a handful of orphaned rows in the vcIntegrity database that trip a new foreign-key constraint introduced in 9.1. Below is the whole story; what the error means, how to read the logs, why it happens, and how to clear it so the upgrade completes. It’s the same shape of problem I wrote about in fixing PostgreSQL on Fleet Management 9.0.1: a schema change colliding with pre-existing data.

This is documented in Broadcom KB 442223. The supported fix uses a script attached to that KB. I’ll walk through the official steps first, then share an unofficial helper of my own for the curious: clearly marked, and not a replacement for the KB script.

The symptom

The upgrade proceeds normally until the vcIntegrity import phase, then halts with the generic schema error shown above. The UI doesn’t tell you what in the schema failed; it just points you at the vcIntegrity migration logs. That’s your cue to SSH in and read the actual exception.

Reading the upgrade log

On the source vCenter, open the vcIntegrity import log:

/var/log/vmware/upgrade/Import_com.vmware.vcIntegrity_####_##_##_##_##.log

The meaningful lines look like this (timestamps and IDs trimmed):

DEBUG vmware-vci-vciInstallUtils [VdbODBCError::VdbODBCError] SQLGetDiagRec
 SQLState='23503' NativeError=1
 MessageText='ERROR: insert or update on table "pm_recommendation_spec"
 violates foreign key constraint "fk_pm_recommendation_spec"
 DETAIL: Key (entity_id, spec_id)=(domain-##, -1000) is not present in
 table "pm_recommendation_info".'

ERROR vmware-vci-vciInstallUtils [installerRunDBCommand 626] [DATABASE]
 Vdb::RunScript, SQL Exec Error: "ODBC error: (23503) - ERROR: insert or
 update on table "pm_recommendation_spec" violates foreign key constraint
 "fk_pm_recommendation_spec" ... " is returned when executing SQL statement
 "ALTER TABLE IF EXISTS PM_RECOMMENDATION_SPEC
 ADD CONSTRAINT FK_PM_RECOMMENDATION_SPEC
 FOREIGN KEY (ENTITY_ID, SPEC_ID)
 REFERENCES PM_RECOMMENDATION_INFO (ENTITY_ID, SPEC_ID)
 ON DELETE CASCADE"

ERROR __main__ Upgrade Phase 'vcIntegrity:Import' failed.
 Exception: Error attempting Upgrade Database Schema
upgrade_errors.PermanentError: Error attempting Upgrade Database Schema

Three things are worth pulling out of that wall of text:

The SQLState is 23503, which in PostgreSQL is foreign_key_violation, not a connectivity, permissions, or disk problem. The database is doing exactly what it was told to do and refusing bad data.

The failing statement is an ALTER TABLE ... ADD CONSTRAINT, not an insert from your workload. The migration is trying to add the new constraint FK_PM_RECOMMENDATION_SPEC to an existing table, and Postgres validates the rows already in that table before it will accept the constraint.

The offending key is (entity_id, spec_id) = (domain-##, -1000). There’s a row in pm_recommendation_spec whose (entity_id, spec_id) pair has no matching parent row in pm_recommendation_info. The -1000 is a sentinel/placeholder value, and domain-## is a managed-object reference, classic leftovers from objects that were removed or never fully written.

Root cause

The pm_recommendation_* tables live in the vcIntegrity database, the component behind vSphere Lifecycle Manager / Update Planner. They store lifecycle “recommendation” data: pm_recommendation_info holds the parent records, and pm_recommendation_spec holds child spec rows that are supposed to reference a parent.

Over the life of an 8.0 environment, it’s possible to end up with child spec rows whose parent info row is gone (or was never created): for example after domains/clusters are removed, or from older code paths that didn’t clean up specs. In 8.0 there was no enforced relationship between the two tables, so these orphans sat there harmlessly.

vCenter 9.1.0.0 hardens the schema. The migration adds a composite foreign key:

ALTER TABLE PM_RECOMMENDATION_SPEC
 ADD CONSTRAINT FK_PM_RECOMMENDATION_SPEC
 FOREIGN KEY (ENTITY_ID, SPEC_ID)
 REFERENCES PM_RECOMMENDATION_INFO (ENTITY_ID, SPEC_ID)
 ON DELETE CASCADE;

When PostgreSQL adds a foreign key, it validates every existing row in the child table against the parent table. Any orphan (a (entity_id, spec_id) pair with no parent) violates the new constraint, so the engine aborts the ALTER TABLE with error 23503. Because the migration treats this as a PermanentError, the whole vcIntegrity import phase fails and the upgrade stops.

In short: 9.1 is correct to demand referential integrity; your 8.0 data just doesn’t satisfy it yet. The fix is to remove the orphaned spec rows so the constraint can be created cleanly.

Broadcom notes that a fix to prevent this will be included in a future VCF release. Until then, the workaround below is the path forward.

The official fix (KB 442223 workaround)

Do this on the source vCenter appliance, before retrying the upgrade.

1. Snapshot the vCenter appliance

Take a snapshot of the source vCenter VM first. Editing the embedded database is low-risk when done correctly, but a snapshot is your guaranteed rollback. (If your environment enforces a snapshot-age policy, see changing the old-snapshot restriction, just remember to remove this snapshot once the upgrade succeeds.)

2. Download the cleanup script from the KB

Download cleanup_rec_orphans.py from the Attachments section of KB 442223. This is the supported script, use it for production work.

3. Copy the script to the appliance

Place it in /tmp on the vCenter server. You can use an SCP client such as WinSCP. If WinSCP fails with a “Received too large … SFTP packet” error, follow KB 326317 (set the shell to bash for the session, or use the SCP fallback).

4. SSH in as root

5. Dry-run audit (read-only)

Run the script with no flags first. It reports orphans without changing anything:

su updatemgr -s /bin/bash -c "python3 /tmp/cleanup_rec_orphans.py"

The su updatemgr matters: the updatemgr account owns the vcIntegrity database connection, so the script authenticates correctly without you handling credentials.

6. Execute the cleanup

Once you’ve reviewed the dry-run output, run it again with --execute to delete the orphaned rows:

su updatemgr -s /bin/bash -c "python3 /tmp/cleanup_rec_orphans.py --execute"

7. Retry the upgrade

Re-run the vCenter upgrade. With the orphans gone, the ALTER TABLE ... ADD CONSTRAINT validates cleanly and the vcIntegrity import phase completes.

What the script actually does (the SQL behind it)

You don’t need to know this to run the KB script, but it helps to understand why it’s safe. The job is purely to find child rows with no parent and remove them. The detection is a straightforward anti-join:

-- Orphans: spec rows whose (entity_id, spec_id) has no parent info row
SELECT s.entity_id, s.spec_id
FROM pm_recommendation_spec s
LEFT JOIN pm_recommendation_info i
 ON i.entity_id = s.entity_id
 AND i.spec_id = s.spec_id
WHERE i.entity_id IS NULL
ORDER BY s.entity_id, s.spec_id;

And the cleanup is the matching delete:

DELETE FROM pm_recommendation_spec s
WHERE NOT EXISTS (
 SELECT 1
 FROM pm_recommendation_info i
 WHERE i.entity_id = s.entity_id
 AND i.spec_id = s.spec_id
);

That’s it. You’re deleting only spec rows that already point at nothing, exactly the rows the new constraint would reject. No valid recommendation data is touched.

Optional: an unofficial audit/cleanup helper

Use the KB script for anything you care about. The Bash helper below is a community convenience I wrote to mirror the same logic for quick auditing in labs. It is unofficial and unsupported, runs read-only by default, and should only be pointed at a vCenter you’ve snapshotted. If in doubt, stop and use cleanup_rec_orphans.py from the KB.

It does three things: locate the database that actually owns the table (so it doesn’t matter what the DB is named in your build), report the orphan count and the offending pairs, and (only with --execute) delete them inside a transaction.

#!/usr/bin/env bash
# audit_rec_orphans.sh: UNOFFICIAL community helper (KB 442223)
#
# Mirrors the intent of Broadcom's cleanup_rec_orphans.py: find/remove
# orphaned rows in pm_recommendation_spec that have no parent in
# pm_recommendation_info and therefore block the FK_PM_RECOMMENDATION_SPEC
# constraint added by the vCenter 9.1.0.0 vcIntegrity schema upgrade.
#
# DRY-RUN by default. Pass --execute to delete.
# ALWAYS snapshot the vCenter appliance first.
# Prefer the official KB script for supported/production environments.
#
# Run it the same way the KB runs its script (as the vcIntegrity DB owner):
# su updatemgr -s /bin/bash -c "/tmp/audit_rec_orphans.sh" # audit
# su updatemgr -s /bin/bash -c "/tmp/audit_rec_orphans.sh --execute" # delete
set -euo pipefail

EXECUTE=0
[ "${1:-}" = "--execute" ] && EXECUTE=1

if [ "$(id -un)" != "updatemgr" ]; then
 echo "[!] Not running as 'updatemgr': DB auth will likely fail."
 echo " Re-run: su updatemgr -s /bin/bash -c \"$0 ${1:-}\""
 exit 1
fi

# Prefer the embedded vPostgres client if present
PSQL="/opt/vmware/vpostgres/current/bin/psql"
[ -x "$PSQL" ] || PSQL="$(command -v psql)"

# Discover which database owns pm_recommendation_spec (override with VCI_DB=...)
VCI_DB="${VCI_DB:-}"
if [ -z "$VCI_DB" ]; then
 for db in $("$PSQL" -Atqc \
 "SELECT datname FROM pg_database WHERE datistemplate=false" 2>/dev/null); do
 if [ -n "$("$PSQL" -Atqd "$db" \
 -c "SELECT to_regclass('pm_recommendation_spec')" 2>/dev/null)" ]; then
 VCI_DB="$db"; break
 fi
 done
fi
[ -n "$VCI_DB" ] || { echo "[x] Could not locate the vcIntegrity DB (set VCI_DB=...)"; exit 1; }
echo "[*] Using database: $VCI_DB"

ORPHAN_FILTER="FROM pm_recommendation_spec s
 LEFT JOIN pm_recommendation_info i
 ON i.entity_id = s.entity_id AND i.spec_id = s.spec_id
 WHERE i.entity_id IS NULL"

COUNT="$("$PSQL" -Atqd "$VCI_DB" -c "SELECT count(*) $ORPHAN_FILTER")"
echo "[*] Orphaned pm_recommendation_spec rows: $COUNT"

if [ "$COUNT" = "0" ]; then
 echo "[+] No orphans: FK_PM_RECOMMENDATION_SPEC will validate cleanly."
 exit 0
fi

echo "[*] Offending (entity_id, spec_id) pairs:"
"$PSQL" -d "$VCI_DB" -c "SELECT s.entity_id, s.spec_id $ORPHAN_FILTER
 ORDER BY s.entity_id, s.spec_id"

if [ "$EXECUTE" -ne 1 ]; then
 echo
 echo "[i] DRY-RUN only. Re-run with --execute to delete the rows above."
 exit 0
fi

echo "[*] Deleting orphaned rows in a transaction…"
"$PSQL" -v ON_ERROR_STOP=1 -d "$VCI_DB" <<'SQL'
BEGIN;
DELETE FROM pm_recommendation_spec s
WHERE NOT EXISTS (
 SELECT 1 FROM pm_recommendation_info i
 WHERE i.entity_id = s.entity_id AND i.spec_id = s.spec_id
);
COMMIT;
SQL
echo "[+] Done. Re-run without --execute to confirm 0 orphans, then retry the upgrade."

Save it to /tmp/audit_rec_orphans.sh, then run the dry-run first and review the output before ever adding --execute. Like the KB script, it’s idempotent, running it again after a successful cleanup simply reports zero orphans.

Verifying the fix

Re-run the audit (dry-run); it should now report 0 orphaned rows. That single number is the whole signal: zero orphans means the new foreign key has nothing left to reject.

Then retry the vCenter upgrade and confirm the vcIntegrity:Import phase passes. If you want belt-and-suspenders confirmation, tail the new import log and check that the ALTER TABLE ... ADD CONSTRAINT FK_PM_RECOMMENDATION_SPEC statement executes without a 23503 error:

grep -i "FK_PM_RECOMMENDATION_SPEC\|23503\|Upgrade Database Schema" \
 /var/log/vmware/upgrade/Import_com.vmware.vcIntegrity_*.log

No matches on 23503 (and a clean run past the schema step) means you’re through it.

Rollback

If anything looks wrong after the cleanup or the retried upgrade, revert to the snapshot you took in step 1 and reassess before trying again. The cleanup itself only removes already-orphaned rows, so a revert puts you exactly back where you started.

Once the upgrade has completed successfully and vCenter is healthy, delete the snapshot: leaving snapshots on a vCenter appliance long-term causes its own problems (disk growth, performance, and consolidation headaches).

Notes & gotchas

Why su updatemgr? That OS account maps to the database role that owns the vcIntegrity connection. Running the script under it means you never handle DB credentials by hand, and the connection has exactly the rights it needs.

Is deleting these rows safe? Yes; you’re only removing pm_recommendation_spec rows that already reference a non-existent parent. They’re dead lifecycle-recommendation data; no live configuration depends on them. Recommendations regenerate as needed after the upgrade.

Does it apply to VVF as well as VCF? The KB lists both VMware Cloud Foundation 9.1 and VMware vSphere Foundation 9.1; the trigger is the vCenter 9.1.0.0 schema change, so it’s the same regardless of which edition you’re upgrading under.

Will it come back? Broadcom has stated a fix is planned for a future VCF release so the orphans don’t block the migration. Until then, audit-then-clean is the routine.

References

Broadcom KB 442223 - “Error attempting Upgrade Database Schema” during vCenter 8.0 U3x to 9.1.0.0 upgrade (symptoms, cause, and the official cleanup_rec_orphans.py workaround).
Broadcom KB 326317 - Connecting to vCenter Server Virtual Appliance using WinSCP fails with “Received too large … SFTP packet” (needed if file transfer to the appliance fails).

Fix: Unable to Update BOM Components, "Compatibility File Is Missing" in SDDC Manager

Cosmin Trif — Fri, 05 Jun 2026 09:15:00 +0000

If you have upgraded your VMware Cloud Foundation (VCF) environment to version 5.x from an earlier release, you may find yourself unable to update BOM (Bill of Materials) components from the SDDC Manager UI. The “Update Now” or “Configure Update” button is greyed out and you see a frustrating error:

“Update cannot be started now as compatibility file is missing”

This post walks through the symptoms, root cause, and the step-by-step fix based on Broadcom KB 396202.

Symptoms

You will typically notice one or more of the following:

In the Updates tab of the relevant Workload Domain (WLD), the “Update Now” or “Configure Update” button is greyed out.
The “Available Updates” section displays the error: “Update cannot be started now as compatibility file is missing”.
Attempting to manually download the VmwareCompatibilityData.json file fails.
In the SDDC Manager LCM debug log (/var/log/vmware/vcf/lcm/lcm-debug.log), you see entries similar to:

Recoverable I/O exception (java.net.SocketException) caught when processing request to {s}->https://storage.googleapis.com:443

VVS Compatibility Data update failed
com.vmware.vcf.compatibility.controllers.internal.vvs.exception.VvsException:
The VVS api call to https://vvs.esp.vmware.com/v1/products/bundles/type/vcf-lcm-bundle?format=json failed.
Response: Connection reset

Root Cause

The compatibility data file VmwareCompatibilityData.json is missing from the expected path on the SDDC Manager appliance:

/nfs/vmware/vcf/nfs-mount/compatibility/VmwareCompatibilityData.json

Why is it missing? The most common reason is that the SDDC Manager was upgraded from a VCF version prior to 5.0.0.0. The VVS compatibility data directory was introduced in VCF 5.0, so environments that were upgraded (rather than freshly deployed) may never have had this directory created.

Step-by-Step Resolution

Step 1: Take a Snapshot

Before making any changes, take an offline snapshot of the SDDC Manager appliance VM. This gives you a safe rollback point.

Step 2: SSH Into SDDC Manager

Connect via SSH using the vcf user, then elevate to root:

ssh vcf@<sddc-manager-fqdn>
su root

Step 3: Create the Compatibility Directory

The directory likely does not exist. Create it and navigate into it:

mkdir -p /nfs/vmware/vcf/nfs-mount/compatibility/
cd /nfs/vmware/vcf/nfs-mount/compatibility/

Step 4: Download the Compatibility Data File

Run the following curl command to pull the VmwareCompatibilityData.json file from Broadcom’s VVS endpoint:

curl --location 'https://vvs.broadcom.com/v1/products/bundles/type/vcf-lcm-v2-bundle?format=json' \
 --header 'X-Vmw-Esp-ClientId: vcf-lcm' \
 > VmwareCompatibilityData.json

Note: If the SDDC Manager does not have internet access, run the curl command on a separate Linux machine that can reach vvs.broadcom.com, then transfer the resulting VmwareCompatibilityData.json file to the SDDC Manager appliance via SCP or another file transfer method.

Step 5: Set Correct Permissions and Ownership

Apply the proper ownership and permissions so the LCM service can read the file:

chown vcf_lcm:vcf -R /nfs/vmware/vcf/nfs-mount/compatibility
chmod 755 -R /nfs/vmware/vcf/nfs-mount/compatibility/

Step 6: Restart the LCM Service

Restart the Lifecycle Manager service for the changes to take effect:

systemctl restart lcm

Wait a couple of minutes for the service to fully come back up, then refresh the SDDC Manager UI. The “Update Now” or “Configure Update” button should no longer be greyed out.

Additional Step: Update Depot URLs

If you are also seeing depot-related errors, the download URLs in the LCM configuration may still reference old repository endpoints that require a valid download token. You should update the depot configuration in:

/opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

Broadcom has a dedicated KB for this issue: Depot connection has issues - SDDC fails to download updates from online repositories (KB 389871).

On a related note, if compatibility or interoperability checks are blocking upgrades in your lab, I covered a script to bypass VCF compatibility and interoperability checks.

Quick Reference: All Commands

For convenience, here are all the commands in sequence:

# SSH in and elevate
ssh vcf@<sddc-manager-fqdn>
su root

# Create directory
mkdir -p /nfs/vmware/vcf/nfs-mount/compatibility/
cd /nfs/vmware/vcf/nfs-mount/compatibility/

# Download compatibility data
curl --location 'https://vvs.broadcom.com/v1/products/bundles/type/vcf-lcm-v2-bundle?format=json' \
 --header 'X-Vmw-Esp-ClientId: vcf-lcm' \
 > VmwareCompatibilityData.json

# Fix permissions
chown vcf_lcm:vcf -R /nfs/vmware/vcf/nfs-mount/compatibility
chmod 755 -R /nfs/vmware/vcf/nfs-mount/compatibility/

# Restart LCM
systemctl restart lcm

Scripted Components Cleanup from VCF Operations 9.1: A Step-by-Step Guide

Cosmin Trif — Wed, 03 Jun 2026 10:30:00 +0000

If you’ve been working with VCF Operations 9.1, you may have noticed that on the Build > Lifecycle > VCF Management > Components page, manually-added components such as VCF Operations for Networks, Log Management, Real-Time Metrics or VCF Automation no longer have a “Delete” option available in the UI.

In VCF Operations 9.0, this was possible on the Fleet Management > Lifecycle > Components page by clicking the three vertical dots next to a component and selecting Delete. That option is gone in 9.1.

Broadcom has published KB article 441333 with an attached Python script (cleanup_component.py) for specific failed or stale component scenarios. This is a destructive cleanup tool, not a general-purpose replacement for the product’s supported backup, restore, upgrade or uninstall workflows.

Prerequisites

Before running the cleanup script, make sure the following are in place:

Python installed on the system where you will run the script (can be any system with network access to the Fleet and VCF Management Services components).
Network access to the Fleet LCM appliance and VCF Services Runtime.
Credentials: use admin@local for the VCF Services Runtime username. Enter the password of vmware-system-user; according to KB 441333, that account and admin@local have the same password.
FQDN values: replace all placeholder FQDNs such as <fleet-lcm-fqdn> with the actual values. These can be obtained from VCF Operations UI -> Build -> Lifecycle -> VCF Management -> Components (e.g., vsp01.example.com for <vcf-services-runtime-fqdn>).

Warning: Before cleanup, verify that usable backups exist for every piece of critical component data that must be retained, including logs and databases. Do not treat this workflow, or a VM snapshot taken outside a product-supported procedure, as a way to restore a deleted component.

Supported Scope and Safety Restrictions

Use the script only for a scenario explicitly covered by the current version of KB 441333, such as a duplicate 9.0 component import, an incorrect FQDN that requires a full redeployment or a component that must be removed before a fresh deployment.

For VCF Automation 9.1, the restrictions are especially important:

The workflow completely removes VCF Automation from the Fleet.
Use it only for permanent removal or to prepare for a fresh deployment.
A fresh deployment performed after this cleanup must not be restored from the removed deployment’s backup as part of this workflow. Use Broadcom’s separate Redeploy and Restore VCF Automation procedure when restoration is required.
Do not use this procedure to clean up a VCF Automation upgrade failure. Contact Broadcom Support for that scenario.
Do not use it to clean up and then import an existing VCF Automation 9.1 system.

For imported 9.0 components, read the component-specific note before deleting anything: some operations remove only the Fleet lifecycle record and leave the existing 9.0 deployment intact. A failed upgrade entry can also remain visible after cleanup; KB 441333 states that this entry does not block subsequent upgrade tasks.

Download and Verify the Script

Download cleanup_component.py from KB 441333 (attached at the bottom of the article).

Verify the SHA256 checksum before running:

sha256sum cleanup_component.py

Expected checksum for the script currently attached to KB 441333:

253bf30d0b33e0c6640aa6cbdf96de41a4a6d75d0265db269b1f44099308ca4c

Do not run the script if the checksum differs. Download the current attachment again and recheck the KB because Broadcom can replace the attachment and publish a new checksum.

What the Script Does

The LCM component cleanup script enables administrators to manage Day-N deployed components visible in the VCF Operations UI:

List components: View all deployed components currently eligible for removal.
Delete components: Remove an eligible component record or deployment according to its component type and version. The effect is not the same for every component, so confirm the selected component ID and the relevant KB section before proceeding.

Important notes:

If a component is running a version older than 9.1, it is removed solely from the Fleet lifecycle. The script does not delete the corresponding infrastructure deployments such as vCenter or VCF Services Runtime.
VSP refers to VCF Management Services.

1. OVA Components

Deleting an OVA-based deployment removes it from Fleet lifecycle and also deletes the associated virtual machines from vCenter.

List Components Deployed on vCenter

python cleanup_component.py list ova-component --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username admin@local

Delete Component Deployed on vCenter

python cleanup_component.py delete ova-component --component-id <ova-component-id> --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username admin@local --vcenter-username <vcenter-username>

The script will prompt for the required passwords. Once confirmed, the OVA component will be removed from Fleet lifecycle and its associated VMs deleted from vCenter.

2. VCF Management Services (VSP) Components

Deleting a VSP-based deployment removes it from Fleet lifecycle and also deletes the associated deployment from VCF Services Runtime.

List Components Deployed on VCF Services Runtime

python cleanup_component.py list vsp-component --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username admin@local

Delete Component Deployed on VCF Services Runtime

python cleanup_component.py delete vsp-component --component-id <vsp-component-id> --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username admin@local

3. VCF Automation

VCF Automation is a multi-component deployment consisting of:

VCF Automation
Migration Service Engine
VCF Automation VCF Services Runtime

Important: You must remove VCF Automation and the Migration Service Engine before you remove the VCF Automation VCF Services Runtime. Follow the order below.

Destructive operation: For VCF Automation 9.1, this sequence completely removes the solution. Continue only for permanent removal or a fresh deployment without restoration. It is not supported for VCF Automation upgrade-failure cleanup. Run the entire VCF Automation 9.1 cleanup sequence in this section as the root user inside the SDDC Manager virtual machine.

Step 1: Remove VCF Automation and Migration Service Engine

List Components Deployed on VCF Services Runtime

python cleanup_component.py list vsp-component --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username admin@local

Delete Component Deployed on the VCF Automation VCF Services Runtime

python cleanup_component.py delete vsp-component --component-id <vsp-component-id> --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username admin@local --vcfa-vcf-services-runtime-fqdn <vcfa-vcf-services-runtime-fqdn> --vcfa-vcf-services-runtime-username <vcfa-vcf-services-runtime-username>

Run the delete command for both the VCF Automation component and the Migration Service Engine component before proceeding to Step 2.

Step 2: Remove the VCF Automation VCF Services Runtime

Warning: Deleting a VCF Services Runtime requires this script to be executed as the root user inside the SDDC Manager virtual machine.

List VCF Services Runtime

python cleanup_component.py list vsp-cluster --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username admin@local

Delete VCF Services Runtime

python cleanup_component.py delete vsp-cluster --component-id <vsp-cluster-id> --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username admin@local --vcenter-username <vcenter-username>

Summary

The cleanup_component.py script is a handy workaround for the missing “Delete” option in VCF Operations 9.1. The key things to remember:

Back up and verify any critical component data before running delete operations; do not assume this cleanup can be reversed with a snapshot or restored deployment.
For OVA components, deletion also removes VMs from vCenter.
For VSP components, deletion also removes the deployment from VCF Services Runtime.
For VCF Automation 9.1, cleanup means full removal: do not use it for a failed upgrade or a restore, and use a fresh deployment only when the documented scenario permits it.
Run the entire VCF Automation 9.1 cleanup sequence as root inside the SDDC Manager VM. Delete VCF Automation and Migration Service Engine first, then delete the VCF Services Runtime.
Always verify the script checksum before running it.

After removing components, also check Fleet password management for leftover or expired password accounts. I covered that remediation in Fixing Expired Password Sync in VCF 9.1 Fleet Management.

For the official script download and additional details, refer to Broadcom KB article 441333.

I hope this helps!