Cosmin.us

Deploying a VCF 9.1 Fleet with the VMware Cloud Foundation Installer: Step-by-Step Instructions

Cosmin Trif — Fri, 07 Aug 2026 09:00:00 +0000

Last year, when VCF 9.0 shipped, I covered the VCF Installer in three posts: deploying the appliance, configuring it, and deploying a VCF 9 instance. With VCF 9.1 generally available since May 12, 2026, it was time to run the whole flow again in the lab — and enough has changed that it earns a fresh walkthrough. This post covers the complete path in one go: downloading and deploying the 9.1 installer appliance, connecting the depot, downloading binaries, and deploying a new VCF fleet on top of an existing vCenter. I used the 9.1.0.0400 installer from the July 2026 refresh.

What changed since the 9.0 posts

The overall shape is the same — deploy the OVA, connect a depot, download binaries, run the deployment wizard — but 9.1 moves a few important pieces, per the official What’s New for the VCF Installer and the depot KBs:

The download token is gone. Depot authentication now uses a Download Service ID + Activation Code obtained from the VCF Business Services console. The token flow from my 9.0 configuration post no longer applies — see KB 443647.
VCF Management Services are deployed by default. The installer now provisions the VCF services runtime, fleet lifecycle, identity broker, software depot, Salt RaaS, telemetry, and the centralized License Server as part of a new deployment.
Integrated planning workflow. The wizard generates CPU, memory, storage, VLAN, and FQDN requirements up front and validates your infrastructure against them — including a Pre-fill generated FQDNs button.
Auto-generated passwords. The installer can generate the component passwords for you, retrievable after deployment.
Dual-stack support. The appliance OVA now asks for an IP Address Version (IPv4 only, or IPv4 and IPv6), and existing dual-stack environments are supported — though wizard-driven deployments on an existing vCenter remain IPv4-only; dual-stack goes through a JSON spec.
Lowercase FQDNs are now formalized. The 9.1 planning docs state do not use capital letters in the FQDN, and uppercase component FQDNs are a documented deployment-failure known issue.

Download the VCF Installer

The appliance OVA comes from the Broadcom Support Portal at support.broadcom.com — select the VMware Cloud Foundations division, go to My Downloads, and pick VMware Cloud Foundation 9 with your target release. If the portal navigation fights you, KB 401497 walks through it.

Select the release — I grabbed 9.1.0.0400 from the July 2026 patch wave
Agree to the Terms and Conditions
Download VCF-SDDC-Manager-Appliance-9.1.0.0400.25570100.ova (2.28 GB) and verify the checksum

Deploy the VCF Installer appliance

Sizing is unchanged from 9.0: 4 vCPUs, 16 GB RAM, and 914 GB storage (thin provisioning keeps that honest), per the official deployment page. The docs describe two placement options: on one of the ESX hosts that will form the management domain (the appliance is converted into SDDC Manager during deployment), or outside the management infrastructure entirely, where one appliance can deploy multiple platforms. As in the 9.0 post, I am deploying the OVA through an existing vCenter.

Log in to vCenter with a user that can deploy an OVF
Right click the cluster and click Deploy OVF Template

Select Local file and upload the installer OVA
Click Next

Give the appliance a name
Select a deployment location
Click Next

Select a compute resource and wait for the compatibility checks
Click Next

Review the deployment details — note the 914 GB thick footprint vs 4.2 GB thin
Click Next

Read and accept the license agreements
Click Next

Select a datastore for the deployment
Click Next

Select the destination network — the appliance needs to reach the ESX hosts, the VM management network, DNS, and NTP
Click Next

Fill in the template customization — root and local user passwords (minimum 15 characters with the usual complexity rules; the local password is what you will use for admin@local in the UI and vcf over SSH), hostname, networking, DNS, and NTP
Click Next

This is also where 9.1 differs from 9.0: the networking section now starts with an IP Address Version selector, with IPv6 address, prefix length, and gateway fields if you go dual-stack.

Review the summary
Click Finish and wait for the deployment to complete

Log in and connect the depot

Once the appliance is up, navigate to https://ip_address/ui or https://hostname/ui. Same as 9.0, the services take a few minutes to start after first boot.

For username use admin@local
Enter the local user password from the OVA deployment
Click Log in

The Get Started page opens with the Download Binaries card flagging No Depot Connection.

Click Depot Settings and Binary Management

Both depot types are available: online (direct or via proxy) and offline, fed by the VCF Download Tool — only one can be connected at a time. For air-gapped environments the official offline depot procedure covers the Download Tool flow; William Lam also documented the new HTTP offline depot support in 9.1.

Click Configure under the Online Depot tile

Here is the big change from 9.0: instead of pasting a download token, the dialog shows the appliance’s Download Service ID (the docs and KB call it the Software Depot ID). Take that ID to the BSC portal at vcf.broadcom.com, register it, and you get an Activation Code back — the full flow is in the official depot procedure and KB 443647. The download token from KB 390098 that we used in the 9.0 post is not used here anymore.

Copy the Download Service ID and register it in the BSC portal to obtain the Activation Code
Enter the Activation Code (enable the proxy toggle if you need one)
Click Authenticate

Download the binaries

With the depot connected, the Binary Management section lists everything available for the release.

Confirm the Online Depot tile shows Depot connection active
In Binary Management, select the product and version — VMware Cloud Foundation 9.1.0.0 in my case
Select the binaries and click Download

The component list is a good illustration of how 9.1 patch waves work: the installer pulls the latest patch of each component rather than a single uniform build — SDDC Manager 9.1.0.0400, NSX 9.1.0.0200, VCF Automation 9.1.0.0200, VCF Operations 9.1.0.0400, and so on. William Lam has a good write-up of this default behavior if you need to pin earlier versions. Also note the sizes — VCF services runtime alone is 17.42 GB and VCF Automation 14.95 GB, so give the downloads some time.

Start the Deployment Wizard

Click Return Home — the Deploy card sits next to Download Binaries. Progress in the wizard is saved locally in the browser after each step, so you can safely step away

Click Deployment Wizard and select VMware Cloud Foundation

The dropdown also offers VMware vSphere Foundation — the 9.1 installer deploys VVF through the same wizard framework, with a reduced flow (no NSX Manager or SDDC Manager steps). That is a post for another day; everything below follows the VCF path.

Deploy VMware Cloud Foundation

The wizard runs through four phases — Introduction, Plan, Prepare, Deploy; the official procedure is Start a New VCF Fleet or a New VCF Instance Deployment.

Introduction

The About page is worth an actual read: it diagrams the VCF fleet concept and calls out that VCF 9.1 introduces VCF management services inside the management domain, next to vCenter, NSX Manager, and SDDC Manager. The installer deploys the management domain; workload domains come later from VCF Operations.

Three deployment paths are offered: a new VCF fleet, a new VCF instance joining an existing fleet’s VCF Operations, or deploying deferred components. Same as in my 9.0 fleet post, this lab is a fresh start.

Select Deploy a new VCF fleet
Click Next

Plan

Check I have an existing vCenter instance — the wizard will collect and validate the details later
Click Next

You can also flag an existing VCF Operations 9.1 instance or an NSX-registered vCenter here. VCF Automation is no longer a hard requirement at deployment time — it can be connected or deployed later as a day-N operation.

Select the Deployment model — I went with Simple, which locks the size to Small
Review the component sizing table and click Next

The sizing table is part of the new integrated planning workflow, and it is worth pausing on: VCF management services alone ask for 40 vCPUs, 82 GB RAM, and 3 TB of disk in this smallest configuration.

Review the Network Options — by default VCF Operations, VCF Automation, and the management services land on the VM management network, and a distributed Transit Gateway is configured to make the domain VPC-ready
Click Next (or Customize if you need separate networks)

One caveat straight from the wizard: with an existing vCenter the wizard supports IPv4-only; dual-stack deployments have to go through a JSON spec.

The Review Prerequisites step is the payoff of the planning workflow: capacity totals (2 hosts, 84 vCPUs, 254 GB RAM, 5,244 GB disk for my configuration), VLAN requirements (none for this path), and the exact FQDN and IP requirements — 12 IPs for the management services pool, 5 for VCF Automation, plus FQDNs for Cloud proxy, NSX Manager, VCF Operations, and the License Server. The Pre-fill generated FQDNs in wizard button carries these into the later steps. Remember: lowercase only, forward and reverse resolvable.

Create the DNS records and reserve the IP ranges
Click Next

Prepare

Select the Version — 9.1.0.0, with the latest downloaded patch of each component applied automatically
Name the VCF instance and the management domain
Click Next

The VCF Management step collects the FQDNs for VCF Operations, the License Server, and the fleet, instance, and services-runtime components — this is where the pre-filled FQDNs from the Plan phase save the typing.

Enter the details of the existing vCenter — FQDN, administrator, and root credentials
Click Connect — the cluster running the vCenter appliance is selected as the deployment destination
Click Next

Provide the IP pools — a single IPv4 range works for both: minimum 12 addresses for the VCF Management Services pool and 5 for the VCF Automation pool
Click Next

Enter the NSX Manager cluster and appliance FQDNs
Decide on the overlay networking mode
Click Next

The Configure overlay using ESX Management VMkernel Networking toggle is one of my favorite quality-of-life items in 9.x: NSX 9 can run overlay over the ESX management VMkernel network without a VLAN-backed transport configuration. Unchecked, NSX is configured in VLAN transport zone mode instead.

Enter the SDDC Manager FQDN and the installer appliance password
Click Next

The info banner explains why the appliance password is requested: when the installer appliance runs on one of the management domain hosts, it is converted into the SDDC Manager appliance during deployment — the same appliance transition the official docs describe, and the reason the prerequisites page noted the SDDC Manager FQDN is optional in that placement.

Deploy

Review the summary — every section from the wizard is expandable here
Optionally click Download JSON Spec
Click Next

The JSON preview tab and the downloadable spec make the whole deployment repeatable — the same file drives API-based deployments, and it is the vehicle for the options the wizard does not expose, like dual-stack networking or an alternative internal cluster CIDR.

Wait for the 11 validation checks to complete — deployment specification, security configuration, DNS resolution, versions and bundles, existing SDDC Manager configuration, password policies, network configuration, existing components, and the capacity calculations
Resolve any errors, acknowledge warnings if appropriate, and click Deploy

From here the installer takes over: management services, VCF Operations, NSX, SDDC Manager conversion, and the rest of the management domain roll out in sequence. Depending on hardware, expect this to run for a few hours.

If validation or deployment fails

A few 9.1-specific failure modes worth knowing before you hit Deploy:

“Deploy and configure VCF Management Platform” fails — the management services use an internal cluster CIDR of 198.18.0.0/15 by default; if that overlaps your network, KB 440541 covers switching it to 240.0.0.0/15 or 250.0.0.0/15 via the JSON spec.
“VCF Management Services Deployment Task” fails on NTP — the NTP server must be reachable over UDP 123 from the deployed appliances, not just from the installer: KB 439240.
vMotion / vSAN network connectivity validations fail — the installer does not recognize standby uplinks; set the vmnics active/active or disable the check per KB 437248.
Uppercase FQDNs — deployment fails for fleet components, identity broker, services runtime, and VCF Automation; lowercase everything (known issues).

For nested labs, William Lam maintains a comprehensive set of installer workarounds for 9.1 — pNIC speed, vSAN HCL, and MTU prechecks included. And a neat trick from the same source: the installer doubles as a standalone pre-check tool — you can run the full validation suite against your infrastructure without deploying anything.

References

Official documentation

Broadcom KBs

My VCF 9.0 installer series

Upgrading VCF 9.1 Management Services to 9.1.0.0400 Step by Step

Cosmin Trif — Mon, 03 Aug 2026 10:00:00 +0000

On July 13, 2026 Broadcom shipped the next VCF 9.1 patch wave, and this post walks through the part of it that lives under Build -> Lifecycle -> VCF Management -> Upgrade: the VCF Management services. The headline releases on this surface are VCF Operations 9.1.0.0400 and VCF Automation 9.1.0.0200, mostly security enhancements plus a batch of fixes. This post is the follow-up to my 9.1.0.0300 walkthrough.

To be clear on scope: this post covers only the management services. SDDC Manager 9.1.0.0400 and the core components — vCenter, NSX, ESX, vSAN — go through the VCF Instances flow and are not covered here; see the short section at the end.

One thing to understand up front: patch releases are not synchronized among all VCF components, and you can pick and choose which components to apply. That is why my target list below is a mix of 0400, 0200, and 0100 builds. It looks odd the first time. It is expected.

Versions in This Update

Here is the full From/To picture from my lab, exactly as the VCF Operations UI reports it.

Component	Current Version	Target Version
Fleet Lifecycle	9.1.0.0300.25536195	9.1.0.0400.25570104
VCF Operations	9.1.0.0300.25531073	9.1.0.0400.25541561
VCF Operations for Networks	9.1.0.0100.25427421	9.1.0.0200.25517220
Identity Broker	9.1.0.0.25368698	9.1.0.0100.25522734
VCF Automation	9.1.0.0100.25429499	9.1.0.0200.25556825
Log Management	9.1.0.0100.25435121	9.1.0.0400.25544947
SDDC Lifecycle	9.1.0.0300.25536194	9.1.0.0400.25570103
Real-Time Metrics	9.1.0.0100.25435036	9.1.0.0400.25544944
Real-Time Metrics Store	9.1.0.0.25370367	9.1.0.0200.25555874
Salt Master	9.1.0.0100.25434834	9.1.0.0400.25544946
Salt RaaS	9.1.0.0100.25434834	9.1.0.0400.25544946
Software Depot	9.1.0.0.25371105	9.1.0.0400.25570105
Migration Service Engine	9.1.0.0.25370929	9.1.0.0200.25556825
VCF Services Runtime	9.1.0.0.25370367	9.1.0.0200.25555874

SDDC Manager 9.1.0.0400 (build 25570100) ships in the same wave but lives outside this component list — see the scope section at the end of the post.

Before You Start

A few rules before clicking anything. Back up your components before patching, and remember only one upgrade or patch operation can run at a time. That single-operation rule matters more than it sounds with fourteen components in the queue — plan for a long session or spread it over a maintenance window. If your depot is online, the binaries pull directly. If you run offline, update the VCF Download Tool to 9.1.0.0400 (build 25550178) first and download the binaries with it.

Two known issues shape how I ran this:

SDDC Lifecycle 9.1.0.0400: the batch precheck can fail with VCF_LCM_FLEET_UPGRADE_400_COMPONENT_NOT_NEEDED_USER_INPUT_SPEC_PROVIDED. Do not use Run Prechecks (ALL). Precheck each component individually.
Software Depot 9.1.0.0400: patching the depot takes it temporarily offline. Do not run any other component patch at the same time, and retry anything that failed afterwards.

Upgrade Fleet Lifecycle

Broadcom’s ordering guidance for maintenance releases starts with Fleet Lifecycle. For patch releases the order is flexible, but I still go Fleet Lifecycle first — it drives patch deployment for the rest of the fleet components.

In VCF Operations, go to Build -> Lifecycle -> VCF Management -> Upgrade. Open the “…” menu on the Fleet Lifecycle card and click Select version.

The Fleet Lifecycle Component Target Version dialog opens. Set the target to 9.1.0.0400.25570104.

Back on the card, the upgrade path now reads 9.1.0.0300.25536195 to 9.1.0.0400.25570104. Click Upgrade.

Kick off the upgrade. The card flips to Upgrade in progress.

Open Upgrade details. The first two subtasks are “Set Fleet lifecycle upgrade context” and “Stage Fleet lifecycle plugin in VCF services runtime”.

The status messages move through Staging binaries and Running Component Stage Prechecks. Nothing to do here but watch.

All 7 subtasks land on Completed. In this run the whole thing took about 45 minutes, 11:08 to 11:54 AM.

Set the VCF Management Target Version

With Fleet Lifecycle done, click CHANGE TARGET VERSION.

The Set Target Version dialog offers the VCF global version 9.1.0.*.

The component list now shows everything Ready for upgrade — with the mixed 0400/0200/0100 targets from the table above. This is the per-component patch numbering at work, not a depot problem.

Because of the SDDC Lifecycle known issue, I did not touch RUN PRECHECKS (ALL). I ran prechecks per component instead, right before each upgrade. It costs a few extra clicks per component, but it avoids the VCF_LCM_FLEET_UPGRADE_400_COMPONENT_NOT_NEEDED_USER_INPUT_SPEC_PROVIDED failure entirely.

Upgrade VCF Operations

First up, VCF Operations itself to 9.1.0.0400.25541561. Note that the cloud proxy and the license server are patched automatically as part of this one — no separate steps needed for either.

The workflow runs through its subtasks; I moved on once it was In Progress.

Upgrade Identity Broker

Next, Identity Broker to 9.1.0.0100.25522734.

This one is a 6-subtask workflow.

Upgrade VCF Automation

VCF Automation goes to 9.1.0.0200.25556825.

The precheck workflow runs first, then the upgrade takes over.

Upgrade VCF Operations for Networks

VCF Operations for Networks moves to 9.1.0.0200.25517220.

Upgrade Log Management

Log Management targets 9.1.0.0400.25544947.

Same pattern: precheck, then the upgrade workflow.

Upgrade SDDC Lifecycle

SDDC Lifecycle goes to 9.1.0.0400.25570103 — the component behind the batch precheck known issue, upgraded here with its own individual precheck.

Upgrade Real-Time Metrics and Metrics Store

Real-Time Metrics targets 9.1.0.0400.25544944.

The Real-Time Metrics row switches to Upgrade in progress.

Then the upgrade workflow itself.

The Real-Time Metrics Store is a separate component; it targets 9.1.0.0200.25555874.

The Real-Time Metrics Store row switches to Upgrade in progress.

And the upgrade workflow follows.

Upgrade Salt Master

Salt Master targets 9.1.0.0400.25544946.

The Salt Master row switches to Upgrade in progress.

The upgrade workflow takes over and works through its subtasks.

Upgrade Software Depot

Software Depot goes to 9.1.0.0400.25570105. Per the known issue in the release notes, the depot goes briefly offline during its own patch, so avoid launching other component patches at the same time and retry anything that failed afterwards.

The Software Depot row switches to Upgrade in progress.

Then the upgrade workflow, while the depot is briefly unavailable.

Upgrade Migration Service Engine

Migration Service Engine targets 9.1.0.0200.25556825.

The Migration Service Engine row switches to Upgrade in progress.

Fun detail in the workflow panel: the staging task exposes the internal component name VCD_MIGRATOR — “Stage VCD_MIGRATOR plugin in VCF services runtime”.

Upgrade VCF Services Runtime

VCF Services Runtime goes to 9.1.0.0200.25555874.

The VCF Services Runtime row switches to Upgrade in progress.

Then the upgrade workflow with its subtasks.

Upgrade Salt RaaS

Salt RaaS ran last in my lab, targeting 9.1.0.0400.25544946.

At the time of this capture the precheck workflow was still running — “Run Salt RaaS upgrade prechecks” shows In Progress.

What About SDDC Manager and the Core Components?

They are out of scope for this post. SDDC Manager 9.1.0.0400 (build 25570100) and the core components from this wave — ESX 9.1.0.0200, vSAN 9.1.0.0200, NSX 9.1.0.0200, and vCenter 9.1.0.0200, later bumped to 9.1.0.0300 on July 29, 2026 — are applied through Build -> Lifecycle -> VCF Instances. That is the same flow I documented in Upgrading VCF 9.1 to 9.1.0.0300 Step by Step.

Validate the Update

Before calling it done:

Check the Component Versions page — every patched management component should show On Target.
Check the task list — no failed workflows left behind.
Confirm the VCF Operations UI is reachable and healthy.
Retry anything that failed while the Software Depot was being patched — the depot was offline during that window.

For SDDC Manager and the core components, see the previous section — the flow in Upgrading VCF 9.1 to 9.1.0.0300 Step by Step still applies one-to-one.

Fixing the Swagger localhost:8600 Server URL in VCF Automation 9.1: Overriding a Spec Baked Into the Image

Cosmin Trif — Fri, 31 Jul 2026 00:00:00 +0000

While putting together the VCF Automation 9.1 API getting started post, I spent some time in the in-product API Help Center — log into the VM Apps tenant, click your username, API Help Center > Explore Automation APIs. The Swagger pages render fine, but the moment you try an actual call, it fails. The Servers box for the VCF Automation (Blueprint) API shows:

http://localhost:8600

Swagger UI builds Try It Out requests from servers[0].url, so the browser dutifully sends every request to port 8600 on your own workstation. Connection refused, every time. The documentation is correct; the advertised host is not.

This post is about finding where that value actually comes from — which took three wrong turns through the appliance’s Kubernetes internals — and the override that finally fixed it without touching a container image. The short version:

The localhost:8600 value is not in any configuration on the appliance. A grep across a full support-bundle cluster dump finds no ConfigMap, Helm value, gateway route, or forwarded-headers setting that produces it.
The spec is a static JSON file generated by springdoc at build time — the giveaway is its "description": "Generated server url" — and it ships as a classpath resource inside the Spring Boot application in the tango-blueprint-service-app pod. Whoever built it had the service listening on port 8600, and the URL froze there.
The api-docs page itself is served by a different pod than the spec, which is why the obvious nginx-level fixes go nowhere.
The fix: mount a corrected copy of the spec from a ConfigMap and prepend file:/override/ to spring.web.resources.static-locations via JAVA_OPTS, so Spring serves the fixed file and never falls through to the copy shipped in the image. Survives pod restarts with nothing to reconcile it away; a product upgrade will wipe it.

Everything below was done on a lab appliance running build 9.1.0.0100.25429499. This is an unsupported modification — more on that at the end.

The Symptom

In the tenant API Help Center, open the VCF Automation API (the Blueprint one, version 2019-09-12). The spec renders, the paths are all correct, but the servers dropdown offers exactly one entry: http://localhost:8600. There is no way to type a different URL — the dropdown only lists what the spec declares, and this spec declares one hardcoded server with no variables.

Browser DevTools (Network tab) shows what the page actually loads. First an index of every API the Help Center knows about:

/tenant/<your-tenant>/api-docs/assets/content/soft/automation-apis.json

That index maps each tile to a spec URL — about sixteen specs across a dozen backend services. The Blueprint entry points at:

/blueprint/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json

And that document contains the problem, verbatim:

"servers": [
 {
 "url": "http://localhost:8600",
 "description": "Generated server url"
 }
]

Two details in that response are worth pausing on, because they shaped the whole investigation:

"Generated server url" is the literal default description springdoc attaches when it auto-computes the server URL from the incoming request. So springdoc generated this — but if it were computed per request, it would show the appliance FQDN, not localhost.
The response carries a last-modified date from months before I ever opened the page, plus accept-ranges: bytes. That is a static file being read off disk (or classpath), not a document rendered per request.

Put together: someone ran the Blueprint service on a build machine, where it listened on localhost:8600, exported the generated spec to a file, and shipped that file in the product. The URL froze at build time.

Ruling Out Configuration

Before touching anything, I wanted to know whether some appliance setting produces or rewrites that value — because if it did, that would be the supported place to fix it.

It does not. I had a full support bundle from this appliance handy (that is a story for another post), and the extracted cluster object dump is conclusive:

The string 8600 appears nowhere in the cluster as a port, URL, or config value. No Service exposes it, no container declares it, no ConfigMap mentions it. The only matches are UID substrings.
There is no springdoc, server.forward-headers-strategy, or server-URL configuration in any ConfigMap in the prelude or vmsp-platform namespaces.
The Envoy gateway routes (Gateway API HTTPRoute objects) that publish the api-docs paths perform path rewrites only — nothing rewrites hosts or response bodies.

So there is no knob. The value lives in content, not configuration, and only Broadcom can fix it properly. Everything from here on is about making the appliance serve a corrected copy.

Wrong Turn #1: The Pod That Serves the Page Doesn’t Serve the Spec

The api-docs page shell — the Angular app with the tiles — is served by the cloud-automation-ui-app pod in the prelude namespace. Its nginx config (ConfigMap cloud-automation-ui) rewrites /tenant/<org>/api-docs and /provider/api-docs into static content under /usr/share/nginx/html/dist. My first instinct was an nginx sub_filter there, rewriting localhost:8600 in responses.

Dead end, for a structural reason: the spec never passes through that pod. The gateway routes the two URLs to different backends entirely:

Path	HTTPRoute	Backend
`/tenant/<org>/api-docs/...` (the page, the index JSON)	`tenant-in-path-automation-ui`	`cloud-automation-ui`
`/blueprint/api-docs/...` (the spec)	`blueprintapi-docs`	`tango-blueprint:8080`, prefix rewritten to `/api-docs/`

A sub_filter in the UI pod would rewrite a response it never sees. The actual owner of the spec is the Blueprint service itself: Deployment/tango-blueprint-service-app, image blueprint-webapp:9.1.0.0100.25429499 — a Spring Boot app, which also explains the Spring Security headers on the spec response.

Wrong Turn #2: There Is No File to Edit

Fine — exec into the right pod and sed the file in place:

kubectl -n prelude exec deploy/tango-blueprint-service-app -- \
 sh -c 'find / -name "api-doc-public-classic-vra-*.json" -not -path "/proc/*" 2>/dev/null'

Nothing. And looking for the application itself:

kubectl -n prelude exec deploy/tango-blueprint-service-app -- \
 sh -c 'find / -maxdepth 3 -name "*.jar" -not -path "/proc/*" 2>/dev/null | head'

/opt/bc-fips/bcutil-fips-2.1.5.jar
/opt/bc-fips/bctls-fips-2.1.22.jar
/opt/bc-fips/bcpkix-fips-2.1.10.jar
/opt/bc-fips/bc-fips-2.1.2.jar
/opt/bc-fips/bcmail-fips-2.1.6.jar
/jdk/lib/jrt-fs.jar

Only the Bouncy Castle FIPS libraries and the JDK — the shallow search doesn’t even turn up the application archive itself. Wherever the application lives, the spec is packaged inside it as a classpath resource: find sees nothing, and sed has nothing to reach. Even if you dug out the archive, patched it, and re-packed it in the running container, the fix would be pointless: the JVM holds it open, so a restart is needed to pick up the change, and the restart is exactly what rebuilds the container filesystem from the pristine image. The fix would erase itself at the moment of application.

Wrong Turn #3: kubectl edit Ate My JAVA_OPTS

The mechanism that does work is Spring’s static resource resolution. Spring Boot serves static content from an ordered list of locations, and the first location that contains a matching file wins:

spring.web.resources.static-locations =
 classpath:/META-INF/resources/, classpath:/resources/, classpath:/static/, classpath:/public/

Setting that property replaces the default list. So if I prepend a filesystem directory and re-append the defaults, a file I control shadows the identically-pathed resource on the application classpath — and everything else keeps being served from the classpath exactly as before.

The path arithmetic works out neatly. After the gateway’s prefix rewrite, the backend sees the request as /api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json. With file:/override/ first in the list, Spring checks /override/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json before falling through to the classpath. That is exactly where a ConfigMap mount can put a corrected copy.

The wrong-turn part: my first attempt applied all of this with kubectl edit deploy. The volume and volumeMount saved; the JAVA_OPTS append silently did not. That env value is one enormous line that already contains $(SERVICE_ACCOUNT_SECRET) — literal Kubernetes variable-expansion syntax — and somewhere in the editor round-trip the appended text was lost; I never pinned down exactly where. (The same string is also a shell command-substitution landmine if you ever round-trip the value through a shell, which is why the fix below patches programmatically.) The pod rolled, the override directory appeared, and the spec still said localhost:8600 because the JVM never got the property. Verify the property reached the Deployment before concluding the approach failed.

The Fix

All commands run as root on the appliance, where kubectl is already configured. Replace vcfa.domain.com with your appliance FQDN.

1. Produce the corrected spec. No spelunking inside the image needed — pull it through the gateway and fix the one string:

curl -sk https://vcfa.domain.com/blueprint/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json \
 | sed 's|http://localhost:8600|https://vcfa.domain.com|g' \
 > /root/api-doc-public-classic-vra-2019-09-12.json

Sanity-check it — zero remaining hits, and still valid JSON:

grep -c localhost:8600 /root/api-doc-public-classic-vra-2019-09-12.json
python3 -m json.tool /root/api-doc-public-classic-vra-2019-09-12.json > /dev/null && echo "valid json"

2. Create the ConfigMap and back up the Deployment:

kubectl -n prelude create configmap blueprint-apidocs-override \
 --from-file=/root/api-doc-public-classic-vra-2019-09-12.json

kubectl -n prelude get deploy tango-blueprint-service-app -o yaml \
 > /root/tango-blueprint-deploy.backup.yaml

3. Add the volume and mount. A strategic merge patch merges named list entries, so this adds without disturbing the existing volumes and mounts:

kubectl -n prelude patch deploy tango-blueprint-service-app --type=strategic -p '
spec:
 template:
 spec:
 volumes:
 - name: apidocs-override
 configMap:
 name: blueprint-apidocs-override
 containers:
 - name: tango-blueprint-service-app
 volumeMounts:
 - name: apidocs-override
 mountPath: /override/api-docs/classic-apis
'

4. Append the properties to JAVA_OPTS — programmatically. This reads the current value, appends both the current (spring.web.resources.static-locations) and the pre-2.4 (spring.resources.static-locations) property names — whichever one this Spring Boot generation ignores does no harm — and writes a JSON patch. It refuses to double-apply:

kubectl -n prelude get deploy tango-blueprint-service-app -o json > /root/d.json

python3 - <<'EOF'
import json
d = json.load(open("/root/d.json"))
env = d["spec"]["template"]["spec"]["containers"][0]["env"]
i = [n for n, e in enumerate(env) if e["name"] == "JAVA_OPTS"][0]
loc = "file:/override/,classpath:/META-INF/resources/,classpath:/resources/,classpath:/static/,classpath:/public/"
add = " -Dspring.web.resources.static-locations=" + loc + " -Dspring.resources.static-locations=" + loc
if "static-locations" in env[i]["value"]:
 json.dump([], open("/root/patch.json", "w"))
 print("already present - nothing to do"); raise SystemExit
json.dump([{"op": "replace",
 "path": "/spec/template/spec/containers/0/env/%d/value" % i,
 "value": env[i]["value"] + add}],
 open("/root/patch.json", "w"))
print("patch written for env index", i)
EOF

kubectl -n prelude patch deploy tango-blueprint-service-app --type=json --patch-file=/root/patch.json

5. Wait for the rollout. The Deployment uses the Recreate strategy with a single replica, so the Blueprint service is briefly down while the pod is replaced:

kubectl -n prelude rollout status deploy/tango-blueprint-service-app --timeout=180s

Verification

Learned from wrong turn #3: verify each layer, not just the end result.

The property is on the Deployment:

kubectl -n prelude get deploy tango-blueprint-service-app \
 -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="JAVA_OPTS")].value}' \
 | tr ' ' '\n' | grep -i static

-Dspring.web.resources.static-locations=file:/override/,classpath:/META-INF/resources/,...
-Dspring.resources.static-locations=file:/override/,classpath:/META-INF/resources/,...

The override file is mounted in the new pod:

kubectl -n prelude exec deploy/tango-blueprint-service-app -c tango-blueprint-service-app -- \
 ls -la /override/api-docs/classic-apis/

And the actual test — the spec as served through the gateway:

curl -sk https://vcfa.domain.com/blueprint/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json \
 | grep -A2 '"servers"'

 "servers": [
 {
 "url": "https://vcfa.domain.com",

Hard-reload the API Help Center page (the SPA caches aggressively) and the Servers box shows the appliance FQDN. Try It Out now sends requests where they belong.

What This Does and Does Not Fix

Worth being precise here, because the Help Center is bigger than one spec:

Fixed: the 2019-09-12 Blueprint spec shown above. Its 2019-01-15 sibling from the same /blueprint/api-docs/classic-apis/ directory — if it shows the same URL — is the same fix: repeat step 1 for it, then rebuild the ConfigMap with both files: kubectl -n prelude create configmap blueprint-apidocs-override --from-file=/root/api-doc-public-classic-vra-2019-09-12.json --from-file=/root/api-doc-public-classic-vra-2019-01-15.json --dry-run=client -o yaml | kubectl -n prelude replace -f -. The mounted volume picks up the change on its own within a minute or so; no restart needed.
Same pattern, different pod: the catalog, deployment, and policies tiles point at api-doc-public-classic-vra-*.yaml files under /catalog/api-docs/classic-apis/ — the same baked-file shape, owned by the catalog service. If those show the same URL, it is the same fix against that Deployment.
Possibly not broken at all: several tiles point at live springdoc endpoints (/identity/doc/v3/api-docs, /iaas-api/swagger/v3/api-docs/..., /project-service/api/swagger/v3/api-docs/...). Those generate their spec per request, so check before assuming. A quick curl ... | jq .servers against each URL from the index tells you which ones need attention.

Caveats

This is an unsupported modification to a Broadcom appliance, and it behaves accordingly:

It survives pod restarts, and nothing reconciles it away. The fix lives in the Deployment spec, so every pod the Deployment creates from now on gets the mount and the property — the verified state above is itself a freshly recreated pod. The tango-blueprint-service Helm release is deployed once at install/upgrade time with nothing reconciling it continuously, so it should come back intact after an appliance reboot as well; re-check the spec once after your next reboot to confirm. (Do not assume this generalizes: several other components in the same namespace are Flux-managed and would revert an edit within minutes.)
A product upgrade will wipe it. The LCM re-renders the Deployment from the new chart. Expect to re-apply, and re-check after any lifecycle operation.
Roll back at any time with the backup from step 2. Not with kubectl apply, though — the backup carries the pre-patch resourceVersion (so the apply is rejected as a conflict), and client-side apply against a Helm-created object has no last-applied-configuration to prune from, so the added volume would survive the merge and the next pod would wedge on the then-deleted ConfigMap. kubectl replace does a full PUT and removes the volume, mount, and JAVA_OPTS additions in one shot; strip the resourceVersion so the PUT is unconditional, and delete the ConfigMap only after the Deployment no longer references it:

sed '/^ resourceVersion:/d' /root/tango-blueprint-deploy.backup.yaml \
 | kubectl -n prelude replace -f -
kubectl -n prelude rollout status deploy/tango-blueprint-service-app --timeout=180s
kubectl -n prelude delete configmap blueprint-apidocs-override

File an SR anyway. As of this writing there is no KB and nothing in the 9.1 release notes for this — the proper fix has to ship in the product, and that only happens if it gets reported. Reference the build number and the "Generated server url" value in the spec.

If you only need a working Try It Out for yourself and would rather not touch the appliance at all, there is a zero-footprint alternative: download the spec, fix servers[0].url in the copy, and import it into Postman — it seeds the collection’s baseUrl from the spec, and auth works as described in the getting started post.

Final Notes

The frustrating part of this bug is how little is actually wrong: every path in the spec is correct, authentication is correct, the API itself works. One string, frozen at build time on some build machine where the Blueprint service happened to listen on port 8600, breaks the one feature the page exists to provide.

The investigation pattern is the reusable part. When a value appears in a rendered page and nothing in the configuration produces it, stop grepping ConfigMaps and follow the bytes instead: DevTools tells you the exact URL, the response headers tell you whether it is static or generated, and the gateway routes tell you which pod actually owns it. In this case each of those three steps invalidated an otherwise-reasonable fix — the nginx rewrite targeted the wrong pod, the in-pod sed targeted a file that does not exist, and the editor-based patch failed silently — before the resource-shadowing approach landed on the one layer where the platform genuinely offers an override.

References

VCF Automation 9.1 API Getting Started — authentication against the same appliance, and where the API docs moved in 9.1
Broadcom TechDocs — What Are the Automation APIs and How Do I Use Them (VCF 9.1)
VM Apps Org APIs on the Broadcom Developer Portal — the hosted alternative to the in-product Help Center
Spring Boot Reference — Serving Static Content — the static-locations resolution order this fix relies on
springdoc-openapi FAQ — how the “Generated server url” value is computed, and why it goes stale behind a proxy

VCF Automation 9.1 API Getting Started

Cosmin Trif — Thu, 30 Jul 2026 07:00:00 +0000

Back in 2022 I wrote vRA 8 API getting started — two curl calls and you had a bearer token. Since then the product has been renamed twice (vRealize Automation → VMware Aria Automation → VCF Automation) and, more importantly for anyone with scripts in production, the authentication flow changed in VCF 9.x. Neither of the two calls from that post works against a 9.1 appliance.

This is the 9.1 version of the same post: the minimum you need to do before you can actually query the API, with curl.

What Changed Since vRA 8

The short version, before the steps:

	vRA 8.x	VCF Automation 9.1 (VM Apps org)
Refresh token	`POST /csp/gateway/am/api/login?access_token` with username/password	Generated in the UI (My Account > API Tokens) — no programmatic call
Bearer token	`POST /iaas/api/login` with the refresh token	`POST /tm/oauth/tenant/{tenant}/token`, standard OAuth `grant_type=refresh_token`
Bearer lifetime	~8 hours	1 hour
Refresh token lifetime	90 days	90 days by default (configurable, at creation time only)
API docs	`https://vra_url/automation-ui/api-docs/`	In-product API Help Center, plus developer.broadcom.com

The two things that break existing scripts:

You cannot get a refresh token with username and password anymore. The docs are explicit: “As of VCF 9.0, the Identity Service API functionality has changed and is no longer used to obtain a refresh token programmatically.” The refresh token is now an API token you generate once in the UI and treat like a credential.
There is a tenant in the URL now. VCF Automation 9 has a provider/tenant model. Everything that used to be vRA lives inside a VM Apps organization, and the token endpoint is scoped to that org name — so step zero is finding out what yours is called.

Step 0: Find Your VM Apps Tenant Name

Log into the Provider Management Portal (https://vcfa.domain.com, the admin account) and go to Organizations. Note two things:

The name of your VM Apps organization — if you upgraded from Aria Automation 8.x, this is the classic tenant that was created during migration (the docs use vcf-e2e-vidm as an example).
The FQDN of the appliance itself.

You need both for every call that follows.

Step 1: Generate an API Token (the New Refresh Token)

This is the part that moved from curl to the UI:

Open the VCF Automation UI at https://vcfa.domain.com and switch the organization to your VM Apps tenant.
Log in with a tenant account (it needs to be an org member with at least the user service role — an org_owner works fine).
Click your name in the top-right corner and select My Account.
Go to the API Tokens tab, click New, give the token a name, and click Create.
Copy the token — it is shown once.

By default the token lives for 129,600 minutes (90 days). If you want a different lifespan, set it before creating the token under General Settings > Timeouts in the Provider Management Portal (0 means unlimited) — the lifetime is baked in at creation and cannot be changed retroactively. To kill a token early, revoke it from the same UI or via DELETE /cloudapi/1.0.0/tokens/urn:vcloud:token:<token_id>.

Step 2: Exchange the API Token for an Access Token

This replaces the old POST /iaas/api/login. The endpoint is a standard OAuth token endpoint, scoped to your tenant:

curl -k --location --request POST 'https://vcfa.domain.com/tm/oauth/tenant/your-vm-apps-tenant/token' \
 --header 'Content-Type: application/x-www-form-urlencoded' \
 --header 'Accept: application/json' \
 --data-urlencode 'grant_type=refresh_token' \
 --data-urlencode 'refresh_token=YOUR_API_TOKEN'

The response contains the bearer token:

{
 "access_token": "eyJhbGciOi...",
 "token_type": "Bearer",
 "expires_in": 3600,
 "refresh_token": "YOUR_API_TOKEN"
}

The access token expires after one hour — noticeably shorter than the ~8 hours you got from /iaas/api/login in 8.x, so long-running scripts should be prepared to re-run the exchange.

For scripting, the whole thing collapses into one line with jq:

ACCESS_TOKEN=$(curl -sk --request POST "https://$VCFA_HOST/tm/oauth/tenant/$VCFA_TENANT/token" \
 -H 'Content-Type: application/x-www-form-urlencoded' \
 -H 'Accept: application/json' \
 --data-urlencode 'grant_type=refresh_token' \
 --data-urlencode "refresh_token=$VCFA_API_TOKEN" | jq -r .access_token)

Step 3: Use It

The good news: once you have the access token, the API surface you know from vRA 8 is still there. The 9.1 programming guide tutorials use the same service endpoints — /iaas/api, /catalog/api, /deployment/api, /blueprint/api, /abx, the Orchestrator gateway — with a plain bearer header:

curl -sk "https://$VCFA_HOST/iaas/api/projects" \
 -H "Authorization: Bearer $ACCESS_TOKEN" | jq .

If a call comes back 403, verify what the account actually is inside the org. The docs still expose the CSP gateway for this (note the csp-auth-token header instead of Authorization):

# your org id
curl -sk "https://$VCFA_HOST/csp/gateway/am/api/loggedin/user/orgs" \
 -H "csp-auth-token: $ACCESS_TOKEN" | jq .

# org role: must be org_owner or org_member
curl -sk "https://$VCFA_HOST/csp/gateway/am/api/loggedin/user/orgs/$ORG_ID/roles" \
 -H "csp-auth-token: $ACCESS_TOKEN" | jq .

# service roles: must include at least "user"
curl -sk "https://$VCFA_HOST/csp/gateway/am/api/loggedin/user/orgs/$ORG_ID/service-roles" \
 -H "csp-auth-token: $ACCESS_TOKEN" | jq .

One caveat from Broadcom: scripts that authenticate against new VM Apps organizations (created in 9.x, not upgraded from 8.x) should use the VCF Automation Provider Management Access Control APIs rather than the CSP APIs — treat the CSP gateway as a compatibility surface, not a foundation for new tooling.

Where the API Docs Moved

The old in-product Swagger at https://vra_url/automation-ui/api-docs/ and the developer.vmware.com portal are both gone. In 9.1:

In-product: log into the VM Apps tenant, click your username in the top-right corner, and open API Help Center > Explore Automation APIs.
Online: the VM Apps Org APIs on the Broadcom Developer Portal.

Also worth knowing before you port scripts from 8.x: the Pipelines/Code Stream APIs (/codestream/api/*), the CMX Kubernetes APIs, and the Migration Assessment APIs are removed in 9.1.

A Note on All Apps Organizations

Everything above covers the VM Apps organization — the classic vRA-shaped experience. The new All Apps side of VCF Automation 9 (the Cloud Director-heritage Tenant Manager) authenticates differently again: provider tokens come from POST /oauth/provider/token, and there is a session endpoint at /cloudapi/1.0.0/sessions/provider that does accept basic auth and returns the bearer in the x-vmware-vcloud-access-token response header. That API family deserves its own post; Christian Ferber has a good writeup linked below in the meantime.

References

Fixing a VCF Operations 9.1 Express Patch Stuck at Installing PAK File: An Empty /etc/yum.repos.d and Four Layers of Upgrade State

Cosmin Trif — Wed, 29 Jul 2026 17:30:00 +0000

The 9.1.0.0300 patch went through this lab without drama at the beginning of July. Three weeks later, the 9.1.0.0400 express patch for VCF Operations hit the same single-node appliance and stopped dead at Installing PAK file — no progress, no error, and a reboot later, still nothing. What followed was a proper archaeology dig through every layer of state the upgrade machinery keeps, and almost every layer turned out to be holding a different version of the truth.

The short version:

The upgrade died two seconds into the OS update phase because the glob over /etc/yum.repos.d/* matched nothing — the directory had been emptied (or removed; the error is identical either way) — and sed exiting 2 on it is fatal to updateCoordinator.py.
The failure was recorded in pakManager’s results file, and from then on a retry guard rejected every new attempt in milliseconds — while CaSA kept retrying roughly every 11–12 seconds, over 1,100 times in the retained logs alone.
The UI said Installing the whole time because CaSA’s own pak history had the APPLY_OS_UPDATE operation with a null completed_time — stale state, not activity.
Rebooting did not retry anything; it just rebuilt the same wedge, with the cluster still offline because offline is persisted state.
The recovery attempts then failed in two brand-new ways: a 409 pak_installation_active raised when the precheck’s state check collided with the assessment install the precheck itself had just launched, and a NullPointerException when LCM asked CaSA for an upgrade task that no longer existed anywhere.

Versions in play: VCF Operations 9.1.0.0300.25531073 upgrading to 9.1.0.0400.25541550, on a single-node deployment, driven by the express patch workflow in fleet management. All timestamps below are UTC, straight from the logs.

The Symptom: Installing PAK File, Forever

The express patch staged cleanly and started applying, then sat at Installing PAK file for hours. A reboot later it still showed the same thing. Time to stop watching the progress bar and go look at what the appliance was actually doing.

The places that matter for any PAK-based upgrade on a VCF Operations node:

What	Where
pakManager action logs	`/storage/log/vcops/log/pakManager/vcopsPakManager.root.<action>.log`
Per-PAK script stdout/stderr	`/storage/log/vcops/log/pakManager/<pak-id>/`
PAK working state + results	`/storage/db/pakRepoLocal/<pak-id>/<pak-id>.results`
pakManager action history	`/storage/db/pakRepoLocal/vcopsPakManagerCommonHistory.json`
CaSA logs + per-PAK history	`/storage/log/vcops/log/casa/`
Staged PAK binaries	`/storage/db/casa/pak/dist_pak_files/VA_LINUX/`

The PAK for this patch is vRealizeOperationsManagerEnterprise-910040025541550 — the ID is the version string with the dots removed, 9.1.0.0400.25541550.

Finding the Loop

The first thing that stood out was not an error message but a rotation pattern. vcopsPakManager.root.apply_os_update.log keeps twenty rotated files of 1 MB each, and every one of them was less than four hours old — the log was burning a full rotation roughly every 11 minutes. Something was running apply_os_update constantly.

Each attempt ended the same way, in milliseconds (log prefixes trimmed for width here and throughout):

[VerifyActionNotRunningOrRun] - apply_os_update result: "Failed"
[VerifyActionNotRunningOrRun] - apply_os_update exit code: "1"
[HandleError] - Exiting with exit code: 1, message: The action: apply_os_update
on the pak file: "vRealizeOperationsManagerEnterprise-910040025541550" previously
failed, cleanup the system and remove_history before attempting again--exiting

That is pakManager’s retry guard: once an action has Failed in the PAK’s .results file, every subsequent invocation is rejected before any script runs. And CaSA — the admin/cluster service that orchestrates PAK operations — was invoking it again roughly every 11–12 seconds, each time via:

/usr/bin/sudo -n /usr/lib/vmware-python-3/bin/python
 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py
 --action apply_os_update --pak vRealizeOperationsManagerEnterprise-910040025541550
 --json --force_content_update true --roles ADMIN,DATA,UI

The retained logs held 1,125 rejections spanning just under four hours — and CaSA’s logs show the loop reaching back to the previous evening. An install loop where every iteration is refused on arrival: the upgrade could not move forward and would never stop trying. So the interesting question became: what failed the first time?

The Root Cause: One sed Against an Empty Directory

CaSA’s operation history gives the setup, and the per-PAK stderr log — /storage/log/vcops/log/pakManager/vRealizeOperationsManagerEnterprise-910040025541550/apply_os_update_stderr.log — gives the kill. The timeline on July 28:

17:42:27 → 17:42:33 — validation runs and passes
17:42:45 → 17:44:29 — BRING_CLUSTER_OFFLINE completes
17:44:37 — APPLY_OS_UPDATE starts
17:44:39 — the very first command of the VA upgrade phase runs, and dies:

[b2b - updateCoordinator - os_upgrade] - Entering VA upgrade phase
[b2b - updateCoordinator - os_upgrade] - Disable existing repositories
[b2b - updateCoordinator - runScript] - Script command:
 "/usr/bin/sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/*"
[b2b - updateCoordinator - runScript] - stderr:
 "/usr/bin/sed: can't read /etc/yum.repos.d/*: No such file or directory"
[b2b - updateCoordinator - runScript] - exit code: 2
[b2b - updateCoordinator - programExit] - Unable to disable existing repositories for updates.

The glob over /etc/yum.repos.d/* matched nothing — the shell passed the literal string through, sed could not open it, and exited 2. That error is byte-identical whether the directory is empty or missing outright; by the time anyone could look, the files were back, so all the log proves is that there was nothing there to match. updateCoordinator.py treats a non-zero exit here as fatal — reasonably, since it is about to drive tdnf package operations and wants external repositories disabled first.

Two details make this sting:

The upgrade does not need working external repos at all. The very next steps in a healthy run copy the PAK’s own local-pak-iso.repo into that directory and install everything from the ISO embedded in the PAK. The sed only needs the glob to match at least one file. A directory with nothing in it is the one state it cannot survive.

This exact command succeeded three weeks earlier. The 0300 patch on July 5 logged the identical sed with an empty stderr and exit 0, then dropped local-pak-iso.repo in place and carried on. So the directory was populated on July 5 and held nothing the glob could match by July 28 — something removed those files in between. More on the suspects at the end.

One more log-reading trap: the failure looks like it took ten minutes, because after the fatal error updateCoordinator.py spends from 17:44:39 to 17:54:47 generating a support bundle before it exits. The Failed result lands in the state files at 17:54:47, ten minutes after the two-second failure. If you correlate timestamps across logs, that gap is the tool being helpful, not the upgrade doing work.

Why It Looked Stuck Instead of Failed

Here is the part that turned a two-second failure into a day of confusion. The upgrade machinery keeps (at least) four separate stores of state, and they disagreed:

Store	Path	Owner	What it said
PAK results file	`/storage/db/pakRepoLocal/<pak-id>/<pak-id>.results`	pakManager	`apply_os_update_result: Failed`
Action history	`/storage/db/pakRepoLocal/vcopsPakManagerCommonHistory.json`	pakManager	`apply_os_update: Failed`
Per-PAK operation history	`/storage/log/vcops/log/casa/casa_pak_history_<pak-id>.json`	CaSA	`APPLY_OS_UPDATE` started, never completed
Upgrade orchestrator documents	CaSA runtime	CaSA	operation active, keep retrying

The UI renders from CaSA’s view. Its per-PAK history file ended like this:

{
 "operation": "APPLY_OS_UPDATE",
 "start_time": 1785260677162,
 "completed_time": null
}

An operation with a start time and a null completion is, as far as the UI is concerned, still installing. pakManager knew the upgrade had failed; CaSA never wrote a terminal result for the operation. Both were internally consistent. Neither was lying. They just were not the same state machine.

Why Rebooting Did Not Help

BRING_CLUSTER_OFFLINE had completed before the failure, and offline is persisted admin state. After the reboot, the platform came up and refused to start its services on purpose — from the support bundle’s sysenv/status.txt, captured while it was down:

vmware-vcops.service - LSB: vRealize Operations Platform
 Active: failed (Result: exit-code)
 vpostgres Cannot be started as the slice is offline
 analytics Cannot be started as the slice is offline
 collector Cannot be started as the slice is offline
 api Cannot be started as the slice is offline

So the reboot traded a stuck upgrade on a running cluster for a stuck upgrade on a down cluster. One genuinely useful side effect: the retry loop did not survive the restart. CaSA’s orchestrator documents came back empty — current_pak_activity all nulls from the first post-reboot poll onward — so the hammering stopped; the last apply_os_update attempt in the logs is from just before the reboot. But nothing resumed, either. The upgrade was now fully inert: no process running, no retries, UI still saying Installing, and monitoring down with it.

Bringing the cluster back online restored the data-collection side of the house — services active, slice ONLINE, still on 9.1.0.0300 — while the upgrade state stayed exactly as wedged as before. Stale state does not age out. Waiting was never going to fix this.

Clearing the PAK State: CLI Archaeology

The guard message says what it wants: “cleanup the system and remove_history before attempting again.” Turning that sentence into working commands took several tries, and the failed attempts are worth documenting because the CLI’s own error messages actively mislead.

First lesson: run it with python3. The script does not behave as a direct executable — invoked by path, the shell tries to interpret Python as shell (abridged, full path shortened):

vcopsPakManager.py: line 6: from: command not found
vcopsPakManager.py: line 10: import: command not found
vcopsPakManager.py: line 31: syntax error near unexpected token `('

python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py

Run bare like that, it prints its full usage — which is where the trap sits. The guard message names remove_history as if it were an action. It is not. It is a parameter of the cleanup action:

cleanup - remove temp and working files
 params:
 --remove_history (ACTION|all) - remove the history for the action. The parameter
 is the action, or "all" to remove all history
 --remove_pak - removes the pak file and all history
 --skip_history_update - removes the pak without updating history

The tour of error messages on the way there:

--action remove_history ... → Action: "remove_history" is not valid
--action cleanup --remove_history --pak <pak> → Action: --pak is not supported for remove_history
--action cleanup --remove_history → option --remove_history requires argument
--action cleanup --remove_pak <pak> → No PAK file specified

The second one is the nasty one: --remove_history requires an argument, so it silently consumed --pak as its value and then complained that --pak is not a valid action name. And the fourth shows --remove_pak is a bare flag, not an option that takes the PAK name — the PAK always goes in --pak. The forms that worked:

python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py \
 --action cleanup --pak vRealizeOperationsManagerEnterprise-910040025541550

python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py \
 --action cleanup --remove_history all --pak vRealizeOperationsManagerEnterprise-910040025541550

And one final surprise from the results file afterwards: --remove_history all flipped a dozen keys to "history removed" — but not the ones that mattered:

apply_adapter_result: history removed
apply_system_update_result: history removed
new_validate_result: history removed
...
validate_result: Failed ← survived
apply_os_update_result: Failed ← survived
apply_os_update_exit_code: 1 ← survived

Several statuses outlived the removal — including, of all things, the apply_os_update Failed/exit-1 pair that is exactly what the retry guard checks. Keep that in mind for the endgame below.

While in there, this is also the moment to fix the actual root cause — put repo definitions back in /etc/yum.repos.d/. A snapshot, a sibling appliance, or the Photon base image all work as a source; strictly speaking the upgrade only needs the glob to match one file, but restoring the stock set is the right fix. Healthy looks like this:

local-pak-iso.repo photon-extras.repo photon-release.repo photon-srpms.repo
photon-debuginfo.repo photon-iso.repo photon.repo photon-updates.repo

The Precheck 409: Colliding With Your Own Assessment

With pakManager’s state cleared and the repo directory restored, the next upgrade precheck failed after about a minute with:

An unexpected error occurred in step ops_precheck. Reference Code: EEF54002.
409: upgrade.verification.pak_installation_active
PreConditionFailedCasaException: key=upgrade.verification.pak_installation_active

My first theory was the obvious one — that null-completed_time entry still sitting in CaSA’s history. The CaSA log killed that theory and replaced it with something stranger. The precheck works by staging and installing an assessment PAK (APUAT-for-814x) on the appliance, and the sequence around the failure reads:

19:03:20 verifyConsistentState PAK_INSTALL → Cluster is in consistent state
19:03:21 APUAT-for-814x staged; validation, then apply_system_update begins
19:04:01 OrchestratorStatusDocument: current_pak_activity =
 {"pak_id":"APUAT-for-814x-910025541563","operation_name":"APPLY_SYSTEM_UPDATE","step":12,...}
19:04:01 verifyConsistentState PAK_INSTALL — again
19:04:02 Cluster state check failed: PreConditionFailedCasaException:
 key=upgrade.verification.pak_installation_active

The same consistency check passed at 19:03:20 with no PAK active, and failed 41 seconds later because a PAK installation was genuinely active — the assessment install the precheck itself had just launched. The stale enterprise-PAK entry was nowhere in it; CaSA was telling the truth about a collision between two halves of the same workflow. Meanwhile the assessment PAK, indifferent to the LCM step failing above it, ran to a clean finish in under twelve minutes — one more case of the layers not sharing a reality.

Two useful tools on the CaSA side regardless. Its deployment state is queryable directly — and note it reported "DONE" while the 409 was still reproducible, so treat it as one more opinion, not an oracle:

curl -k -u admin https://localhost/casa/deployment/cluster/installationstate

And the non-destructive reset is restarting the service:

systemctl restart vmware-casa

I restarted CaSA and re-ran the precheck; this run went clean end to end — the assessment PAK staged, validated, installed, and cleaned up in thirteen minutes, and the step reported success. Whether the restart mattered or the retry simply avoided the earlier collision, I cannot prove from the logs — what the evidence does show is that the null-completion history entry was still in place, untouched, during the passing run. It was never the thing the 409 keyed on.

The NullPointerException: Asking an Empty Orchestrator for a Task

Precheck green, one more attempt at the actual upgrade — and a new reference code:

An unexpected error occurred in step ops_apply_upgrade. Reference Code: 90AD4093.
Detail: Cannot invoke "com.broadcom.vcf.lcm.sddc.upgrade.sdk.client.appliance.model.Task.getId()"
because "task" is null

An NPE in the LCM client is a product bug in the sense that it hides the real condition — but the underlying state is easy to read. CaSA’s orchestrator, logged on every status poll, was completely empty:

UpdateOrchestratorDocument={"casa_webapp_pak_id":null,"casa_webapp_force_content_update":false,
 "casa_pak_status":{}}
OrchestratorStatusDocument: {"current_pak_activity":{"pak_id":null,"operation_name":null,
 "step":0,"total_steps":0,...},...,"queued_pak_ids":[],"is_upgrade_orchestrator_active":false,...}

And the PAK’s working directory was down to two files:

/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-910040025541550/
├── manifest.txt
└── vRealizeOperationsManagerEnterprise-910040025541550.results ← still says Failed

Two separate subtractions added up to this. The reboot had emptied CaSA’s runtime documents and queue — that happened back at 12:43, and it is what stopped the retry loop. The cleanup action had then removed the extracted PAK content from pakRepoLocal — which is its job. Between the two, nothing on the appliance still described an in-flight upgrade, so when LCM told CaSA to proceed with the apply step, there was no registered PAK, no queued operation, and nothing to build a task from. CaSA returned no task; the LCM client dereferenced it; NPE. The pendulum had swung all the way from state says failed, refuse to run to no state at all, nothing to run.

The original binary survives, though — cleanup did not touch the staged PAK repository:

locate .pak | grep 0400
/storage/db/casa/pak/dist_pak_files/VA_LINUX/vRealizeOperationsManagerEnterprise-910040025541550.pak
(plus the 0400 license-server and cloud-proxy PAKs, trimmed)

The Clean Path Out

⚠️ What follows is the planned remediation from this point, not a procedure I have verified end to end — which is exactly why step 1 exists. It is a lab; on a production deployment, this is the moment to take the reference codes (EEF54002, 90AD4093) and the support bundle to an SR instead of editing state by hand.

At this point the environment is: cluster online and collecting, on 9.1.0.0300; repo directory restored; precheck passing; PAK binary staged; and two pieces of stale state left — the Failed survivor keys in the results file and the null-completion entry in CaSA’s per-PAK history. The way out is to stop half-clearing and reset the PAK completely, then let the workflow rebuild everything from the binary:

Snapshot the appliance. The cluster is healthy right now; protect that before another attempt.
Remove the PAK and all of its history. Per the usage text, --remove_pak “removes the pak file and all history” — note this is the one cleanup variant I have not exercised, and given that --remove_history all left the Failed keys standing, do not take the usage text on faith:
```
python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py \
 --action cleanup --remove_pak --pak vRealizeOperationsManagerEnterprise-910040025541550
```
Afterwards, verify the pakRepoLocal directory (with its .results file) is actually gone, and that the staged binary in /storage/db/casa/pak/dist_pak_files/VA_LINUX/ is still there — if --remove_pak reaches into the staged repository too, the workflow will need to re-download the PAK, which is inconvenient rather than fatal.
Clear CaSA’s per-PAK history so the UI stops rendering the ghost Installing operation and nothing can ever mistake it for an active install again — with the service stopped, move the stale file aside:
```
systemctl stop vmware-casa
mv /storage/log/vcops/log/casa/casa_pak_history_vRealizeOperationsManagerEnterprise-910040025541550.json /root/
systemctl start vmware-casa
```
Re-run the express patch from the beginning, not as a resume. With the staged .pak in place there is nothing to re-download; the workflow re-stages, re-validates, and — with /etc/yum.repos.d/ populated — should get past the two-second mark it never survived before. Expect the cluster to go offline again for the apply; that is the designed behavior (bring_cluster_offline: true in the manifest), so schedule it.

Verification

What success looks like once the re-run goes through:

The appliance reports 9.1.0.0400.25541550 and the express patch workflow completes without a reference code.
curl -k -u admin https://localhost/casa/deployment/cluster/installationstate returns "DONE" — with no failed step behind it this time.
The new casa_pak_history_vRealizeOperationsManagerEnterprise-910040025541550.json shows every operation, APPLY_OS_UPDATE included, with a non-null completed_time.
The cluster slice is ONLINE and collecting, and no apply_os_update invocations appear in the pakManager logs after completion — the loop stays dead.
/etc/yum.repos.d/ is still populated. If it is empty again, find the culprit before the next patch does this all over.

What Actually Emptied /etc/yum.repos.d?

The unresolved thread. The directory was demonstrably populated on July 5 (the 0300 patch’s sed succeeded and copied local-pak-iso.repo into it) and demonstrably held nothing the glob could match on July 28. Neither observed upgrade run removed those files, and the PAK cleanup actions do not reach into /etc. Something else did, in the three-week window between patches.

The usual suspects are STIG/hardening remediation, a compliance scan “fixing” findings, or an overzealous cleanup script — and the hardening theory is less hypothetical than it sounds: the appliance ships a STIG-hardening Ansible role that the upgrade itself runs during apply_system_update, and its task list includes a literal “Find yum repos” step. A framework that polices exactly this directory demonstrably lives on the box; the July runs used it non-destructively, but whatever emptied the directory operated on the same territory. I have not caught the culprit in the act, so the case stays open.

If that something is scheduled, it will do it again, and the next express patch will die the same two-second death. Before every PAK-based upgrade on a VCF Operations appliance, it now costs me one command to check:

ls /etc/yum.repos.d/

Empty directory, no upgrade. It is that direct.

Takeaways

A stuck progress bar is a claim, not a fact. The UI renders CaSA’s operation history; a null completed_time displays as Installing forever, even when the failure happened in two seconds and nothing has run since. The pakManager logs are where the truth lives.
Know the four state stores — per-PAK results file, vcopsPakManagerCommonHistory.json, CaSA’s casa_pak_history_<pak>.json, and CaSA’s runtime orchestrator documents. Every confusing symptom in this saga was two of them disagreeing — including the 409, where the precheck disagreed with the assessment install it had launched itself.
The retry guard message is honest but the CLI is not literal about it — remove_history is a parameter of --action cleanup, not an action, and a missing argument makes it eat the next flag with a misleading error.
Verify what state-clearing commands actually cleared. --remove_history all reported nothing wrong while leaving the apply_os_update: Failed keys — the exact ones the guard checks — untouched. After any cleanup, read the .results file; do not assume.
cleanup without the root cause fixed is a trap. The guard fires before the script runs; clear the state while /etc/yum.repos.d/ is still empty and the next attempt fails in two seconds and re-arms it — from an offline cluster.
Reboots do not clear persisted upgrade state. Offline stays offline, Failed stays Failed, and the null completion stays null. What a reboot did do here is wipe CaSA’s runtime orchestrator — which killed the retry loop, made the wedge quieter and easier to misread as idle, and set up the eventual task-is-null NPE.
Fatal-error support bundles skew timelines. Nearly all of the ten minutes this “upgrade” ran were generateSupportBundle.py executing after the fatal error. Read the stderr log inward from the first ERROR, not the timestamps outward.
Check /etc/yum.repos.d/ before any PAK upgrade, and find out what is emptying it — whatever removed those files will break the next patch identically if it runs again.

Final Notes

Every stage of this one was the same lesson wearing different clothes: the upgrade machinery is several loosely-coupled state machines, and each error message was one machine reporting its own local truth. pakManager said previously failed; CaSA said still installing; the precheck said installation active — correctly, about the assessment install it had just started itself; LCM said task is null. Every statement was locally accurate, none of them described the system, and the actual root cause — a directory that a three-week-old patch run proves used to be populated — never appeared in any of them.

The two-second failure was trivial. The day was spent on state.

VCF Operations for Networks 9.1 Search Queries: The vRNI Query Language Lives On

Cosmin Trif — Mon, 27 Jul 2026 07:00:00 +0000

If you spent years typing flows where firewall action = deny into vRealize Network Insight, then watched the product get renamed to Aria Operations for Networks and then absorbed into VCF Operations as VCF Operations for Networks, you probably have one practical question: did the search language survive?

It did — completely. I compared the VCF Operations for Networks 9.1 search documentation against the legacy Aria Operations for Networks 6.13 docs page by page, and the query grammar is identical: same entity types, same where operators, same group by / order by / limit clauses, same planning and path queries. Even the REST API still answers under the old /api/ni/ (Network Insight) namespace. Your saved searches and your muscle memory carry over as-is.

The short version:

VCF Operations 9.1 has one console but at least four separate query languages: the networks entity search (vRNI heritage), the Enhanced Search metric queries (vROps heritage), log search under Operate > Logs, and the new-in-9.1 PromQL Custom Query for real-time metrics. Knowing which bar you are typing into matters more than any syntax detail.
The networks search DSL is unchanged from vRNI/AON: <projection> of <entity> where <condition> group by <property> order by <field> limit N, plus top N, in last N days, path queries (VM 'a' to VM 'b'), and plan queries for micro-segmentation.
The 9.1 differences are cosmetic or peripheral: NSX Advanced Load Balancer is renamed Avi Load Balancer in the docs, “VMware Cloud (VMC)” prose became “VMware Cloud on AWS”, the VMware SD-WAN (VeloCloud) search-queries page is gone from the 9.1 doc tree, and AWS/Azure data sources are deprecated in 9.1.
The classic community query lists — the ones we all bookmarked in the vRNI days — still apply. I have collected the best of them below, together with the official examples, into one cookbook.

Everything here is based on the 9.1 documentation set and verified against the legacy 6.13 docs; sources are linked at the end.

One Console, Four Query Languages

The single most confusing thing about search in VCF Operations 9.1 is that “search” is not one feature. The console consolidates products that each brought their own query engine, and none of them were merged:

Query surface	Where it lives in 9.1	Grammar	Lineage
Networks entity search	Inside the VCF Operations for Networks section (Operate pillar)	`entity where ... group by ...`	vRNI / Aria Operations for Networks
Enhanced Search (metric queries)	The global search bar in the top menu, on every page	`Metric of Object Type where ... childOf ...`	vROps / Aria Operations
Log search	Operate > Logs	Log filtering, alert queries, field extraction	Aria Operations for Logs
Custom Query (new in 9.1)	Operate > Real-Time Metrics / Troubleshooting Workbench	PromQL	New real-time metrics pipeline

This post is mainly about the first one, because that is where the network troubleshooting value is. The second gets its own section further down, since the two are easy to conflate — the same word where appears in both grammars but the operator sets are different. The 9.1 release notes only call out one genuinely new query mechanism, and it is the fourth: “Use PromQL-based search queries to derive operational insights.” Neither the networks DSL nor the Enhanced Search grammar changed in 9.1.

A note on architecture, since the rebrand hides it: VCF Operations for Networks is still a separate, pre-hardened appliance, deployed as a Day-2 operation from VCF Operations and lifecycle-managed by Fleet Management — not code merged into the core VCF Operations binary. Only one instance can be integrated per fleet. Licensing is automatic starting with VCF 9: the component is licensed through the vCenter instance it is connected to, and there is nothing to license manually anymore.

Anatomy of a Networks Search Query

The documentation splits queries into four categories: structured queries, entity queries, planning queries, and path queries. In practice you will spend most of your time on structured queries, which read like a relaxed SQL:

sum(bytes) of flows where Flow Type = 'Routed' group by Source Subnet Network order by sum(bytes)

The pieces, in the order the parser expects them:

Entity Type (Mandatory)

Every structured query is anchored on an entity type: vms, hosts, flows, firewall rules, switch ports, routes, nsx managers, kubernetes pods, and many more. Singular and plural both work, and synonyms are accepted (virtual machine = vm). Queries are not case-sensitive.

Typing just the entity type is already a valid query — vms lists every VM. The search bar validates as you type: a check mark means the query is valid, a cross means it is not. An old vRNI trick from the community lists that is still worth trying: type help into the search field to bring up the search help.

Filters: `where` Plus Operators

Conditions are property, operator, value, combined with and / or and parentheses:

Operator	Example
`=` / `!=`	`flows where source ip address = '10.16.240.0/24'`
`>` `<` `>=` `<=`	`vms where memory >= 4096 mb`, `vms where cpu usage rate < 70%`
`like` / `not like`	`vms where name like 'app'`
`in` / `not in`	`flows where port in (22, 23, 80, 443)`
`is set` / `is not set`	`vms where firewall rule is set`
`matches` / `not matches`	`vm where name matches '[a-z]vm-delta[0-9]'` (regex)
grouping `()`	`flows where (src tier = 'App' and destination tier = 'DB') OR (destination tier = 'App' and source tier = 'DB')`

The underrated one is the nested in — a sub-query as the right-hand side:

vm where host in (host where Blade like 'sys/chassis-1')
vm where vlan in (vlan of vm where os in ('Microsoft Windows Server 2003', 'Microsoft Windows Server 2008'))

The second example is a trimmed version of the documented “total attack surface” query (the full one lists five vulnerable operating systems and ends with group by Vlan): every VM that shares a VLAN with a VM running a vulnerable OS.

Projections: What to Show

By default a query returns the entity’s standard field set. A projection in front changes that:

Property list: os of vms, cpu cores of vms, or several at once — CPU Usage Rate, Network Rate, mac address, ip, host of vm 'Quality control-VM26'
Count: count of vms, count of vm where operating system like 'Microsoft Windows Server 2008'
List(): fetch one entity while filtering on another — List(host) of vms where memory <= 2gb
Aggregates (sum, max, min, avg): sum(memory) of hosts, sum(bytes) of flows
Series for time-series graphs: series(avg(cpu usage)) of vms where cpu cores = 4

Grouping, Ordering, Limiting

vms group by host
sum(bytes) of flows group by dest vm order by sum(bytes)
vms order by memory limit 5

Three rules worth memorizing, because they are the usual cause of the red X:

limit must be preceded by order by.
order by accepts exactly one field; descending is the default (asc flips it).
When a group by query has a projection or an order by, the projection must contain an aggregation function. sum(bytes) of flows group by dest vm is valid; bytes of flows group by dest vm is explicitly called out in the docs as invalid.

Time: `top N` and `in last N days`

The examples in the official advanced-queries page lean heavily on two inline constructs:

top 7 vm group by name, Vlan order by sum(Total Network Traffic) in last 7 days
Kubernetes alerts where alert code = 'ImagePullBackOff' in last 24 hours
New firewall rules in last 24 hours

There is also a UI time control next to the search bar with presets (“last 24 hours”, “last 3 days”), an At option for a point in time, and a Between option for a range — useful for re-running the same query against last Tuesday’s state, which remains one of this product’s superpowers.

Path and Planning Queries

Path queries draw the hop-by-hop topology between two endpoints, including the underlay:

VMware VM 'VM01' to VMware VM 'VM02'
vm 'app1-web' to Internet

Planning queries feed the micro-segmentation donut — the flow analysis view you use before writing distributed-firewall rules. The documented forms take a bare argument:

plan security in last 7 days
plan securitygroup1
plan host1

The community lists extend these with filters — plan vm where folder = 'Production' and plan flows where vm = 'app1-web' and traffic type = 'INTERNET_TRAFFIC' both come from mtellin’s gist rather than the official grammar reference, but they follow the same where rules as everything else.

Entity and Free-Text Searches

Plain strings work too: 'prod-68-1' (exact name, quoted), prod (partial name), vm app1 (entity type plus partial name). And a community favorite from the gist: paste a bare MAC address like 00:50:56:a6:c9:6a into the search bar to find the machine that owns it — a great party trick when someone shows up with nothing but a switch log.

vCenter Tags

vm where tag = 'Environment:Production'

The syntax is '{category}:{tagname}'. Tags are also available as a Group-by criterion in flow analysis (up to three at once).

The bookmark icon next to the search bar saves any valid query; saved searches are per-user, and six defaults ship with the product (All Flows, Applications, Azure, Kubernetes Dashboard, Top Trends, NSX). Two features worth more attention than they get:

Help > Useful Searches is a curated, tagged library of pre-built queries — parameterized ones open pre-filled with sample values you swap out.
View Search Query on dashboard widgets (VM Details, Host Details, NSX Manager, Flow Insights, and a few others) reveals the query powering the widget, which you can copy into the search bar and modify. This is the fastest way I know to learn the DSL: find a widget that almost shows what you want, steal its query, adjust.

And almost any search can become an alert: run the query, then create a notification from the results page. Threshold queries (flows where firewall action = deny, packet-drop queries) make surprisingly good custom alerts.

The Query Cookbook

The queries below are collected from the official 9.1 search-queries reference and the two classic community lists — Pat’s vRNI search queries post and mtellin’s gist — deduplicated and grouped by what I actually use them for. Since the grammar is unchanged from vRNI, the community-era queries remain valid in 9.1 unless they depend on a data source that is gone (more on that below).

Flows and Traffic Patterns

What it answers	Query
Denied flows, environment-wide	`flows where firewall action = deny`
Internet traffic by source VM	`sum(bytes) of flows where Flow Type = 'Internet' group by src vm`
Top talkers, limit 10	`sum(bytes) of flows group by src vm order by sum(bytes) limit 10`
Top ports by volume	`sum(bytes) of flow group by port order by sum(bytes)`
Sum and average per port	`sum(bytes), avg(bytes) of flows group by port`
Top routed subnet pairs (last 7 days)	`top 7 flow where Flow Type = 'Routed' group by Source Subnet Network, Destination Subnet Network order by sum(Bytes) in last 7 days`
Traffic that never leaves its subnet — micro-segmentation candidates	`top 7 flow where Flow Type = 'Switched' group by Subnet Network order by sum(Bytes) in last 7 days`
VMs talking outside the country	`top 7 flow where Destination Country != 'United States' group by Source VM, Destination Country order by sum(Bytes) in last 7 days`
Traffic between two application tiers, either direction	`Flows where (src tier = 'App' and destination Tier = 'DB') OR (destination tier = 'App' and source tier = 'DB')`
Data center interconnect traffic between two sites	`sum(bytes) of flows where (Dst Manager = 'site2' AND src manager = 'site1') OR (Dst Manager = 'site1' AND src manager = 'site2')`

NSX Firewall and Security

What it answers	Query
Effective rules between two VMs	`firewall rules from 'Prod-Midtier-1' to 'Prod-Db-1'`
Any/any allow rules — segmentation debt	`firewall rule where action = allow and service any = true`
Masked (shadowed) rules	`Firewall Rule Masked Alert`
Rules created recently	`New firewall rules in last 24 hours`
Rules no flow has ever hit	`Firewall rule where flow is not set`
Flow count per rule	`Flow group by firewall rule`
Flows hitting specific rule IDs	`Flow where rule id in (1011, 1012, 1013)`
VMs excluded from the DFW	`vm where firewall status = 'Excluded'`
VMs covered by a given rule	`vm where Firewall Rule = 'Prod MidTier to Prod DB - DBService'`
Security groups referenced by no rule at all	`Security group where Indirect Incoming Rules is not set and Indirect Outgoing Rules is not set and Direct Incoming Rules is not set and Direct Outgoing Rules is not set`
Rule and group membership churn	`firewall rule membership change`, `security group membership change`

VM and Host Inventory

What it answers	Query
VM count per OS	`vms group by operating system`
Segments with the most VMs	`vm group by l2 network`
Datastores with the most VMs	`vm group by datastore`
Hosts per vSphere version / build	`host group by version`, `host group by OS`
Network identity card for every VM	`l2 network, vlan, ip address, default gateway of vms`
VMs still running vulnerable operating systems	`count of vm where Operating System like 'Microsoft Windows Server 2003' or Operating System like 'Microsoft Windows Server 2008'`
Everything about one VM in a single line	`CPU Usage Rate, Network Rate, Memory Usage Rate, mac address, ip, vxlan, host of vm 'Quality control-VM26'`

Physical Network and Routing

What it answers	Query
All switches from one vendor	`switches where vendor = Cisco`
Switch ports dropping packets, worst first	`switchport where vendor = 'Cisco' order by total packet drops`
Routes in a VRF	`routes where vrf = 'Provider Edge 3'`
Routes via a given next hop	`routes where NextHop Router = 'California-Edge'`
BGP sessions that are not up	`BGP Connection where BGP Status != 'ACTIVE'`
BGP neighbors of a router	`list(BGP Neighbor) of NSX-T Router 'Router Name'`
ECMP-enabled routers on a manager	`NSX-T Router where ECMP = 'Enabled' and manager = 'NSX-T Manager Name'`

Kubernetes (VKS / Antrea)

What it answers	Query
Pod distribution per node	`kubernetes Pods group by Kubernetes Node`
Nodes not in Ready state	`kubernetes nodes where Ready != 'True'`
Pods created in the last hour	`new kubernetes pod in last 1 hour`
Dropped flows per service	`flows where firewall action = 'DROP' group by Kubernetes Service`
Cross-namespace compliance check	`flows from Kubernetes Namespace 'PCI' to Kubernetes Namespace 'Non-PCI'`

This category got more relevant in 9.1: container IPFIX flows from VKS clusters running Antrea now export to the VCF Operations for Networks collector, with pod-to-pod and pod-to-external traffic dashboards.

Dashboard Shortcuts

Not structured queries, but single keywords from the gist that jump straight to built-in views: problem, change (recent changes in the environment), topn, storage, and pci compliance of VC Manager '192.168.10.10' (the PCI dashboard — which, per the gist, required an Enterprise license back in the vRNI days).

A Caveat on Community Shorthand

The community lists include a few compact forms you will not find in the official grammar reference — vm by VLAN, show VLAN, vm by network rate, top 10 vms by max packet drops, flows where flow type = 'VM-Physical' by bytes. They come from vRNI-era posts that predate the current documented grammar, which only uses group by / order by for these jobs. I have kept them out of the tables above for that reason. If you rely on one of them, the in-product validation check mark will tell you immediately whether your build still accepts it.

The Other Search Bar: Enhanced Search Metric Queries

The global search bar at the top of the VCF Operations console — the one that is on every page — is a different animal with its own documented query language, under the section Broadcom titles “Enhanced Search Capability”. It searches the vROps-side world: objects, metrics, properties, dashboards, reports, views, alert definitions, symptom definitions, recommendations, notifications, IP addresses, super metrics, and (since recent releases) features — type li and select Feature to jump straight to Licenses, navigation path included.

Its metric query grammar looks like this:

CPU|Usage % of Virtual Machine
CPU|Usage % of Virtual Machine where CPU|Usage % > 15 and Memory|Usage % > 15
CPU|Demand MHz of Virtual Machine where Configuration|Name contains 'Centos'
CPU|Usage % of Virtual Machine where CPU|Usage % > 15 or Memory|Usage % > 15 childOf vSphere World

The differences from the networks DSL are exactly where you would trip:

The pattern is Metric of Object Type, with the pipe character separating metric group and metric name (CPU|Usage %). If you start with a metric name, naming the object type is mandatory.
String operators are camelCase keywords: equals, notEquals, contains, notContains, startsWith, notStartsWith — there is no like, no in, no matches here. Numeric operators (>, >=, <, <=, =, !=) plus arithmetic (+ - * / with parentheses) and and/or round out the set. String values are case-insensitive.
childOf at the end of the query scopes results to descendants of an ancestor object — the closest thing this grammar has to the networks DSL’s nested in.
Hard limits: 300 characters per query, at most five different metric names. Break either and a red banner explains the issue; for syntax errors specifically it reads “The query contains syntax error(s). Please modify it to get suggestions.”
There is no group by or order by in the query itself; slicing happens on the Metric Search Results page instead, via Transformation (Min/Max/Average/Sum/First/Last/Current), Roll Up (hour through year), Parents, value-distribution buckets, and time settings. The Transformation filter only activates when the query has a where condition and the object type has at most 200 objects over at most one month. From any result row, the vertical-ellipsis menu jumps into the Troubleshooting Workbench.

One honest note: none of this is new in 9.1. I compared the 9.0 and 9.1 pages — the content is identical — and the same grammar exists in Aria Operations 8.x. What is new in 9.1 on the metrics side is the PromQL-based Custom Query in Real-Time Metrics, which is a separate topic (and a separate language) altogether.

What Actually Changed in 9.1 for Operations for Networks

Since search itself did not move, here is what did — the 9.1 items relevant to a networks operator, from the release notes and product support notes:

Network Assessment and Value report — evaluates your current VDS-based networking and produces a CAPEX/OPEX analysis for VCF networking capabilities. It replaces the vCenter Assessment report, which was removed.
VPC Planning — guided vSphere-to-VPC transition planning, with subnet tables and a CSV export of the generated VPC design.
Expanded IPFIX for VKS clusters — Antrea container flows land in the collector; pod-to-pod and pod-to-external dashboards.
Expanded migration planning — automatic migration wave generation, clustering workloads by flow, latency, and bandwidth data.
Network health dashboards — NSX Edge appliance and ESX host networking capacity monitoring surfaced through VCF Operations Health and Diagnostics.
Deprecations: AWS and Azure data sources are deprecated in 9.1, and the networks API had 22 operations marked deprecated — the largest deprecation set in the Operations component. The VMware SD-WAN (VeloCloud) search-queries reference page is also absent from the 9.1 documentation tree.

That deprecation list is worth acting on if you have search-based alerting or scripts built on AWS/Azure entity queries (AWS EC2 where Security Group = ..., Azure NSG Rule where Action = 'ALLOW') — the syntax still exists in the 9.1 reference today, but the data sources feeding those entities are on the way out.

Getting There from Standalone 6.x

If you are still on a standalone appliance — I have covered upgrading Aria Operations for Networks to 6.14 via the upgrade bundle and via Aria Suite Lifecycle before — the path into 9.x is: get to 6.14.0 Patch 6 or 6.14.1+, import the instance into VCF Operations Fleet Management, upgrade it in place to VCF Operations for Networks 9.0.x, and from there directly to 9.1. Only one instance can be imported per fleet. Every search query in this post works the same on both sides of that migration, which is precisely the point.

The API Corner

For automation, the search subsystem is exposed over REST, still under the Network Insight namespace:

POST /api/ni/search/ql — takes the same query-language strings you type in the search bar
POST /api/ni/search — entity search
POST /api/ni/search/aggregation — aggregations
POST /api/ni/search/groupby — group-by queries

The schemas are on the VCF Operations for Networks API reference. The ql endpoint is the interesting one: anything you prototype in the search bar can be lifted verbatim into a script.

References

Fixing an NSX Service Account Rotation Loop in VCF 9.1: PASSWORD_MANAGER_NSX_USER_API_FAILED and the 128-Character Password

Cosmin Trif — Wed, 15 Jul 2026 17:30:00 +0000

Continuing the credential cleanup in the same 9.1 environment from the expired password sync and VCFOPSPWID03 posts, one credential refused to settle: the service account SDDC Manager uses to talk to the NSX Manager API, svc-sddcmanager-a-nsx-mgmt--8199 in my lab. It showed up as disconnected, every remediation attempt failed, and the environment kept trying to rotate it in a loop.

This one took several wrong turns before the actual root cause surfaced, and almost every wrong turn taught me something worth writing down. The short version:

SDDC Manager generated a 128-character password for the service account, but its own remediate path enforces a 127-character maximum — an off-by-one between the generator and the validator that keeps the rotation failing.
The service account is an OS-level PAM user on the NSX appliance: invisible to the NSX API, UI, and CLI, manageable only as root with standard Linux tools.
Failed rotation attempts had locked the account at the NSX API level.
And the final boss: SDDC Manager’s rotation machinery was re-changing the password within a couple of seconds of any manual reset, so every fix I tested was already stale by the time I tested it.

The Symptom: A Service Account Stuck Disconnected

The credential in question, as SDDC Manager tracks it:

Field	Value
Username	`svc-sddcmanager-a-nsx-mgmt--8199`
Resource type	`NSXT_MANAGER`
Credential type	`API`
Account type	`SERVICE`

The rotation failures are visible on the SDDC Manager appliance in the operations manager log:

tail -f /var/log/vmware/vcf/operationsmanager/operationsmanager.log |
 grep -Ei 'svc-sddcmanager|ROTATE|REMEDIATE|diagnosticMessage'

The recurring entry that matters (flattened here):

errorCode: PASSWORD_MANAGER_NSX_USER_API_FAILED
diagnosticMessage: Response from NSX <nsx-manager-fqdn>: 403 -
The credentials were incorrect or the account specified has been locked.

A 403 for a service account that only SDDC Manager itself manages. Nobody had touched this password by hand — SDDC Manager rotated it, and then could no longer authenticate with the value it had just set.

Note: The examples in this post use curl -k, which skips TLS certificate verification — fine in a lab where the appliances still run self-signed certificates. In production, drop -k and trust the appliance CA instead. If any password you type contains !, disable history expansion first (set +H in bash, unsetopt BANG_HIST in zsh) so the shell does not mangle it. And the read -rp / read -rsp prompts use bash syntax — on the appliances that is the shell you get, but if you run any of this locally from zsh, start bash first or use the zsh form (read -rs 'VAR?prompt: '); in zsh, read -rsp silently leaves the variable empty, and an empty password produces confusing downstream errors instead of a clear failure.

The Off-by-One: 128 Generated vs 127 Accepted

Digging through the failed rotation tasks showed the first real clue: the password SDDC Manager had generated for the service account was 128 characters long, while the remediate path rejects anything over 127 characters.

The generator takes its cue from the NSX node authentication policy. NSX advertises a maximum_password_length of 128 by default, SDDC Manager generates a password at that maximum, and then its own validation refuses to remediate with it. The account ends up in the worst possible state: the rotation half-succeeds often enough to keep changing the password, but remediation can never complete, so the credential is marked disconnected and the platform keeps retrying.

Dead Ends Worth Knowing About

Before the fix, the detours — each one reasonable, each one wrong.

Remediating from the UI

The obvious first move. The remediation workflow failed with the same 403, and the UI adds nothing to the diagnosis — the useful error text only exists in operationsmanager.log.

Guessing the Credential Tuple

My first API attempts failed validation before doing anything, because a PATCH /v1/credentials call must reference the exact tuple SDDC Manager has on file: resourceName, resourceType, credentialType, and username all have to match. Do not guess any of them — read them back first.

Get a token (I ran these directly on the SDDC Manager appliance against localhost, but the FQDN works the same):

SDDC="https://localhost"
read -rp "SSO username: " SSO_USER
read -rsp "SSO password: " SSO_PASS; echo

TOKEN=$(jq -nc --arg u "$SSO_USER" --arg p "$SSO_PASS" \
 '{username:$u,password:$p}' |
 curl -sk -H 'Content-Type: application/json' -d @- \
 "$SDDC/v1/tokens" | jq -r .accessToken)

unset SSO_PASS
[ -n "$TOKEN" ] && [ "$TOKEN" != "null" ] || echo "Token acquisition failed" >&2

The block prints nothing on success — the token lands silently in $TOKEN. Test it with a cheap authenticated call before building anything on top of it:

curl -sk -o /dev/null -w "%{http_code}\n" \
 -H "Authorization: Bearer $TOKEN" "$SDDC/v1/domains"

A 200 means the token works. Anything else, start over at the token block.

List the NSX credentials and pull the exact tuple:

curl -sk -H "Authorization: Bearer $TOKEN" \
 "$SDDC/v1/credentials?resourceType=NSXT_MANAGER" |
 jq -r '.elements[] | select(.accountType=="SERVICE") |
 [.resource.resourceName, .resource.resourceType, .credentialType, .username] | @tsv'

That returns the values used in the remediation calls later in this post.

If jq fails here with “Cannot iterate over null”, the response was not a credentials list — it is almost always an authentication error object instead. Check the HTTP status and the raw body:

curl -sk -o /tmp/creds.json -w "%{http_code}\n" \
 -H "Authorization: Bearer $TOKEN" \
 "$SDDC/v1/credentials?resourceType=NSXT_MANAGER"

jq . /tmp/creds.json

A 401 means the token is empty, expired (they are short-lived), or the literal string null from a failed acquisition — re-run the token block above and mind its failure check. A 401 body complaining that “jwt strings must contain exactly 2 period characters” is the same problem stated more directly: whatever is in $TOKEN is not a JWT at all, and the usual cause is a rejected /v1/tokens login quietly turning into null — check that the username is a full SSO identity such as administrator@vsphere.local, not a bare username.

Capture the service account username while you are here — the remediation and rotation calls later run in this same SDDC Manager session and reuse it:

SVC=$(curl -sk -H "Authorization: Bearer $TOKEN" \
 "$SDDC/v1/credentials?resourceType=NSXT_MANAGER" |
 jq -r '.elements[] | select(.accountType=="SERVICE") | .username')

echo "$SVC"

If you spend time in this API, the /v1/credentials overview covers the basic operations.

Changing the VCF Operations Password Policies

VCF Operations 9.1 exposes password policy settings, and lowering a maximum length there looked like the clean fix for the 128-character problem. It did nothing. Those policies do not govern what SDDC Manager generates for service accounts — the generator follows the NSX node authentication policy, not VCF Operations. Worth knowing before you spend an afternoon proving it.

The First Fix: Lower the NSX Maximum Password Length

Since the generator follows the NSX policy maximum and the validator caps at 127, the workaround is to lower the NSX maximum_password_length so generated passwords land under the limit. I went to 100 — comfortably below 127, still far longer than any human-managed password.

One trap first: the setting lives in the auth-policy endpoint. There is no /api/v1/node/aaa/password-complexity endpoint — that name belongs to the NSX CLI command (get/set password-complexity), and the API returns 404 for it. The REST path you want is /api/v1/node/aaa/auth-policy, the same endpoint from the password reuse post.

Back up the current policy before changing anything:

read -rsp "NSX admin password: " NSXPASS; echo

curl -sk -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy |
 tee /tmp/auth-policy-backup.json | jq .

{
 "api_failed_auth_lockout_period": 900,
 "api_max_auth_failures": 5,
 "minimum_password_length": 12,
 "maximum_password_length": 128,
 ...
}

Lower the maximum (maximum_password_length accepts 8 to 128):

curl -sk -u admin:"$NSXPASS" -X PUT \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy \
 -H 'Content-Type: application/json' \
 -d '{"maximum_password_length": 100}'

Policy changes take a short while to apply and synchronize across the manager cluster nodes in 9.x — read the value back before moving on:

curl -sk -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy | jq .maximum_password_length

New rotations now generate passwords of at most 100 characters, comfortably under the 127-character validation. Problem solved — except the account was still disconnected, and remediation still failed with the same 403.

Plot Twist: TEST_BEFORE_REMEDIATE

The remediation task logs showed the failure happening in a TEST_BEFORE_REMEDIATE stage. Before SDDC Manager remediates a credential, it first authenticates against NSX as the service account with the password it believes is current. If that test fails — wrong password or locked account — the whole remediation aborts with PASSWORD_MANAGER_NSX_USER_API_FAILED, and nothing is remediated.

That explains the loop: the earlier failed rotations had burned through NSX’s API authentication failure allowance (api_max_auth_failures: 5 in the policy above), the account got locked, and from then on even a correct password could not pass the pre-remediation test.

So before anything else could work, I had to look at the account itself on the NSX side — which is where this got interesting.

The Hidden User: An API-Only PAM Account

The service account does not exist as far as the NSX management plane is concerned:

GET /api/v1/node/users — not listed.
NSX UI under user management — not there.
NSX CLI get user / set user — unknown user.

But on the NSX Manager appliance as root, it is a perfectly ordinary Linux account:

grep svc- /etc/passwd

svc-sddcmanager-a-nsx-mgmt--8199:x:...:...:/home/svc-sddcmanager-a-nsx-mgmt--8199:...

The account authenticates to the NSX API through PAM, has password history in /etc/security/opasswd, and is managed by exactly nothing except root and the standard Linux toolbox. No SSH access, no UI, API-only. That means the diagnostics you would normally run through the NSX API happen with faillock and chage instead:

SVC="svc-sddcmanager-a-nsx-mgmt--8199"

faillock --user "$SVC"
chage -l "$SVC"

faillock confirmed the lockout — a stack of recent authentication failures, courtesy of the rotation loop. Clear it:

faillock --user "$SVC" --reset

While in there, two more root-level tools are worth knowing for this account type:

passwd "$SVC" # set a known password, as root
echo "" > /etc/security/opasswd # wipe PAM password history — see warning

⚠️ /etc/security/opasswd holds the PAM password history for every user on the appliance, not just the service account. Emptying it is a blunt lab move to get past “password has been already used” rejections during repeated testing. Do not do this on a production appliance without a support ticket telling you to.

Keeping the Account from Locking Again

The lockout pattern here matches Broadcom KB 420736: SDDC Manager trips the NSX API lockout with its own failed authentication attempts, and remediation can never win against a locked account. The durable fix is to add SDDC Manager’s IP to the NSX lockout_immune_addresses list, so its authentication failures never lock anything.

Run this on the SDDC Manager appliance, so hostname -i resolves to the right IP; the jq filter appends it to the existing list without clobbering other entries:

Note: PUT /api/v1/cluster/api-service replaces the entire cluster-wide API service configuration document, and NSX restarts the API service to apply it — expect a brief API interruption, so run this when nothing else is mid-call. The pipeline below reads the full current document and keeps a copy in /tmp/api-service-backup.json before anything is overwritten.

curl -sk -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/cluster/api-service |
 tee /tmp/api-service-backup.json |
 jq --arg ip "$(hostname -i)" \
 '.lockout_immune_addresses += [$ip] | .lockout_immune_addresses |= unique' |
 curl -sk -u admin:"$NSXPASS" -X PUT \
 -H 'Content-Type: application/json' -d @- \
 https://<nsx-manager-fqdn>/api/v1/cluster/api-service

With the lockout cleared and immunity in place, I reset the service account password as root, confirmed it worked with a quick authentication test…

read -rsp "Service account password you just set: " SVCPASS; echo

curl -sk -o /dev/null -w "%{http_code}\n" \
 -u "$SVC:$SVCPASS" https://<nsx-manager-fqdn>/api/v1/cluster/status

unset SVCPASS

…and got 403. Seconds after setting it. With a password I had just set moments earlier.

The Real Root Cause: Racing the Rotation Engine

This was the point where the investigation stopped being about policies and started being about timing. The sequence kept repeating:

As root on NSX: passwd the service account to a known value.
Test authentication with that value: 200 — for a moment.
Test again a few seconds later: 403.

Nothing was “wrong” with any password. SDDC Manager’s rotation machinery was re-changing the service account password within a couple of seconds of any reset. The disconnected credential kept the retry loop hot, every retry rotated the password on the appliance, and every manual test raced the engine and lost. The vault was not stale, the policy was not blocking anything anymore — my known-good password simply stopped being the current password almost immediately after I set it.

Watching the log in one terminal makes the race visible:

tail -f /var/log/vmware/vcf/operationsmanager/operationsmanager.log |
 grep -Ei 'svc-sddcmanager|ROTATE|REMEDIATE'

The fix is to stop fighting the engine and synchronize with it instead: set the password and hand it to SDDC Manager inside that couple-second window, so the value on the appliance and the value being remediated converge before the next rotation fires.

⚠️ What follows is a lab maneuver. Manually resetting a platform-managed service account and racing the rotation engine is recoverable here; on a production NSX Manager, capture operationsmanager.log, the failed task IDs, and the faillock output and open a support request first — see the last bullet in Hardening Takeaways.

In practice, that means staging everything in advance:

Terminal 1 — NSX Manager as root, with a non-interactive password set ready to run:

read -rsp "New service account password: " SVCPASS; echo
echo "$SVC:$SVCPASS" | chpasswd
unset SVCPASS

Terminal 2 — the same SDDC Manager session from earlier, so $SDDC, $TOKEN, and $SVC are still set, with the remediation payload pre-built — the only thing left is to press Enter:

read -rsp "Same password as set on NSX: " SVCPASS; echo

TASK_ID=$(jq -nc \
 --arg fqdn "<nsx-manager-fqdn>" \
 --arg user "$SVC" \
 --arg pw "$SVCPASS" \
 '{operationType:"REMEDIATE",
 elements:[{resourceName:$fqdn, resourceType:"NSXT_MANAGER",
 credentials:[{credentialType:"API", username:$user, password:$pw}]}]}' |
 curl -sk -X PATCH \
 -H "Authorization: Bearer $TOKEN" \
 -H 'Content-Type: application/json' -d @- \
 "$SDDC/v1/credentials" | jq -r .id)

unset SVCPASS

Run terminal 1, then terminal 2 immediately — within the window, before the engine rotates again. Poll the task:

curl -sk -H "Authorization: Bearer $TOKEN" \
 "$SDDC/v1/credentials/tasks/$TASK_ID" | jq '{name, status}'

This time TEST_BEFORE_REMEDIATE authenticated successfully — the password it tested was, at that moment, actually the password on the appliance — and the remediation completed. The credential flipped back to healthy, and the rotation loop stopped, because there was nothing left to retry.

To finish cleanly, trigger one supervised rotation so the account ends up on a fresh SDDC Manager-generated password (now capped at 100 characters by the lowered policy) instead of a hand-typed one:

jq -nc \
 --arg fqdn "<nsx-manager-fqdn>" \
 --arg user "$SVC" \
 '{operationType:"ROTATE",
 elements:[{resourceName:$fqdn, resourceType:"NSXT_MANAGER",
 credentials:[{credentialType:"API", username:$user}]}]}' |
curl -sk -X PATCH \
 -H "Authorization: Bearer $TOKEN" \
 -H 'Content-Type: application/json' -d @- \
 "$SDDC/v1/credentials" | jq .

Poll the task the same way as before; with the loop broken and the policy cap in place, the rotation completes cleanly.

Verification

The rotation task in SDDC Manager completes successfully, and no new ROTATE/REMEDIATE retries appear in operationsmanager.log for the service account.
The credential status is back to healthy (GET /v1/credentials shows no error state for the tuple).
faillock --user "$SVC" on the NSX appliance stays empty.
lockout_immune_addresses in GET /api/v1/cluster/api-service contains the SDDC Manager IP.
The NSX auth-policy holds the values you intend to keep — in my case maximum_password_length: 100 stays, since the 128/127 mismatch would otherwise come straight back on the next rotation.

Clean up the working files, since the backups describe your security posture and the lockout-immune IP list:

rm -f /tmp/auth-policy-backup.json /tmp/api-service-backup.json
unset NSXPASS

Hardening Takeaways

A few things I am keeping from this one, beyond the fix itself:

Add SDDC Manager to lockout_immune_addresses proactively. Any credential workflow that can fail more times than api_max_auth_failures allows can lock its own account and wedge remediation. KB 420736 describes the pattern.
Back up auth-policy before any PUT — tee to a file costs nothing, and the endpoint carries every knob (lockout periods, complexity, history) in one document, so an overzealous edit has a wide blast radius.
Check NTP while you are in the logs. This environment’s operationsmanager.log also showed intermittent JWT clock-skew errors between components — harmless-looking noise that can turn into real authentication failures. Time sync issues and credential troubleshooting mix badly; rule them out early.
Know when to stop and open an SR. Manually resetting a service account password as root and racing the rotation engine is recoverable in a lab. In production, the moment a platform-managed service account is in a rotation loop, capture operationsmanager.log, the failed task IDs, and the faillock output, and open a support request — KB 420736 describes the lockout pattern, and KB 423038 is a related reference worth attaching alongside it.

Final Notes

Every layer of this problem produced the same useless symptom — a 403 — for four different reasons: a password too long to remediate, a locked account, a pre-remediation test that authenticates as the account it is about to fix, and finally a rotation engine overwriting the password faster than I could test it. The 128-versus-127 mismatch set the loop in motion, but the thing that made it unfixable-looking was the timing: every diagnostic I ran was measuring a password that had already been replaced.

Two habits would have shortened this significantly. First, tail -f the operations manager log before forming a theory — the rotation attempts were right there, timestamped, showing the engine acting seconds after every manual change. Second, when a platform manages an account, assume it will keep managing it while you troubleshoot: either pause the machinery or synchronize with it. Fighting an automated rotation engine with manual password resets is a race you lose every time — until you make the race the plan.

Fixing VCFOPSPWID03: NSX Password Update Failed as Call from NSX Received Invalid Response (VCF 9.1)

Cosmin Trif — Sat, 11 Jul 2026 17:30:00 +0000

While rotating credentials in the same 9.1 environment I covered in the expired password sync post, an Update of the NSX Manager admin password (credential type API) from VCF Operations kept failing after about a minute with error code VCFOPSPWID03 and a spectacularly unhelpful message: “call from NSX received invalid response.”

The actual cause turned out to be simple — NSX rejected the new password because it had already been used on that manager. This being a lab, I was deliberately recycling a previous password. That goes against best practices, and NSX enforces it. But the way the error surfaces (and the way the reuse policy behaves in this build) is worth writing down.

The Error: “Call from NSX Received Invalid Response”

The password update banner in VCF Operations:

The failed request under Management Tasks, with the task failing in Stage 1 after roughly a minute:

The full error text:

Error Code: VCFOPSPWID03
Password task UPDATE for <nsx-manager-fqdn> has failed. Error message:
NSX password update failed at <nsx-manager-fqdn>: Failed to update password
for NSX user admin for <nsx-manager-fqdn>, as call from NSX received
invalid response.

The same failure is visible on the VCF Operations node in the component-manager-plugin-*.log (in a support bundle it is under logs/; on the appliance under /storage/log/vcops/log/):

[...UpdatePasswordTask.execute] - Executing Update password task...
[...PasswordTaskUtil.monitorPasswordTask] - Password Task : <task-id> Status: IN_PROGRESS
[...PasswordTaskUtil.monitorPasswordTask] - Password Task : <task-id> Status: FAILED
[...Task.injectFailedEventWithLocale] - Injecting task failure event. Error Code : 'VCFOPSPWID03', Retry : 'false'
java.lang.RuntimeException: Password task UPDATE for <nsx-manager-fqdn> has failed. [...]
as call from NSX received invalid response.

“Invalid response” suggests connectivity or certificate problems, and that is exactly the wrong place to go digging. NSX was up, reachable, and answering — it just did not like the request.

Finding the Real Error

Drill into the failed password task and expand the details. The Cause field carries the message that the top-level error flattened away:

Description: Password update for resource : <nsx-manager-fqdn>,
 user : admin and credential type : API
Progress Messages: Failed to get NSX user details.
Cause: Specified password invalid: Password has been already
 used. Choose another.

There it is. Not a connectivity issue, not a certificate issue — password reuse. NSX checked the new password against its password history and refused it.

The NSX Password Reuse Policy

Password history on NSX is not the classic PAM /etc/security/opasswd mechanism (mine was empty — that was my first dead end). It is governed by the password_remembrance setting in the node authentication policy, alongside the complexity settings.

Prompt for the admin password once instead of putting it inline — passwords containing ! get mangled by history expansion in interactive shells (set +H disables that in bash; in zsh use unsetopt BANG_HIST), and inline credentials linger in shell history:

set +H
read -rsp "NSX admin password: " NSXPASS; echo

curl -k -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy

{
 "api_failed_auth_lockout_period": 900,
 "api_max_auth_failures": 5,
 "minimum_password_length": 12,
 "password_remembrance": 5,
 ...
}

Or on the NSX CLI: get password-complexity.

Per the VCF 9.1 NSX documentation, password_remembrance is the number of previous passwords a user cannot reuse. Three details matter here:

NSX supports one password policy — you cannot exempt a single user like admin from history checks — and the manager password-policy settings synchronize across the management cluster nodes.
In VCF 9.1 the default value is 5 and the documented valid range is 1 to 12 for new configurations. Out of the box, the last five passwords are blocked — which is why my recycled lab password was refused without anyone ever hardening this environment. (The same doc notes that policy settings customized before an upgrade are retained afterward, so upgraded environments may be carrying older values.)
Older NSX 4.x documentation describes 0 as “check disabled”. This 9.1 build rejects 0 outright, as I found out next.

Note: These examples use curl -k, which skips TLS certificate verification — fine in a lab where the manager still has a self-signed certificate. In production, drop -k and trust the NSX CA instead. When you are done, unset NSXPASS.

Why You Cannot Just Disable Password History

My first instinct was to turn the history check off entirely, since 0 was the documented disabled/default value in NSX 4.x. Both the CLI (set password-complexity) and the API refuse it in this build:

curl -k -u admin:"$NSXPASS" -X PUT \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy \
 -H "Content-Type: application/json" \
 -d '{"password_remembrance": 0}'

{
 "error_code": 255,
 "error_message": "Request body not valid - PasswordComplexityProperties.password_remembrance: value '0' must be greater than or equal to '1'",
 "module_name": "common-services"
}

That matches the 9.1 documentation, which lists a valid range of 1 to 12 for new password_remembrance configurations. In this build there is no accepted way to set 0 — the lowest you can configure is 1, which blocks only the single most recent password.

Recommended Fix: Use a New Compliant Password

In production — and honestly in most labs too — the right response to “Password has been already used” is the boring one: leave the history policy alone, pick a new password that satisfies the policy (minimum length, character classes, no dictionary-based words, not among the remembered previous passwords), and re-run the Update action in VCF Operations. The workflow completes, and the appliance password, the stored credential, and the expiration metadata all stay in sync.

The next section covers the lab case, where re-using one specific password was the whole point.

Lab-Only Workaround: Temporarily Lower Password History to 1

Treat this as a reversible change, not a new setting:

Confirm the cause first. The failed task’s Cause must actually say “Password has been already used” — this workaround fixes nothing else.
Record the current value, so you can restore it exactly:

curl -sk -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy | jq .password_remembrance

Lower it to 1 — the minimum of the valid range — so NSX only blocks reuse of the immediately previous password; anything older becomes acceptable again:

curl -k -u admin:"$NSXPASS" -X PUT \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy \
 -H "Content-Type: application/json" \
 -d '{"password_remembrance": 1}'

Read the value back before retrying. API policy changes take about 20 seconds to apply and synchronize across the manager cluster nodes — rerun the GET from step 2 and confirm it returns 1.
Re-run the Update action in VCF Operations. In my case the task completed with the recycled password, and because the change went through the supported workflow, the appliance password, the stored credential, and the expiration metadata all ended up in sync — no manual vault surgery required.
Restore the exact value you recorded in step 2 — not “whatever the default is”:

curl -k -u admin:"$NSXPASS" -X PUT \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy \
 -H "Content-Type: application/json" \
 -d '{"password_remembrance": <original-value>}'

Two follow-ups worth knowing:

Even with password_remembrance: 1, you can never “change” to the password that is currently set — the workflow requires current and new to differ — and the most recent entry in the history may still be blocked. If you absolutely must land back on a very recently used password (labs again), rotate through one or two throwaway passwords first.
Keep an eye on api_max_auth_failures — enough failed update attempts with a wrong current password will lock the account on top of everything else.

If Reuse Is Not Your Problem

The same flattened “call from NSX received invalid response” error can hide any password policy rejection, not just reuse. Start with the failed task’s Cause field — in this environment it carried NSX’s real error text, and reading it costs nothing.

If the Cause is still too vague, you can provoke NSX into stating its verbatim reason by attempting the change directly against the node API — with one important caveat:

⚠️ This is not a validation call. PUT /api/v1/node/users/<userid> is the real password-change API — there is no dry-run. If the request succeeds, the NSX admin password has just changed outside VCF Operations: the Fleet vault and SDDC Manager still hold the old credential, and every workflow that authenticates to NSX inherits that drift until you rotate once more through the VCF Operations Update action (entering the just-set password as the current one). Only run this with a password you actually intend to keep.

First find the user ID (the built-in admin is typically 10000):

curl -k -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/users

Then attempt the change, letting jq build the payload so special characters survive the shell:

read -rsp "New password: " NEWPASS; echo

jq -nc --arg old "$NSXPASS" --arg new "$NEWPASS" \
 '{password: $new, old_password: $old}' |
curl -k -u admin:"$NSXPASS" -X PUT \
 -H "Content-Type: application/json" -d @- \
 https://<nsx-manager-fqdn>/api/v1/node/users/<userid>

unset NEWPASS

Whatever check is failing comes back in plain text: reuse, minimum length, character classes, or the Linux PAM dictionary baseline. That last one deserves a mention: the docs state the default password complexity rules “as enforced by the Linux PAM module” always apply on top of the configurable settings, and there is no auth-policy knob to turn them off. Dictionary-based passwords — including the <VendorName>1!-style classics every lab inherits — get rejected on API-driven changes even when they satisfy length and character-class rules. If the error says “dictionary word” or “too simplistic,” no password_remembrance setting will save you; pick a password without recognizable words.

If the direct call succeeds but the VCF Operations workflow still fails, then you genuinely have a workflow-side problem, and the component-manager-plugin-*.log on the VCF Operations node is the place to look — but remember the vault is now behind, so plan one more rotation through Update once the workflow is fixed.

Verification

After the update task completes:

The Update Password request in Management Tasks completes successfully.
The new password authenticates against the NSX API — a quick policy read doubles as a credential check (prompt again so the variable holds the new password, and unset NSXPASS when done):

read -rsp "New NSX admin password: " NSXPASS; echo
curl -k -u admin:"$NSXPASS" \
 https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy

UI login to NSX Manager with the new password works.
password_remembrance is restored to the exact value you recorded before the workaround, not whatever the troubleshooting session left behind.
The password entry in Fleet shows a healthy status and refreshed expiration metadata — the account query from the expired password sync post works for this.

Final Notes

The error message is the villain of this story. “Call from NSX received invalid response” sent me looking at connectivity and certificates, when the task details had the real answer one click deeper: “Password has been already used. Choose another.” When a VCF Operations password task fails with VCFOPSPWID03, expand the task and read the Cause before touching anything else.

The second lesson is that the NSX build in VCF 9.1 rejects disabling password history — new password_remembrance values are limited to 1 through 12, with a default of 5. In a lab, temporarily dropping it to 1 is a reasonable escape hatch for recycling a password, as long as you restore the original value afterward. In production, reusing passwords is the actual problem — use a new compliant password and keep the history policy strict.

Fixing Expired Password Sync in VCF 9.1 Fleet Management

Cosmin Trif — Tue, 07 Jul 2026 17:30:00 +0000

In VCF 9.1, Fleet Password Management accounts can get stuck in EXPIRED status, and the password workflows changed compared to 9.0. In 9.0, you may have seen UI actions such as Disconnected or Remediate. In the 9.1 environment I was working on, those actions were not available. The UI exposed Update, and the Fleet Password Management API provides the same kind of workflow: query password accounts, then update or rotate the password.

Symptoms

The issue in this case was simple to describe but easy to chase in the wrong place:

Some password accounts showed as EXPIRED in Fleet password management.
The VCF Operations root password had already been changed outside Fleet.
The 9.1 UI did not expose the older remediation-style options, so the useful paths were the Update action for single accounts and the API for repeatable remediation.

This post walks through the recovery path I used for VCF 9.1.

What Changed in VCF 9.1 Password Management

The important realization is that VCF 9.1 Fleet Password Management does not expose a documented “store this already changed password in the vault” workflow.

The UI exposes Update for a password account. For automation or for working through a larger list of expired entries, the official API exposes:

POST /suite-api/api/fleet-management/password-management/accounts/query
PUT /suite-api/api/fleet-management/password-management/accounts/{passwordAccountKey}/password

The update workflow requires both the current password and the new password. In the API these fields are currentPassword and newPassword; in the UI they are entered through the Update action. Either way, this is a rotation workflow. If a password was already changed directly on the appliance, you need to provide that current real password and rotate once more through Fleet so the appliance password, the vault, and the expiration metadata are back in sync.

References:

Get a VCF Operations API Token

The VCF Operations API supports OpsToken authorization. Set the VCF Operations API host first:

OPS="https://<vcf-operations-fqdn>"

Note: These examples use curl -k, which skips TLS certificate verification — common in labs where the appliance still has its self-signed certificate. With -k you are sending credentials to whatever answers on that FQDN. In production, drop -k and trust the VCF Operations CA instead, for example curl --cacert /path/to/vcf-ops-ca.pem ..., or add the CA to the system trust store.

Use an account that is authorized to call the Fleet password management APIs. Avoid echoing the password back to the terminal. The read -rp / read -rsp prompts in this post use bash syntax; if your interactive shell is zsh, either start bash first or use the zsh form, for example read -rs 'VCFOPS_PASSWORD?VCF Operations password: '.

Acquire a token:

read -rp "VCF Operations username: " VCFOPS_USERNAME
read -rsp "VCF Operations password: " VCFOPS_PASSWORD; echo

TOKEN=$(jq -nc \
 --arg u "$VCFOPS_USERNAME" \
 --arg p "$VCFOPS_PASSWORD" \
 '{username:$u,password:$p}' |
 curl -sk -H 'Content-Type: application/json' -d @- \
 "$OPS/suite-api/api/auth/token/acquire" | jq -r .token)

unset VCFOPS_PASSWORD
[ -n "$TOKEN" ] && [ "$TOKEN" != "null" ] || echo "Token acquisition failed" >&2

Test the token:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 "$OPS/suite-api/api/versions/current" | jq .

Query Expired Password Accounts

Now query the expired password entries:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{"status":"EXPIRED"}' \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=200" \
 | tee /tmp/expired-password-accounts.json | jq .

For a cleaner table:

jq -r '.vcfPasswordAccounts[] |
 [.appliance, .applianceFqdn, .userName, .credentialType, .expiryDate, .passwordAccountKey] |
 @tsv' /tmp/expired-password-accounts.json

I usually sort the output into a tracking table like this:

Appliance	FQDN	User	Credential Type	Action
VCF_OPERATIONS	`<vcf-operations-fqdn>`	root	SSH	Update in UI or rotate through API
SDDC_MANAGER	`<sddc-manager-fqdn>`	root	SSH	Confirm current password, then rotate
VCF_SERVICES_RUNTIME	`<vcf-services-runtime-fqdn>`	vmware-system-user	SSH	Confirm ownership before rotating

Do not blindly rotate everything at once. Some accounts may be service users, some may belong to appliances that are temporarily unreachable, and some may be stale entries for components that no longer exist. Also note that domain-level credentials such as ESXi, vCenter, and NSX are owned by SDDC Manager rather than the Fleet vault — those rotate through the SDDC Manager /v1/credentials API instead.

Update an Expired Password Account

For a single account, you can do this from the VCF Operations UI by selecting the expired password account and using Update. Enter the current working password and the new final password.

For repeatable remediation, use the API equivalent below.

One important CLI note: do not paste passwords with special characters directly into an inline curl or shell variable assignment. Passwords that contain an exclamation mark can be changed or rejected by the shell before they ever reach the API because ! may trigger history expansion in interactive shells.

If you are using Bash, disable history expansion for the session:

set +H

If you are using Zsh, disable bang history expansion:

unsetopt BANG_HIST

The safer pattern is to prompt for the password and let jq build the JSON payload.

Pick one expired entry and export the key:

KEY="<passwordAccountKey>"

Run the update call. The currentPassword must be the password that works on the target right now. The newPassword is the new final password you want Fleet to set and store.

read -rsp "Current password: " CUR; echo
read -rsp "New password: " NEW; echo

jq -nc --arg c "$CUR" --arg n "$NEW" \
 '{currentPassword:$c,newPassword:$n}' |
curl -sk -X PUT \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d @- \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/${KEY}/password" \
 | tee /tmp/pwupdate-task.json | jq .

unset CUR NEW

The response should create a password update workflow request. Track the task in VCF Operations, or use the returned request ID if your environment exposes the workflow request endpoint.

After the task completes, query the account again. Filter by the FQDN of the appliance you just updated — it is in your tracking table or /tmp/expired-password-accounts.json — so the result does not depend on how many password accounts the fleet has:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{"applianceFqdn":"<appliance-fqdn>"}' \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=200" \
 | jq --arg KEY "$KEY" '.vcfPasswordAccounts[] | select(.passwordAccountKey==$KEY)'

You want to see the account move out of EXPIRED and show a new expiryDate.

If the Password Was Already Changed Outside Fleet

This was the awkward case in my environment.

If the password has already been changed manually on the appliance, the Fleet vault may still have the old value. Since 9.1 does not provide a documented vault-only update, use the actual current appliance password as the current password and rotate once more to a new final password through the UI Update action or the API.

That gives Fleet a successful workflow and lets it update:

The appliance password
The stored credential
The expiration metadata
The password account status

If you want to keep the exact same password that was already set manually, that is where the product gap shows up. The documented workflow wants a change from current to new.

If the Current Password Is Unknown

If nobody knows the current password, Fleet cannot update it because the workflow requires the current password.

Use the native recovery method for that appliance first. For example:

Open the VM console.
Reset the account to a temporary known password.
Confirm SSH or console login works.
Immediately rotate from temporary to final through the UI Update action or the Fleet API.

Do not leave the temporary password as the final state. The point is to re-enter the supported Fleet workflow as quickly as possible.

If an Entry Looks Stale

Before rotating, verify that the appliance still exists and belongs to the environment:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{"applianceFqdn":"<appliance-fqdn>"}' \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=50" \
 | jq .

Also check DNS and reachability from the shell where you are running the remediation:

getent hosts <appliance-fqdn>
curl -k -I https://<appliance-fqdn>

If the component has been removed but the password entry remains, do not delete database rows manually. Open a support case or use the supported inventory cleanup path for that component.

Verification Checklist

After the expired entries are addressed, verify the following:

The expired account query returns fewer entries, ideally zero for the scope you remediated.
Updated accounts show a refreshed expiryDate.
The password update workflow completed successfully.
Direct login to the target appliance works with the new password.

Query all remaining expired entries:

curl -sk \
 -H "Authorization: OpsToken $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{"status":"EXPIRED"}' \
 "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=200" \
 | jq '.vcfPasswordAccounts'

Finally, clean up the working files, since they contain fleet inventory details such as appliance FQDNs, usernames, and password account keys:

rm -f /tmp/expired-password-accounts.json /tmp/pwupdate-task.json

Final Notes

The main lesson is that VCF 9.1 password management needs to be treated as an update or rotation workflow. The password account state is exposed through VCF Operations Fleet Password Management, and expired entries need to be handled through the UI Update action or the documented account query and password update API calls. This is the same rotate-only philosophy the platform has had since the vRSLCM Locker password management days — the tooling changed, the workflow shape did not.

For this kind of issue, I would avoid two shortcuts:

Do not use VCF 9.0 UI guidance for VCF 9.1.
Do not update Fleet LCM database rows by hand.

The reliable path is to update expired accounts through the UI or API and verify the resulting password account state.

Upgrading VCF 9.1 to 9.1.0.0300 Step by Step

Cosmin Trif — Sun, 05 Jul 2026 17:30:00 +0000

The 9.1.0.0300 patch train for VMware Cloud Foundation 9.1 is applied through a few different lifecycle views. In my lab, the update covered the Fleet Lifecycle component, VCF Operations, SDDC Lifecycle, and SDDC Manager.

Before starting, review the official Broadcom release notes:

Broadcom describes these patch releases as time-sensitive fixes that can be applied between major, minor, and maintenance releases. That also means every component may not move at the same time, so check what is actually offered in your environment instead of assuming the whole VCF bill of materials changes together.

Versions in This Upgrade

Here is what was upgraded in this run.

Component	From	To
Fleet Lifecycle	9.1.0.0.25371109	9.1.0.0300.25536195
VCF Operations	9.1.0.0200.25500350	9.1.0.0300.25531073
SDDC Lifecycle	9.1.0.0200.25495903	9.1.0.0300.25536194
SDDC Manager	9.1.0.0200 build 25496021	9.1.0.0300 build 25536191

As always, take backups or snapshots that match your normal rollback procedure, confirm that the Broadcom depot is reachable, and make sure no other lifecycle operations are running before you start.

Start in VCF Management

Log in to VCF Operations and go to Build -> Lifecycle -> VCF Management -> Upgrade. If the lifecycle metadata has not synced recently, use Sync before checking for available target versions.

The Fleet Lifecycle card is the first item I updated.

Click the target version selector and choose the 9.1.0.0300 target.

After selecting the target, the Fleet Lifecycle card should show the upgrade path to 9.1.0.0300.25536195. Click Upgrade.

Upgrade Fleet Lifecycle

Once the task starts, the Fleet Lifecycle card changes to Upgrade in progress.

Open Upgrade details to monitor the subtask list. In my run, the task staged binaries, ran component stage prechecks, and initialized the VCF component stage.

When this finishes, the Fleet Lifecycle card should show the new current version.

Set the VCF Management Target Version

Next, set the management component target version. In the Target Version 9.1.0.* section, click Change Target Version.

In my lab I used the wildcard 9.1.0.* global version. Select the patch release for VCF Operations and any other management components that might need to be upgraded.

Click Set Version and then run the prechecks. The page should show the components that are ready for upgrade.

Upgrade VCF Operations

Start with VCF Operations. The row in my environment moved from 9.1.0.0200.25500350 to 9.1.0.0300.25531073.

Click Upgrade for the VCF Operations row.

Use Upgrade details to monitor the precheck and update process. In this run, the task started an appliance precheck, initiated the VCF Ops precheck, and then began applying the system update.

When the VCF Operations update completes, the Tasks tab should show the refresh and upgrade tasks as completed.

Upgrade SDDC Lifecycle

After VCF Operations is done, return to the upgrade list. SDDC Lifecycle should still show as ready if you did not upgrade everything at once.

In my environment, SDDC Lifecycle moved from 9.1.0.0200.25495903 to 9.1.0.0300.25536194.

Click Upgrade for SDDC Lifecycle and monitor the upgrade details.

The details panel should show the SDDC Lifecycle plugin being staged in the VCF services runtime.

Wait for the task to complete before moving on to the SDDC Manager upgrade plan.

Download the SDDC Manager Patch Binary

Now go to Build -> Lifecycle -> VCF Instances -> instance-a -> Binary Management.

Set VCF Version to 9.1, select Patch Binaries, and confirm the SDDC Manager 9.1.0.0300 binary is downloaded. In this run the binary was released on Jun 29, 2026 and was 2.41 GB.

If it is not downloaded yet, select the binary and click Download.

Edit the Workload Domain Upgrade Plan

Go to the management domain under the VCF instance, then open the Upgrades tab. In this example the management domain is mgmt-a.

Click Edit Plan.

In Plan Component Upgrade, select SDDC Manager and choose target version 9.1.0.0300. In my run, ESX, NSX, and vCenter were already on 9.1.0.0100 and were not selected for this 0300 SDDC Manager patch plan.

Click Next and review the plan. The wizard notes that SDDC Manager is upgraded first so it can continue orchestrating the remaining component lifecycle work.

Click Submit Plan.

Start or Schedule the SDDC Manager Upgrade

The upgrade sequence should now show the SDDC Manager 9.1.0.0300 upgrade as the available step. You can run it immediately with Upgrade Now or schedule it for a maintenance window.

After starting the upgrade, monitor the VMware Cloud Foundation Upgrade Status page. You can also click View Upgrade Activity for deeper task progress.

In my run, the visible steps included:

Run VCF Prerequisite
Setup Common Appliance Platform
Validate Services Before Upgrade
Remove Packages Pre Upgrade

Do not refresh services or reboot appliances manually while the upgrade is running. Let the lifecycle workflow complete.

Validate the Upgrade

After the workflow completes, return to the management domain and open Component Versions.

The SDDC Manager row should show the same current and target version, with status On Target.

I also recommend checking:

VCF Management -> Tasks for failed or partially completed tasks
VCF Operations health and adapters
SDDC Manager service health
Fleet password management for accounts showing EXPIRED — I covered the remediation in Fixing Expired Password Sync in VCF 9.1 Fleet Management
Workload domain upgrade status
Depot/binary cleanup after you are comfortable with the rollback window

If you keep older bundles around for too long, they can consume a lot of space. I covered bundle cleanup separately in Cleaning up VCF Bundles.

Exporting and Restoring vCenter 8.x Permissions Before Breaking Enhanced Linked Mode

Cosmin Trif — Tue, 30 Jun 2026 15:45:00 +0000

Breaking Enhanced Linked Mode is one of those changes where the technical command is not the hardest part. The hard part is making sure administrators, service accounts, and external products can still log in and do their jobs after the vCenters no longer share the same SSO domain.

In this post I will walk through a repeatable way to export and restore vCenter 8.x RBAC data before splitting Enhanced Linked Mode. The focus is on three permission layers:

Custom roles
Explicit object or inventory permissions
Global permissions

The short version is this: object permissions usually survive the split because they live in the vCenter database, but global permissions and SSO-domain objects need special handling. If you depend on global permissions, do not start the split until you have exported them and have a plan to recreate or import them afterward.

What Changes When You Split ELM

Enhanced Linked Mode lets multiple vCenter Server instances share one vCenter Single Sign-On domain. When you break that relationship, the vCenters stop sharing the same SSO namespace.

That matters because not all permissions are stored the same way.

Item	What to expect
Custom roles	Stored in the vCenter database, so they usually survive the split. Export them anyway and re-import if anything drifted.
Object permissions	Usually remain on the vCenter inventory objects, but the principals must still resolve after the split.
Global permissions	Must be exported and recreated/imported after the split.
Local SSO users/groups	Must be recreated if they are still needed.
AD/LDAP identity sources	Must be recreated or verified on the standalone vCenter.
Solution users/plugins	Usually need product-specific re-registration or repair.

If the environment is managed by VMware Cloud Foundation / SDDC Manager, stop and validate supportability first. Cross-domain repointing is not something I would treat as a casual vCenter-only operation in a VCF managed environment.

Download the Helper Scripts

I use three helper scripts for this workflow:

Script	Purpose
Export-VCenterRbac.ps1	Exports custom roles and explicit object permissions from a vCenter.
Convert-AuthzDoctorPermissions.ps1	Converts `authz-doctor` output into a clean CSV for global permissions.
Import-VCenterRbac.ps1	Imports roles, optional global permissions, and object permissions after the split.

The scripts assume PowerShell 7+ and PowerCLI:

Install-Module VMware.PowerCLI -Scope CurrentUser
Set-PowerCLIConfiguration -InvalidCertificateAction Warn -Confirm:$false

Run the import with -WhatIf first. This is especially important when restoring global permissions.

Before You Export Anything

Do the boring checks first. This is where most bad outcomes are avoided.

Confirm which vCenters are staying linked and which vCenters are being split.
Confirm you have working SSO administrator access.
Confirm you have root access to each VCSA so you can run authz-doctor.
Take a file-based backup of each vCenter.
Take powered-off snapshots of all ELM nodes at the same point in time if that is your rollback method.
If vCenter HA is enabled, remove it before the split.
Identify any local SSO users/groups that need to exist after the split.
Identify all AD/LDAP identity sources that need to be recreated.
Identify solution accounts for NSX, SRM, Aria, backup products, monitoring tools, and plugins.

I also like to create at least one explicit vCenter-root permission for a known AD admin group on each vCenter before the change. Do not rely only on global permissions for your break-glass path.

Step 1: Export Roles and Object Permissions

Create one export folder per vCenter. Do not mix them. The object paths and managed object references are specific to the vCenter they came from.

cd C:\temp\vcenter-elm-rbac

.\Export-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -OutDir .\exports\vcsa01

.\Export-VCenterRbac.ps1 `
 -Server vcsa02.example.com `
 -OutDir .\exports\vcsa02

.\Export-VCenterRbac.ps1 `
 -Server vcsa03.example.com `
 -OutDir .\exports\vcsa03

Each export folder should contain:

manifest.json
roles.json
object-permissions.json
object-permissions.csv
global-permissions.template.csv

roles.json is used for import. object-permissions.json is also used for import. The CSV is there so you can review the permissions quickly in Excel or another editor.

Open manifest.json and make sure the count looks reasonable:

Get-Content .\exports\vcsa01\manifest.json | ConvertFrom-Json

Step 2: Export Global Permissions with authz-doctor

Global permissions are the part I care about most in an ELM split. They are easy to forget because they live under Administration > Access Control > Global Permissions, not on the normal inventory tree.

The authz-doctor tool is preinstalled on recent vCenter builds, but not on all of them. If the path below does not exist on your appliance, download the RPM attached to Broadcom KB 369938 and install it with rpm -i --force first, which is the install command the KB uses.

On each vCenter appliance, SSH in as root and run:

/usr/lib/vmware-vpx/scripts/authz-doctor/authz-doctor.py permission_check > /tmp/authz-doctor-permissions.txt

Copy the file back to the matching export folder:

scp root@vcsa01.example.com:/tmp/authz-doctor-permissions.txt ./exports/vcsa01/

The output is a bordered ASCII table. The permission rows look like this:

+-----------------------------------+-------+---------+-----------+-----------+--------+
| Principal | Group | Role Id | Role Name | Propagate | Entity |
+-----------------------------------+-------+---------+-----------+-----------+--------+
| VSPHERE.LOCAL\vpxd-extension-xxxx | False | -1 | Admin | True | Global |
| DOMAIN\vSphere-Admins | True | -1 | Admin | True | Global |
+-----------------------------------+-------+---------+-----------+-----------+--------+

The converter detects permission rows by their six-column layout and skips the border and header lines, so you can feed it the raw permission_check output without any cleanup.

Then convert the global permission rows into CSV:

.\Convert-AuthzDoctorPermissions.ps1 `
 -InputFile .\exports\vcsa01\authz-doctor-permissions.txt `
 -OutputFile .\exports\vcsa01\global-permissions.csv `
 -GlobalOnly

Repeat this for each vCenter.

The resulting file should have this shape:

"Principal","IsGroup","RoleId","RoleName","Propagate","Entity"
"DOMAIN\vSphere-Admins","True","-1","Admin","True","Global"

Step 3: Review Principals Before the Split

Do not blindly import the CSV later. Review it first.

Pay special attention to:

administrator@vsphere.local or users from the old SSO domain
Local SSO groups that will not exist after the split
Solution users such as vpxd-*, vsphere-ui-*, NSX, SRM, backup, or monitoring accounts
AD groups whose domain name or identity source alias may change
Custom roles that exist on one vCenter but not another

If you are moving from a shared SSO domain to standalone SSO domains, a principal name that made sense before the split may not resolve afterward. The permission entry can exist, but it is useless if the identity source cannot resolve the user or group.

Step 4: Add Temporary Object-Level Admin Access

Before breaking ELM, add a direct permission at the root of each vCenter inventory for the admin group you will use after the split.

In the vSphere Client:

Select the vCenter object at the top of the inventory.
Go to Permissions.
Add your AD admin group.
Assign the appropriate role, usually Administrator for the migration window.
Enable propagation.

This gives you a local object permission that is independent of Global Permissions. It is your practical safety net.

Step 5: Break Enhanced Linked Mode

Follow the supported vCenter 8.x ELM split procedure for your environment. At a high level, the flow is:

Shut down the vCenter being split from the SSO domain.
From one of the remaining linked vCenters, unregister the powered-off node.
Power on the vCenter being split.
Repoint it to a standalone SSO domain.
Validate that the vCenter no longer shows the other linked vCenters.

Be aware that the unregister step (cmsso-util unregister) restarts all vCenter services on the node where you run it, so plan for a short vSphere Client outage on that remaining vCenter, not just on the one being split.

One extra step if your ELM group has only two nodes: after the repoint, Broadcom KB 370062 says to also run cmsso-util unregister on the remaining node, so the stale partner information for the node that left is released.

The key point for this post is timing: do not import permissions until the standalone vCenter has its identity source and SSO domain in the final state.

Step 6: Recreate Identity Sources and Local SSO Objects

After the vCenter is standalone, log in with the new SSO administrator and recreate or verify:

AD/LDAP identity sources
Local SSO users
Local SSO groups
Any SSO group nesting used by the exported permissions
Certificates or trust needed for LDAPS

Also check Global Permissions for stale entries that still reference the previous SSO domain and clean them up. The split KB links a dedicated procedure for removing stale global permissions after a domain repoint, and the import is much easier to reason about when the old domain’s dead entries are gone first.

Before importing permissions, test that the principal resolves. For example, search for the AD group in the vSphere Client permission picker. If the picker cannot resolve it, the import will not give you meaningful access.

Step 7: Dry-Run the Import

Start with a dry run:

.\Import-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -InDir .\exports\vcsa01 `
 -GlobalPermissionsCsv .\exports\vcsa01\global-permissions.csv `
 -UsePrivateMobForGlobalPermissions `
 -WhatIf

The import order is:

Create or update custom roles.
Import global permissions if a CSV is provided.
Import object permissions.

The global permission import uses the vCenter MOB endpoint for AuthorizationService.AddGlobalAccessControlList. That is why the switch is intentionally named -UsePrivateMobForGlobalPermissions. I want it to be obvious that this is different from the normal object permission path through AuthorizationManager.

The MOB route is specifically the vCenter 8.x answer. vSphere 9.0 added a supported REST API for global permissions (POST /api/vcenter/authorization/permissions with a GlobalAcl DynamicID), so if you are doing this on 9.x, use that instead.

If you only want to restore roles and global permissions first, skip object permissions:

.\Import-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -InDir .\exports\vcsa01 `
 -GlobalPermissionsCsv .\exports\vcsa01\global-permissions.csv `
 -UsePrivateMobForGlobalPermissions `
 -SkipObjectPermissions `
 -WhatIf

Step 8: Run the Import

Once the dry run looks right, run it without -WhatIf:

.\Import-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -InDir .\exports\vcsa01 `
 -GlobalPermissionsCsv .\exports\vcsa01\global-permissions.csv `
 -UsePrivateMobForGlobalPermissions

If you changed principal names during the split, edit global-permissions.csv before running the import. For example, change:

OLDSSO\vSphere Admins

to:

CORP\vSphere Admins

or whatever the correct identity source now exposes.

Step 9: Verify Access

Do not stop after the script completes. Verify the permission model from both the API side and the user side.

Run authz-doctor again:

/usr/lib/vmware-vpx/scripts/authz-doctor/authz-doctor.py permission_check > /tmp/authz-doctor-permissions-after.txt

Check Global Permissions in the vSphere Client:

Administration > Access Control > Global Permissions

Then test real logins:

Log out of the SSO administrator account.
Log in as a user from the restored AD admin group.
Confirm you can see the expected inventory.
Confirm you can perform a low-risk administrative action.
Test at least one non-admin role if you have delegated access.

For object permissions, you can also re-export and compare the count:

.\Export-VCenterRbac.ps1 `
 -Server vcsa01.example.com `
 -OutDir .\exports\vcsa01-after

Compare-Object `
 (Import-Csv .\exports\vcsa01\object-permissions.csv) `
 (Import-Csv .\exports\vcsa01-after\object-permissions.csv) `
 -Property Principal,RoleName,Propagate,EntityPath

Some differences are expected if you intentionally changed principals or cleaned up stale entries. Unexpected missing admin groups are not expected.

Common Gotchas

The import cannot fix an identity source that does not exist. Recreate AD/LDAP first.

Custom role IDs can change. The import script matches role names first because role IDs are not always stable across systems.

Object permissions can point to objects that no longer exist. The script tries the original managed object reference first and then the inventory path. If neither resolves, review the CSV and decide whether the permission is still needed.

Solution users are not normal human permissions. If NSX, SRM, Aria, a backup product, or a monitoring tool owns a registration, use that product’s supported reconnect or repair procedure instead of blindly restoring stale solution-user permissions.

Global permissions are broad. Review them carefully before bringing them back into a standalone vCenter.

References

The goal is not just to preserve a list of permissions. The goal is to preserve access that still resolves, still matches the right role, and still works after each vCenter stands on its own.

Clearing a Full vSAN Trace Ramdisk Across ESXi Hosts in Parallel

Cosmin Trif — Wed, 24 Jun 2026 10:30:00 +0000

While reviewing an SOS support bundle from a VMware Cloud Foundation environment, I noticed every ESXi host in the management cluster was logging the same warning, over and over, right up to the moment the bundle was collected:

[vob.visorfs.ramdisk.full] Cannot extend visorfs file
/vsantraces/vsantracesLSOMVerbose--...zst because its ramdisk
(vsantraceFailover) is full.

These are -INFO level VOB events, not errors, and they do not touch your data or VMs. But when the same symptom appears on all hosts at once, fires continuously, and never clears on its own, it is worth understanding what is actually happening and fixing it cluster-wide rather than logging into each host by hand. This post walks through the diagnosis and a small Bash script that queries, fixes, and reclaims space on every host in parallel.

Understanding the vSAN Trace Ramdisk

vSAN writes diagnostic traces (DOM, LSOM, CLOM, PLOG, and others) to /vsantraces, an in-memory location backed by a ramdisk. Under normal operation these traces rotate into compressed .zst archives and the ramdisk stays well under capacity. ESXi also keeps a secondary vsantraceFailover ramdisk that catches trace writes when the primary cannot be extended.

The warning above means the failover ramdisk has hit 100% and the trace daemon can no longer write new trace data. Because tracing is purely diagnostic, the cluster keeps running normally — but you lose trace history, the logs fill with noise, and a genuinely useful troubleshooting tool is effectively offline.

A few things are easy to get wrong here, so it is worth being precise:

This is not caused by verbose tracing being left on. On a healthy vSAN ESA cluster, LSOMVerbose is enabled by default. Confirm the configured level before assuming someone changed it.
A vsantraced restart re-initializes the daemon but does not purge files already sitting on the failover ramdisk. If the ramdisk is full, restarting alone will often leave it full.
The real reclaim comes from removing the old rotated .zst archives, which are the bulk of the consumed space.

Diagnosing Before You Touch Anything

The first job is to confirm the state on every host with read-only commands. Three pieces of information tell you almost everything:

# What trace level is actually configured?
esxcli vsan trace get

# Is the failover ramdisk full?
vdf -h | grep -i vsantrace

# What is consuming the space?
ls -lhS /vsantraces/ | head

If vdf shows a line like vsantraceFailover 300M 300M 0B 100%, that is your smoking gun. The ls output will typically show several large vsantraces--*.zst archives (often the configured max file size each) as the dominant consumers.

A note on SSH: ESXi SSH is disabled by default, and a recent SOS bundle will reflect that. Enable it per host first (vCenter > Host > Configure > Services > SSH > Start) before running anything below, and disable it again when you are done. If you would rather keep SSH off entirely, the same commands can be issued through vCenter with PowerCLI Get-EsxCli.

Querying Every Host at Once

Logging into hosts one at a time does not scale, and the whole point is that this condition tends to hit the entire cluster together. The script below fans out over SSH: it launches one background job per host, waits for all of them, and writes each host’s output to its own file. A per-connection timeout keeps a single unreachable host from hanging the whole run.

#!/usr/bin/env bash
#
# query-vsan-traces.sh
# Query (and optionally fix) all ESXi hosts in parallel for the
# vSAN "vsantraceFailover ramdisk full" condition.
#
# Usage:
# ./query-vsan-traces.sh # QUERY only (read-only, default)
# ./query-vsan-traces.sh --fix # QUERY, then restart vsantraced
# ./query-vsan-traces.sh --clean # QUERY, then delete OLD .zst archives
# ./query-vsan-traces.sh --clean host1 # restrict the action to specific hosts
# ./query-vsan-traces.sh --clean --yes # skip the confirmation prompt
# SSH_USER=root ./query-vsan-traces.sh # override the SSH user (default: root)

set -u

# --- config -----------------------------------------------------------------
HOSTS_DEFAULT=(
 esx-01.example.lab
 esx-02.example.lab
 esx-03.example.lab
 esx-04.example.lab
)
SSH_USER="${SSH_USER:-root}"
# Per-connection timeouts so one unreachable host can't hang the whole run.
SSH_OPTS=(-o ConnectTimeout=10 -o BatchMode=yes -o StrictHostKeyChecking=accept-new)
OUTDIR="vsan-trace-report-$(date +%Y%m%d-%H%M%S)"
# ---------------------------------------------------------------------------

# --- arg parsing ------------------------------------------------------------
DO_FIX=0
DO_CLEAN=0
ASSUME_YES=0
HOSTS=()
for arg in "$@"; do
 case "$arg" in
 --fix) DO_FIX=1 ;;
 --clean) DO_CLEAN=1 ;;
 --yes|-y) ASSUME_YES=1 ;;
 -*) echo "Unknown option: $arg" >&2; exit 2 ;;
 *) HOSTS+=("$arg") ;;
 esac
done
[ ${#HOSTS[@]} -eq 0 ] && HOSTS=("${HOSTS_DEFAULT[@]}")

mkdir -p "$OUTDIR"

# The diagnostic commands run on each host. Keep them read-only.
read -r -d '' QUERY_CMDS <<'EOF'
echo "===== $(hostname) ====="
echo "--- esxcli vsan trace get ---"
esxcli vsan trace get 2>&1
echo "--- ramdisk usage (vsantrace) ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
echo "--- top trace files by size ---"
ls -lhS /vsantraces/ 2>/dev/null | head -n 15 || echo "(/vsantraces not present)"
echo "--- recent ramdisk-full VOBs (last 20) ---"
grep "vob.visorfs.ramdisk.full" /var/log/vobd.log 2>/dev/null | tail -n 20 || echo "(none)"
EOF

# Remediation A: restart vsantraced and show before/after ramdisk usage.
read -r -d '' FIX_CMDS <<'EOF'
echo "===== $(hostname) ====="
echo "--- BEFORE: ramdisk usage ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
echo "--- restarting vsantraced ---"
/etc/init.d/vsantraced restart 2>&1
sleep 3
echo "--- AFTER: ramdisk usage ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
EOF

# Remediation B: delete OLD .zst archives, keeping the newest 3 per host.
# Active (non-.zst) trace files are never touched.
read -r -d '' CLEAN_CMDS <<'EOF'
echo "===== $(hostname) ====="
echo "--- BEFORE: ramdisk usage ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
echo "--- deleting all but the newest 3 .zst archives ---"
ls -t /vsantraces/*.zst 2>/dev/null | tail -n +4 | while read -r f; do
 echo "rm $f"
 rm -f "$f"
done
echo "--- AFTER: ramdisk usage ---"
vdf -h 2>/dev/null | grep -i vsantrace || echo "(no vsantrace ramdisk line)"
EOF

run_on_host() {
 local host="$1" cmds="$2" suffix="$3"
 local out="$OUTDIR/${host}${suffix}.txt"
 if ssh "${SSH_OPTS[@]}" "${SSH_USER}@${host}" "$cmds" >"$out" 2>&1; then
 echo "OK $host -> $out"
 else
 echo "FAIL $host (see $out for error)"
 fi
}

fan_out() {
 local cmds="$1" suffix="$2"
 local pids=()
 for h in "${HOSTS[@]}"; do
 run_on_host "$h" "$cmds" "$suffix" &
 pids+=("$!")
 done
 wait "${pids[@]}"
}

confirm() {
 [ "$ASSUME_YES" -eq 1 ] && return 0
 printf "Proceed on all %d hosts? [y/N] " "${#HOSTS[@]}"
 read -r reply
 case "$reply" in y|Y|yes|YES) return 0 ;; *) echo "Aborted."; exit 0 ;; esac
}

# --- 1. QUERY (always) ------------------------------------------------------
echo "Querying ${#HOSTS[@]} hosts in parallel as user '$SSH_USER'..."
echo "Output dir: $OUTDIR"
echo
fan_out "$QUERY_CMDS" ""
echo
echo "Hosts with the failover ramdisk at 100%:"
grep -l "vsantraceFailover.*100%" "$OUTDIR"/*.txt 2>/dev/null \
 || echo " (none — ramdisks have headroom)"

# --- 2. FIX (only with --fix) -----------------------------------------------
if [ "$DO_FIX" -eq 1 ]; then
 echo; echo "FIX MODE: restart 'vsantraced' on ${#HOSTS[@]} host(s)."
 echo "Only affects diagnostic tracing — no vSAN data, VMs, or I/O touched."
 confirm
 fan_out "$FIX_CMDS" "-fix"
fi

# --- 3. CLEAN (only with --clean) -------------------------------------------
if [ "$DO_CLEAN" -eq 1 ]; then
 echo; echo "CLEAN MODE: delete old .zst archives on ${#HOSTS[@]} host(s)."
 echo "Keeps the newest 3 archives per host; active trace files untouched."
 confirm
 fan_out "$CLEAN_CMDS" "-clean"
fi

Step 1: Run the Query

With your hostnames filled into HOSTS_DEFAULT (or passed on the command line), run it with no arguments first. This is read-only and safe:

./query-vsan-traces.sh

You get one output file per host plus a summary line listing exactly which hosts are sitting at 100%. Confirm from the esxcli vsan trace get output that the trace level is the default before going further — if it is, you know this is a stuck-ramdisk problem, not a misconfiguration.

Step 2: Try the Restart

The lightest-touch remediation is to restart the trace daemon, which re-initializes the trace directories:

./query-vsan-traces.sh --fix

The script prints before/after vdf output for each host. In my case the restart completed cleanly on all hosts but the failover ramdisk stayed at 100% — the daemon came back, but the files already on the ramdisk were not purged. That is the expected outcome when the ramdisk is already full, and it is exactly why the script does not stop here.

Step 3: Reclaim the Space

The reclaim step deletes the old rotated .zst archives, which are what actually fills the ramdisk. The script keeps the newest three archives per host so you do not lose all recent history, and it never touches the active (non-.zst) trace files:

./query-vsan-traces.sh --clean

Each remediation mode prompts for confirmation once, listing the hosts it is about to act on, and you can skip the prompt with --yes in an automation context. After the cleanup, the AFTER vdf line should show the failover ramdisk back under capacity, and the ramdisk.full VOBs should stop appearing within a minute or two.

Why This Order Matters

It is tempting to jump straight to deleting files, but running the query first buys you two things. You confirm the trace configuration is actually default (ruling out a real misconfiguration), and you capture a record of the pre-change state in the per-host output files. The restart is offered before the cleanup simply because it is the lower-impact action; when it does not free space, the cleanup is the definitive fix.

If a host is still at 100% after a cleanup, that points to something actively re-filling the ramdisk faster than rotation can drain it — a stuck writer, or a persistent trace target that is unwritable. At that point you are past the mechanical fixes and it is worth opening a support request rather than looping on the same commands.

Conclusion

The “vsantraceFailover ramdisk full” warning looks alarming because it repeats endlessly across every host, but it is a contained, diagnostic-only condition with a safe, repeatable fix. The key is to treat the whole cluster as a unit: query every host in parallel, confirm the configuration is default, attempt the cheap restart, and reclaim space by clearing old trace archives when the restart is not enough. A small Bash wrapper turns what would be a tedious host-by-host chore into three commands you can run from your workstation, with a written record of each host’s state along the way.

Fixing "Error attempting Upgrade Database Schema" When Upgrading vCenter 8.0 U3x to 9.1.0.0

Cosmin Trif — Tue, 16 Jun 2026 12:45:00 +0000

If you’re moving vCenter Server from 8.0 U3x to 9.1.0.0 — typically as part of a VCF 9.1 deployment or a VVF 9.1 upgrade — the run can fail partway through with:

Error attempting Upgrade Database Schema
Please check vcintegrity migration logs for details.

This is one of those failures that looks scary in the UI but has a clean, well-understood root cause: a handful of orphaned rows in the vcIntegrity database that trip a new foreign-key constraint introduced in 9.1. Below is the whole story — what the error means, how to read the logs, why it happens, and how to clear it so the upgrade completes. It’s the same shape of problem I wrote about in fixing PostgreSQL on Fleet Management 9.0.1: a schema change colliding with pre-existing data.

This is documented in Broadcom KB 442223. The supported fix uses a script attached to that KB. I’ll walk through the official steps first, then share an unofficial helper of my own for the curious — clearly marked, and not a replacement for the KB script.

The symptom

The upgrade proceeds normally until the vcIntegrity import phase, then halts with the generic schema error shown above. The UI doesn’t tell you what in the schema failed — it just points you at the vcIntegrity migration logs. That’s your cue to SSH in and read the actual exception.

Reading the upgrade log

On the source vCenter, open the vcIntegrity import log:

/var/log/vmware/upgrade/Import_com.vmware.vcIntegrity_####_##_##_##_##.log

The meaningful lines look like this (timestamps and IDs trimmed):

DEBUG vmware-vci-vciInstallUtils [VdbODBCError::VdbODBCError] SQLGetDiagRec
 SQLState='23503' NativeError=1
 MessageText='ERROR: insert or update on table "pm_recommendation_spec"
 violates foreign key constraint "fk_pm_recommendation_spec"
 DETAIL: Key (entity_id, spec_id)=(domain-##, -1000) is not present in
 table "pm_recommendation_info".'

ERROR vmware-vci-vciInstallUtils [installerRunDBCommand 626] [DATABASE]
 Vdb::RunScript, SQL Exec Error: "ODBC error: (23503) - ERROR: insert or
 update on table "pm_recommendation_spec" violates foreign key constraint
 "fk_pm_recommendation_spec" ... " is returned when executing SQL statement
 "ALTER TABLE IF EXISTS PM_RECOMMENDATION_SPEC
 ADD CONSTRAINT FK_PM_RECOMMENDATION_SPEC
 FOREIGN KEY (ENTITY_ID, SPEC_ID)
 REFERENCES PM_RECOMMENDATION_INFO (ENTITY_ID, SPEC_ID)
 ON DELETE CASCADE"

ERROR __main__ Upgrade Phase 'vcIntegrity:Import' failed.
 Exception: Error attempting Upgrade Database Schema
upgrade_errors.PermanentError: Error attempting Upgrade Database Schema

Three things are worth pulling out of that wall of text:

The SQLState is 23503, which in PostgreSQL is foreign_key_violation — not a connectivity, permissions, or disk problem. The database is doing exactly what it was told to do and refusing bad data.

The failing statement is an ALTER TABLE ... ADD CONSTRAINT, not an insert from your workload. The migration is trying to add the new constraint FK_PM_RECOMMENDATION_SPEC to an existing table, and Postgres validates the rows already in that table before it will accept the constraint.

The offending key is (entity_id, spec_id) = (domain-##, -1000). There’s a row in pm_recommendation_spec whose (entity_id, spec_id) pair has no matching parent row in pm_recommendation_info. The -1000 is a sentinel/placeholder value, and domain-## is a managed-object reference — classic leftovers from objects that were removed or never fully written.

Root cause

The pm_recommendation_* tables live in the vcIntegrity database — the component behind vSphere Lifecycle Manager / Update Planner. They store lifecycle “recommendation” data: pm_recommendation_info holds the parent records, and pm_recommendation_spec holds child spec rows that are supposed to reference a parent.

Over the life of an 8.0 environment, it’s possible to end up with child spec rows whose parent info row is gone (or was never created) — for example after domains/clusters are removed, or from older code paths that didn’t clean up specs. In 8.0 there was no enforced relationship between the two tables, so these orphans sat there harmlessly.

vCenter 9.1.0.0 hardens the schema. The migration adds a composite foreign key:

ALTER TABLE PM_RECOMMENDATION_SPEC
 ADD CONSTRAINT FK_PM_RECOMMENDATION_SPEC
 FOREIGN KEY (ENTITY_ID, SPEC_ID)
 REFERENCES PM_RECOMMENDATION_INFO (ENTITY_ID, SPEC_ID)
 ON DELETE CASCADE;

When PostgreSQL adds a foreign key, it validates every existing row in the child table against the parent table. Any orphan — a (entity_id, spec_id) pair with no parent — violates the new constraint, so the engine aborts the ALTER TABLE with error 23503. Because the migration treats this as a PermanentError, the whole vcIntegrity import phase fails and the upgrade stops.

In short: 9.1 is correct to demand referential integrity; your 8.0 data just doesn’t satisfy it yet. The fix is to remove the orphaned spec rows so the constraint can be created cleanly.

Broadcom notes that a fix to prevent this will be included in a future VCF release. Until then, the workaround below is the path forward.

The official fix (KB 442223 workaround)

Do this on the source vCenter appliance, before retrying the upgrade.

1. Snapshot the vCenter appliance

Take a snapshot of the source vCenter VM first. Editing the embedded database is low-risk when done correctly, but a snapshot is your guaranteed rollback. (If your environment enforces a snapshot-age policy, see changing the old-snapshot restriction — just remember to remove this snapshot once the upgrade succeeds.)

2. Download the cleanup script from the KB

Download cleanup_rec_orphans.py from the Attachments section of KB 442223. This is the supported script — use it for production work.

3. Copy the script to the appliance

Place it in /tmp on the vCenter server. You can use an SCP client such as WinSCP. If WinSCP fails with a “Received too large … SFTP packet” error, follow KB 326317 (set the shell to bash for the session, or use the SCP fallback).

4. SSH in as root

5. Dry-run audit (read-only)

Run the script with no flags first. It reports orphans without changing anything:

su updatemgr -s /bin/bash -c "python3 /tmp/cleanup_rec_orphans.py"

The su updatemgr matters: the updatemgr account owns the vcIntegrity database connection, so the script authenticates correctly without you handling credentials.

6. Execute the cleanup

Once you’ve reviewed the dry-run output, run it again with --execute to delete the orphaned rows:

su updatemgr -s /bin/bash -c "python3 /tmp/cleanup_rec_orphans.py --execute"

7. Retry the upgrade

Re-run the vCenter upgrade. With the orphans gone, the ALTER TABLE ... ADD CONSTRAINT validates cleanly and the vcIntegrity import phase completes.

What the script actually does (the SQL behind it)

You don’t need to know this to run the KB script, but it helps to understand why it’s safe. The job is purely to find child rows with no parent and remove them. The detection is a straightforward anti-join:

-- Orphans: spec rows whose (entity_id, spec_id) has no parent info row
SELECT s.entity_id, s.spec_id
FROM pm_recommendation_spec s
LEFT JOIN pm_recommendation_info i
 ON i.entity_id = s.entity_id
 AND i.spec_id = s.spec_id
WHERE i.entity_id IS NULL
ORDER BY s.entity_id, s.spec_id;

And the cleanup is the matching delete:

DELETE FROM pm_recommendation_spec s
WHERE NOT EXISTS (
 SELECT 1
 FROM pm_recommendation_info i
 WHERE i.entity_id = s.entity_id
 AND i.spec_id = s.spec_id
);

That’s it. You’re deleting only spec rows that already point at nothing — exactly the rows the new constraint would reject. No valid recommendation data is touched.

Optional: an unofficial audit/cleanup helper

Use the KB script for anything you care about. The Bash helper below is a community convenience I wrote to mirror the same logic for quick auditing in labs. It is unofficial and unsupported, runs read-only by default, and should only be pointed at a vCenter you’ve snapshotted. If in doubt, stop and use cleanup_rec_orphans.py from the KB.

It does three things: locate the database that actually owns the table (so it doesn’t matter what the DB is named in your build), report the orphan count and the offending pairs, and — only with --execute — delete them inside a transaction.

#!/usr/bin/env bash
# audit_rec_orphans.sh — UNOFFICIAL community helper (KB 442223)
#
# Mirrors the intent of Broadcom's cleanup_rec_orphans.py: find/remove
# orphaned rows in pm_recommendation_spec that have no parent in
# pm_recommendation_info and therefore block the FK_PM_RECOMMENDATION_SPEC
# constraint added by the vCenter 9.1.0.0 vcIntegrity schema upgrade.
#
# DRY-RUN by default. Pass --execute to delete.
# ALWAYS snapshot the vCenter appliance first.
# Prefer the official KB script for supported/production environments.
#
# Run it the same way the KB runs its script (as the vcIntegrity DB owner):
# su updatemgr -s /bin/bash -c "/tmp/audit_rec_orphans.sh" # audit
# su updatemgr -s /bin/bash -c "/tmp/audit_rec_orphans.sh --execute" # delete
set -euo pipefail

EXECUTE=0
[ "${1:-}" = "--execute" ] && EXECUTE=1

if [ "$(id -un)" != "updatemgr" ]; then
 echo "[!] Not running as 'updatemgr' — DB auth will likely fail."
 echo " Re-run: su updatemgr -s /bin/bash -c \"$0 ${1:-}\""
 exit 1
fi

# Prefer the embedded vPostgres client if present
PSQL="/opt/vmware/vpostgres/current/bin/psql"
[ -x "$PSQL" ] || PSQL="$(command -v psql)"

# Discover which database owns pm_recommendation_spec (override with VCI_DB=...)
VCI_DB="${VCI_DB:-}"
if [ -z "$VCI_DB" ]; then
 for db in $("$PSQL" -Atqc \
 "SELECT datname FROM pg_database WHERE datistemplate=false" 2>/dev/null); do
 if [ -n "$("$PSQL" -Atqd "$db" \
 -c "SELECT to_regclass('pm_recommendation_spec')" 2>/dev/null)" ]; then
 VCI_DB="$db"; break
 fi
 done
fi
[ -n "$VCI_DB" ] || { echo "[x] Could not locate the vcIntegrity DB (set VCI_DB=...)"; exit 1; }
echo "[*] Using database: $VCI_DB"

ORPHAN_FILTER="FROM pm_recommendation_spec s
 LEFT JOIN pm_recommendation_info i
 ON i.entity_id = s.entity_id AND i.spec_id = s.spec_id
 WHERE i.entity_id IS NULL"

COUNT="$("$PSQL" -Atqd "$VCI_DB" -c "SELECT count(*) $ORPHAN_FILTER")"
echo "[*] Orphaned pm_recommendation_spec rows: $COUNT"

if [ "$COUNT" = "0" ]; then
 echo "[+] No orphans — FK_PM_RECOMMENDATION_SPEC will validate cleanly."
 exit 0
fi

echo "[*] Offending (entity_id, spec_id) pairs:"
"$PSQL" -d "$VCI_DB" -c "SELECT s.entity_id, s.spec_id $ORPHAN_FILTER
 ORDER BY s.entity_id, s.spec_id"

if [ "$EXECUTE" -ne 1 ]; then
 echo
 echo "[i] DRY-RUN only. Re-run with --execute to delete the rows above."
 exit 0
fi

echo "[*] Deleting orphaned rows in a transaction…"
"$PSQL" -v ON_ERROR_STOP=1 -d "$VCI_DB" <<'SQL'
BEGIN;
DELETE FROM pm_recommendation_spec s
WHERE NOT EXISTS (
 SELECT 1 FROM pm_recommendation_info i
 WHERE i.entity_id = s.entity_id AND i.spec_id = s.spec_id
);
COMMIT;
SQL
echo "[+] Done. Re-run without --execute to confirm 0 orphans, then retry the upgrade."

Save it to /tmp/audit_rec_orphans.sh, then run the dry-run first and review the output before ever adding --execute. Like the KB script, it’s idempotent — running it again after a successful cleanup simply reports zero orphans.

Verifying the fix

Re-run the audit (dry-run) — it should now report 0 orphaned rows. That single number is the whole signal: zero orphans means the new foreign key has nothing left to reject.

Then retry the vCenter upgrade and confirm the vcIntegrity:Import phase passes. If you want belt-and-suspenders confirmation, tail the new import log and check that the ALTER TABLE ... ADD CONSTRAINT FK_PM_RECOMMENDATION_SPEC statement executes without a 23503 error:

grep -i "FK_PM_RECOMMENDATION_SPEC\|23503\|Upgrade Database Schema" \
 /var/log/vmware/upgrade/Import_com.vmware.vcIntegrity_*.log

No matches on 23503 (and a clean run past the schema step) means you’re through it.

Rollback

If anything looks wrong after the cleanup or the retried upgrade, revert to the snapshot you took in step 1 and reassess before trying again. The cleanup itself only removes already-orphaned rows, so a revert puts you exactly back where you started.

Once the upgrade has completed successfully and vCenter is healthy, delete the snapshot — leaving snapshots on a vCenter appliance long-term causes its own problems (disk growth, performance, and consolidation headaches).

Notes & gotchas

Why su updatemgr? That OS account maps to the database role that owns the vcIntegrity connection. Running the script under it means you never handle DB credentials by hand, and the connection has exactly the rights it needs.

Is deleting these rows safe? Yes — you’re only removing pm_recommendation_spec rows that already reference a non-existent parent. They’re dead lifecycle-recommendation data; no live configuration depends on them. Recommendations regenerate as needed after the upgrade.

Does it apply to VVF as well as VCF? The KB lists both VMware Cloud Foundation 9.1 and VMware vSphere Foundation 9.1 — the trigger is the vCenter 9.1.0.0 schema change, so it’s the same regardless of which edition you’re upgrading under.

Will it come back? Broadcom has stated a fix is planned for a future VCF release so the orphans don’t block the migration. Until then, audit-then-clean is the routine.

References

Broadcom KB 442223 — “Error attempting Upgrade Database Schema” during vCenter 8.0 U3x to 9.1.0.0 upgrade (symptoms, cause, and the official cleanup_rec_orphans.py workaround).
Broadcom KB 326317 — Connecting to vCenter Server Virtual Appliance using WinSCP fails with “Received too large … SFTP packet” (needed if file transfer to the appliance fails).

Fix: Unable to Update BOM Components — "Compatibility File Is Missing" in SDDC Manager

Cosmin Trif — Fri, 05 Jun 2026 09:15:00 +0000

If you have upgraded your VMware Cloud Foundation (VCF) environment to version 5.x from an earlier release, you may find yourself unable to update BOM (Bill of Materials) components from the SDDC Manager UI. The “Update Now” or “Configure Update” button is greyed out and you see a frustrating error:

“Update cannot be started now as compatibility file is missing”

This post walks through the symptoms, root cause, and the step-by-step fix based on Broadcom KB 396202.

Symptoms

You will typically notice one or more of the following:

In the Updates tab of the relevant Workload Domain (WLD), the “Update Now” or “Configure Update” button is greyed out.
The “Available Updates” section displays the error: “Update cannot be started now as compatibility file is missing”.
Attempting to manually download the VmwareCompatibilityData.json file fails.
In the SDDC Manager LCM debug log (/var/log/vmware/vcf/lcm/lcm-debug.log), you see entries similar to:

Recoverable I/O exception (java.net.SocketException) caught when processing request to {s}->https://storage.googleapis.com:443

VVS Compatibility Data update failed
com.vmware.vcf.compatibility.controllers.internal.vvs.exception.VvsException:
The VVS api call to https://vvs.esp.vmware.com/v1/products/bundles/type/vcf-lcm-bundle?format=json failed.
Response: Connection reset

Root Cause

The compatibility data file VmwareCompatibilityData.json is missing from the expected path on the SDDC Manager appliance:

/nfs/vmware/vcf/nfs-mount/compatibility/VmwareCompatibilityData.json

Why is it missing? The most common reason is that the SDDC Manager was upgraded from a VCF version prior to 5.0.0.0. The VVS compatibility data directory was introduced in VCF 5.0, so environments that were upgraded (rather than freshly deployed) may never have had this directory created.

Step-by-Step Resolution

Step 1 — Take a Snapshot

Before making any changes, take an offline snapshot of the SDDC Manager appliance VM. This gives you a safe rollback point.

Step 2 — SSH Into SDDC Manager

Connect via SSH using the vcf user, then elevate to root:

ssh vcf@<sddc-manager-fqdn>
su root

Step 3 — Create the Compatibility Directory

The directory likely does not exist. Create it and navigate into it:

mkdir -p /nfs/vmware/vcf/nfs-mount/compatibility/
cd /nfs/vmware/vcf/nfs-mount/compatibility/

Step 4 — Download the Compatibility Data File

Run the following curl command to pull the VmwareCompatibilityData.json file from Broadcom’s VVS endpoint:

curl --location 'https://vvs.broadcom.com/v1/products/bundles/type/vcf-lcm-v2-bundle?format=json' \
 --header 'X-Vmw-Esp-ClientId: vcf-lcm' \
 > VmwareCompatibilityData.json

Note: If the SDDC Manager does not have internet access, run the curl command on a separate Linux machine that can reach vvs.broadcom.com, then transfer the resulting VmwareCompatibilityData.json file to the SDDC Manager appliance via SCP or another file transfer method.

Step 5 — Set Correct Permissions and Ownership

Apply the proper ownership and permissions so the LCM service can read the file:

chown vcf_lcm:vcf -R /nfs/vmware/vcf/nfs-mount/compatibility
chmod 755 -R /nfs/vmware/vcf/nfs-mount/compatibility/

Step 6 — Restart the LCM Service

Restart the Lifecycle Manager service for the changes to take effect:

systemctl restart lcm

Wait a couple of minutes for the service to fully come back up, then refresh the SDDC Manager UI. The “Update Now” or “Configure Update” button should no longer be greyed out.

Additional Step — Update Depot URLs

If you are also seeing depot-related errors, the download URLs in the LCM configuration may still reference old repository endpoints that require a valid download token. You should update the depot configuration in:

/opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

Broadcom has a dedicated KB for this issue: Depot connection has issues — SDDC fails to download updates from online repositories (KB 389871).

On a related note, if compatibility or interoperability checks are blocking upgrades in your lab, I covered a script to bypass VCF compatibility and interoperability checks.

Quick Reference — All Commands

For convenience, here are all the commands in sequence:

# SSH in and elevate
ssh vcf@<sddc-manager-fqdn>
su root

# Create directory
mkdir -p /nfs/vmware/vcf/nfs-mount/compatibility/
cd /nfs/vmware/vcf/nfs-mount/compatibility/

# Download compatibility data
curl --location 'https://vvs.broadcom.com/v1/products/bundles/type/vcf-lcm-v2-bundle?format=json' \
 --header 'X-Vmw-Esp-ClientId: vcf-lcm' \
 > VmwareCompatibilityData.json

# Fix permissions
chown vcf_lcm:vcf -R /nfs/vmware/vcf/nfs-mount/compatibility
chmod 755 -R /nfs/vmware/vcf/nfs-mount/compatibility/

# Restart LCM
systemctl restart lcm

Scripted Components Cleanup from VCF Operations 9.1: A Step-by-Step Guide

Cosmin Trif — Wed, 03 Jun 2026 10:30:00 +0000

If you’ve been working with VCF Operations 9.1, you may have noticed that on the Build > Lifecycle > VCF Management > Components page, manually-added components such as VCF Operations for Networks, Log Management, Real-Time Metrics or VCF Automation no longer have a “Delete” option available in the UI.

In VCF Operations 9.0, this was possible on the Fleet Management > Lifecycle > Components page by clicking the three vertical dots next to a component and selecting Delete. That option is gone in 9.1.

Broadcom has published KB article 441333 with an attached Python script (cleanup_component.py) that fills this gap. In this post I walk through how to use it for each component type.

Prerequisites

Before running the cleanup script, make sure the following are in place:

Python installed on the system where you will run the script (can be any system with network access to the Fleet and VCF Management Services components).
Network access to the Fleet LCM appliance and VCF Services Runtime.
Credentials — use username admin as the value for the VCF Services Runtime username. The password is the same as the vmware-system-user password.
FQDN values — replace all placeholder FQDNs such as <fleet-lcm-fqdn> with the actual values. These can be obtained from VCF Operations UI -> Build -> Lifecycle -> VCF Management -> Components (e.g., vsp01.example.com for <vcf-services-runtime-fqdn>).

Warning: Prior to deletion, verify that you have a backup of any critical component data which needs to be retained, such as logs or database backups.

Download and Verify the Script

Download cleanup_component.py from KB 441333 (attached at the bottom of the article).

Verify the SHA256 checksum before running:

sha256sum cleanup_component.py

Expected checksum:

27A82D393333D17261EF1E8629A69AD6C40659A8537DF961DE95E545FE656F5B

What the Script Does

The LCM component cleanup script enables administrators to manage Day-N deployed components visible in the VCF Operations UI:

List components — View all deployed components currently eligible for removal.
Delete components — Safely remove inoperable or misconfigured components from the system so they can be redeployed later through VCF Operations.

Important notes:

If a component is running a version older than 9.1, it is removed solely from the Fleet lifecycle. The script does not delete the corresponding infrastructure deployments such as vCenter or VCF Services Runtime.
VSP refers to VCF Management Services.

1. OVA Components

Deleting an OVA-based deployment removes it from Fleet lifecycle and also deletes the associated virtual machines from vCenter.

List Components Deployed on vCenter

python cleanup_component.py list ova-component --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username>

Delete Component Deployed on vCenter

python cleanup_component.py delete ova-component --component-id <ova-component-id> --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username> --vcenter-username <vcenter-username>

The script will prompt for the required passwords. Once confirmed, the OVA component will be removed from Fleet lifecycle and its associated VMs deleted from vCenter.

2. VCF Management Services (VSP) Components

Deleting a VSP-based deployment removes it from Fleet lifecycle and also deletes the associated deployment from VCF Services Runtime.

List Components Deployed on VCF Services Runtime

python cleanup_component.py list vsp-component --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username>

Delete Component Deployed on VCF Services Runtime

python cleanup_component.py delete vsp-component --component-id <vsp-component-id> --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username>

3. VCF Automation

VCF Automation is a multi-component deployment consisting of:

VCF Automation
Migration Service Engine
VCF Automation VCF Services Runtime

Important: You must remove VCF Automation and the Migration Service Engine before you remove the VCF Automation VCF Services Runtime. Follow the order below.

Step 1: Remove VCF Automation and Migration Service Engine

List Components Deployed on VCF Services Runtime

python cleanup_component.py list vsp-component --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username>

Delete Component Deployed on the VCF Automation VCF Services Runtime

python cleanup_component.py delete vsp-component --component-id <vsp-component-id> --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username> --vcfa-vcf-services-runtime-fqdn <vcfa-vcf-services-runtime-fqdn> --vcfa-vcf-services-runtime-username <vcfa-vcf-services-runtime-username>

Run the delete command for both the VCF Automation component and the Migration Service Engine component before proceeding to Step 2.

Step 2: Remove the VCF Automation VCF Services Runtime

Warning: Deleting a VCF Services Runtime requires this script to be executed as the root user inside the SDDC Manager virtual machine.

List VCF Services Runtime

python cleanup_component.py list vsp-cluster --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username>

Delete VCF Services Runtime

python cleanup_component.py delete vsp-cluster --component-id <vsp-cluster-id> --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username> --vcenter-username <vcenter-username>

Summary

The cleanup_component.py script is a handy workaround for the missing “Delete” option in VCF Operations 9.1. The key things to remember:

Back up any critical component data before running delete operations.
For OVA components, deletion also removes VMs from vCenter.
For VSP components, deletion also removes the deployment from VCF Services Runtime.
For VCF Automation, always delete VCF Automation and Migration Service Engine first, then delete the VCF Services Runtime — and run that last step as root on the SDDC Manager VM.
Always verify the script checksum before running it.

After removing components, also check Fleet password management for leftover or expired password accounts — I covered that remediation in Fixing Expired Password Sync in VCF 9.1 Fleet Management.

For the official script download and additional details, refer to Broadcom KB article 441333.

I hope this helps!

Installing and Configuring Cloudbase-Init on Windows for VCF Automation Guest Customization

Cosmin Trif — Thu, 16 Apr 2026 15:00:00 +0000

Customizing Windows guest instances in VMware environments has traditionally been handled by the VMware Guest Customization Specification (Sysprep-based). While this approach works, it has limitations — especially when you need cloud-agnostic blueprints that work across vCenter, Azure, AWS, and GCP cloud accounts.

Enter Cloudbase-Init — the Windows equivalent of Cloud-Init. It provides powerful guest customization capabilities including user creation, password injection, hostname configuration, SSH public keys, and user-data script execution. This guide walks through the full process of installing and configuring Cloudbase-Init on a Windows Server template for use with VCF Automation (formerly VMware Aria Automation / vRealize Automation).

What is Cloudbase-Init?

Cloudbase-Init is an open-source Windows guest initialization service, designed to be the Cloud-Init equivalent for Windows operating systems. Key features include:

User creation and password management — Create local accounts, inject passwords, and enforce password change on first logon
Hostname customization — Set the Windows computer name during provisioning
SSH public key injection — Deploy SSH keys for remote access
User-data scripts — Execute PowerShell, batch, or cloud-config YAML scripts at first boot
Disk extension — Automatically extend volumes to use all available disk space
MTU configuration — Set network MTU values via DHCP

When paired with VCF Automation’s cloudConfig and remoteAccess blueprint properties, Cloudbase-Init enables fully automated Windows provisioning with zero manual intervention.

Prerequisites

Before you begin, ensure you have:

A Windows Server ISO (2016, 2019, 2022, or 2025) mounted in vCenter
vCenter access with permissions to create VMs and templates
VCF Automation (or Aria Automation / vRealize Automation 8.x+) deployed and configured
An RDP client for accessing the Windows VM during setup
A vCenter Cloud Account configured in VCF Automation (ideally using a custom vCenter role)

Part 1: Preparing the Windows Template

Step 1: Create a Base Windows VM

Log in to vCenter and create a new Virtual Machine
Mount your Windows Server ISO and complete the OS installation
Install VMware Tools (required for OVF metadata transport)
Apply all Windows Updates and install any additional software packages your organization requires (e.g., monitoring agents, antivirus)
Power on the VM and log in via RDP

Tip: Install all prerequisite software before installing Cloudbase-Init. Once Cloudbase-Init runs Sysprep, the VM will shut down and should be converted to a template immediately.

Step 2: Download Cloudbase-Init

Download the Cloudbase-Init installation binaries from one of these locations:

GitHub: https://github.com/cloudbase/cloudbase-init
Official Downloads: https://cloudbase.it/cloudbase-init/#download

Important: Ensure you are using version 0.9.12.dev72 or greater, which includes the OvfService metadata provider required for vCenter integration. The OvfService is what allows Cloudbase-Init to receive customization data from VCF Automation via VMware Tools.

Step 3: Run the Cloudbase-Init Installer

Run the CloudbaseInitSetup_x64.msi installer on the Windows VM
Accept the license agreement and click Next
On the configuration page, set the following options:
- Username: Administrator
- Check: ✅ Run Cloudbase-Init service as LocalSystem
- Leave all other defaults as-is

Click Next, then click Install

Note: Some advanced user-data scripts may require Cloudbase-Init to run under a dedicated administrator account rather than LocalSystem. If this applies to your environment, create and select a dedicated service account at install time.

Do NOT finalize the installer yet — when prompted with the “Completing the Cloudbase-Init Setup Wizard” dialog, leave the checkboxes unchecked for now. We need to edit the configuration files first.

Step 4: Configure cloudbase-init-unattend.conf

Navigate to the Cloudbase-Init installation directory (typically C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\) and edit the file cloudbase-init-unattend.conf.

This configuration file controls the behavior during the specialize phase of Windows setup (before the first user logon). Replace its contents with:

[DEFAULT]
username=Administrator
groups=Administrators
inject_user_password=true
config_drive_raw_hhd=true
config_drive_cdrom=true
config_drive_vfat=true
bsdtar_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\bsdtar.exe
mtools_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\
verbose=true
debug=true
logdir=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\
logfile=cloudbase-init-unattend.log
default_log_levels=comtypes=INFO,suds=INFO,iso8601=WARN,requests=WARN
logging_serial_port_settings=
mtu_use_dhcp_config=true
ntp_use_dhcp_config=true
local_scripts_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts\
metadata_services=cloudbaseinit.metadata.services.ovfservice.OvfService
plugins=cloudbaseinit.plugins.common.mtu.MTUPlugin,cloudbaseinit.plugins.common.sethostname.SetHostNamePlugin,cloudbaseinit.plugins.windows.extendvolumes.ExtendVolumesPlugin
allow_reboot=false
stop_service_on_exit=false
check_latest_version=false

Key settings explained:

Setting	Value	Purpose
`metadata_services`	`OvfService`	Only use the OVF metadata provider (vCenter). Prevents Cloudbase-Init from iterating through unsupported providers and cluttering logs
`plugins`	MTU, SetHostName, ExtendVolumes	Specialize-phase plugins: configure network MTU, set hostname, extend disk volumes
`allow_reboot`	`false`	Prevent unexpected reboots during the specialize phase
`stop_service_on_exit`	`false`	Keep the service running for the next phase

Step 5: Configure cloudbase-init.conf

In the same conf directory, edit cloudbase-init.conf. This file controls the behavior during the first logon phase.

Replace its contents with:

[DEFAULT]
username=Administrator
groups=Administrators
inject_user_password=true
first_logon_behaviour=always
config_drive_raw_hhd=true
config_drive_cdrom=true
config_drive_vfat=true
bsdtar_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\bsdtar.exe
mtools_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\
verbose=true
debug=true
logdir=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\
logfile=cloudbase-init.log
default_log_levels=comtypes=INFO,suds=INFO,iso8601=WARN,requests=WARN
logging_serial_port_settings=
mtu_use_dhcp_config=true
ntp_use_dhcp_config=true
local_scripts_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts\
metadata_services=cloudbaseinit.metadata.services.ovfservice.OvfService
plugins=cloudbaseinit.plugins.windows.createuser.CreateUserPlugin,cloudbaseinit.plugins.windows.setuserpassword.SetUserPasswordPlugin,cloudbaseinit.plugins.common.sshpublickeys.SetUserSSHPublicKeysPlugin,cloudbaseinit.plugins.common.userdata.UserDataPlugin

Key settings explained:

Setting	Value	Purpose
`first_logon_behaviour`	`always`	Forces the user to change the password after first logon
`metadata_services`	`OvfService`	Same as unattend — only use OVF metadata for vCenter
`plugins`	CreateUser, SetUserPassword, SetUserSSHPublicKeys, UserData	First-logon plugins: create user account, set password, inject SSH keys, execute user-data scripts

Important: The username=Administrator setting combined with SetUserPasswordPlugin and the remoteAccess password in the blueprint allows Cloudbase-Init to change the initial blank password created after Sysprep.

Step 6: Understanding the Plugin Order

Cloudbase-Init executes plugins in the order they are listed in the configuration. Here is a breakdown of both phases:

Specialize Phase (cloudbase-init-unattend.conf):

MTUPlugin — Configures network MTU based on DHCP
SetHostNamePlugin — Sets the Windows computer name
ExtendVolumesPlugin — Extends disk volumes to use all available space

First Logon Phase (cloudbase-init.conf):

CreateUserPlugin — Creates the local user account
SetUserPasswordPlugin — Sets the user password from metadata
SetUserSSHPublicKeysPlugin — Injects SSH public keys
UserDataPlugin — Executes user-data scripts (PowerShell, batch, cloud-config)

Tip: Only list plugins with capabilities supported by OvfService. This keeps logs clean and avoids error messages from plugins that cannot find their expected metadata.

Step 7: Complete the Installation with Sysprep

Now return to the Cloudbase-Init installer “Completing” dialog and check both boxes:

✅ Run Sysprep to create a generalized image
✅ Shutdown when Sysprep terminates

Click Finish. The system will run Windows Sysprep to generalize the image and then automatically shut down the VM.

Note: If Sysprep appears to hang or the VM does not shut down, try uninstalling Cloudbase-Init, re-downloading the installer, and repeating the process. This can occur with certain Windows Server 2016 builds.

Step 8: Convert to Template

Once the VM has shut down:

In vCenter, right-click the VM
Select Template → Convert to Template

Your Windows template is now ready for use with VCF Automation.

Silent Installation (Alternative)

For automated template builds (e.g., with Packer or other automation tools), Cloudbase-Init supports silent installation:

# Download Cloudbase-Init
Invoke-WebRequest -Uri "https://cloudbase.it/downloads/CloudbaseInitSetup_x64.msi" -OutFile "C:\CloudbaseInitSetup_x64.msi"

# Silent install
msiexec /i C:\CloudbaseInitSetup_x64.msi /qn /l*v C:\cloudbase-init-install.log ^
 LOGGINGSERIALPORTNAME=COM1 ^
 USERNAME=Administrator ^
 INJECTMETADATAPASSWORD=TRUE ^
 LOCALSYSTEMSERVICE=TRUE

# Copy custom configuration files
Copy-Item "cloudbase-init.conf" "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\cloudbase-init.conf" -Force
Copy-Item "cloudbase-init-unattend.conf" "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\cloudbase-init-unattend.conf" -Force

# Run Sysprep
& "C:\Windows\System32\Sysprep\sysprep.exe" /generalize /oobe /shutdown /unattend:"C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\Unattend.xml"

Part 2: Deploying and Customizing Windows in VCF Automation

Now that the template is prepared, let’s create a cloud-agnostic blueprint in VCF Automation to deploy and customize Windows instances.

Step 1: Verify Infrastructure Setup

Ensure the following are configured in VCF Automation:

Cloud Account — A vCenter Cloud Account is connected

Cloud Zone — A Cloud Zone is associated with the Cloud Account

Project — A Project is linked to the Cloud Zone with appropriate users/groups

Flavor Mapping — At least one flavor mapping (e.g., small = 1 vCPU, 2 GB RAM)

Image Mapping — An image mapping pointing to your Cloudbase-Init Windows template

Tip: If you don’t see your template in the Image Mapping dropdown, navigate to Cloud Accounts and click Sync Images. By default, image collection runs automatically every 24 hours.

Storage Profile — A storage profile for your vCenter datastore(s)

Step 2: Create the Blueprint

Navigate to Design → Cloud Templates and create a new template for your project.

Here is an example cloud-agnostic blueprint that deploys the Windows image with guest customization:

formatVersion: 1
inputs:
 hostname:
 type: string
 title: Hostname
 default: win-demo
 password:
 type: string
 title: Administrator Password
 encrypted: true
 default: P@ssw0rd!2024
resources:
 Cloud_Machine_1:
 type: Cloud.Machine
 properties:
 image: win2022
 flavor: small
 remoteAccess:
 authentication: usernamePassword
 username: Administrator
 password: ${input.password}
 cloudConfig: |
 #cloud-config
 set_hostname: ${input.hostname}
 write_files:
 - content: |
 Cloudbase-Init deployment successful!
 Deployed by VCF Automation
 Hostname: ${input.hostname}
 path: C:\deployment-info.txt

If you also need extra data disks in your Windows blueprints, I covered adding custom disks with drive letters and labels in Aria Automation separately.

Understanding the Blueprint Properties

remoteAccess

The remoteAccess section provides credentials to the guest instance:

Property	Description
`authentication`	Authentication type: `usernamePassword`, `generatedPublicPrivateKey`, or `publicPrivateKey`
`username`	The local account name (matches `username` in cloudbase-init.conf)
`password`	The password to set (must meet Windows password policy requirements)

The OVF metadata service picks up these values and exposes them to the CreateUserPlugin and SetUserPasswordPlugin.

cloudConfig

The cloudConfig property sets the user-data payload. The metadata service picks it up and passes it to the UserDataPlugin. The plugin interprets the data based on the first line:

First Line	Script Type
`#cloud-config`	Cloud-config YAML (hostname, write_files, etc.)
`#ps1` or `#ps1_sysnative`	PowerShell script
`#cmd`	Windows batch/cmd script
`rem cmd`	Windows batch/cmd script (alternative)

Example: PowerShell user-data

cloudConfig: |
 #ps1_sysnative
 # Install Chocolatey
 Set-ExecutionPolicy Bypass -Scope Process -Force
 [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
 iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

 # Install common tools
 choco install notepadplusplus 7zip git -y

 # Create a marker file
 Set-Content -Path "C:\deployment-complete.txt" -Value "Deployment completed at $(Get-Date)"

Example: Cloud-config with multiple write_files

cloudConfig: |
 #cloud-config
 set_hostname: myserver
 write_files:
 - content: |
 Server provisioned by VCF Automation
 Date: 2026-04-16
 path: C:\info.txt
 - content: |
 # Custom PowerShell profile
 Set-Alias ll Get-ChildItem
 path: C:\Users\Administrator\Documents\WindowsPowerShell\profile.ps1

Step 3: Deploy the Blueprint

Click Deploy (or Test)
Provide a Deployment Name
Fill in the input parameters (hostname, password)
Select Current Draft as the version
Click Deploy

VCF Automation will:

Clone the Windows template
Power on the VM
VMware Tools will transport the OVF metadata to the guest
Cloudbase-Init will pick up the metadata and execute the configured plugins
The hostname will be set, user account configured, and user-data scripts executed

Step 4: Verify the Customization

Once the deployment completes successfully:

Open an RDP client and connect to the new VM using the credentials from the blueprint
On first login, you will be prompted to change the password (due to first_logon_behaviour=always)

After changing the password, verify:
- Hostname — Open a command prompt and run hostname to confirm the custom name was applied
- Deployment file — Check C:\deployment-info.txt (or your custom file path) for the content specified in write_files
- User account — Confirm the Administrator account is active with the correct group membership

Troubleshooting

Cloudbase-Init Logs

If customization doesn’t work as expected, check the logs at:

C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log
C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init-unattend.log

Common Issues

Issue	Cause	Solution
Hostname not set	SetHostNamePlugin not in unattend plugins list	Add `SetHostNamePlugin` to `cloudbase-init-unattend.conf` plugins
Password not working	Password doesn’t meet Windows policy	Use a password with uppercase, lowercase, number, and special character
User-data script not executing	Wrong first-line header	Ensure the first line is `#ps1_sysnative`, `#cloud-config`, etc.
Metadata not found	VMware Tools not installed	Install VMware Tools on the template before Cloudbase-Init
Sysprep hangs	Installer corruption	Re-download and reinstall Cloudbase-Init
Logs show “iterating metadata services”	Multiple metadata_services listed	Set only `OvfService` in the metadata_services line

Checking the OVF Environment

To verify that VMware Tools is properly transporting OVF metadata, you can check the OVF environment XML on the guest:

# Check if OVF environment is available
Get-Content "C:\Program Files\VMware\VMware Tools\OvfEnvironment.xml" -ErrorAction SilentlyContinue

Advanced Configurations

Adding Custom Scripts to LocalScripts

You can place PowerShell or batch scripts in the LocalScripts directory that Cloudbase-Init will execute at boot:

C:\Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts\

Scripts in this directory are executed in alphabetical order. This is useful for scripts that need to run on every boot rather than just the first boot.

Using Cloudbase-Init with Packer

For fully automated template creation with HashiCorp Packer, add a provisioner step:

provisioner "powershell" {
 inline = [
 "Invoke-WebRequest -Uri 'https://cloudbase.it/downloads/CloudbaseInitSetup_x64.msi' -OutFile 'C:\\CloudbaseInitSetup_x64.msi'",
 "Start-Process msiexec -ArgumentList '/i C:\\CloudbaseInitSetup_x64.msi /qn LOGGINGSERIALPORTNAME=COM1 USERNAME=Administrator INJECTMETADATAPASSWORD=TRUE LOCALSYSTEMSERVICE=TRUE' -Wait",
 ]
}

provisioner "file" {
 source = "conf/cloudbase-init.conf"
 destination = "C:\\Program Files\\Cloudbase Solutions\\Cloudbase-Init\\conf\\cloudbase-init.conf"
}

provisioner "file" {
 source = "conf/cloudbase-init-unattend.conf"
 destination = "C:\\Program Files\\Cloudbase Solutions\\Cloudbase-Init\\conf\\cloudbase-init-unattend.conf"
}

Multi-Cloud Blueprints

One of the biggest advantages of Cloudbase-Init is cloud-agnostic blueprints. The same blueprint and cloudConfig work across vCenter, Azure, AWS, and GCP cloud accounts — Cloudbase-Init automatically detects the appropriate metadata service for each cloud provider.

For vCenter-only deployments, specifying OvfService alone in the configuration is recommended for cleaner logs and faster initialization.

Summary

Cloudbase-Init is a powerful and flexible tool for Windows guest customization in VCF Automation environments. By following this guide, you have:

✅ Installed Cloudbase-Init on a Windows Server VM
✅ Configured both cloudbase-init-unattend.conf (specialize phase) and cloudbase-init.conf (first logon phase) for OvfService
✅ Prepared a generalized Windows template via Sysprep
✅ Created a cloud-agnostic blueprint with remoteAccess and cloudConfig properties
✅ Deployed and verified the customized Windows instance

This approach gives you consistent, repeatable Windows provisioning with the flexibility to customize user accounts, hostnames, file creation, and script execution — all without manual intervention.

References

Querying Aria Operations for Logs API by Hostname – A Complete Guide

Cosmin Trif — Thu, 02 Apr 2026 14:20:00 +0000

If you need to pull log events programmatically from VMware Aria Operations for Logs (formerly vRealize Log Insight), filtering by a specific hostname, this guide walks you through the entire process — from authentication to querying the powerful internal API endpoint.

This approach is inspired by Brock Peterson’s excellent series on the Aria Operations for Logs API, expanded here with full session authentication steps and hostname-specific query examples.

Prerequisites

Before we begin, you’ll need:

Aria Operations for Logs deployed and accessible (FQDN or IP)
A local admin account or Active Directory / vIDM credentials
curl installed on your workstation
Access to port 9543 on the Aria Operations for Logs appliance

Note: All examples use the -k flag to skip SSL certificate verification. In production, you should use proper certificate trust chains instead.

Step 1: Authenticate and Retrieve a Session Token

Every API interaction starts with obtaining a Bearer token. You do this by posting your credentials to the /api/v2/sessions endpoint.

Local Authentication

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN:9543/api/v2/sessions" \
 -H "Content-Type: application/json" \
 -d '{
 "username": "admin",
 "password": "YourPasswordHere",
 "provider": "Local"
 }'

Active Directory Authentication

If your environment uses Active Directory, change the provider value:

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN:9543/api/v2/sessions" \
 -H "Content-Type: application/json" \
 -d '{
 "username": "your_ad_user",
 "password": "YourPasswordHere",
 "provider": "ActiveDirectory"
 }'

vIDM (Workspace ONE Access) Authentication

For federated identity via vIDM:

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN:9543/api/v2/sessions" \
 -H "Content-Type: application/json" \
 -d '{
 "username": "user@domain.com",
 "password": "YourPasswordHere",
 "provider": "vIDM"
 }'

Understanding the Response

All three methods return the same JSON structure:

{
 "userId": "3c1b81cc-418e-44c0-b91a-54e10a87b1d3",
 "sessionId": "1a2b3c4d-your-session-token-here==",
 "ttl": 1800
}

Field	Description
`userId`	Unique identifier for the authenticated user
`sessionId`	This is your Bearer token — save it for all subsequent API calls
`ttl`	Token time-to-live in seconds (default: 1800 = 30 minutes)

Copy the sessionId value — this is the Bearer token you’ll use in every subsequent request.

Step 2: Verify Your Session (Optional)

Before running queries, you can verify your token is valid:

curl -k "https://YOUR_LOG_INSIGHT_FQDN:9543/api/v2/sessions/current" \
 -H "Authorization: Bearer YOUR_SESSION_ID_HERE"

A successful response confirms your session is active and returns your user details.

Step 3: Discover the Internal API

The standard /api/v2/events endpoint works but is unintuitive and poorly documented. The internal API is far more powerful and flexible.

You can browse the internal API documentation at:

https://YOUR_LOG_INSIGHT_FQDN/rest-api/internal

Scroll down to the events section and you’ll find POST /events/query — this is the endpoint we’ll use for all our hostname queries.

Step 4: Query Events by Hostname

Now for the main event. The internal POST /events/query endpoint accepts a JSON body with a constraints field that uses the same query language as the Explore Logs UI.

Basic Hostname Query (Last 5 Minutes)

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN/api/v1/events/query" \
 -H "Authorization: Bearer YOUR_SESSION_ID_HERE" \
 -H "Content-Type: application/json" \
 -d '{
 "constraints": "hostname:\"esxi01.lab.local\" & LastFiveMinutes",
 "aggregations": "",
 "groups": "",
 "order": "",
 "resultsFrom": 1,
 "resultsTo": 500,
 "timeout": 30000
 }'

Hostname Query (Last 24 Hours)

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN/api/v1/events/query" \
 -H "Authorization: Bearer YOUR_SESSION_ID_HERE" \
 -H "Content-Type: application/json" \
 -d '{
 "constraints": "hostname:\"esxi01.lab.local\" & LastDay",
 "aggregations": "",
 "groups": "",
 "order": "",
 "resultsFrom": 1,
 "resultsTo": 500,
 "timeout": 30000
 }'

Hostname + Text Search

Search for the word “error” on a specific host:

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN/api/v1/events/query" \
 -H "Authorization: Bearer YOUR_SESSION_ID_HERE" \
 -H "Content-Type: application/json" \
 -d '{
 "constraints": "hostname:\"esxi01.lab.local\" & '\''error'\'' & LastDay",
 "aggregations": "",
 "groups": "",
 "order": "",
 "resultsFrom": 1,
 "resultsTo": 500,
 "timeout": 30000
 }'

Multiple Hostnames with OR

Query logs from two different hosts simultaneously:

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN/api/v1/events/query" \
 -H "Authorization: Bearer YOUR_SESSION_ID_HERE" \
 -H "Content-Type: application/json" \
 -d '{
 "constraints": "(hostname:\"esxi01.lab.local\" | hostname:\"esxi02.lab.local\") & LastDay",
 "aggregations": "",
 "groups": "",
 "order": "",
 "resultsFrom": 1,
 "resultsTo": 500,
 "timeout": 30000
 }'

Hostname + Specific Field Exists

Find vCenter events logged from a specific host:

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN/api/v1/events/query" \
 -H "Authorization: Bearer YOUR_SESSION_ID_HERE" \
 -H "Content-Type: application/json" \
 -d '{
 "constraints": "hostname:\"vcsa01.lab.local\" & EXISTS(vc_event_type) & LastDay",
 "aggregations": "",
 "groups": "",
 "order": "",
 "resultsFrom": 1,
 "resultsTo": 500,
 "timeout": 30000
 }'

Hostname + Specific Field Value

Filter a host for specific vCenter event types (e.g., user login events):

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN/api/v1/events/query" \
 -H "Authorization: Bearer YOUR_SESSION_ID_HERE" \
 -H "Content-Type: application/json" \
 -d '{
 "constraints": "hostname:\"vcsa01.lab.local\" & vc_event_type:\"com.vmware.vim25.userloginsessionevent\" & Last48",
 "aggregations": "",
 "groups": "",
 "order": "",
 "resultsFrom": 1,
 "resultsTo": 20000,
 "timeout": 30000
 }'

Hostname + Epoch Timestamp Range

For precise time ranges, use seconds-since-epoch timestamps:

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN/api/v1/events/query" \
 -H "Authorization: Bearer YOUR_SESSION_ID_HERE" \
 -H "Content-Type: application/json" \
 -d '{
 "constraints": "hostname:\"esxi01.lab.local\" & (timestamp > 1743552000 & timestamp < 1743638400)",
 "aggregations": "",
 "groups": "",
 "order": "",
 "resultsFrom": 1,
 "resultsTo": 500,
 "timeout": 30000
 }'

Tip: Use a tool like epochconverter.com to convert human-readable dates to epoch timestamps.

Hostname with Aggregation (Event Count by Type)

Get a count of events grouped by type for a specific host:

curl -k -X POST "https://YOUR_LOG_INSIGHT_FQDN/api/v1/events/query" \
 -H "Authorization: Bearer YOUR_SESSION_ID_HERE" \
 -H "Content-Type: application/json" \
 -d '{
 "constraints": "hostname:\"esxi01.lab.local\" & LastDay",
 "aggregations": "COUNT(event)",
 "groups": "event_type",
 "order": "event ASC",
 "resultsFrom": 1,
 "resultsTo": 20000,
 "timeout": 30000
 }'

Quick Reference: Constraint Syntax

Syntax	Description
`hostname:\"value\"`	Filter by exact hostname
`'text string'`	Search for text in log messages
`&`	AND operator
`\|`	OR operator
`EXISTS(field)`	Check if a field exists in the log entry
`field:\"value\"`	Filter by field value
`LastFiveMinutes`	Last 5 minutes of logs
`LastDay`	Last 24 hours
`Last48`	Last 48 hours
`timestamp > epoch`	Custom epoch-based time range

Quick Reference: Request Body Fields

Field	Description	Notes
`constraints`	Your query (filters, time range)	Required
`aggregations`	Aggregate function (e.g., `COUNT(event)`)	Leave empty for raw logs
`groups`	Group-by field	Leave empty for raw logs
`order`	Sort order for aggregations	Leave empty for raw logs
`resultsFrom`	Starting result index	Starts at 1
`resultsTo`	Ending result index	Maximum is 20,000
`timeout`	Query timeout in milliseconds	30000 = 30 seconds

Putting It All Together: A Complete Script

Here’s a one-liner that authenticates and queries in a single pipeline:

# Authenticate and store the token
TOKEN=$(curl -sk -X POST "https://YOUR_LOG_INSIGHT_FQDN:9543/api/v2/sessions" \
 -H "Content-Type: application/json" \
 -d '{"username":"admin","password":"YourPasswordHere","provider":"Local"}' \
 | python3 -c "import sys,json; print(json.load(sys.stdin)['sessionId'])")

# Query events for a specific hostname
curl -sk -X POST "https://YOUR_LOG_INSIGHT_FQDN/api/v1/events/query" \
 -H "Authorization: Bearer $TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "constraints": "hostname:\"esxi01.lab.local\" & LastDay",
 "aggregations": "",
 "groups": "",
 "order": "",
 "resultsFrom": 1,
 "resultsTo": 500,
 "timeout": 30000
 }' | python3 -m json.tool

Important Notes

Field name verification: The hostname field is the standard syslog source field in Aria Operations for Logs. However, depending on your ingestion method and content packs, the field could be named source, vmw_esxi_hostname, or something custom. Check the field list in the Explore Logs UI sidebar to confirm.
Token expiration: The default TTL is 30 minutes (1800 seconds). For longer-running scripts, re-authenticate periodically.
Result limits: The maximum value for resultsTo is 20,000. For larger result sets, use the UI export-to-NFS feature as described in Brock Peterson’s export guide.
Internal API caveat: The POST /events/query endpoint is part of the internal API surface. While it’s far more powerful than the public /api/v2/events endpoint, internal APIs may change between versions without notice.

Custom Rightsizing Guardrails in VCF Operations 9 with Super Metrics

Cosmin Trif — Tue, 31 Mar 2026 23:30:00 +0000

In a previous post I covered how to configure VCF Operations 9 to use 90 days of usage data for rightsizing recommendations. That gives you a solid foundation — but once you start reviewing the recommendations, you will quickly notice that the engine does not enforce any organizational sizing standards.

For example, the engine might recommend reducing a VM from 8 vCPUs to 5, or suggest removing just 1 GB of RAM. Both are technically valid from a demand perspective, but in practice:

Odd vCPU counts can cause NUMA alignment issues, conflict with per-socket licensing models, and break internal VM sizing templates.
Sub-2GB memory changes create noise in your rightsizing reports without delivering meaningful capacity savings.

VCF Operations does not have native policy settings for these guardrails. But you can build them using Super Metrics — custom formulas that calculate on every collection cycle and produce metrics you can use in dashboards, views, and reports.

Before going further, I want to credit Brock Peterson’s operations_supermetrics repository on GitHub. It contains dozens of production-ready Super Metrics in the correct VCF Operations JSON import format. The rightsizing Super Metrics in this post build on his work — specifically his vCPU Recommended (oversized VMs) and Memory Recommended (oversized VMs) metrics.

Understanding the Super Metric Format

Before building anything, it helps to understand the structure. When you export a Super Metric from VCF Operations, the JSON looks like this (taken from Brock Peterson’s repo):

{"b3ac88b8-09b6-4fc3-a4b2-35e615f340c9": {
 "resourceKinds": [{
 "resourceKindKey": "VirtualMachine",
 "adapterKindKey": "VMWARE"
 }],
 "modificationTime": 1694040789186,
 "name": "vCPU Recommended (oversized VMs)",
 "formula": "${this, metric=cpu|corecount_provisioned} - ${this, metric=summary|oversized|vcpus}",
 "description": "vCPU Recommended (oversized VMs)",
 "unitId": "",
 "modifiedBy": "287ed085-c6ca-4255-b69c-956f1665ed51"
}}

Key fields:

The outer key is a UUID — VCF Operations generates this on import, so you can use any valid UUID.
resourceKinds — defines which object type the metric applies to (here: VirtualMachine from the VMWARE adapter).
formula — the expression. Uses ${this, metric=...} to reference metrics on the same object.
unitId — the unit of measurement ("" for counts, "kb" for kilobytes, "gb" for gigabytes).

The key metric paths for rightsizing are:

cpu|corecount_provisioned — current allocated vCPUs
summary|oversized|vcpus — recommended vCPU reduction count
summary|undersized|vcpus — recommended vCPU addition count
config|hardware|memoryKB — current allocated memory in KB
summary|oversized|memory — recommended memory reduction in KB
summary|undersized|memory — recommended memory addition in KB

Super Metric 1: Even vCPU Recommendations (Oversized VMs)

The Problem

Brock’s original formula ${this, metric=cpu|corecount_provisioned} - ${this, metric=summary|oversized|vcpus} gives you the target vCPU count, but it can be odd. If a VM has 8 vCPUs and the engine recommends reducing by 3, the result is 5 — which does not align with standard sizing templates.

The Formula

To force even numbers, we divide by 2, truncate to an integer, and multiply back:

(${this, metric=cpu|corecount_provisioned} - ${this, metric=summary|oversized|vcpus}) / 2 * 2

How it works: VCF Operations Super Metric formulas perform integer truncation when dividing integers. If the raw target is 5, then 5 / 2 = 2 (truncated), and 2 * 2 = 4. If the raw target is 6, then 6 / 2 = 3, and 3 * 2 = 6. The result is always even, rounded down.

Downloadable JSON

📥 Download: Recommended vCPUs Even (Oversized VMs).json

{"a1c2d3e4-5f6a-7b8c-9d0e-1f2a3b4c5d6e": {
 "resourceKinds": [{
 "resourceKindKey": "VirtualMachine",
 "adapterKindKey": "VMWARE"
 }],
 "modificationTime": 1711929600000,
 "name": "vCPU Recommended Even (Oversized VMs)",
 "formula": "(${this, metric=cpu|corecount_provisioned} - ${this, metric=summary|oversized|vcpus}) / 2 * 2",
 "description": "Recommended vCPU count for oversized VMs, rounded down to the nearest even number. Based on Brock Peterson's original vCPU Recommended super metric.",
 "unitId": "",
 "modifiedBy": ""
}}

Super Metric 2: Even vCPU Recommendations (Undersized VMs)

The same logic applies to undersized VMs, but we add the recommended increase instead of subtracting:

(${this, metric=cpu|corecount_provisioned} + ${this, metric=summary|undersized|vcpus}) / 2 * 2

If a VM has 2 vCPUs and the engine recommends adding 3, the raw target is 5 → 5 / 2 * 2 = 4. Rounded down to the next even number.

Downloadable JSON

📥 Download: Recommended vCPUs Even (Undersized VMs).json

{"b2d3e4f5-6a7b-8c9d-0e1f-2a3b4c5d6e7f": {
 "resourceKinds": [{
 "resourceKindKey": "VirtualMachine",
 "adapterKindKey": "VMWARE"
 }],
 "modificationTime": 1711929600000,
 "name": "vCPU Recommended Even (Undersized VMs)",
 "formula": "(${this, metric=cpu|corecount_provisioned} + ${this, metric=summary|undersized|vcpus}) / 2 * 2",
 "description": "Recommended vCPU count for undersized VMs, rounded up to the nearest even number. Based on Brock Peterson's original vCPU Recommended super metric.",
 "unitId": "",
 "modifiedBy": ""
}}

Super Metric 3: Memory Reduction — 2 GB Minimum (Oversized VMs)

The Problem

The engine might recommend removing 512 MB or 1 GB of RAM from a VM. On a 16 GB VM, that is not worth the operational overhead of a change window. You want to filter the noise and only see reductions of 2 GB or more.

The Formula

VCF Operations Super Metrics support the ternary operator (condition ? true_value : false_value) for conditional logic. The summary|oversized|memory metric is in KB, so 2 GB = 2,097,152 KB:

${this, metric=summary|oversized|memory} >= 2097152 ? ${this, metric=config|hardware|memoryKB} - ${this, metric=summary|oversized|memory} : 0

This returns the recommended new memory value (in KB) only if the reduction is 2 GB or more. Otherwise it returns 0.

Downloadable JSON

📥 Download: Memory Recommended 2GB Min (Oversized VMs).json

{"c3e4f5a6-7b8c-9d0e-1f2a-3b4c5d6e7f8a": {
 "resourceKinds": [{
 "resourceKindKey": "VirtualMachine",
 "adapterKindKey": "VMWARE"
 }],
 "modificationTime": 1711929600000,
 "name": "Memory Recommended 2GB Min (Oversized VMs)",
 "formula": "${this, metric=summary|oversized|memory} >= 2097152 ? ${this, metric=config|hardware|memoryKB} - ${this, metric=summary|oversized|memory} : 0",
 "description": "Recommended memory for oversized VMs. Only returns a value when the recommended reduction is 2 GB (2097152 KB) or more. Returns 0 for reductions below 2 GB to reduce noise. Based on Brock Peterson's original Memory Recommended super metric.",
 "unitId": "kb",
 "modifiedBy": ""
}}

Super Metric 4: Memory Addition — 2 GB Minimum (Undersized VMs)

Same logic for undersized VMs — only surface memory additions of 2 GB or more:

${this, metric=summary|undersized|memory} >= 2097152 ? ${this, metric=config|hardware|memoryKB} + ${this, metric=summary|undersized|memory} : 0

Downloadable JSON

📥 Download: Memory Recommended 2GB Min (Undersized VMs).json

{"d4f5a6b7-8c9d-0e1f-2a3b-4c5d6e7f8a9b": {
 "resourceKinds": [{
 "resourceKindKey": "VirtualMachine",
 "adapterKindKey": "VMWARE"
 }],
 "modificationTime": 1711929600000,
 "name": "Memory Recommended 2GB Min (Undersized VMs)",
 "formula": "${this, metric=summary|undersized|memory} >= 2097152 ? ${this, metric=config|hardware|memoryKB} + ${this, metric=summary|undersized|memory} : 0",
 "description": "Recommended memory for undersized VMs. Only returns a value when the recommended increase is 2 GB (2097152 KB) or more. Returns 0 for increases below 2 GB to reduce noise. Based on Brock Peterson's original Memory Recommended super metric.",
 "unitId": "kb",
 "modifiedBy": ""
}}

How to Import

Download the JSON files above.
In VCF Operations, navigate to Infrastructure Operations > Configuration > Super Metrics.
Click the ⋯ (ellipsis) menu and select Import.
Upload the JSON file. The Super Metric will appear in your list.
Repeat for each file.

After importing, the Super Metrics will NOT start calculating until you activate them in a policy. See the next section.

Activating in Your Policy

Navigate to Infrastructure Operations > Configuration > Policies > Policy Definition.
Select your production policy.
Scroll to the Super Metrics section.
Find the imported Super Metrics and toggle them to Enabled.
Save the policy.

After the next collection cycle (typically 5 minutes), the Super Metrics will begin calculating for all VMs governed by that policy.

Tip: If you have different policies for production vs. dev/test, you may only want these guardrails on the production policy. Dev/test environments might intentionally allow odd vCPU counts or smaller memory increments.

Using the Custom Metrics

Once activated, these Super Metrics appear as regular metrics on VM objects. You can:

Add them as columns in a custom View or Dashboard alongside the native rightsizing metrics.
Include them in scheduled Reports for quarterly capacity reviews.
Compare the native recommended value against your guardrail-adjusted value to quantify the difference.

A useful dashboard layout:

Column	Source	Purpose
VM Name	Built-in	Identifies the VM
Current vCPUs	`cpu	corecount_provisioned`
Native Recommended vCPUs	Brock’s original super metric	What the engine says
Even Recommended vCPUs	Your imported super metric	Guardrail-adjusted target
Current Memory (KB)	`config	hardware
Memory Recommended (2GB+ only)	Your imported super metric	Only meaningful changes

Adjusting the Thresholds

Guardrail	Current Value	To Adjust
vCPU rounding	Nearest even (÷2 ×2)	Change the divisor to 4 for multiples of 4 (÷4 ×4)
Memory threshold	2 GB (2,097,152 KB)	Change `2097152` to `4194304` for a 4 GB minimum

Edit the formula directly in the Super Metric editor and click Validate to confirm.

Downloads

File	Description
Recommended vCPUs Even (Oversized VMs).json	Even vCPU target for oversized VMs
Recommended vCPUs Even (Undersized VMs).json	Even vCPU target for undersized VMs
Memory Recommended 2GB Min (Oversized VMs).json	Memory target with 2 GB minimum reduction
Memory Recommended 2GB Min (Undersized VMs).json	Memory target with 2 GB minimum addition

Import all four into Infrastructure Operations > Configuration > Super Metrics > ⋯ > Import.

References (VCF 9)

Importing a vCenter 8.x as a Workload Domain into VCF 9

Cosmin Trif — Tue, 31 Mar 2026 23:00:00 +0000

As organizations adopt VMware Cloud Foundation 9.0, a common question comes up: what do I do with my existing vCenter 8.x environments? The good news is that VCF 9.0 supports backwards compatibility with 8.x vCenter instances. You can import them as workload domains directly from the VCF Operations console, bringing your legacy infrastructure under VCF management without rebuilding from scratch.

In this post I will walk through the full import process — from prerequisites and networking requirements to NSX deployment and validation — so you know exactly what to expect before you hit the “Import” button.

Note on terminology: When the action is performed within the VCF Operations console, it is referred to as an Import. When using the VCF Installer appliance, the same action is called a Converge. This post covers the Import workflow.

Prerequisites

Before starting, make sure your existing vSphere environment meets the following requirements. Missing any of these will cause the prechecks to fail.

Minimum Versions

Component	Minimum Version
VCF Instance	9.0.x
vCenter	8.0 Update 1 or later
ESXi Hosts	8.0 Update 1 or later
NSX Manager (if existing)	4.1.0.2 or later

Infrastructure Requirements

Shared Datastore — At least one datastore shared across all hosts in the cluster, accessible and writable by all hosts. VCF supports vSAN, NFS v3, VMFS, NFS 4.1, iSCSI, and vVols. If multiple datastore types are present, VCF determines the principal storage using this priority: vSAN > NFS v3 > VMFS > NFS 4.1 > iSCSI > vVols.
vSphere Distributed Switch (VDS) — Version 8.0 or later with at least 2 uplinks attached from all hosts. Standard switches alone are not supported — if you are still using a VSS, migrate to VDS first.
DRS — Must be set to Fully Automated. Partially automated or manual DRS is not supported for import. You can change this in the cluster settings before starting.
Static VMkernel IPs — All VMkernel interfaces must use statically assigned IP addresses (except NSX Host TEPs, which can use DHCP).
Dedicated vMotion Network — vMotion traffic must be on its own dedicated network, not shared with management.
DNS Records — Forward and reverse DNS records must be configured for all VCF management components. All FQDNs must resolve to unique IPs. If deploying a new NSX instance, you will need 4 DNS records and 3 reserved IPs for the NSX cluster.
SSH Enabled — SSH must be enabled on the vCenter being imported.

What Is NOT Supported

Before you start, be aware of these blockers:

Cisco virtual switches
vCenter instances without a VDS
Custom vCenter ports for client connections
Enhanced Linked Mode (ELM) — you must break ELM before importing (see Broadcom TechDocs for the deactivation procedure)
Clusters managed by Dell VxRail
Multiple NSX Manager instances on a single vCenter
NSX Bare Metal Edge nodes

Supported with Caveats

These configurations are supported but require preparation:

Baseline-based lifecycle management — Clusters using baselines can be imported, but you will get a warning during prechecks. SDDC Manager will take over lifecycle management post-import, and you should plan to convert to vSphere Lifecycle Manager images before upgrading to ESXi 9.0.
vCenter VM on the imported cluster — If the vCenter appliance VM lives on one of the clusters being imported, this is supported. However, if it resides on the default management domain cluster, you will need to vMotion it to a supported location first.
Standalone or single-host clusters — Supported only if your vCenter has an additional cluster that meets the import requirements.

Step 1 — Initiate the Import from VCF Operations

All import actions are performed from the VCF Operations 9 console.

Log in to the VCF Operations home page.
Navigate to Inventory > Detailed View.
Expand VCF Instances and select the Organization (VCF Instance) into which you want to import the workload domain.
Click the Add Workload Domain dropdown and select Import a vCenter.

Step 2 — Define the Workload Domain

Enter a unique name for the new Workload Domain (e.g., WD-Prod-DC2).
Click Next.

Step 3 — Specify the vCenter

You have two options here:

Select an existing standalone vCenter — If the vCenter is already visible in the VCF inventory, select it from the table.
Specify an external vCenter — For a vCenter that is not yet registered, enter the following:
- vCenter Server FQDN
- vCenter Server Root Password
- SSO Username
- SSO Password

If the vCenter is connected to an existing NSX Manager instance, toggle the NSX option and provide:

NSX Manager VIP FQDN
NSX Administrator Password
NSX Root Password
NSX Audit Password

If your existing NSX Manager is version 9.0 or later and has Edge clusters, VCF will discover and import the Edge node VMs, including their credentials. The passwords will be reset during import — you can retrieve the new ones from the VCF credential store afterward.

Confirm the certificate thumbprints for vCenter and NSX Manager (if applicable) and click Next.

Step 4 — Run Prechecks

Once you click Next, VCF Operations will run a series of automated prechecks against your environment. These verify:

Network connectivity and port requirements
DNS resolution for all FQDNs
DRS configuration
Datastore accessibility
VDS compatibility

Tip: You may see warnings related to upgrade policy settings if you are importing an 8.x environment. These are informational and typically allow you to resume. If you see errors, resolve them before proceeding.

Step 5 — Configure NSX Networking

If your vCenter is not already connected to an NSX instance, VCF will need to deploy one. For a vCenter 8.x import, NSX 4.x is required, which means a 3-node NSX Manager cluster (High-Availability deployment).

NSX Manager Cluster Configuration

Setting	Value
Deployment Size	High-Availability (3 nodes, recommended) — Standard (single node) is only available for vCenter 9.0+ imports
Appliance Size	Medium, Large, or Extra Large depending on your environment size
Appliance FQDNs	3 unique FQDNs for each NSX Manager node
Cluster FQDN	The VIP FQDN for the NSX Manager cluster
Network	The NSX Manager nodes will deploy on the same management network as the vCenter

Passwords

Set passwords for the NSX Administrator, Root, and Audit accounts. The password requirements are strict:

Minimum 12 characters
At least one uppercase, one lowercase, one digit, one special character
At least 5 different characters
No dictionary words, no palindromes, no more than 4 sequential characters

NSX Overlay Networking

Choose whether to use the existing management VMkernel for NSX overlay networking or to configure a dedicated overlay transport. For most production environments, keeping the default overlay options is reasonable for the initial import — you can reconfigure overlay networking after the import is complete.

Step 6 — Download Required Binaries

This is a step that can catch you off guard. During validation, you may encounter errors indicating that specific NSX binaries are not available.

Navigate to Fleet Management in the VCF Operations console.
In the version dropdown, select VCF 5.2 (yes, 5.2 — this is the version that maps to NSX 4.x binaries for 8.x vCenter imports).
Look inside Patch Binaries for the specific NSX version required.
Start the download and wait for it to complete before re-running validation.

Important: This step is version-specific. The binary version required depends on your target NSX version, which is determined by the vCenter version being imported. Check the VCF 9 Bill of Materials if you are unsure which binary to download.

Step 7 — Validate and Review

Run the final validation — this verifies all networking, binaries, and configuration.
Review the full configuration summary.
Click Finish to start the import.

The import process will:

Deploy the NSX Manager cluster (if no existing NSX was specified)
Configure networking and overlay transport
Register all clusters and hosts with SDDC Manager
Add the workload domain to your VCF Organization

You can monitor the progress under Tasks in the Fleet Management section. Once complete, the new workload domain will appear in your Organization inventory.

Post-Import Considerations

After the import completes, there are a few things to address:

Lifecycle Management

SDDC Manager now manages lifecycle for the imported workload domain. If your clusters were using baselines for host patching, plan to convert them to vSphere Lifecycle Manager images before upgrading to ESXi 9.0. See the Broadcom TechDocs on converting from baselines to images for the procedure.

Password Management

ESXi host root credentials for imported hosts may not automatically appear in the VCF credential store. If you need SDDC Manager to manage these passwords, follow the workaround in Broadcom KB 388859 to add the missing credential entries.

Upgrading to VCF 9.0.x

Once the 8.x environment is imported, you can plan an upgrade path to bring the vCenter and ESXi hosts up to 9.0.x. This is a separate workflow managed through SDDC Manager and is covered in detail in the Broadcom TechDocs upgrade guides.

Quick Reference: Import Checklist

Item	Status
vCenter 8.0 U1+ with SSH enabled	☐
ESXi hosts 8.0 U1+	☐
VDS 8.0+ with 2+ uplinks	☐
DRS set to Fully Automated	☐
Static VMkernel IPs	☐
Dedicated vMotion network	☐
Forward + reverse DNS records for all components	☐
4 DNS records + 3 IPs reserved for NSX cluster	☐
No ELM, no Cisco switches, no VxRail	☐
Required NSX binaries downloaded in Fleet Management	☐
Shared datastore accessible from all hosts	☐

References (VCF 9)

Using 90 Days of Usage for Rightsizing Recommendations in VCF Operations 9

Cosmin Trif — Mon, 30 Mar 2026 15:30:00 +0000

In a previous post I walked through configuring VCF Operations 9 capacity policies to align with a 90-day planning window. If you followed that guide, the good news is that rightsizing recommendations share the same underlying policy — Risk Level, Time Remaining thresholds, and buffers all feed directly into the rightsizing engine.

So why a separate post? Because rightsizing is a different workflow with its own nuances. Capacity planning asks “when will my clusters run out of room?” Rightsizing asks “are my individual VMs sized correctly for what they actually use?” The policy is the same foundation, but how you consume and act on the results is completely different.

Note on naming: VCF Operations 9 is the successor to VMware Aria Operations (formerly vRealize Operations). The same rightsizing concepts apply across versions but some UI labels may differ.

How the Rightsizing Engine Uses Your 90-Day Policy

When you configured your capacity policy with a Conservative risk level and a 90-day critical threshold, you also configured the rightsizing engine. Here is how those settings translate:

Risk Level = Conservative → The rightsizing engine uses the upper bound of projected VM utilization. This means it will not recommend downsizing a VM unless it is genuinely underutilized even during peak periods within your 90-day window. This prevents the classic “we rightsized the VM and now month-end processing is slow” problem.
Time Remaining thresholds → The “Recommended Size” metric is calculated to ensure the VM maintains a healthy state for the duration defined by the Green Time Remaining Score Threshold, plus an additional 30 days. With a 90-day critical threshold, the engine is projecting VM resource needs well beyond the immediate term.
Peak Focused = Enabled → The engine accounts for utilization spikes in its recommendations. A VM that idles at 10% CPU for 85 days but spikes to 80% during month-end will not be flagged as massively oversized.

If you have not yet configured these policy settings, start with the capacity policy guide first — those same settings power rightsizing.

What’s Different for Rightsizing: Business Hours

One policy setting that is particularly important for rightsizing but less so for cluster capacity is Business Hours. Where capacity planning looks at the cluster as a whole around the clock, rightsizing focuses on individual VM behavior — and many VMs have workloads that only run during specific hours.

If your line-of-business applications are only active during 8 AM – 6 PM on weekdays, you want the rightsizing engine to project demand based on those hours, not the idle weekend utilization that would drag the average down and lead to undersized recommendations.

To configure Business Hours:

Open your policy in Infrastructure Operations > Configuration > Policies > Policy Definition.
Select your production policy (e.g., Production Clusters – 90 Day Capacity Window).
Navigate to the Capacity section.
Look for the Business Hours setting.
Define your operational hours (e.g., Monday–Friday, 08:00–18:00).
Save the policy.

With Business Hours enabled, the capacity engine and rightsizing engine will weight the utilization data from those hours more heavily, which directly improves the accuracy of the recommendations for workloads that follow a predictable schedule.

Tip: If you have 24/7 workloads (databases, web servers), keep Business Hours disabled for those VMs by applying a separate policy without Business Hours defined.

Understanding the Recommendation Caps

Before you act on any rightsizing results, it is important to understand that VCF Operations applies built-in safety limits to its recommendations:

Direction	Cap	What it Means
Oversized (downsize)	50% of current allocation	The engine will never recommend reducing a VM’s CPU or Memory by more than half in a single cycle
Undersized (upsize)	100% of current allocation	The engine will never recommend more than doubling the current configuration in a single cycle

These caps exist to prevent drastic changes that could destabilize workloads. If a VM is severely oversized — for example, allocated 16 vCPUs but only ever using 2 — the tool will recommend reducing to 8 vCPUs on the first pass. After the change is applied and another 90-day observation period passes, it may recommend a further reduction.

This is why rightsizing is an iterative process, not a one-time event. Plan for at least two or three cycles to fully right-size an environment that has significant sprawl.

Consuming the Rightsizing Results

Now that your policy is tuned for 90 days of usage data, here is how to find and act on the recommendations.

Finding Oversized and Undersized VMs

Navigate to Capacity > Optimize > Rightsize.
The view will show VMs flagged as Oversized, Undersized, or Powered Off based on your policy settings.
Filter by cluster or custom group to focus on your production workloads.

Key Metrics to Review

For each VM, pay attention to these metrics:

Recommended vCPUs — The engine’s suggestion based on 90 days of projected peak demand (Conservative) or average demand (Aggressive).
Recommended Memory — Same projection logic applied to RAM.
CPU Demand % — What the VM is actually using relative to what is allocated. A VM with 8 vCPUs at 5% demand is a strong downsizing candidate.
Memory Active % — Active memory versus allocated. Be cautious here — memory rightsizing is more sensitive than CPU because memory pressure can cause ballooning or swapping.

Exporting Rightsizing Data for Quarterly Reviews

For quarterly capacity reviews, export the rightsizing recommendations directly from the Rightsize page:

Navigate to Capacity > Optimize > Rightsize.
Apply your filters (cluster, custom group, or object type) to scope the results to your production workloads.
Use the Export option to download the rightsizing data as a CSV or PDF.
For a recurring view, use the out-of-the-box VM Rightsizing Details dashboard — search for it under Dashboards — which provides a consolidated view of potential savings and reclaimable capacity.

This export becomes the artifact you bring to your quarterly capacity review meeting, alongside the capacity projections from the first blog post.

The Rightsizing Workflow: A Practical Approach

Rightsizing an entire environment in one pass is risky. Here is a phased approach that works well with a 90-day observation window:

Phase 1 — Identify Quick Wins (Week 1–2)

Focus on the obvious outliers first:

VMs with CPU Demand < 10% and 8+ vCPUs allocated.
VMs with Memory Active < 20% and 16+ GB allocated.
Powered-off VMs that have been off for 30+ days — these are candidates for decommissioning, not rightsizing.

Phase 2 — Apply CPU Changes First (Week 3–4)

CPU rightsizing is generally safer than memory rightsizing because:

vCPU changes can often be performed with a scheduled reboot.
The performance impact of reducing vCPUs is more predictable.
Over-allocation of vCPUs can actually hurt performance due to CPU scheduling overhead (co-stop and ready time).

Apply the recommended CPU changes to your quick-win VMs and let the environment run for at least one full collection cycle before moving on.

Phase 3 — Evaluate Memory Changes (Week 5+)

Memory is trickier:

If a VM loses memory headroom, it may start ballooning or swapping, which causes latent performance issues that do not show up immediately.
Always leave a 10–15% memory buffer above the recommended amount.
Monitor the Balloon and Swap Used metrics for at least a week after making changes.

Phase 4 — Observe and Iterate (90 Days)

After the initial round of changes, let the environment run for a full 90-day cycle. The engine will re-evaluate the VMs based on their new configuration and may recommend further adjustments. This is normal — rightsizing is iterative by design.

Bringing It All Together

If you configured your capacity policy using the previous guide, you already have 90% of the rightsizing foundation in place. The additional steps for rightsizing are:

Setting	Value	Notes
Risk Level	Conservative	Same as capacity policy — shared setting
Time Remaining Thresholds	Warning: 120 days, Critical: 90 days	Same as capacity policy — shared setting
Peak Focused	Enabled	Same as capacity policy — shared setting
Business Hours	Defined per workload type	Rightsizing-specific — critical for accuracy
Recommendation Caps	50% down / 100% up	Built-in, not configurable
Rightsizing Approach	Iterative (2–3 cycles)	Plan for 6–9 months of refinement

Why This Matters

Every oversized VM is wasted capacity that could defer a hardware purchase. Every undersized VM is a performance risk that could lead to an outage.

With a 90-day rightsizing model you can:

Reclaim capacity without guessing — recommendations are backed by 90 days of real usage data.
Avoid month-end surprises — Conservative + Peak Focused ensures cyclical workloads are respected.
Build a repeatable process — quarterly rightsizing reviews become a standard part of your capacity management cadence.
Quantify savings — the delta between current allocation and recommended allocation is the capacity you can reclaim or redeploy.

References (VCF 9)

Configuring VCF Operations 9 Capacity Calculations to Consider 90 Days of Usage

Cosmin Trif — Mon, 30 Mar 2026 11:00:00 +0000

As a solutions architect managing a VCF 9 Operations instance, one of the first things I tune after initial deployment is the capacity engine. Out of the box, VCF Operations uses an exponential decay model that heavily weights the most recent data points. That is great for catching sudden spikes, but it can misrepresent your true capacity posture if your environment has cyclical workloads — think month-end batch jobs, quarterly reporting bursts, or seasonal traffic patterns.

If your planning cycles are based on 90 days of usage, you need to make sure the capacity engine’s projections actually reflect that window. In this post I will walk through the policy settings that influence how VCF Operations calculates capacity, and how to configure them so 90 days of historical utilization are meaningfully represented in the results.

Note on naming: VCF Operations 9 is the successor to VMware Aria Operations (formerly vRealize Operations). If you are upgrading from Aria Operations, the same policy concepts apply but some UI labels may differ.

Why 90 Days Matters

Most enterprise procurement and capacity planning processes operate on a quarterly cadence. When leadership asks “do we need to buy more hosts next quarter?”, the answer needs to be rooted in at least 90 days of observed demand — not just the last week of usage.

By default, the VCF Operations capacity engine:

Uses an exponential decay window for historical data, which means it considers all available history but gives significantly more weight to recent data points.
Projects 1 year into the future for trend analysis.
Calculates Capacity Remaining based on the projected peak usage over the next 3 days.
Determines Recommended Size by looking at the maximum projected utilization from now until 30 days past the warning threshold.

The problem? If you had a one-time spike last Tuesday, the decay model may over-represent that event. Conversely, if you had a massive batch processing window 60 days ago that will repeat, the decay model may under-represent it. We need to tune the policy so the engine balances both recent and historical data appropriately.

Step 1 — Clone Your Base Policy

Never modify the default vSphere Solution policy directly. Always clone it first so your customizations survive product upgrades.

Navigate to Infrastructure Operations > Configuration > Policies > Policy Definition.
Click ADD to create a new policy.

Select vSphere Solution’s Default Policy (VCF Operations) as the base policy to inherit from.
Name it something descriptive, e.g., Production Clusters – 90 Day Capacity Window.
Click CREATE POLICY.

Tip: Create separate policies for production and non-production workloads. Production should use conservative settings while dev/test can safely use aggressive settings.

Step 2 — Configure the Capacity Settings

Once inside the policy, navigate to the Capacity settings section. This is where we align the engine with our 90-day planning horizon.

Risk Level: Conservative vs. Aggressive

This setting directly affects how the engine interprets historical data:

Conservative — Uses the upper bound of the projected utilization range. This means the engine will factor in higher peaks from your 90-day history and project them forward. Use this for production.
Aggressive — Uses the mean of the upper and lower bound projections. This smooths out peaks but can underestimate future demand. Use this for dev/test environments where higher utilization is acceptable.

For a 90-day capacity planning model, set this to Conservative. This ensures that the periodic spikes captured within that 90-day window are not averaged away.

Time Remaining Thresholds

These thresholds define when VCF Operations triggers capacity alerts. Align them with your 90-day planning cycle:

Threshold	Recommended Value	Rationale
Warning	120 days	Gives you a full quarter plus a 30-day buffer to initiate procurement
Critical	90 days	Aligns exactly with your quarterly planning boundary

Setting the Critical threshold to 90 days means that when the capacity engine projects resource exhaustion within the next 90 days, you will receive a critical alert — directly aligning the tool with your procurement lead time.

Capacity Buffers

Capacity buffers reserve a percentage of total usable capacity as headroom. These are subtracted from the total before usage percentages are calculated:

Resource	Recommended Buffer	Notes
CPU	10–15%	Accounts for HA failover and burst headroom
Memory	10–15%	Memory is typically the first resource to constrain
Disk	15–20%	vSAN environments need additional headroom for rebalancing operations

These buffers ensure that when the engine says you have 90 days of capacity remaining, it actually means 90 days with comfortable headroom — not 90 days until the cluster is at 100% and VMs start ballooning.

Peak Focused Setting

If your environment has significant periodic peaks (month-end processing, backup windows, etc.), enable the Peak Focused setting. This tells the capacity engine to give higher weight to utilization spikes when projecting future needs. Combined with the Conservative risk level and a 90-day critical threshold, this creates a capacity model that respects your historical peak demand patterns.

Step 3 — Use Maintenance Schedules to Protect Your 90-Day Baseline

When a non-recurring event is happening in your environment — a one-time data migration, a stress test, a host rebuild — the resulting utilization data will feed into the capacity engine and skew your 90-day projections. VCF Operations provides Maintenance Schedules to prevent this.

A Maintenance Schedule pauses data collection and suppresses alerts for selected objects during a defined window. Because the capacity engine cannot project on data it never collected, those anomalous periods are effectively excluded from your 90-day baseline.

Navigate to Infrastructure Operations > Configuration > Inventory Management.
Select the object(s) that will be affected by the non-standard activity (e.g., the ESXi hosts involved in a migration).
Click Start Maintenance.
Define the start and end times of the maintenance window, or set it to run indefinitely if you are unsure when the activity will complete and plan to stop it manually.

Tip: If you know the maintenance window in advance, schedule it before the activity begins. The engine will not collect any metrics during that window, keeping your historical data clean for capacity planning.

Once the maintenance window ends, data collection resumes automatically and the capacity engine continues its projections based only on valid operational data.

Step 4 — Apply the Policy to Your Clusters

The policy only takes effect when it is attached to the correct objects:

Still in the Policy workspace, navigate to the Groups and Objects section.
Apply the policy to a Custom Group or directly to your production vSphere clusters.
If you have not created a Custom Group yet, go to Infrastructure Operations > Configuration > Custom Groups and create one that includes your production clusters.

Important: Policy precedence matters. If multiple policies apply to the same object, the most specific one wins. Verify your policy is the active one for the target clusters by checking Policies > Active Policies.

Step 5 — Validate the Configuration

After applying the policy, wait for at least one full collection cycle (typically 5 minutes) and then validate:

Navigate to Capacity > Optimize > Assess.
Select one of your production clusters.
Review the Time Remaining and Capacity Remaining metrics. They should now reflect the configured thresholds (warning at 120 days, critical at 90 days).
Check the Demand Trend chart — the projection curve should show a conservative (upper bound) trend line that accounts for historical peaks.

If the values look unreasonably aggressive or conservative, revisit your buffer percentages and risk level settings.

Bringing It All Together

Here is a summary of the full configuration for a 90-day capacity planning model:

Setting	Value
Risk Level	Conservative
Warning Threshold	120 days
Critical Threshold	90 days
CPU Buffer	10–15%
Memory Buffer	10–15%
Disk Buffer	15–20%
Peak Focused	Enabled
Maintenance Schedules	Used during non-standard events
Policy Applied To	Production Clusters (Custom Group)

Why This Matters Beyond Dashboards

Getting the capacity engine right is not about making the dashboards look green. It is about answering the question your CTO is going to ask: “Will we run out of capacity before our next hardware refresh?”

With a 90-day model you can:

Justify hardware purchases with data rooted in 90 days of real demand, not just last week’s averages.
Catch cyclical patterns that shorter windows miss — month-end, quarter-end, year-end processing.
Reduce overprovisioning by using conservative projections that account for peaks without panic-buying.
Align IT and finance by speaking the same quarterly language.

Capacity is only half of the story — I covered making VM-level recommendations honor the same window in Using 90 Days of Usage for Rightsizing Recommendations in VCF Operations 9.

References (VCF 9)

How I Migrated My WordPress Blog to Hugo on Cloudflare Workers (And Why You Should Too)

Cosmin Trif — Tue, 24 Mar 2026 13:00:00 -0400

After years of running my tech blog on WordPress, I finally pulled the trigger and migrated everything to Hugo — a blazing-fast static site generator — hosted on Cloudflare Workers. The result? A site that loads in milliseconds, costs nothing to host, and requires zero server maintenance.

Here’s exactly how I did it, what went wrong, and what I’d do differently.

Why Leave WordPress?

WordPress served me well for years. But as my post count grew to 278 articles (mostly VMware VCF guides with dozens of screenshots each), the cracks started showing:

Performance: Even with caching plugins, page loads were sluggish. Each request hit PHP and MySQL.
Maintenance overhead: WordPress core updates, plugin updates, PHP version bumps, SSL renewals, database backups — it never ends.
Security surface: Every plugin is a potential vulnerability. I was running a firewall plugin just to block brute-force login attempts.
Cost: Hosting a WordPress site with decent performance isn’t free. My VPS bill added up.

A static site eliminates all of these problems. No database, no PHP, no server — just HTML files served from a CDN.

Choosing the Stack

Hugo over Jekyll, Gatsby, Next.js

I evaluated several static site generators:

Generator	Pros	Cons
Jekyll	Ruby-based, GitHub Pages native	Slow builds for large sites
Gatsby	React-based, rich plugin ecosystem	Heavy, complex, Node dependency hell
Next.js	Full-stack capable	Overkill for a blog
Hugo	Single binary, sub-second builds, Go templating	Steeper template learning curve

Hugo won because of build speed. With 278 posts and 13,000+ images, Hugo builds the entire site in under 7 seconds. Jekyll would take minutes.

PaperMod Theme

I went with PaperMod — a clean, fast, SEO-optimized Hugo theme. It ships with:

Dark/light mode toggle
Built-in search (via Fuse.js)
Table of contents
Share buttons
Reading time estimates
Archive and taxonomy pages

I customized it with a dark-first infrastructure theme — muted teal accents, JetBrains Mono for metadata, and Inter for body text.

Cloudflare Workers over GitHub Pages, Netlify, Vercel

Cloudflare Workers (formerly Cloudflare Pages) was the obvious choice:

Free tier handles my traffic easily
Global CDN — content served from 300+ edge locations
Git integration — push to GitHub, site deploys automatically
Custom domain + SSL — handled natively
Security headers — configured via _headers file

The Migration Process

Step 1: Extract the WordPress Backup

My WordPress backup was in the proprietary .wpress format (from the All-in-One WP Migration plugin). This isn’t a standard ZIP — it has a custom binary header format.

I wrote a Python script to parse the binary headers and extract the contents:

def extract_wpress(filepath, output_dir):
 with open(filepath, 'rb') as f:
 while True:
 header = f.read(HEADER_SIZE)
 if len(header) < HEADER_SIZE:
 break
 name = header[0:255].replace(b'\x00', b'').decode('utf-8')
 size = int(header[255:269].replace(b'\x00', b'').decode('utf-8'))
 # ... extract file data

This gave me two critical files:

database.sql — the full WordPress MySQL dump
uploads/ — all 13,000+ media files

Step 2: Convert Posts to Markdown

The core of the migration was a Python script that:

Parsed the SQL dump to extract posts (title, content, slug, date, categories, tags, excerpt)
Converted HTML content to Markdown using the markdownify library
Generated Hugo-compatible front matter with proper YAML escaping
Rewrote image URLs from wp-content/uploads/ to Hugo’s /images/ path

The trickiest part was YAML front matter sanitization. WordPress post titles and descriptions contain all sorts of special characters — quotes, ampersands, colons, tabs — that break YAML parsing. I wrote a yaml_safe_string() helper:

def yaml_safe_string(value):
 if not value:
 return '""'
 value = value.replace('\t', ' ').strip()
 if any(c in value for c in ':{}[]&*?|>!%@`"\',#'):
 escaped = value.replace('\\', '\\\\').replace('"', '\\"')
 return f'"{escaped}"'
 return f'"{value}"'

Step 3: Scaffold the Hugo Site

hugo new site cosmin-gq
cd cosmin-gq
git submodule add https://github.com/adityatelange/hugo-PaperMod themes/PaperMod

Then I configured hugo.toml with:

SEO metadata (description, keywords, author, Open Graph)
JSON-LD structured data via a custom extend_head.html partial
Sitemap configuration (304 URLs generated)
RSS feed (278 items)
Security headers (_headers file for HSTS, CSP, X-Frame-Options)
Feed redirects (/feed/ → /index.xml for WordPress RSS subscribers)

Step 4: Deploy to Cloudflare

Cloudflare recently merged Pages into Workers, so the deployment uses Wrangler:

Build settings:

Build command: hugo --minify
Deploy command: npx wrangler deploy

wrangler.jsonc:

{
 "name": "cosmin-gq",
 "compatibility_date": "2025-09-27",
 "assets": {
 "directory": "public"
 }
}

The workflow is simple: git push → Cloudflare clones the repo → Hugo builds → Wrangler uploads 14,000+ static files to the edge.

Gotchas and Lessons Learned

1. The `paginate` Deprecation

Hugo v0.128.0+ removed the paginate config key. Cloudflare’s build environment uses a newer Hugo version, so this broke the build:

ERROR deprecated: site config key paginate was deprecated in Hugo v0.128.0

Fix: Replace paginate = 10 with:

[pagination]
 pagerSize = 10

2. Cloudflare Workers Don’t Support 404 Redirects

My _redirects file had rules to block old WordPress admin paths:

/wp-admin/* /404.html 404
/wp-login.php /404.html 404

Workers only supports redirect status codes (200, 301, 302, 303, 307, 308). Hugo’s built-in 404 page handles missing routes anyway.

3. The `baseURL` Matters

Hugo generates all links using the baseURL from hugo.toml. If your production domain isn’t set up yet and you’re testing on the .workers.dev URL, all links will point to the wrong place. Set baseURL to match wherever the site is currently served.

4. Don’t Commit the `public/` Directory

Hugo’s build output goes into public/. Since Cloudflare rebuilds this on every deploy, there’s no reason to track it in git. Add it to .gitignore:

public/
.DS_Store
resources/
node_modules/
.wrangler/

If you add LinkedIn share buttons, be aware that most ad blockers (uBlock Origin, AdGuard) will hide them because the share URL contains linkedin.com. Nothing you can do about this from the site side.

The Results

Metric	WordPress	Hugo + Cloudflare
Build time	N/A (dynamic)	~7 seconds
Page load	2-4 seconds	< 200ms
Hosting cost	~$20/month	Free
Server maintenance	Weekly	None
Security updates	Constant	None needed
Uptime	99.5%	99.99% (Cloudflare SLA)

The site now loads almost instantly from anywhere in the world, costs nothing to run, and the only “deployment” step is git push.

What I’d Do Differently

Start with .gitignore — I accidentally committed the public/ directory on the first push (13,000+ files). Set up .gitignore before the first commit.
Use relativeURLs — This avoids the baseURL mismatch issue entirely during testing.
Test the Cloudflare build first — The Workers deployment flow has quirks (no 404 redirects, Wrangler auto-detection). A quick test deploy before migrating all content saves debugging time.

Final Thoughts

If you’re running a technical blog on WordPress and it’s mostly content + screenshots (no e-commerce, no user accounts, no dynamic forms), there’s very little reason to stay on WordPress. Hugo gives you better performance, better security, and zero ongoing costs.

The migration took about a day for 278 posts and 13,000 images. The hardest part was sanitizing the YAML front matter — everything else was straightforward.

The site is live at cosmin.us. All the old content is preserved, SEO metadata is intact, and the RSS feed still works at the same path.

If you have questions about the migration process, feel free to reach out on X (@cosm0_).

Bulk Delete Alerts in VCF Operations (formerly know as Aria Operations)

Cosmin Trif — Tue, 17 Mar 2026 13:53:09 +0000

If you manage a VMware Aria Operations (formerly vROps) environment, you know how quickly alert fatigue can set in. In this guide I will focus specifically on the “Guest OS Experiencing CPU queue” alert. While helpful in identifying CPU-constrained VMs, it can quickly generate thousands of entries if you have a heavily over provisioned environment.

Inspired by Brock Peterson’s excellent guide on Alert Hygiene, we are going to look at exactly how to cancel and delete these specific CPU queue alerts, both via the User Interface and programmatically via the API.

Why is this Alert So Noisy? (The False Positive Bug)

According to Broadcom (KB 376641), Aria Operations versions 8.18.X and earlier suffer from a low sensitivity threshold for this specific alert.

By default, the alert triggers if the Peak vCPU Queue exceeds a threshold of 10 and the overall CPU Usage goes beyond a mere 250 MHz. In modern environments, a VM can easily hit 250 MHz without actually experiencing any tangible performance degradation, resulting in massive alert spam. (Note: Broadcom notes this underlying issue is resolved in VCF Operations 9.0).

Before we go nuclear and delete the existing alerts, let’s look at how to tune it properly.

The Official Fix: Tuning the Alert Definition

Instead of disabling the alert entirely, Broadcom recommends tuning the alert symptoms so it only triggers when there is sizable CPU utilization.

Broadcom provides an XML file in KB 376641 that replaces the CPU Usage > 250 Mhz symptom with CPU Net Run > 75%. It also increases the Wait Cycle to 10 minutes (to avoid alerting on quick CPU spikes) and drops the Cancel Cycle to 1.

How to apply the fix:

Download the Guest OS CPU Queue Alert.xml file from the Broadcom KB article.
In Aria Operations, navigate to Configure > Alerts > Alert Definitions.
Click the three dots (…) menu next to Add and select Import.
Browse and select the XML file you downloaded.
Check the box for “Overwrite existing Alert Definition” and click Import.

Now that the alert is fixed going forward, let’s clean up the thousands of old alerts sitting in your database.

Here is your step-by-step guide to cleaning up the noise.

Method 1: Clearing the Alerts via the UI

If you just want to do a quick manual clean-up, the UI is the easiest path.

Step 1: Log into your VMware Aria Operations console.
Step 2: Navigate to the Troubleshoot tab on the left-hand menu and select Alerts.
Step 3: To isolate our noisy alerts, use the Group by dropdown menu and select Alert Definition.
Step 4: Scroll down or use the quick filter to find the Guest OS Experiencing CPU queue grouping.
Step 5: Select that specific grouping.
Step 6: Click on the ACTIONS dropdown menu at the top of the list and select Cancel Alert. Note: Once canceled, you will notice the lightbulb icon in the Status column turns gray, indicating the alerts are now Inactive.

Step 7: With the grouping still selected, click on the ACTIONS menu one more time and select Delete Canceled Alert.

Just like that, all your historical CPU queue alerts are gone!

Method 2: Clearing the Alerts via the API (Programmatic Approach)

If you have tens of thousands of alerts or want to build a scheduled script to handle this automatically, you can use the Aria Operations REST API. For ease of use I will use the swagger API. the swagger API can be found by going to the Operations URL https://vra_master/suite-api/doc/swagger-ui.html

Step 1: Identify the Alert UUIDs

First, we need to find the active alerts to cancel them. You will use the POST /api/alerts/query endpoint. The response will provide you with a list of alerts, each containing an alertId string.

As an example my JSON for Querry looks like this:

{
 "activeOnly" : true,
 "alertCriticality": [ "CRITICAL", "IMMEDIATE", "WARNING", "INFORMATION" ]
}

Step 2: Cancel the Alerts the Alerts

Now that you have your list of alertId values, you need to make them Inactive. You will use the POST /api/alerts endpoint. In your JSON payload, you will include the alertId strings within the uuids field and set the action to cancel.

Step 3: Bulk Delete Inactive Alerts (Targeted vs. Global)

Once the alerts are canceled, they sit in the database as “Inactive”. You have two ways to delete them depending on your goal:

Option A: Targeted Delete (Safest): If you only want to delete the CPU queue alerts you just canceled, send a DELETE request to the standard /api/alerts endpoint and pass the exact same list of alertId UUIDs you gathered in Step 1.

{
 "alert-query": {
 "alertStatus": {
 "alertStatus" : "CANCELED",
 "alertId": "alert-uuid"
 }
 }
}

Option B: The Global Purge: If you want to use this opportunity to clean house and delete ALL inactive alerts across your entire Aria Operations environment, simply run an empty DELETE request against the /api/alerts/bulk endpoint. Warning: This will wipe out all canceled alerts, regardless of the alert type!

{
 "alert-query": {
 "alertStatus": {
 "alertStatus" : "CANCELED"
 }
 }
}

Pro-Tip: Adjusting Global Retention:

If you don’t want to manually do this every week, you can instruct Aria Operations to clean up Inactive alerts faster.

Go to Administration > Global Settings > Data Retention.
Look for the Symptoms/Alerts setting.
By default, Aria Operations keeps canceled/inactive alerts for 30 days. You can safely lower this (Brock recommends 10 days) to keep your database lean and your troubleshoot tabs clean.

Bonus: How to Stop the Alert from Triggering Altogether

If you don’t care about “Guest OS Experiencing CPU queue” alerts at all and want to stop them from coming back permanently, you should disable them in your active policy:

Navigate to Configure > Policies.
Edit your active/default policy.
Click on the Alerts and Symptoms tile.
Search for “Guest OS Experiencing CPU queue”.
Change the state to Deactivated (represented by a crossed-out circle) and save the policy.

If host maintenance windows are a big source of your alert noise, also check out my guide on how to suppress ESXi host alerts in VCF Operations 9 when a host enters maintenance mode.

Unstuck Your VCF Lab: A Script to Bypass Compatibility & Interoperability Checks

Cosmin Trif — Mon, 09 Feb 2026 02:58:43 +0000

If you spend enough time building out VMware Cloud Foundation (VCF) in a lab, Proof of Concept, or nested environment, you will eventually hit “The Wall.”

You know the hardware works. You know the ESXi versions will play nicely together. But SDDC Manager’s automated guardrails, the strict Hardware Compatibility List (HCL) checks and VVS (VMware Validated Solutions) interoperability validations throw up a red flag and stop your deployment or upgrade dead in its tracks.

VCF is designed to be rigid for stability in production. But in a lab, sometimes you just need to force things through to test a specific feature or scenario.

Today, I’m sharing a quick shell script that modifies the LCM and Domain Manager configurations on the SDDC Manager appliance to bypass these strict validation checks.

The Big Disclaimer

STOP. READ THIS FIRST.

This procedure is not supported by VMware GSS for production environments. These guardrails exist to prevent you from building an unstable or unsupportable environment.

By running this script, you are telling SDDC Manager to ignore safety checks.

Do not run this in production.
Only use this in labs, POCs, or nested environments where you accept the risk of instability.
Always take a snapshot of your SDDC Manager VM before proceeding.

What Does The Script Do?

The script targets two main guardrails within the SDDC Manager appliance:

1. The LCM Compatibility Check

The Lifecycle Manager (LCM) consults the VMware Compatibility Guide to ensure your hardware (servers, I/O cards, drives) is validated for the target VCF version.

The config modified: /opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag
The action: By adding the flag vcf.compatibility.controllers.compatibilityCheckEnabled=false, we tell LCM to skip this HCL lookup.

2. The Domain Manager VVS Validation

The Domain Manager handles the creation and expansion of Workload Domains. It has internal checks to ensure your configuration adheres to VMware Validated Solutions (VVS) interoperability standards.

The config modified: /etc/vmware/vcf/domainmanager/application-prod.properties
The action: By adding vcf.domainmanager.validation.enableVvsInteropValidation=false, we loosen constraints related to interop validation during domain operations.

The “Quick & Dirty” Commands

If you just want to copy-paste the raw commands right into your SSH session as root, here is what you need to do. It appends the necessary flags to the config files and restarts the relevant services to apply the changes.

echo "vcf.compatibility.controllers.compatibilityCheckEnabled=false" >> /opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag
systemctl restart lcm
echo "vcf.domainmanager.validation.enableVvsInteropValidation=false" >> /etc/vmware/vcf/domainmanager/application-prod.properties
systemctl restart domainmanager.service

The “Better” Way (Reusable Script)

If you are frequently tearing down and rebuilding labs, it’s better to wrap this into a reusable script with a few basic safety checks (like ensuring you are root and creating backups of the config files before modifying them).

Here is a version of the process script:

#!/bin/bash

# Title: disable-vcf-validations.sh
# Description: Disables LCM compatibility check and Domain Manager VVS interop validation on SDDC Manager.
# NOTE: FOR LAB USE ONLY. NOT SUPPORTED IN PRODUCTION.

# 1. Ensure running as Root
if [ "$EUID" -ne 0 ]; then
 echo "Please run as root"
 exit 1
fi

echo "--- Starting VCF Validation Bypass ---"

# 2. Disable LCM Compatibility Check
LCM_CONF="/opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag"

if [ -f "$LCM_CONF" ]; then
 echo "Backing up LCM config..."
 cp "$LCM_CONF" "${LCM_CONF}.bak"
 echo "Disabling LCM Compatibility Check flag..."
 echo "vcf.compatibility.controllers.compatibilityCheckEnabled=false" >> "$LCM_CONF"
 echo "Restarting LCM service..."
 systemctl restart lcm
else
 echo "Error: LCM config file not found at $LCM_CONF"
fi

echo "------------------------------------"

# 3. Disable Domain Manager VVS Validation
DM_PROPS="/etc/vmware/vcf/domainmanager/application-prod.properties"

if [ -f "$DM_PROPS" ]; then
 echo "Backing up Domain Manager properties..."
 cp "$DM_PROPS" "${DM_PROPS}.bak"
 echo "Disabling VVS Interop Validation flag..."
 echo "vcf.domainmanager.validation.enableVvsInteropValidation=false" >> "$DM_PROPS"
 echo "Restarting Domain Manager service..."
 systemctl restart domainmanager.service
else
 echo "Error: Domain Manager properties not found at $DM_PROPS"
fi

echo "--- Finished. Validations disabled. ---"

How to use it:

SSH into your SDDC Manager appliance using the vcf user and su to root.
Create a file named disable-vcf-validations.sh.
Paste the script code above into the file.
Make it executable: chmod +x disable-vcf-validations.sh.
Run it: ./disable-vcf-validations.sh.

Once the services restart, retry the operation that was previously blocked in the SDDC Manager UI. If the failed workflow is still sitting there in a running state, you may first need to stop the running task in SDDC Manager. Happy labbing!

Salt vs. BigFix: The Speed of Light vs. The Heavyweight Champion

Cosmin Trif — Tue, 27 Jan 2026 18:40:05 +0000

In the world of infrastructure management, we are often forced to choose between the legacy reliability of the past and the high-speed automation of the future.

For years, HCL BigFix (formerly IBM BigFix) has been the “Heavyweight Champion”—a massive, unmovable force in endpoint management. But a challenger has matured into a dominant player: Salt (SaltStack).

If you are building a modern, self-healing infrastructure, the conversation shouldn’t just be about “patching.” It should be about reaction time, extensibility, and developer experience.

Here is why Salt doesn’t just compete with BigFix—it changes the game entirely.

The Old Guard: Where BigFix Fits

Let’s be fair: BigFix is a tank. It is legendary for its ability to manage hundreds of thousands of endpoints, even over slow satellite links. Its “Relevance Language” is incredibly efficient at querying endpoint status without draining CPU.

But BigFix shows its age in the modern DevOps world:

The Learning Curve: “Relevance Language” is proprietary and complex. It is a skill you learn only for BigFix.
The Paradigm: BigFix is primarily a polling-based system. It asks, “Are you compliant?” and waits for an answer.
The Closed Box: It is a commercial product with a “black box” feel compared to open-source ecosystems.

Why Salt Stands Out: The “React vs. Poll” Revolution

Salt was built differently. It wasn’t designed just to manage servers; it was designed to control them at the speed of thought.

1. The Event Bus (The Killer Feature)

This is where Salt leaves BigFix in the dust. BigFix relies on agents checking in. Salt relies on a bi-directional Event Bus.

BigFix says: “I will check this server every few minutes to see if the service is down.”
Salt says: “The service just died. The minion fired an event to the master. The master triggered a Reactor state. The service is already restarting.”

This allows for Event-Driven Automation. You aren’t just patching; you are building a self-healing infrastructure that reacts to security threats, file changes, or load spikes in milliseconds.

2. Speed (ZeroMQ)

Salt is built on ZeroMQ, a high-performance asynchronous messaging library. It is capable of parallel execution across thousands of nodes nearly instantly.

Remote Execution: Want to check disk usage on 10,000 servers? With Salt, you get the results back in seconds, not minutes.
The “Feel”: Using the Salt CLI feels like you are typing on the local machine, even when you are controlling a fleet of thousands.

3. Python vs. Relevance

BigFix requires you to learn Relevance Language (which looks like Q: name of operating system).

Salt is built on Python.

Extensibility: If Salt can’t do something out of the box, you can write a standard Python module to do it.
Hiring: It is infinitely easier to find an engineer who knows Python than one who is a certified BigFix Relevance expert.

The Code Face-Off: Auto-Healing a Service

Let’s look at a real-world scenario: Nginx crashes on a server. How do these two tools handle it?

The BigFix Method: “The Loop”

BigFix relies on a Fixlet that contains Relevance (the condition) and Action Script (the fix). The client evaluates this loop periodically.

Relevance (Condition):

Bash

exists service "nginx" whose (state of it = "Stopped")

Action Script (The Fix):

Bash

waithidden /usr/sbin/service nginx start

The Pro: It works reliably even if the network is flaky.
The Con: Latency. You are at the mercy of the client’s polling interval. If your loop is set to 15 minutes, your web server is down for 15 minutes.

The Salt Method: “The Reactor”

Salt pushes intelligence to the edge. The Minion monitors the service and fires an event the moment it dies. The Master catches that event and fires a reaction immediately.

The Beacon (On the Minion):

Configured to watch the service and fire an event only when it changes.

YAML

beacons:
 service:
 - services:
 nginx:
 onchangeonly: True

The Reactor (On the Master):

Listens for the event and triggers the fix instantly.

YAML

restart_nginx:
 local.service.start:
 - tgt: {{ data['id'] }}
 - arg:
 - nginx

The Result: The moment Nginx crashes, Salt restarts it. The downtime is measured in milliseconds, not minutes.

The Head-to-Head

Feature	BigFix	Salt (SaltStack)
Primary Architecture	Relay-based (Polling/UDP)	Master/Minion (ZeroMQ Event Bus)
Language	Relevance & Action Script (Proprietary)	Python & YAML (Universal)
Reaction Time	Minutes (Polling Interval)	Milliseconds (Event-Driven)
Automation Style	Compliance Enforcement	Reactive & Orchestrated
Community	Vendor-locked	Open Source (Vibrant ecosystem)

The Verdict:

BigFix is an excellent tool for “Endpoint Management”—keeping a laptop fleet patched and compliant.

But if you are running servers, cloud infrastructure, or applications, you need more than management. You need Orchestration.

Salt stands out because it bridges the gap between configuration management (like Ansible) and remote execution. It allows you to:

Define your infrastructure as code (State files).
React to problems instantly (Beacons/Reactors).
Scale without losing speed (ZeroMQ).

For the modern engineer, the choice is clear. Stop polling your infrastructure and asking it how it’s doing. Start listening to it with Salt. If you want to try it yourself, start with my step-by-step guide to deploying Salt open source.

Cosmin.us

Deploying a VCF 9.1 Fleet with the VMware Cloud Foundation Installer: Step-by-Step Instructions

What changed since the 9.0 posts

Download the VCF Installer

Deploy the VCF Installer appliance

Log in and connect the depot

Download the binaries

Start the Deployment Wizard

Deploy VMware Cloud Foundation

Introduction

Plan

Prepare

Deploy

If validation or deployment fails

References

Upgrading VCF 9.1 Management Services to 9.1.0.0400 Step by Step

Versions in This Update

Before You Start

Upgrade Fleet Lifecycle

Set the VCF Management Target Version

Upgrade VCF Operations

Upgrade Identity Broker

Upgrade VCF Automation

Upgrade VCF Operations for Networks

Upgrade Log Management

Upgrade SDDC Lifecycle

Upgrade Real-Time Metrics and Metrics Store

Upgrade Salt Master

Upgrade Software Depot

Upgrade Migration Service Engine

Upgrade VCF Services Runtime

Upgrade Salt RaaS

What About SDDC Manager and the Core Components?

Validate the Update

Fixing the Swagger localhost:8600 Server URL in VCF Automation 9.1: Overriding a Spec Baked Into the Image

The Symptom

Ruling Out Configuration

Wrong Turn #1: The Pod That Serves the Page Doesn’t Serve the Spec

Wrong Turn #2: There Is No File to Edit

Wrong Turn #3: kubectl edit Ate My JAVA_OPTS

The Fix

Verification

What This Does and Does Not Fix

Caveats

Final Notes

References

VCF Automation 9.1 API Getting Started

What Changed Since vRA 8

Step 0: Find Your VM Apps Tenant Name

Step 1: Generate an API Token (the New Refresh Token)

Step 2: Exchange the API Token for an Access Token

Step 3: Use It

Where the API Docs Moved

A Note on All Apps Organizations

References

Fixing a VCF Operations 9.1 Express Patch Stuck at Installing PAK File: An Empty /etc/yum.repos.d and Four Layers of Upgrade State

The Symptom: Installing PAK File, Forever

Finding the Loop

The Root Cause: One sed Against an Empty Directory

Why It Looked Stuck Instead of Failed

Why Rebooting Did Not Help

Clearing the PAK State: CLI Archaeology

The Precheck 409: Colliding With Your Own Assessment

The NullPointerException: Asking an Empty Orchestrator for a Task

The Clean Path Out

Verification

What Actually Emptied /etc/yum.repos.d?

Takeaways

Final Notes

VCF Operations for Networks 9.1 Search Queries: The vRNI Query Language Lives On

One Console, Four Query Languages

Anatomy of a Networks Search Query

Entity Type (Mandatory)

Filters: where Plus Operators

Projections: What to Show

Grouping, Ordering, Limiting

Time: top N and in last N days

Path and Planning Queries

Entity and Free-Text Searches

vCenter Tags

Filters: `where` Plus Operators

Time: `top N` and `in last N days`