Troubleshooting

The 9.1.0.0300 patch went through this lab without drama at the beginning of July. Three weeks later, the 9.1.0.0400 express patch for VCF Operations hit the same single-node appliance and stopped dead at Installing PAK file — no progress, no error, and a reboot later, still nothing. What followed was a proper archaeology dig through every layer of state the upgrade machinery keeps, and almost every layer turned out to be holding a different version of the truth. ...

Continuing the credential cleanup in the same 9.1 environment from the expired password sync and VCFOPSPWID03 posts, one credential refused to settle: the service account SDDC Manager uses to talk to the NSX Manager API, svc-sddcmanager-a-nsx-mgmt--8199 in my lab. It showed up as disconnected, every remediation attempt failed, and the environment kept trying to rotate it in a loop. This one took several wrong turns before the actual root cause surfaced, and almost every wrong turn taught me something worth writing down. The short version: ...

While rotating credentials in the same 9.1 environment I covered in the expired password sync post, an Update of the NSX Manager admin password (credential type API) from VCF Operations kept failing after about a minute with error code VCFOPSPWID03 and a spectacularly unhelpful message: “call from NSX received invalid response.” The actual cause turned out to be simple — NSX rejected the new password because it had already been used on that manager. This being a lab, I was deliberately recycling a previous password. That goes against best practices, and NSX enforces it. But the way the error surfaces (and the way the reuse policy behaves in this build) is worth writing down. ...

In VCF 9.1, Fleet Password Management accounts can get stuck in EXPIRED status, and the password workflows changed compared to 9.0. In 9.0, you may have seen UI actions such as Disconnected or Remediate. In the 9.1 environment I was working on, those actions were not available. The UI exposed Update, and the Fleet Password Management API provides the same kind of workflow: query password accounts, then update or rotate the password. ...

Breaking Enhanced Linked Mode is one of those changes where the technical command is not the hardest part. The hard part is making sure administrators, service accounts, and external products can still log in and do their jobs after the vCenters no longer share the same SSO domain. In this post I will walk through a repeatable way to export and restore vCenter 8.x RBAC data before splitting Enhanced Linked Mode. The focus is on three permission layers: ...

If you’re moving vCenter Server from 8.0 U3x to 9.1.0.0 — typically as part of a VCF 9.1 deployment or a VVF 9.1 upgrade — the run can fail partway through with: Error attempting Upgrade Database Schema Please check vcintegrity migration logs for details. This is one of those failures that looks scary in the UI but has a clean, well-understood root cause: a handful of orphaned rows in the vcIntegrity database that trip a new foreign-key constraint introduced in 9.1. Below is the whole story — what the error means, how to read the logs, why it happens, and how to clear it so the upgrade completes. It’s the same shape of problem I wrote about in fixing PostgreSQL on Fleet Management 9.0.1: a schema change colliding with pre-existing data. ...

If you have upgraded your VMware Cloud Foundation (VCF) environment to version 5.x from an earlier release, you may find yourself unable to update BOM (Bill of Materials) components from the SDDC Manager UI. The “Update Now” or “Configure Update” button is greyed out and you see a frustrating error: “Update cannot be started now as compatibility file is missing” This post walks through the symptoms, root cause, and the step-by-step fix based on Broadcom KB 396202. ...