Fixing Expired Password Sync in VCF 9.1 Fleet Management

In VCF 9.1, Fleet Password Management accounts can get stuck in EXPIRED status, and the password workflows changed compared to 9.0. In 9.0, you may have seen UI actions such as Disconnected or Remediate. In the 9.1 environment I was working on, those actions were not available. The UI exposed Update, and the Fleet Password Management API provides the same kind of workflow: query password accounts, then update or rotate the password.

Symptoms

The issue in this case was simple to describe but easy to chase in the wrong place:

• Some password accounts showed as EXPIRED in Fleet password management.
• The VCF Operations root password had already been changed outside Fleet.
• The 9.1 UI did not expose the older remediation-style options, so the useful paths were the Update action for single accounts and the API for repeatable remediation.

This post walks through the recovery path I used for VCF 9.1.

What Changed in VCF 9.1 Password Management

The important realization is that VCF 9.1 Fleet Password Management does not expose a documented “store this already changed password in the vault” workflow.

The UI exposes Update for a password account. For automation or for working through a larger list of expired entries, the official API exposes:

• POST /suite-api/api/fleet-management/password-management/accounts/query
• PUT /suite-api/api/fleet-management/password-management/accounts/{passwordAccountKey}/password

The update workflow requires both the current password and the new password. In the API these fields are currentPassword and newPassword; in the UI they are entered through the Update action. Either way, this is a rotation workflow. If a password was already changed directly on the appliance, you need to provide that current real password and rotate once more through Fleet so the appliance password, the vault, and the expiration metadata are back in sync.

References:

• Get VCF Password Accounts
• Update Password
• VCF Operations API Authentication

Get a VCF Operations API Token

The VCF Operations API supports OpsToken authorization. Set the VCF Operations API host first:

OPS="https://<vcf-operations-fqdn>"

Note: These examples use curl -k, which skips TLS certificate verification — common in labs where the appliance still has its self-signed certificate. With -k you are sending credentials to whatever answers on that FQDN. In production, drop -k and trust the VCF Operations CA instead, for example curl --cacert /path/to/vcf-ops-ca.pem ..., or add the CA to the system trust store.

Use an account that is authorized to call the Fleet password management APIs. Avoid echoing the password back to the terminal. The read -rp / read -rsp prompts in this post use bash syntax; if your interactive shell is zsh, either start bash first or use the zsh form, for example read -rs 'VCFOPS_PASSWORD?VCF Operations password: '.

Acquire a token:

read -rp "VCF Operations username: " VCFOPS_USERNAME
read -rsp "VCF Operations password: " VCFOPS_PASSWORD; echo

TOKEN=$(jq -nc \
  --arg u "$VCFOPS_USERNAME" \
  --arg p "$VCFOPS_PASSWORD" \
  '{username:$u,password:$p}' |
  curl -sk -H 'Content-Type: application/json' -d @- \
  "$OPS/suite-api/api/auth/token/acquire" | jq -r .token)

unset VCFOPS_PASSWORD
[ -n "$TOKEN" ] && [ "$TOKEN" != "null" ] || echo "Token acquisition failed" >&2

Test the token:

curl -sk \
  -H "Authorization: OpsToken $TOKEN" \
  "$OPS/suite-api/api/versions/current" | jq .

Query Expired Password Accounts

Now query the expired password entries:

curl -sk \
  -H "Authorization: OpsToken $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"status":"EXPIRED"}' \
  "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=200" \
  | tee /tmp/expired-password-accounts.json | jq .

For a cleaner table:

jq -r '.vcfPasswordAccounts[] |
  [.appliance, .applianceFqdn, .userName, .credentialType, .expiryDate, .passwordAccountKey] |
  @tsv' /tmp/expired-password-accounts.json

I usually sort the output into a tracking table like this:

Do not blindly rotate everything at once. Some accounts may be service users, some may belong to appliances that are temporarily unreachable, and some may be stale entries for components that no longer exist. Also note that domain-level credentials such as ESXi, vCenter, and NSX are owned by SDDC Manager rather than the Fleet vault — those rotate through the SDDC Manager /v1/credentials API instead.

Update an Expired Password Account

For a single account, you can do this from the VCF Operations UI by selecting the expired password account and using Update. Enter the current working password and the new final password.

For repeatable remediation, use the API equivalent below.

One important CLI note: do not paste passwords with special characters directly into an inline curl or shell variable assignment. Passwords that contain an exclamation mark can be changed or rejected by the shell before they ever reach the API because ! may trigger history expansion in interactive shells.

If you are using Bash, disable history expansion for the session:

set +H

If you are using Zsh, disable bang history expansion:

unsetopt BANG_HIST

The safer pattern is to prompt for the password and let jq build the JSON payload.

Pick one expired entry and export the key:

KEY="<passwordAccountKey>"

Run the update call. The currentPassword must be the password that works on the target right now. The newPassword is the new final password you want Fleet to set and store.

read -rsp "Current password: " CUR; echo
read -rsp "New password: " NEW; echo

jq -nc --arg c "$CUR" --arg n "$NEW" \
  '{currentPassword:$c,newPassword:$n}' |
curl -sk -X PUT \
  -H "Authorization: OpsToken $TOKEN" \
  -H 'Content-Type: application/json' \
  -d @- \
  "$OPS/suite-api/api/fleet-management/password-management/accounts/${KEY}/password" \
  | tee /tmp/pwupdate-task.json | jq .

unset CUR NEW

The response should create a password update workflow request. Track the task in VCF Operations, or use the returned request ID if your environment exposes the workflow request endpoint.

After the task completes, query the account again. Filter by the FQDN of the appliance you just updated — it is in your tracking table or /tmp/expired-password-accounts.json — so the result does not depend on how many password accounts the fleet has:

curl -sk \
  -H "Authorization: OpsToken $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"applianceFqdn":"<appliance-fqdn>"}' \
  "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=200" \
  | jq --arg KEY "$KEY" '.vcfPasswordAccounts[] | select(.passwordAccountKey==$KEY)'

You want to see the account move out of EXPIRED and show a new expiryDate.

If the Password Was Already Changed Outside Fleet

This was the awkward case in my environment.

If the password has already been changed manually on the appliance, the Fleet vault may still have the old value. Since 9.1 does not provide a documented vault-only update, use the actual current appliance password as the current password and rotate once more to a new final password through the UI Update action or the API.

That gives Fleet a successful workflow and lets it update:

• The appliance password
• The stored credential
• The expiration metadata
• The password account status

If you want to keep the exact same password that was already set manually, that is where the product gap shows up. The documented workflow wants a change from current to new.

If the Current Password Is Unknown

If nobody knows the current password, Fleet cannot update it because the workflow requires the current password.

Use the native recovery method for that appliance first. For example:

1. Open the VM console.
2. Reset the account to a temporary known password.
3. Confirm SSH or console login works.
4. Immediately rotate from temporary to final through the UI Update action or the Fleet API.

Do not leave the temporary password as the final state. The point is to re-enter the supported Fleet workflow as quickly as possible.

If an Entry Looks Stale

Before rotating, verify that the appliance still exists and belongs to the environment:

curl -sk \
  -H "Authorization: OpsToken $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"applianceFqdn":"<appliance-fqdn>"}' \
  "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=50" \
  | jq .

Also check DNS and reachability from the shell where you are running the remediation:

getent hosts <appliance-fqdn>
curl -k -I https://<appliance-fqdn>

If the component has been removed but the password entry remains, do not delete database rows manually. Open a support case or use the supported inventory cleanup path for that component.

Verification Checklist

After the expired entries are addressed, verify the following:

• The expired account query returns fewer entries, ideally zero for the scope you remediated.
• Updated accounts show a refreshed expiryDate.
• The password update workflow completed successfully.
• Direct login to the target appliance works with the new password.

Query all remaining expired entries:

curl -sk \
  -H "Authorization: OpsToken $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"status":"EXPIRED"}' \
  "$OPS/suite-api/api/fleet-management/password-management/accounts/query?pageSize=200" \
  | jq '.vcfPasswordAccounts'

Finally, clean up the working files, since they contain fleet inventory details such as appliance FQDNs, usernames, and password account keys:

rm -f /tmp/expired-password-accounts.json /tmp/pwupdate-task.json

Final Notes

The main lesson is that VCF 9.1 password management needs to be treated as an update or rotation workflow. The password account state is exposed through VCF Operations Fleet Password Management, and expired entries need to be handled through the UI Update action or the documented account query and password update API calls. This is the same rotate-only philosophy the platform has had since the vRSLCM Locker password management days — the tooling changed, the workflow shape did not.

For this kind of issue, I would avoid two shortcuts:

• Do not use VCF 9.0 UI guidance for VCF 9.1.
• Do not update Fleet LCM database rows by hand.

The reliable path is to update expired accounts through the UI or API and verify the resulting password account state.