Fixing an NSX Service Account Rotation Loop in VCF 9.1: PASSWORD_MANAGER_NSX_USER_API_FAILED and the 128-Character Password

Continuing the credential cleanup in the same 9.1 environment from the expired password sync and VCFOPSPWID03 posts, one credential refused to settle: the service account SDDC Manager uses to talk to the NSX Manager API, svc-sddcmanager-a-nsx-mgmt--8199 in my lab. It showed up as disconnected, every remediation attempt failed, and the environment kept trying to rotate it in a loop.

This one took several wrong turns before the actual root cause surfaced, and almost every wrong turn taught me something worth writing down. The short version:

SDDC Manager generated a 128-character password for the service account, but its own remediate path enforces a 127-character maximum — an off-by-one between the generator and the validator that keeps the rotation failing.
The service account is an OS-level PAM user on the NSX appliance: invisible to the NSX API, UI, and CLI, manageable only as root with standard Linux tools.
Failed rotation attempts had locked the account at the NSX API level.
And the final boss: SDDC Manager’s rotation machinery was re-changing the password within a couple of seconds of any manual reset, so every fix I tested was already stale by the time I tested it.

The Symptom: A Service Account Stuck Disconnected

The credential in question, as SDDC Manager tracks it:

Field	Value
Username	`svc-sddcmanager-a-nsx-mgmt--8199`
Resource type	`NSXT_MANAGER`
Credential type	`API`
Account type	`SERVICE`

The rotation failures are visible on the SDDC Manager appliance in the operations manager log:

tail -f /var/log/vmware/vcf/operationsmanager/operationsmanager.log |
  grep -Ei 'svc-sddcmanager|ROTATE|REMEDIATE|diagnosticMessage'

The recurring entry that matters (flattened here):

errorCode: PASSWORD_MANAGER_NSX_USER_API_FAILED
diagnosticMessage: Response from NSX <nsx-manager-fqdn>: 403 -
The credentials were incorrect or the account specified has been locked.

A 403 for a service account that only SDDC Manager itself manages. Nobody had touched this password by hand — SDDC Manager rotated it, and then could no longer authenticate with the value it had just set.

Note: The examples in this post use curl -k, which skips TLS certificate verification — fine in a lab where the appliances still run self-signed certificates. In production, drop -k and trust the appliance CA instead. If any password you type contains !, disable history expansion first (set +H in bash, unsetopt BANG_HIST in zsh) so the shell does not mangle it. And the read -rp / read -rsp prompts use bash syntax — on the appliances that is the shell you get, but if you run any of this locally from zsh, start bash first or use the zsh form (read -rs 'VAR?prompt: '); in zsh, read -rsp silently leaves the variable empty, and an empty password produces confusing downstream errors instead of a clear failure.

The Off-by-One: 128 Generated vs 127 Accepted

Digging through the failed rotation tasks showed the first real clue: the password SDDC Manager had generated for the service account was 128 characters long, while the remediate path rejects anything over 127 characters.

The generator takes its cue from the NSX node authentication policy. NSX advertises a maximum_password_length of 128 by default, SDDC Manager generates a password at that maximum, and then its own validation refuses to remediate with it. The account ends up in the worst possible state: the rotation half-succeeds often enough to keep changing the password, but remediation can never complete, so the credential is marked disconnected and the platform keeps retrying.

Dead Ends Worth Knowing About

Before the fix, the detours — each one reasonable, each one wrong.

Remediating from the UI

The obvious first move. The remediation workflow failed with the same 403, and the UI adds nothing to the diagnosis — the useful error text only exists in operationsmanager.log.

Guessing the Credential Tuple

My first API attempts failed validation before doing anything, because a PATCH /v1/credentials call must reference the exact tuple SDDC Manager has on file: resourceName, resourceType, credentialType, and username all have to match. Do not guess any of them — read them back first.

Get a token (I ran these directly on the SDDC Manager appliance against localhost, but the FQDN works the same):

SDDC="https://localhost"
read -rp "SSO username: " SSO_USER
read -rsp "SSO password: " SSO_PASS; echo

TOKEN=$(jq -nc --arg u "$SSO_USER" --arg p "$SSO_PASS" \
  '{username:$u,password:$p}' |
  curl -sk -H 'Content-Type: application/json' -d @- \
  "$SDDC/v1/tokens" | jq -r .accessToken)

unset SSO_PASS
[ -n "$TOKEN" ] && [ "$TOKEN" != "null" ] || echo "Token acquisition failed" >&2

The block prints nothing on success — the token lands silently in $TOKEN. Test it with a cheap authenticated call before building anything on top of it:

curl -sk -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $TOKEN" "$SDDC/v1/domains"

A 200 means the token works. Anything else, start over at the token block.

List the NSX credentials and pull the exact tuple:

curl -sk -H "Authorization: Bearer $TOKEN" \
  "$SDDC/v1/credentials?resourceType=NSXT_MANAGER" |
  jq -r '.elements[] | select(.accountType=="SERVICE") |
    [.resource.resourceName, .resource.resourceType, .credentialType, .username] | @tsv'

That returns the values used in the remediation calls later in this post.

If jq fails here with “Cannot iterate over null”, the response was not a credentials list — it is almost always an authentication error object instead. Check the HTTP status and the raw body:

curl -sk -o /tmp/creds.json -w "%{http_code}\n" \
  -H "Authorization: Bearer $TOKEN" \
  "$SDDC/v1/credentials?resourceType=NSXT_MANAGER"

jq . /tmp/creds.json

A 401 means the token is empty, expired (they are short-lived), or the literal string null from a failed acquisition — re-run the token block above and mind its failure check. A 401 body complaining that “jwt strings must contain exactly 2 period characters” is the same problem stated more directly: whatever is in $TOKEN is not a JWT at all, and the usual cause is a rejected /v1/tokens login quietly turning into null — check that the username is a full SSO identity such as administrator@vsphere.local, not a bare username.

Capture the service account username while you are here — the remediation and rotation calls later run in this same SDDC Manager session and reuse it:

SVC=$(curl -sk -H "Authorization: Bearer $TOKEN" \
  "$SDDC/v1/credentials?resourceType=NSXT_MANAGER" |
  jq -r '.elements[] | select(.accountType=="SERVICE") | .username')

echo "$SVC"

If you spend time in this API, the /v1/credentials overview covers the basic operations.

Changing the VCF Operations Password Policies

VCF Operations 9.1 exposes password policy settings, and lowering a maximum length there looked like the clean fix for the 128-character problem. It did nothing. Those policies do not govern what SDDC Manager generates for service accounts — the generator follows the NSX node authentication policy, not VCF Operations. Worth knowing before you spend an afternoon proving it.

The First Fix: Lower the NSX Maximum Password Length

Since the generator follows the NSX policy maximum and the validator caps at 127, the workaround is to lower the NSX maximum_password_length so generated passwords land under the limit. I went to 100 — comfortably below 127, still far longer than any human-managed password.

One trap first: the setting lives in the auth-policy endpoint. There is no /api/v1/node/aaa/password-complexity endpoint — that name belongs to the NSX CLI command (get/set password-complexity), and the API returns 404 for it. The REST path you want is /api/v1/node/aaa/auth-policy, the same endpoint from the password reuse post.

Back up the current policy before changing anything:

read -rsp "NSX admin password: " NSXPASS; echo

curl -sk -u admin:"$NSXPASS" \
  https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy |
  tee /tmp/auth-policy-backup.json | jq .

{
  "api_failed_auth_lockout_period": 900,
  "api_max_auth_failures": 5,
  "minimum_password_length": 12,
  "maximum_password_length": 128,
  ...
}

Lower the maximum (maximum_password_length accepts 8 to 128):

curl -sk -u admin:"$NSXPASS" -X PUT \
  https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy \
  -H 'Content-Type: application/json' \
  -d '{"maximum_password_length": 100}'

Policy changes take a short while to apply and synchronize across the manager cluster nodes in 9.x — read the value back before moving on:

curl -sk -u admin:"$NSXPASS" \
  https://<nsx-manager-fqdn>/api/v1/node/aaa/auth-policy | jq .maximum_password_length

New rotations now generate passwords of at most 100 characters, comfortably under the 127-character validation. Problem solved — except the account was still disconnected, and remediation still failed with the same 403.

Plot Twist: TEST_BEFORE_REMEDIATE

The remediation task logs showed the failure happening in a TEST_BEFORE_REMEDIATE stage. Before SDDC Manager remediates a credential, it first authenticates against NSX as the service account with the password it believes is current. If that test fails — wrong password or locked account — the whole remediation aborts with PASSWORD_MANAGER_NSX_USER_API_FAILED, and nothing is remediated.

That explains the loop: the earlier failed rotations had burned through NSX’s API authentication failure allowance (api_max_auth_failures: 5 in the policy above), the account got locked, and from then on even a correct password could not pass the pre-remediation test.

So before anything else could work, I had to look at the account itself on the NSX side — which is where this got interesting.

The Hidden User: An API-Only PAM Account

The service account does not exist as far as the NSX management plane is concerned:

GET /api/v1/node/users — not listed.
NSX UI under user management — not there.
NSX CLI get user / set user — unknown user.

But on the NSX Manager appliance as root, it is a perfectly ordinary Linux account:

grep svc- /etc/passwd

svc-sddcmanager-a-nsx-mgmt--8199:x:...:...:/home/svc-sddcmanager-a-nsx-mgmt--8199:...

The account authenticates to the NSX API through PAM, has password history in /etc/security/opasswd, and is managed by exactly nothing except root and the standard Linux toolbox. No SSH access, no UI, API-only. That means the diagnostics you would normally run through the NSX API happen with faillock and chage instead:

SVC="svc-sddcmanager-a-nsx-mgmt--8199"

faillock --user "$SVC"
chage -l "$SVC"

faillock confirmed the lockout — a stack of recent authentication failures, courtesy of the rotation loop. Clear it:

faillock --user "$SVC" --reset

While in there, two more root-level tools are worth knowing for this account type:

passwd "$SVC"                        # set a known password, as root
echo "" > /etc/security/opasswd     # wipe PAM password history — see warning

⚠️ /etc/security/opasswd holds the PAM password history for every user on the appliance, not just the service account. Emptying it is a blunt lab move to get past “password has been already used” rejections during repeated testing. Do not do this on a production appliance without a support ticket telling you to.

Keeping the Account from Locking Again

The lockout pattern here matches Broadcom KB 420736: SDDC Manager trips the NSX API lockout with its own failed authentication attempts, and remediation can never win against a locked account. The durable fix is to add SDDC Manager’s IP to the NSX lockout_immune_addresses list, so its authentication failures never lock anything.

Run this on the SDDC Manager appliance, so hostname -i resolves to the right IP; the jq filter appends it to the existing list without clobbering other entries:

Note: PUT /api/v1/cluster/api-service replaces the entire cluster-wide API service configuration document, and NSX restarts the API service to apply it — expect a brief API interruption, so run this when nothing else is mid-call. The pipeline below reads the full current document and keeps a copy in /tmp/api-service-backup.json before anything is overwritten.

curl -sk -u admin:"$NSXPASS" \
  https://<nsx-manager-fqdn>/api/v1/cluster/api-service |
  tee /tmp/api-service-backup.json |
  jq --arg ip "$(hostname -i)" \
    '.lockout_immune_addresses += [$ip] | .lockout_immune_addresses |= unique' |
  curl -sk -u admin:"$NSXPASS" -X PUT \
    -H 'Content-Type: application/json' -d @- \
    https://<nsx-manager-fqdn>/api/v1/cluster/api-service

With the lockout cleared and immunity in place, I reset the service account password as root, confirmed it worked with a quick authentication test…

read -rsp "Service account password you just set: " SVCPASS; echo

curl -sk -o /dev/null -w "%{http_code}\n" \
  -u "$SVC:$SVCPASS" https://<nsx-manager-fqdn>/api/v1/cluster/status

unset SVCPASS

…and got 403. Seconds after setting it. With a password I had just set moments earlier.

The Real Root Cause: Racing the Rotation Engine

This was the point where the investigation stopped being about policies and started being about timing. The sequence kept repeating:

As root on NSX: passwd the service account to a known value.
Test authentication with that value: 200 — for a moment.
Test again a few seconds later: 403.

Nothing was “wrong” with any password. SDDC Manager’s rotation machinery was re-changing the service account password within a couple of seconds of any reset. The disconnected credential kept the retry loop hot, every retry rotated the password on the appliance, and every manual test raced the engine and lost. The vault was not stale, the policy was not blocking anything anymore — my known-good password simply stopped being the current password almost immediately after I set it.

Watching the log in one terminal makes the race visible:

tail -f /var/log/vmware/vcf/operationsmanager/operationsmanager.log |
  grep -Ei 'svc-sddcmanager|ROTATE|REMEDIATE'

The fix is to stop fighting the engine and synchronize with it instead: set the password and hand it to SDDC Manager inside that couple-second window, so the value on the appliance and the value being remediated converge before the next rotation fires.

⚠️ What follows is a lab maneuver. Manually resetting a platform-managed service account and racing the rotation engine is recoverable here; on a production NSX Manager, capture operationsmanager.log, the failed task IDs, and the faillock output and open a support request first — see the last bullet in Hardening Takeaways.

In practice, that means staging everything in advance:

Terminal 1 — NSX Manager as root, with a non-interactive password set ready to run:

read -rsp "New service account password: " SVCPASS; echo
echo "$SVC:$SVCPASS" | chpasswd
unset SVCPASS

Terminal 2 — the same SDDC Manager session from earlier, so $SDDC, $TOKEN, and $SVC are still set, with the remediation payload pre-built — the only thing left is to press Enter:

read -rsp "Same password as set on NSX: " SVCPASS; echo

TASK_ID=$(jq -nc \
  --arg fqdn "<nsx-manager-fqdn>" \
  --arg user "$SVC" \
  --arg pw "$SVCPASS" \
  '{operationType:"REMEDIATE",
    elements:[{resourceName:$fqdn, resourceType:"NSXT_MANAGER",
      credentials:[{credentialType:"API", username:$user, password:$pw}]}]}' |
  curl -sk -X PATCH \
    -H "Authorization: Bearer $TOKEN" \
    -H 'Content-Type: application/json' -d @- \
    "$SDDC/v1/credentials" | jq -r .id)

unset SVCPASS

Run terminal 1, then terminal 2 immediately — within the window, before the engine rotates again. Poll the task:

curl -sk -H "Authorization: Bearer $TOKEN" \
  "$SDDC/v1/credentials/tasks/$TASK_ID" | jq '{name, status}'

This time TEST_BEFORE_REMEDIATE authenticated successfully — the password it tested was, at that moment, actually the password on the appliance — and the remediation completed. The credential flipped back to healthy, and the rotation loop stopped, because there was nothing left to retry.

To finish cleanly, trigger one supervised rotation so the account ends up on a fresh SDDC Manager-generated password (now capped at 100 characters by the lowered policy) instead of a hand-typed one:

jq -nc \
  --arg fqdn "<nsx-manager-fqdn>" \
  --arg user "$SVC" \
  '{operationType:"ROTATE",
    elements:[{resourceName:$fqdn, resourceType:"NSXT_MANAGER",
      credentials:[{credentialType:"API", username:$user}]}]}' |
curl -sk -X PATCH \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d @- \
  "$SDDC/v1/credentials" | jq .

Poll the task the same way as before; with the loop broken and the policy cap in place, the rotation completes cleanly.

Verification

The rotation task in SDDC Manager completes successfully, and no new ROTATE/REMEDIATE retries appear in operationsmanager.log for the service account.
The credential status is back to healthy (GET /v1/credentials shows no error state for the tuple).
faillock --user "$SVC" on the NSX appliance stays empty.
lockout_immune_addresses in GET /api/v1/cluster/api-service contains the SDDC Manager IP.
The NSX auth-policy holds the values you intend to keep — in my case maximum_password_length: 100 stays, since the 128/127 mismatch would otherwise come straight back on the next rotation.

Clean up the working files, since the backups describe your security posture and the lockout-immune IP list:

rm -f /tmp/auth-policy-backup.json /tmp/api-service-backup.json
unset NSXPASS

Hardening Takeaways

A few things I am keeping from this one, beyond the fix itself:

Add SDDC Manager to lockout_immune_addresses proactively. Any credential workflow that can fail more times than api_max_auth_failures allows can lock its own account and wedge remediation. KB 420736 describes the pattern.
Back up auth-policy before any PUT — tee to a file costs nothing, and the endpoint carries every knob (lockout periods, complexity, history) in one document, so an overzealous edit has a wide blast radius.
Check NTP while you are in the logs. This environment’s operationsmanager.log also showed intermittent JWT clock-skew errors between components — harmless-looking noise that can turn into real authentication failures. Time sync issues and credential troubleshooting mix badly; rule them out early.
Know when to stop and open an SR. Manually resetting a service account password as root and racing the rotation engine is recoverable in a lab. In production, the moment a platform-managed service account is in a rotation loop, capture operationsmanager.log, the failed task IDs, and the faillock output, and open a support request — KB 420736 describes the lockout pattern, and KB 423038 is a related reference worth attaching alongside it.

Final Notes

Every layer of this problem produced the same useless symptom — a 403 — for four different reasons: a password too long to remediate, a locked account, a pre-remediation test that authenticates as the account it is about to fix, and finally a rotation engine overwriting the password faster than I could test it. The 128-versus-127 mismatch set the loop in motion, but the thing that made it unfixable-looking was the timing: every diagnostic I ran was measuring a password that had already been replaced.

Two habits would have shortened this significantly. First, tail -f the operations manager log before forming a theory — the rotation attempts were right there, timestamped, showing the engine acting seconds after every manual change. Second, when a platform manages an account, assume it will keep managing it while you troubleshoot: either pause the machinery or synchronize with it. Fighting an automated rotation engine with manual password resets is a race you lose every time — until you make the race the plan.

The Symptom: A Service Account Stuck Disconnected#

The Off-by-One: 128 Generated vs 127 Accepted#

Dead Ends Worth Knowing About#

Remediating from the UI#

Guessing the Credential Tuple#

Changing the VCF Operations Password Policies#

The First Fix: Lower the NSX Maximum Password Length#

Plot Twist: TEST_BEFORE_REMEDIATE#

The Hidden User: An API-Only PAM Account#

Keeping the Account from Locking Again#

The Real Root Cause: Racing the Rotation Engine#

Verification#

Hardening Takeaways#

Final Notes#

Related posts