Fixing a VCF Operations 9.1 Express Patch Stuck at Installing PAK File: An Empty /etc/yum.repos.d and Four Layers of Upgrade State

The 9.1.0.0300 patch went through this lab without drama at the beginning of July. Three weeks later, the 9.1.0.0400 express patch for VCF Operations hit the same single-node appliance and stopped dead at Installing PAK file — no progress, no error, and a reboot later, still nothing. What followed was a proper archaeology dig through every layer of state the upgrade machinery keeps, and almost every layer turned out to be holding a different version of the truth.

The short version:

• The upgrade died two seconds into the OS update phase because the glob over /etc/yum.repos.d/* matched nothing — the directory had been emptied (or removed; the error is identical either way) — and sed exiting 2 on it is fatal to updateCoordinator.py.
• The failure was recorded in pakManager’s results file, and from then on a retry guard rejected every new attempt in milliseconds — while CaSA kept retrying roughly every 11–12 seconds, over 1,100 times in the retained logs alone.
• The UI said Installing the whole time because CaSA’s own pak history had the APPLY_OS_UPDATE operation with a null completed_time — stale state, not activity.
• Rebooting did not retry anything; it just rebuilt the same wedge, with the cluster still offline because offline is persisted state.
• The recovery attempts then failed in two brand-new ways: a 409 pak_installation_active raised when the precheck’s state check collided with the assessment install the precheck itself had just launched, and a NullPointerException when LCM asked CaSA for an upgrade task that no longer existed anywhere.

Versions in play: VCF Operations 9.1.0.0300.25531073 upgrading to 9.1.0.0400.25541550, on a single-node deployment, driven by the express patch workflow in fleet management. All timestamps below are UTC, straight from the logs.

The Symptom: Installing PAK File, Forever

The express patch staged cleanly and started applying, then sat at Installing PAK file for hours. A reboot later it still showed the same thing. Time to stop watching the progress bar and go look at what the appliance was actually doing.

The places that matter for any PAK-based upgrade on a VCF Operations node:

The PAK for this patch is vRealizeOperationsManagerEnterprise-910040025541550 — the ID is the version string with the dots removed, 9.1.0.0400.25541550.

Finding the Loop

The first thing that stood out was not an error message but a rotation pattern. vcopsPakManager.root.apply_os_update.log keeps twenty rotated files of 1 MB each, and every one of them was less than four hours old — the log was burning a full rotation roughly every 11 minutes. Something was running apply_os_update constantly.

Each attempt ended the same way, in milliseconds (log prefixes trimmed for width here and throughout):

[VerifyActionNotRunningOrRun] - apply_os_update result: "Failed"
[VerifyActionNotRunningOrRun] - apply_os_update exit code: "1"
[HandleError] - Exiting with exit code: 1, message: The action: apply_os_update
on the pak file: "vRealizeOperationsManagerEnterprise-910040025541550" previously
failed, cleanup the system and remove_history before attempting again--exiting

That is pakManager’s retry guard: once an action has Failed in the PAK’s .results file, every subsequent invocation is rejected before any script runs. And CaSA — the admin/cluster service that orchestrates PAK operations — was invoking it again roughly every 11–12 seconds, each time via:

/usr/bin/sudo -n /usr/lib/vmware-python-3/bin/python
  /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py
  --action apply_os_update --pak vRealizeOperationsManagerEnterprise-910040025541550
  --json --force_content_update true --roles ADMIN,DATA,UI

The retained logs held 1,125 rejections spanning just under four hours — and CaSA’s logs show the loop reaching back to the previous evening. An install loop where every iteration is refused on arrival: the upgrade could not move forward and would never stop trying. So the interesting question became: what failed the first time?

The Root Cause: One sed Against an Empty Directory

CaSA’s operation history gives the setup, and the per-PAK stderr log — /storage/log/vcops/log/pakManager/vRealizeOperationsManagerEnterprise-910040025541550/apply_os_update_stderr.log — gives the kill. The timeline on July 28:

• 17:42:27 → 17:42:33 — validation runs and passes
• 17:42:45 → 17:44:29 — BRING_CLUSTER_OFFLINE completes
• 17:44:37 — APPLY_OS_UPDATE starts
• 17:44:39 — the very first command of the VA upgrade phase runs, and dies:

[b2b - updateCoordinator - os_upgrade] - Entering VA upgrade phase
[b2b - updateCoordinator - os_upgrade] - Disable existing repositories
[b2b - updateCoordinator - runScript] - Script command:
    "/usr/bin/sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/*"
[b2b - updateCoordinator - runScript] - stderr:
    "/usr/bin/sed: can't read /etc/yum.repos.d/*: No such file or directory"
[b2b - updateCoordinator - runScript] - exit code: 2
[b2b - updateCoordinator - programExit] - Unable to disable existing repositories for updates.

The glob over /etc/yum.repos.d/* matched nothing — the shell passed the literal string through, sed could not open it, and exited 2. That error is byte-identical whether the directory is empty or missing outright; by the time anyone could look, the files were back, so all the log proves is that there was nothing there to match. updateCoordinator.py treats a non-zero exit here as fatal — reasonably, since it is about to drive tdnf package operations and wants external repositories disabled first.

Two details make this sting:

The upgrade does not need working external repos at all. The very next steps in a healthy run copy the PAK’s own local-pak-iso.repo into that directory and install everything from the ISO embedded in the PAK. The sed only needs the glob to match at least one file. A directory with nothing in it is the one state it cannot survive.

This exact command succeeded three weeks earlier. The 0300 patch on July 5 logged the identical sed with an empty stderr and exit 0, then dropped local-pak-iso.repo in place and carried on. So the directory was populated on July 5 and held nothing the glob could match by July 28 — something removed those files in between. More on the suspects at the end.

One more log-reading trap: the failure looks like it took ten minutes, because after the fatal error updateCoordinator.py spends from 17:44:39 to 17:54:47 generating a support bundle before it exits. The Failed result lands in the state files at 17:54:47, ten minutes after the two-second failure. If you correlate timestamps across logs, that gap is the tool being helpful, not the upgrade doing work.

Why It Looked Stuck Instead of Failed

Here is the part that turned a two-second failure into a day of confusion. The upgrade machinery keeps (at least) four separate stores of state, and they disagreed:

The UI renders from CaSA’s view. Its per-PAK history file ended like this:

{
    "operation": "APPLY_OS_UPDATE",
    "start_time": 1785260677162,
    "completed_time": null
}

An operation with a start time and a null completion is, as far as the UI is concerned, still installing. pakManager knew the upgrade had failed; CaSA never wrote a terminal result for the operation. Both were internally consistent. Neither was lying. They just were not the same state machine.

Why Rebooting Did Not Help

BRING_CLUSTER_OFFLINE had completed before the failure, and offline is persisted admin state. After the reboot, the platform came up and refused to start its services on purpose — from the support bundle’s sysenv/status.txt, captured while it was down:

vmware-vcops.service - LSB: vRealize Operations Platform
   Active: failed (Result: exit-code)
     vpostgres Cannot be started as the slice is offline
     analytics Cannot be started as the slice is offline
     collector Cannot be started as the slice is offline
     api       Cannot be started as the slice is offline

So the reboot traded a stuck upgrade on a running cluster for a stuck upgrade on a down cluster. One genuinely useful side effect: the retry loop did not survive the restart. CaSA’s orchestrator documents came back empty — current_pak_activity all nulls from the first post-reboot poll onward — so the hammering stopped; the last apply_os_update attempt in the logs is from just before the reboot. But nothing resumed, either. The upgrade was now fully inert: no process running, no retries, UI still saying Installing, and monitoring down with it.

Bringing the cluster back online restored the data-collection side of the house — services active, slice ONLINE, still on 9.1.0.0300 — while the upgrade state stayed exactly as wedged as before. Stale state does not age out. Waiting was never going to fix this.

Clearing the PAK State: CLI Archaeology

The guard message says what it wants: “cleanup the system and remove_history before attempting again.” Turning that sentence into working commands took several tries, and the failed attempts are worth documenting because the CLI’s own error messages actively mislead.

First lesson: run it with python3. The script does not behave as a direct executable — invoked by path, the shell tries to interpret Python as shell (abridged, full path shortened):

vcopsPakManager.py: line 6: from: command not found
vcopsPakManager.py: line 10: import: command not found
vcopsPakManager.py: line 31: syntax error near unexpected token `('

python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py

Run bare like that, it prints its full usage — which is where the trap sits. The guard message names remove_history as if it were an action. It is not. It is a parameter of the cleanup action:

cleanup - remove temp and working files
  params:
    --remove_history (ACTION|all) - remove the history for the action.  The parameter
                                    is the action, or "all" to remove all history
    --remove_pak - removes the pak file and all history
    --skip_history_update - removes the pak without updating history

The tour of error messages on the way there:

--action remove_history ...                      → Action: "remove_history" is not valid
--action cleanup --remove_history --pak <pak>    → Action: --pak is not supported for remove_history
--action cleanup --remove_history                → option --remove_history requires argument
--action cleanup --remove_pak <pak>              → No PAK file specified

The second one is the nasty one: --remove_history requires an argument, so it silently consumed --pak as its value and then complained that --pak is not a valid action name. And the fourth shows --remove_pak is a bare flag, not an option that takes the PAK name — the PAK always goes in --pak. The forms that worked:

python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py \
  --action cleanup --pak vRealizeOperationsManagerEnterprise-910040025541550

python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py \
  --action cleanup --remove_history all --pak vRealizeOperationsManagerEnterprise-910040025541550

And one final surprise from the results file afterwards: --remove_history all flipped a dozen keys to "history removed" — but not the ones that mattered:

apply_adapter_result:        history removed
apply_system_update_result:  history removed
new_validate_result:         history removed
...
validate_result:             Failed        ← survived
apply_os_update_result:      Failed        ← survived
apply_os_update_exit_code:   1             ← survived

Several statuses outlived the removal — including, of all things, the apply_os_update Failed/exit-1 pair that is exactly what the retry guard checks. Keep that in mind for the endgame below.

While in there, this is also the moment to fix the actual root cause — put repo definitions back in /etc/yum.repos.d/. A snapshot, a sibling appliance, or the Photon base image all work as a source; strictly speaking the upgrade only needs the glob to match one file, but restoring the stock set is the right fix. Healthy looks like this:

local-pak-iso.repo     photon-extras.repo   photon-release.repo  photon-srpms.repo
photon-debuginfo.repo  photon-iso.repo      photon.repo          photon-updates.repo

The Precheck 409: Colliding With Your Own Assessment

With pakManager’s state cleared and the repo directory restored, the next upgrade precheck failed after about a minute with:

An unexpected error occurred in step ops_precheck. Reference Code: EEF54002.
409: upgrade.verification.pak_installation_active
PreConditionFailedCasaException: key=upgrade.verification.pak_installation_active

My first theory was the obvious one — that null-completed_time entry still sitting in CaSA’s history. The CaSA log killed that theory and replaced it with something stranger. The precheck works by staging and installing an assessment PAK (APUAT-for-814x) on the appliance, and the sequence around the failure reads:

19:03:20  verifyConsistentState PAK_INSTALL → Cluster is in consistent state
19:03:21  APUAT-for-814x staged; validation, then apply_system_update begins
19:04:01  OrchestratorStatusDocument: current_pak_activity =
            {"pak_id":"APUAT-for-814x-910025541563","operation_name":"APPLY_SYSTEM_UPDATE","step":12,...}
19:04:01  verifyConsistentState PAK_INSTALL — again
19:04:02  Cluster state check failed: PreConditionFailedCasaException:
            key=upgrade.verification.pak_installation_active

The same consistency check passed at 19:03:20 with no PAK active, and failed 41 seconds later because a PAK installation was genuinely active — the assessment install the precheck itself had just launched. The stale enterprise-PAK entry was nowhere in it; CaSA was telling the truth about a collision between two halves of the same workflow. Meanwhile the assessment PAK, indifferent to the LCM step failing above it, ran to a clean finish in under twelve minutes — one more case of the layers not sharing a reality.

Two useful tools on the CaSA side regardless. Its deployment state is queryable directly — and note it reported "DONE" while the 409 was still reproducible, so treat it as one more opinion, not an oracle:

curl -k -u admin https://localhost/casa/deployment/cluster/installationstate

And the non-destructive reset is restarting the service:

systemctl restart vmware-casa

I restarted CaSA and re-ran the precheck; this run went clean end to end — the assessment PAK staged, validated, installed, and cleaned up in thirteen minutes, and the step reported success. Whether the restart mattered or the retry simply avoided the earlier collision, I cannot prove from the logs — what the evidence does show is that the null-completion history entry was still in place, untouched, during the passing run. It was never the thing the 409 keyed on.

The NullPointerException: Asking an Empty Orchestrator for a Task

Precheck green, one more attempt at the actual upgrade — and a new reference code:

An unexpected error occurred in step ops_apply_upgrade. Reference Code: 90AD4093.
Detail: Cannot invoke "com.broadcom.vcf.lcm.sddc.upgrade.sdk.client.appliance.model.Task.getId()"
because "task" is null

An NPE in the LCM client is a product bug in the sense that it hides the real condition — but the underlying state is easy to read. CaSA’s orchestrator, logged on every status poll, was completely empty:

UpdateOrchestratorDocument={"casa_webapp_pak_id":null,"casa_webapp_force_content_update":false,
  "casa_pak_status":{}}
OrchestratorStatusDocument: {"current_pak_activity":{"pak_id":null,"operation_name":null,
  "step":0,"total_steps":0,...},...,"queued_pak_ids":[],"is_upgrade_orchestrator_active":false,...}

And the PAK’s working directory was down to two files:

/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-910040025541550/
├── manifest.txt
└── vRealizeOperationsManagerEnterprise-910040025541550.results   ← still says Failed

Two separate subtractions added up to this. The reboot had emptied CaSA’s runtime documents and queue — that happened back at 12:43, and it is what stopped the retry loop. The cleanup action had then removed the extracted PAK content from pakRepoLocal — which is its job. Between the two, nothing on the appliance still described an in-flight upgrade, so when LCM told CaSA to proceed with the apply step, there was no registered PAK, no queued operation, and nothing to build a task from. CaSA returned no task; the LCM client dereferenced it; NPE. The pendulum had swung all the way from state says failed, refuse to run to no state at all, nothing to run.

The original binary survives, though — cleanup did not touch the staged PAK repository:

locate .pak | grep 0400
/storage/db/casa/pak/dist_pak_files/VA_LINUX/vRealizeOperationsManagerEnterprise-910040025541550.pak
(plus the 0400 license-server and cloud-proxy PAKs, trimmed)

The Clean Path Out

⚠️ What follows is the planned remediation from this point, not a procedure I have verified end to end — which is exactly why step 1 exists. It is a lab; on a production deployment, this is the moment to take the reference codes (EEF54002, 90AD4093) and the support bundle to an SR instead of editing state by hand.

At this point the environment is: cluster online and collecting, on 9.1.0.0300; repo directory restored; precheck passing; PAK binary staged; and two pieces of stale state left — the Failed survivor keys in the results file and the null-completion entry in CaSA’s per-PAK history. The way out is to stop half-clearing and reset the PAK completely, then let the workflow rebuild everything from the binary:

1. Snapshot the appliance. The cluster is healthy right now; protect that before another attempt.

2. Remove the PAK and all of its history. Per the usage text, --remove_pak “removes the pak file and all history” — note this is the one cleanup variant I have not exercised, and given that --remove_history all left the Failed keys standing, do not take the usage text on faith:

   python3 /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py \
     --action cleanup --remove_pak --pak vRealizeOperationsManagerEnterprise-910040025541550
   

   Afterwards, verify the pakRepoLocal directory (with its .results file) is actually gone, and that the staged binary in /storage/db/casa/pak/dist_pak_files/VA_LINUX/ is still there — if --remove_pak reaches into the staged repository too, the workflow will need to re-download the PAK, which is inconvenient rather than fatal.

3. Clear CaSA’s per-PAK history so the UI stops rendering the ghost Installing operation and nothing can ever mistake it for an active install again — with the service stopped, move the stale file aside:

   systemctl stop vmware-casa
   mv /storage/log/vcops/log/casa/casa_pak_history_vRealizeOperationsManagerEnterprise-910040025541550.json /root/
   systemctl start vmware-casa
   

4. Re-run the express patch from the beginning, not as a resume. With the staged .pak in place there is nothing to re-download; the workflow re-stages, re-validates, and — with /etc/yum.repos.d/ populated — should get past the two-second mark it never survived before. Expect the cluster to go offline again for the apply; that is the designed behavior (bring_cluster_offline: true in the manifest), so schedule it.

Verification

What success looks like once the re-run goes through:

• The appliance reports 9.1.0.0400.25541550 and the express patch workflow completes without a reference code.
• curl -k -u admin https://localhost/casa/deployment/cluster/installationstate returns "DONE" — with no failed step behind it this time.
• The new casa_pak_history_vRealizeOperationsManagerEnterprise-910040025541550.json shows every operation, APPLY_OS_UPDATE included, with a non-null completed_time.
• The cluster slice is ONLINE and collecting, and no apply_os_update invocations appear in the pakManager logs after completion — the loop stays dead.
• /etc/yum.repos.d/ is still populated. If it is empty again, find the culprit before the next patch does this all over.

What Actually Emptied /etc/yum.repos.d?

The unresolved thread. The directory was demonstrably populated on July 5 (the 0300 patch’s sed succeeded and copied local-pak-iso.repo into it) and demonstrably held nothing the glob could match on July 28. Neither observed upgrade run removed those files, and the PAK cleanup actions do not reach into /etc. Something else did, in the three-week window between patches.

The usual suspects are STIG/hardening remediation, a compliance scan “fixing” findings, or an overzealous cleanup script — and the hardening theory is less hypothetical than it sounds: the appliance ships a STIG-hardening Ansible role that the upgrade itself runs during apply_system_update, and its task list includes a literal “Find yum repos” step. A framework that polices exactly this directory demonstrably lives on the box; the July runs used it non-destructively, but whatever emptied the directory operated on the same territory. I have not caught the culprit in the act, so the case stays open.

If that something is scheduled, it will do it again, and the next express patch will die the same two-second death. Before every PAK-based upgrade on a VCF Operations appliance, it now costs me one command to check:

ls /etc/yum.repos.d/

Empty directory, no upgrade. It is that direct.

Takeaways

• A stuck progress bar is a claim, not a fact. The UI renders CaSA’s operation history; a null completed_time displays as Installing forever, even when the failure happened in two seconds and nothing has run since. The pakManager logs are where the truth lives.
• Know the four state stores — per-PAK results file, vcopsPakManagerCommonHistory.json, CaSA’s casa_pak_history_<pak>.json, and CaSA’s runtime orchestrator documents. Every confusing symptom in this saga was two of them disagreeing — including the 409, where the precheck disagreed with the assessment install it had launched itself.
• The retry guard message is honest but the CLI is not literal about it — remove_history is a parameter of --action cleanup, not an action, and a missing argument makes it eat the next flag with a misleading error.
• Verify what state-clearing commands actually cleared. --remove_history all reported nothing wrong while leaving the apply_os_update: Failed keys — the exact ones the guard checks — untouched. After any cleanup, read the .results file; do not assume.
• cleanup without the root cause fixed is a trap. The guard fires before the script runs; clear the state while /etc/yum.repos.d/ is still empty and the next attempt fails in two seconds and re-arms it — from an offline cluster.
• Reboots do not clear persisted upgrade state. Offline stays offline, Failed stays Failed, and the null completion stays null. What a reboot did do here is wipe CaSA’s runtime orchestrator — which killed the retry loop, made the wedge quieter and easier to misread as idle, and set up the eventual task-is-null NPE.
• Fatal-error support bundles skew timelines. Nearly all of the ten minutes this “upgrade” ran were generateSupportBundle.py executing after the fatal error. Read the stderr log inward from the first ERROR, not the timestamps outward.
• Check /etc/yum.repos.d/ before any PAK upgrade, and find out what is emptying it — whatever removed those files will break the next patch identically if it runs again.

Final Notes

Every stage of this one was the same lesson wearing different clothes: the upgrade machinery is several loosely-coupled state machines, and each error message was one machine reporting its own local truth. pakManager said previously failed; CaSA said still installing; the precheck said installation active — correctly, about the assessment install it had just started itself; LCM said task is null. Every statement was locally accurate, none of them described the system, and the actual root cause — a directory that a three-week-old patch run proves used to be populated — never appeared in any of them.

The two-second failure was trivial. The day was spent on state.