While putting together the VCF Automation 9.1 API getting started post, I spent some time in the in-product API Help Center — log into the VM Apps tenant, click your username, API Help Center > Explore Automation APIs. The Swagger pages render fine, but the moment you try an actual call, it fails. The Servers box for the VCF Automation (Blueprint) API shows:

http://localhost:8600

Swagger UI builds Try It Out requests from servers[0].url, so the browser dutifully sends every request to port 8600 on your own workstation. Connection refused, every time. The documentation is correct; the advertised host is not.

This post is about finding where that value actually comes from — which took three wrong turns through the appliance’s Kubernetes internals — and the override that finally fixed it without touching a container image. The short version:

  • The localhost:8600 value is not in any configuration on the appliance. A grep across a full support-bundle cluster dump finds no ConfigMap, Helm value, gateway route, or forwarded-headers setting that produces it.
  • The spec is a static JSON file generated by springdoc at build time — the giveaway is its "description": "Generated server url" — and it ships as a classpath resource inside the Spring Boot application in the tango-blueprint-service-app pod. Whoever built it had the service listening on port 8600, and the URL froze there.
  • The api-docs page itself is served by a different pod than the spec, which is why the obvious nginx-level fixes go nowhere.
  • The fix: mount a corrected copy of the spec from a ConfigMap and prepend file:/override/ to spring.web.resources.static-locations via JAVA_OPTS, so Spring serves the fixed file and never falls through to the copy shipped in the image. Survives pod restarts with nothing to reconcile it away; a product upgrade will wipe it.

Everything below was done on a lab appliance running build 9.1.0.0100.25429499. This is an unsupported modification — more on that at the end.

The Symptom

In the tenant API Help Center, open the VCF Automation API (the Blueprint one, version 2019-09-12). The spec renders, the paths are all correct, but the servers dropdown offers exactly one entry: http://localhost:8600. There is no way to type a different URL — the dropdown only lists what the spec declares, and this spec declares one hardcoded server with no variables.

Browser DevTools (Network tab) shows what the page actually loads. First an index of every API the Help Center knows about:

/tenant/<your-tenant>/api-docs/assets/content/soft/automation-apis.json

That index maps each tile to a spec URL — about sixteen specs across a dozen backend services. The Blueprint entry points at:

/blueprint/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json

And that document contains the problem, verbatim:

"servers": [
  {
    "url": "http://localhost:8600",
    "description": "Generated server url"
  }
]

Two details in that response are worth pausing on, because they shaped the whole investigation:

  • "Generated server url" is the literal default description springdoc attaches when it auto-computes the server URL from the incoming request. So springdoc generated this — but if it were computed per request, it would show the appliance FQDN, not localhost.
  • The response carries a last-modified date from months before I ever opened the page, plus accept-ranges: bytes. That is a static file being read off disk (or classpath), not a document rendered per request.

Put together: someone ran the Blueprint service on a build machine, where it listened on localhost:8600, exported the generated spec to a file, and shipped that file in the product. The URL froze at build time.

Ruling Out Configuration

Before touching anything, I wanted to know whether some appliance setting produces or rewrites that value — because if it did, that would be the supported place to fix it.

It does not. I had a full support bundle from this appliance handy (that is a story for another post), and the extracted cluster object dump is conclusive:

  • The string 8600 appears nowhere in the cluster as a port, URL, or config value. No Service exposes it, no container declares it, no ConfigMap mentions it. The only matches are UID substrings.
  • There is no springdoc, server.forward-headers-strategy, or server-URL configuration in any ConfigMap in the prelude or vmsp-platform namespaces.
  • The Envoy gateway routes (Gateway API HTTPRoute objects) that publish the api-docs paths perform path rewrites only — nothing rewrites hosts or response bodies.

So there is no knob. The value lives in content, not configuration, and only Broadcom can fix it properly. Everything from here on is about making the appliance serve a corrected copy.

Wrong Turn #1: The Pod That Serves the Page Doesn’t Serve the Spec

The api-docs page shell — the Angular app with the tiles — is served by the cloud-automation-ui-app pod in the prelude namespace. Its nginx config (ConfigMap cloud-automation-ui) rewrites /tenant/<org>/api-docs and /provider/api-docs into static content under /usr/share/nginx/html/dist. My first instinct was an nginx sub_filter there, rewriting localhost:8600 in responses.

Dead end, for a structural reason: the spec never passes through that pod. The gateway routes the two URLs to different backends entirely:

PathHTTPRouteBackend
/tenant/<org>/api-docs/... (the page, the index JSON)tenant-in-path-automation-uicloud-automation-ui
/blueprint/api-docs/... (the spec)blueprintapi-docstango-blueprint:8080, prefix rewritten to /api-docs/

A sub_filter in the UI pod would rewrite a response it never sees. The actual owner of the spec is the Blueprint service itself: Deployment/tango-blueprint-service-app, image blueprint-webapp:9.1.0.0100.25429499 — a Spring Boot app, which also explains the Spring Security headers on the spec response.

Wrong Turn #2: There Is No File to Edit

Fine — exec into the right pod and sed the file in place:

kubectl -n prelude exec deploy/tango-blueprint-service-app -- \
  sh -c 'find / -name "api-doc-public-classic-vra-*.json" -not -path "/proc/*" 2>/dev/null'

Nothing. And looking for the application itself:

kubectl -n prelude exec deploy/tango-blueprint-service-app -- \
  sh -c 'find / -maxdepth 3 -name "*.jar" -not -path "/proc/*" 2>/dev/null | head'

/opt/bc-fips/bcutil-fips-2.1.5.jar
/opt/bc-fips/bctls-fips-2.1.22.jar
/opt/bc-fips/bcpkix-fips-2.1.10.jar
/opt/bc-fips/bc-fips-2.1.2.jar
/opt/bc-fips/bcmail-fips-2.1.6.jar
/jdk/lib/jrt-fs.jar

Only the Bouncy Castle FIPS libraries and the JDK — the shallow search doesn’t even turn up the application archive itself. Wherever the application lives, the spec is packaged inside it as a classpath resource: find sees nothing, and sed has nothing to reach. Even if you dug out the archive, patched it, and re-packed it in the running container, the fix would be pointless: the JVM holds it open, so a restart is needed to pick up the change, and the restart is exactly what rebuilds the container filesystem from the pristine image. The fix would erase itself at the moment of application.

Wrong Turn #3: kubectl edit Ate My JAVA_OPTS

The mechanism that does work is Spring’s static resource resolution. Spring Boot serves static content from an ordered list of locations, and the first location that contains a matching file wins:

spring.web.resources.static-locations =
  classpath:/META-INF/resources/, classpath:/resources/, classpath:/static/, classpath:/public/

Setting that property replaces the default list. So if I prepend a filesystem directory and re-append the defaults, a file I control shadows the identically-pathed resource on the application classpath — and everything else keeps being served from the classpath exactly as before.

The path arithmetic works out neatly. After the gateway’s prefix rewrite, the backend sees the request as /api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json. With file:/override/ first in the list, Spring checks /override/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json before falling through to the classpath. That is exactly where a ConfigMap mount can put a corrected copy.

The wrong-turn part: my first attempt applied all of this with kubectl edit deploy. The volume and volumeMount saved; the JAVA_OPTS append silently did not. That env value is one enormous line that already contains $(SERVICE_ACCOUNT_SECRET) — literal Kubernetes variable-expansion syntax — and somewhere in the editor round-trip the appended text was lost; I never pinned down exactly where. (The same string is also a shell command-substitution landmine if you ever round-trip the value through a shell, which is why the fix below patches programmatically.) The pod rolled, the override directory appeared, and the spec still said localhost:8600 because the JVM never got the property. Verify the property reached the Deployment before concluding the approach failed.

The Fix

All commands run as root on the appliance, where kubectl is already configured. Replace vcfa.domain.com with your appliance FQDN.

1. Produce the corrected spec. No spelunking inside the image needed — pull it through the gateway and fix the one string:

curl -sk https://vcfa.domain.com/blueprint/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json \
  | sed 's|http://localhost:8600|https://vcfa.domain.com|g' \
  > /root/api-doc-public-classic-vra-2019-09-12.json

Sanity-check it — zero remaining hits, and still valid JSON:

grep -c localhost:8600 /root/api-doc-public-classic-vra-2019-09-12.json
python3 -m json.tool /root/api-doc-public-classic-vra-2019-09-12.json > /dev/null && echo "valid json"

2. Create the ConfigMap and back up the Deployment:

kubectl -n prelude create configmap blueprint-apidocs-override \
  --from-file=/root/api-doc-public-classic-vra-2019-09-12.json

kubectl -n prelude get deploy tango-blueprint-service-app -o yaml \
  > /root/tango-blueprint-deploy.backup.yaml

3. Add the volume and mount. A strategic merge patch merges named list entries, so this adds without disturbing the existing volumes and mounts:

kubectl -n prelude patch deploy tango-blueprint-service-app --type=strategic -p '
spec:
  template:
    spec:
      volumes:
      - name: apidocs-override
        configMap:
          name: blueprint-apidocs-override
      containers:
      - name: tango-blueprint-service-app
        volumeMounts:
        - name: apidocs-override
          mountPath: /override/api-docs/classic-apis
'

4. Append the properties to JAVA_OPTS — programmatically. This reads the current value, appends both the current (spring.web.resources.static-locations) and the pre-2.4 (spring.resources.static-locations) property names — whichever one this Spring Boot generation ignores does no harm — and writes a JSON patch. It refuses to double-apply:

kubectl -n prelude get deploy tango-blueprint-service-app -o json > /root/d.json

python3 - <<'EOF'
import json
d = json.load(open("/root/d.json"))
env = d["spec"]["template"]["spec"]["containers"][0]["env"]
i = [n for n, e in enumerate(env) if e["name"] == "JAVA_OPTS"][0]
loc = "file:/override/,classpath:/META-INF/resources/,classpath:/resources/,classpath:/static/,classpath:/public/"
add = " -Dspring.web.resources.static-locations=" + loc + " -Dspring.resources.static-locations=" + loc
if "static-locations" in env[i]["value"]:
    json.dump([], open("/root/patch.json", "w"))
    print("already present - nothing to do"); raise SystemExit
json.dump([{"op": "replace",
            "path": "/spec/template/spec/containers/0/env/%d/value" % i,
            "value": env[i]["value"] + add}],
          open("/root/patch.json", "w"))
print("patch written for env index", i)
EOF

kubectl -n prelude patch deploy tango-blueprint-service-app --type=json --patch-file=/root/patch.json

5. Wait for the rollout. The Deployment uses the Recreate strategy with a single replica, so the Blueprint service is briefly down while the pod is replaced:

kubectl -n prelude rollout status deploy/tango-blueprint-service-app --timeout=180s

Verification

Learned from wrong turn #3: verify each layer, not just the end result.

The property is on the Deployment:

kubectl -n prelude get deploy tango-blueprint-service-app \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="JAVA_OPTS")].value}' \
  | tr ' ' '\n' | grep -i static

-Dspring.web.resources.static-locations=file:/override/,classpath:/META-INF/resources/,...
-Dspring.resources.static-locations=file:/override/,classpath:/META-INF/resources/,...

The override file is mounted in the new pod:

kubectl -n prelude exec deploy/tango-blueprint-service-app -c tango-blueprint-service-app -- \
  ls -la /override/api-docs/classic-apis/

And the actual test — the spec as served through the gateway:

curl -sk https://vcfa.domain.com/blueprint/api-docs/classic-apis/api-doc-public-classic-vra-2019-09-12.json \
  | grep -A2 '"servers"'

  "servers": [
    {
      "url": "https://vcfa.domain.com",

Hard-reload the API Help Center page (the SPA caches aggressively) and the Servers box shows the appliance FQDN. Try It Out now sends requests where they belong.

What This Does and Does Not Fix

Worth being precise here, because the Help Center is bigger than one spec:

  • Fixed: the 2019-09-12 Blueprint spec shown above. Its 2019-01-15 sibling from the same /blueprint/api-docs/classic-apis/ directory — if it shows the same URL — is the same fix: repeat step 1 for it, then rebuild the ConfigMap with both files: kubectl -n prelude create configmap blueprint-apidocs-override --from-file=/root/api-doc-public-classic-vra-2019-09-12.json --from-file=/root/api-doc-public-classic-vra-2019-01-15.json --dry-run=client -o yaml | kubectl -n prelude replace -f -. The mounted volume picks up the change on its own within a minute or so; no restart needed.
  • Same pattern, different pod: the catalog, deployment, and policies tiles point at api-doc-public-classic-vra-*.yaml files under /catalog/api-docs/classic-apis/ — the same baked-file shape, owned by the catalog service. If those show the same URL, it is the same fix against that Deployment.
  • Possibly not broken at all: several tiles point at live springdoc endpoints (/identity/doc/v3/api-docs, /iaas-api/swagger/v3/api-docs/..., /project-service/api/swagger/v3/api-docs/...). Those generate their spec per request, so check before assuming. A quick curl ... | jq .servers against each URL from the index tells you which ones need attention.

Caveats

This is an unsupported modification to a Broadcom appliance, and it behaves accordingly:

  • It survives pod restarts, and nothing reconciles it away. The fix lives in the Deployment spec, so every pod the Deployment creates from now on gets the mount and the property — the verified state above is itself a freshly recreated pod. The tango-blueprint-service Helm release is deployed once at install/upgrade time with nothing reconciling it continuously, so it should come back intact after an appliance reboot as well; re-check the spec once after your next reboot to confirm. (Do not assume this generalizes: several other components in the same namespace are Flux-managed and would revert an edit within minutes.)
  • A product upgrade will wipe it. The LCM re-renders the Deployment from the new chart. Expect to re-apply, and re-check after any lifecycle operation.
  • Roll back at any time with the backup from step 2. Not with kubectl apply, though — the backup carries the pre-patch resourceVersion (so the apply is rejected as a conflict), and client-side apply against a Helm-created object has no last-applied-configuration to prune from, so the added volume would survive the merge and the next pod would wedge on the then-deleted ConfigMap. kubectl replace does a full PUT and removes the volume, mount, and JAVA_OPTS additions in one shot; strip the resourceVersion so the PUT is unconditional, and delete the ConfigMap only after the Deployment no longer references it:
sed '/^  resourceVersion:/d' /root/tango-blueprint-deploy.backup.yaml \
  | kubectl -n prelude replace -f -
kubectl -n prelude rollout status deploy/tango-blueprint-service-app --timeout=180s
kubectl -n prelude delete configmap blueprint-apidocs-override
  • File an SR anyway. As of this writing there is no KB and nothing in the 9.1 release notes for this — the proper fix has to ship in the product, and that only happens if it gets reported. Reference the build number and the "Generated server url" value in the spec.

If you only need a working Try It Out for yourself and would rather not touch the appliance at all, there is a zero-footprint alternative: download the spec, fix servers[0].url in the copy, and import it into Postman — it seeds the collection’s baseUrl from the spec, and auth works as described in the getting started post.

Final Notes

The frustrating part of this bug is how little is actually wrong: every path in the spec is correct, authentication is correct, the API itself works. One string, frozen at build time on some build machine where the Blueprint service happened to listen on port 8600, breaks the one feature the page exists to provide.

The investigation pattern is the reusable part. When a value appears in a rendered page and nothing in the configuration produces it, stop grepping ConfigMaps and follow the bytes instead: DevTools tells you the exact URL, the response headers tell you whether it is static or generated, and the gateway routes tell you which pod actually owns it. In this case each of those three steps invalidated an otherwise-reasonable fix — the nginx rewrite targeted the wrong pod, the in-pod sed targeted a file that does not exist, and the editor-based patch failed silently — before the resource-shadowing approach landed on the one layer where the platform genuinely offers an override.

References