One-Host Supervisor on VCF 9.1, Part 4: Enabling a Supervisor with One Control Plane VM

In Part 3 we gave NSX a way out without edges. Now we enable the Supervisor on the one-host cluster, with a single control plane VM instead of the usual three. Two things stalled it along the way, and I headed off a third before it could. I’ll go through all of them, then prove the result with a test pod.

Before you begin

• vSphere HA and DRS on, HA admission control off (Part 2)
• The FTT=0 policy compatible with the vSAN datastore (Part 2)
• Five consecutive free IPs on the management network for the Supervisor (mine: 10.1.1.85 to 10.1.1.89)
• DNS servers that answer from the address they were queried on (more on that below)

Then ask vCenter whether the cluster can take a Supervisor on VPC networking:

GET /api/vcenter/namespace-management/cluster-compatibility?network_provider=NSX_VPC

My cluster (domain-c44) came back compatible: true. Keep the network_provider filter: without it, vCenter lists edge cluster reasons that do not apply to a VPC Supervisor.

Step 1: Enable the Supervisor

POST /api/vcenter/namespace-management/supervisors/domain-c44?action=enable_on_compute_cluster

{
  "name": "supervisor-mgmt-a",
  "zone": "zone-mgmt-02a",
  "control_plane": {
    "count": 1,
    "size": "SMALL",
    "storage_policy": "<FTT=0 policy id>",
    "network": {
      "backing": { "backing": "NETWORK", "network": "<moref of mgmt-vds01-mgmt-02a>" },
      "ip_management": {
        "dhcp_enabled": false,
        "gateway_address": "10.1.1.1/24",
        "ip_assignments": [ { "assignee": "NODE", "ranges": [ { "address": "10.1.1.85", "count": 5 } ] } ]
      },
      "services": {
        "dns": { "servers": [ "10.1.1.1" ], "search_domains": [ "site-a.vcf.lab" ] },
        "ntp": { "servers": [ "10.1.1.1" ] }
      }
    }
  },
  "workloads": {
    "edge": { "provider": "NSX_VPC" },
    "network": {
      "network_type": "NSX_VPC",
      "nsx_vpc": {
        "nsx_project": "/orgs/default/projects/default",
        "vpc_connectivity_profile": "/orgs/default/projects/default/vpc-connectivity-profiles/default",
        "default_private_cidrs": [ { "address": "172.16.200.0", "prefix": "24" } ]
      },
      "ip_management": {
        "dhcp_enabled": false,
        "gateway_address": "",
        "ip_assignments": [ { "assignee": "SERVICE", "ranges": [ { "address": "10.96.0.0", "count": 512 } ] } ]
      },
      "services": {
        "dns": { "servers": [ "10.1.1.1" ], "search_domains": [ "site-a.vcf.lab" ] },
        "ntp": { "servers": [ "10.1.1.1" ] }
      }
    },
    "storage": {
      "ephemeral_storage_policy": "<FTT=0 policy id>",
      "image_storage_policy": "<FTT=0 policy id>"
    }
  }
}

A few notes on the spec:

• control_plane.count: 1 is the whole point: one control plane VM.
• The enable call creates the vSphere Zone named in zone.
• The management network is a portgroup on the new cluster’s VDS, from Part 2.
• The same FTT=0 policy covers the control plane, ephemeral and image storage.

Then watch it:

GET /api/vcenter/namespace-management/supervisors/summaries

You want config_status: RUNNING and kubernetes_status: READY.

What is normal while it configures

These showed up in my lab and went away by themselves. Don’t chase them on the first pass:

• CONFIGURING with ERROR or WARNING for a long stretch
• Need 2 net devices, but found only 1 net devices while the control plane VM was coming up
• signature verification result not found for the embedded velero and VKS (tkg.vsphere.vmware.com) services, retried until they passed

One message kept repeating and did not go away: WCP could not create the Kubernetes Service Content Library. Its log said Unable to connect to kubernetes content library <depot URL> because Error creating kubernetes subscribed library 400 Bad Request. I did not chase the cause. It did not block the Supervisor, but you need a working image library before you deploy any VKS cluster.

Stall 1: the snapshot from Part 2

About two hours after the enable, the control plane VM deployment failed and was cleaned up:

The System VM for solution <supervisor id> on cluster domain-c44 has not been deployed because of unexpected error

The Supervisor itself stayed in place, waiting to retry. The cause was the snapshot I took of the nested esx-05a VM before creating the cluster. A nested ESXi running on a snapshot writes through the snapshot’s delta disk, and on my physical host that was painfully slow:

Nested vSAN on esx-05a also kept losing access to its volume and getting it back every few minutes. The host events read Lost access to volume <uuid> ... due to connectivity issues, then Successfully restored access to volume <uuid> ... following connectivity issues. I deleted the snapshot, write latency dropped to 0 ms, and WCP picked the deployment up again on its own. No disable and re-enable needed.

This one only bites nested labs. A physical GPU host has no outer snapshot. If your host is nested, delete its snapshot as soon as Part 2 checks out.

Stall 2: the vLCM health gate

With the control plane VM up, the Supervisor started flipping between CONFIGURING and RUNNING. The API VIP was up and VKS and velero were running, but the cci-ns core service needed a controller pod running as a vSphere Pod on esx-05a, and Kubernetes had only one node: the control plane VM.

esx-05a was not a node because its spherelet agent was installed but not running. Every minute, WCP asked vLCM to finish setting it up (an apply on the cluster’s solutions-internal software), and every minute vLCM refused:

Health Check for 'esx-05a.site-a.vcf.lab' failed

That health check is vSAN health. On my nested NVMe disks it showed two yellow tests: nvmeonhcl (hardware compatibility) and vsanesacompliancecheck. In the lab I silenced the same seven checks Holodeck already silences on the 4-host cluster. In PowerCLI:

$cl = Get-Cluster -Name 'cluster-mgmt-02a'
$hs = Get-VsanView -Id 'VsanVcClusterHealthSystem-vsan-cluster-health-system'
$checks = 'nvmeonhcl', 'controllerfirmware', 'autohclupdate', 'controlleronhcl',
          'hcldiskclaimcheck', 'vsanesacompliancecheck', 'vsanesacompliancecloudcheck'
$hs.VsanHealthSetVsanClusterSilentChecks($cl.ExtensionData.MoRef, [string[]]$checks, $null)
$hs.VsanHealthGetVsanClusterSilentChecks($cl.ExtensionData.MoRef)

About two and a half minutes after the silence, WCP’s next retry went through. spherelet started, esx-05a joined as a node, and the Supervisor went to RUNNING / READY.

On hardware that is on the vSAN HCL, I would expect these tests to pass and nothing to need silencing. Check vSAN health on the new cluster before you enable the Supervisor. If a test is yellow or red, fix it, or at least understand it, before you reach for this.

The one I headed off: DNS answering from the wrong address

VPC workloads reach your DNS servers through SNAT on the VNAs. If a DNS server sits on a router with several addresses and listens on 0.0.0.0, it can answer from a different address than the one it was asked on, and the client drops the reply. That broke VKS DNS in an earlier build of my lab. The Technitium DNS server on the Holodeck router was answering from the wrong address.

This time I bound the DNS server to explicit addresses before Part 3. If your pods resolve nothing and DNS just times out, check this first.

Step 2: Prove it with a test pod

A RUNNING / READY status is not the same as a working path. To prove it, I ran one throwaway vSphere Pod that resolves a name and fetches an HTTPS page through the VPC.

Create a namespace:

POST /api/vcenter/namespaces/instances/v2

{ "namespace": "kit-smoke", "supervisor": "<supervisor id>", "storage_specs": [ { "policy": "<FTT=0 policy id>" } ] }

Then get onto the control plane VM. On the vCenter appliance, as root, /usr/lib/vmware-wcp/decryptK8Pwd.py prints the control plane IP and root password. This is a lab shortcut; normally you log in with the VCF CLI or kubectl vsphere login.

ssh root@10.1.1.85
export KUBECONFIG=/etc/kubernetes/admin.conf

kubectl -n kit-smoke run kit-smoke --image=docker.io/library/busybox:1.36 --restart=Never --command -- \
  sh -c "nslookup vc-mgmt-a.site-a.vcf.lab 10.1.1.1 && echo DNS_OK; wget -q -T 15 -O /dev/null https://www.google.com/ && echo EGRESS_OK"

# wait for the pod to finish before reading its output
kubectl -n kit-smoke wait pod/kit-smoke --for=jsonpath='{.status.phase}'=Succeeded --timeout=300s
kubectl -n kit-smoke logs kit-smoke

# the Supervisor API VIP, from the management network
curl -sk -o /dev/null -w '%{http_code}\n' https://10.1.7.197:6443/version

In my case:

• The pod reached Succeeded, and printed DNS_OK and EGRESS_OK. busybox wget also prints a note that it does not validate TLS certificates; that is expected.
• The VIP answered 401. That is the result we want: the API is reachable and asks for credentials.

The LoadBalancer services all got their addresses from the external IP block in Part 3:

Clean up with kubectl -n kit-smoke delete pod kit-smoke, then delete the namespace in vCenter.

Where we ended up

• supervisor-mgmt-a is RUNNING / READY, with one control plane VM
• Two Kubernetes nodes: the control plane VM and esx-05a
• VKS 3.6.1, velero and cci-ns running
• The API VIP on 10.1.7.197, served through the VNAs

Last, delete the SDDC Manager snapshot from Part 1 (and the nested host snapshot, if you have not already).

What’s next

Two things before the GPU work. First, a VKS image library, since the Supervisor could not create its own. Then VCF Private AI Services. William’s PAIS 3.0 post lists what it needs: a Supervisor on VPC networking, which we now have, the PAIS Supervisor Service, a Harbor registry for the models, and an NVIDIA GPU through DirectPath I/O or vGPU. His post is on 9.1.1, so check the interoperability matrix for your Supervisor version first.