One-Host Supervisor on Cosmin.us

One-Host Supervisor on VCF 9.1, Part 4: Enabling a Supervisor with One Control Plane VM

Cosmin Trif — Wed, 21 Oct 2026 08:30:00 +0000

In Part 3 we gave NSX a way out without edges. Now we enable the Supervisor on the one-host cluster, with a single control plane VM instead of the usual three. Two things stalled it along the way, and I headed off a third before it could. I’ll go through all of them, then prove the result with a test pod.

Before you begin

vSphere HA and DRS on, HA admission control off (Part 2)
The FTT=0 policy compatible with the vSAN datastore (Part 2)
Five consecutive free IPs on the management network for the Supervisor (mine: 10.1.1.85 to 10.1.1.89)
DNS servers that answer from the address they were queried on (more on that below)

Then ask vCenter whether the cluster can take a Supervisor on VPC networking:

GET /api/vcenter/namespace-management/cluster-compatibility?network_provider=NSX_VPC

My cluster (domain-c44) came back compatible: true. Keep the network_provider filter: without it, vCenter lists edge cluster reasons that do not apply to a VPC Supervisor.

Step 1: Enable the Supervisor

POST /api/vcenter/namespace-management/supervisors/domain-c44?action=enable_on_compute_cluster

{
 "name": "supervisor-mgmt-a",
 "zone": "zone-mgmt-02a",
 "control_plane": {
 "count": 1,
 "size": "SMALL",
 "storage_policy": "<FTT=0 policy id>",
 "network": {
 "backing": { "backing": "NETWORK", "network": "<moref of mgmt-vds01-mgmt-02a>" },
 "ip_management": {
 "dhcp_enabled": false,
 "gateway_address": "10.1.1.1/24",
 "ip_assignments": [ { "assignee": "NODE", "ranges": [ { "address": "10.1.1.85", "count": 5 } ] } ]
 },
 "services": {
 "dns": { "servers": [ "10.1.1.1" ], "search_domains": [ "site-a.vcf.lab" ] },
 "ntp": { "servers": [ "10.1.1.1" ] }
 }
 }
 },
 "workloads": {
 "edge": { "provider": "NSX_VPC" },
 "network": {
 "network_type": "NSX_VPC",
 "nsx_vpc": {
 "nsx_project": "/orgs/default/projects/default",
 "vpc_connectivity_profile": "/orgs/default/projects/default/vpc-connectivity-profiles/default",
 "default_private_cidrs": [ { "address": "172.16.200.0", "prefix": "24" } ]
 },
 "ip_management": {
 "dhcp_enabled": false,
 "gateway_address": "",
 "ip_assignments": [ { "assignee": "SERVICE", "ranges": [ { "address": "10.96.0.0", "count": 512 } ] } ]
 },
 "services": {
 "dns": { "servers": [ "10.1.1.1" ], "search_domains": [ "site-a.vcf.lab" ] },
 "ntp": { "servers": [ "10.1.1.1" ] }
 }
 },
 "storage": {
 "ephemeral_storage_policy": "<FTT=0 policy id>",
 "image_storage_policy": "<FTT=0 policy id>"
 }
 }
}

A few notes on the spec:

control_plane.count: 1 is the whole point: one control plane VM.
The enable call creates the vSphere Zone named in zone.
The management network is a portgroup on the new cluster’s VDS, from Part 2.
The same FTT=0 policy covers the control plane, ephemeral and image storage.

Then watch it:

GET /api/vcenter/namespace-management/supervisors/summaries

You want config_status: RUNNING and kubernetes_status: READY.

What is normal while it configures

These showed up in my lab and went away by themselves. Don’t chase them on the first pass:

CONFIGURING with ERROR or WARNING for a long stretch
Need 2 net devices, but found only 1 net devices while the control plane VM was coming up
signature verification result not found for the embedded velero and VKS (tkg.vsphere.vmware.com) services, retried until they passed

One message kept repeating and did not go away: WCP could not create the Kubernetes Service Content Library. Its log said Unable to connect to kubernetes content library <depot URL> because Error creating kubernetes subscribed library 400 Bad Request. I did not chase the cause. It did not block the Supervisor, but you need a working image library before you deploy any VKS cluster.

Stall 1: the snapshot from Part 2

About two hours after the enable, the control plane VM deployment failed and was cleaned up:

The System VM for solution <supervisor id> on cluster domain-c44 has not been deployed because of unexpected error

The Supervisor itself stayed in place, waiting to retry. The cause was the snapshot I took of the nested esx-05a VM before creating the cluster. A nested ESXi running on a snapshot writes through the snapshot’s delta disk, and on my physical host that was painfully slow:

Nested host	Write latency	Write IOPS
`esx-05a`, on the snapshot	172 ms average, 814 ms peak	about 20
the other nested hosts, no snapshot	0 ms	250 to 340

Nested vSAN on esx-05a also kept losing access to its volume and getting it back every few minutes. The host events read Lost access to volume <uuid> ... due to connectivity issues, then Successfully restored access to volume <uuid> ... following connectivity issues. I deleted the snapshot, write latency dropped to 0 ms, and WCP picked the deployment up again on its own. No disable and re-enable needed.

This one only bites nested labs. A physical GPU host has no outer snapshot. If your host is nested, delete its snapshot as soon as Part 2 checks out.

Stall 2: the vLCM health gate

With the control plane VM up, the Supervisor started flipping between CONFIGURING and RUNNING. The API VIP was up and VKS and velero were running, but the cci-ns core service needed a controller pod running as a vSphere Pod on esx-05a, and Kubernetes had only one node: the control plane VM.

esx-05a was not a node because its spherelet agent was installed but not running. Every minute, WCP asked vLCM to finish setting it up (an apply on the cluster’s solutions-internal software), and every minute vLCM refused:

Health Check for 'esx-05a.site-a.vcf.lab' failed

That health check is vSAN health. On my nested NVMe disks it showed two yellow tests: nvmeonhcl (hardware compatibility) and vsanesacompliancecheck. In the lab I silenced the same seven checks Holodeck already silences on the 4-host cluster. In PowerCLI:

$cl = Get-Cluster -Name 'cluster-mgmt-02a'
$hs = Get-VsanView -Id 'VsanVcClusterHealthSystem-vsan-cluster-health-system'
$checks = 'nvmeonhcl', 'controllerfirmware', 'autohclupdate', 'controlleronhcl',
 'hcldiskclaimcheck', 'vsanesacompliancecheck', 'vsanesacompliancecloudcheck'
$hs.VsanHealthSetVsanClusterSilentChecks($cl.ExtensionData.MoRef, [string[]]$checks, $null)
$hs.VsanHealthGetVsanClusterSilentChecks($cl.ExtensionData.MoRef)

About two and a half minutes after the silence, WCP’s next retry went through. spherelet started, esx-05a joined as a node, and the Supervisor went to RUNNING / READY.

On hardware that is on the vSAN HCL, I would expect these tests to pass and nothing to need silencing. Check vSAN health on the new cluster before you enable the Supervisor. If a test is yellow or red, fix it, or at least understand it, before you reach for this.

The one I headed off: DNS answering from the wrong address

VPC workloads reach your DNS servers through SNAT on the VNAs. If a DNS server sits on a router with several addresses and listens on 0.0.0.0, it can answer from a different address than the one it was asked on, and the client drops the reply. That broke VKS DNS in an earlier build of my lab. The Technitium DNS server on the Holodeck router was answering from the wrong address.

This time I bound the DNS server to explicit addresses before Part 3. If your pods resolve nothing and DNS just times out, check this first.

Step 2: Prove it with a test pod

A RUNNING / READY status is not the same as a working path. To prove it, I ran one throwaway vSphere Pod that resolves a name and fetches an HTTPS page through the VPC.

Create a namespace:

POST /api/vcenter/namespaces/instances/v2

{ "namespace": "kit-smoke", "supervisor": "<supervisor id>", "storage_specs": [ { "policy": "<FTT=0 policy id>" } ] }

Then get onto the control plane VM. On the vCenter appliance, as root, /usr/lib/vmware-wcp/decryptK8Pwd.py prints the control plane IP and root password. This is a lab shortcut; normally you log in with the VCF CLI or kubectl vsphere login.

ssh root@10.1.1.85
export KUBECONFIG=/etc/kubernetes/admin.conf

kubectl -n kit-smoke run kit-smoke --image=docker.io/library/busybox:1.36 --restart=Never --command -- \
 sh -c "nslookup vc-mgmt-a.site-a.vcf.lab 10.1.1.1 && echo DNS_OK; wget -q -T 15 -O /dev/null https://www.google.com/ && echo EGRESS_OK"

# wait for the pod to finish before reading its output
kubectl -n kit-smoke wait pod/kit-smoke --for=jsonpath='{.status.phase}'=Succeeded --timeout=300s
kubectl -n kit-smoke logs kit-smoke

# the Supervisor API VIP, from the management network
curl -sk -o /dev/null -w '%{http_code}\n' https://10.1.7.197:6443/version

In my case:

The pod reached Succeeded, and printed DNS_OK and EGRESS_OK. busybox wget also prints a note that it does not validate TLS certificates; that is expected.
The VIP answered 401. That is the result we want: the API is reachable and asks for credentials.

The LoadBalancer services all got their addresses from the external IP block in Part 3:

Service	Address
`kube-apiserver-lb-svc` (the Supervisor API VIP)	`10.1.7.197`
`kube-dns-lb`	`10.1.7.193`
`docker-registry`	`10.1.7.194`
`mgmt-image-proxy`	`10.1.7.196`

Clean up with kubectl -n kit-smoke delete pod kit-smoke, then delete the namespace in vCenter.

Where we ended up

supervisor-mgmt-a is RUNNING / READY, with one control plane VM
Two Kubernetes nodes: the control plane VM and esx-05a
VKS 3.6.1, velero and cci-ns running
The API VIP on 10.1.7.197, served through the VNAs

Last, delete the SDDC Manager snapshot from Part 1 (and the nested host snapshot, if you have not already).

What’s next

Two things before the GPU work. First, a VKS image library, since the Supervisor could not create its own. Then VCF Private AI Services. William’s PAIS 3.0 post lists what it needs: a Supervisor on VPC networking, which we now have, the PAIS Supervisor Service, a Harbor registry for the models, and an NVIDIA GPU through DirectPath I/O or vGPU. His post is on 9.1.1, so check the interoperability matrix for your Supervisor version first.

One-Host Supervisor on VCF 9.1, Part 3: NSX VPC Networking Without Edges

Cosmin Trif — Wed, 21 Oct 2026 08:20:00 +0000

In Part 2 we built the one-host cluster. Before we can enable a Supervisor on it, NSX needs a way to get VPC traffic in and out. My management domain has NSX with an overlay transport zone, but no edge cluster, no Tier-0 and no Tier-1. In VCF 9.1 that is fine: a Supervisor on VPC networking can use a distributed transit gateway and a small cluster of Virtual Network Appliances (VNAs) instead of edges.

This post covers four new NSX objects and two changes to existing ones, in the order I made them.

How it fits together

Distributed VLAN connection: connects NSX to one VLAN. The gateway of that VLAN lives on your router.
Transit gateway attachment: attaches that VLAN connection to the project’s default transit gateway.
External IP block: the addresses NSX hands out for the Supervisor API VIP, SNAT and LoadBalancer services. I used a range inside the VLAN’s subnet, so the router reaches them directly with no routes to add.
VNA cluster: two appliances that give the VPCs NAT and load balancing. The VNA rides on the TEP network of the host it runs on, so it needs no network of its own beyond a management address. Two is the minimum (vStellar’s VNA post covers the VNA in depth).
VPC connectivity profile (a change to an existing object): tells every VPC in the project to use the VNA cluster as its service gateway and the external block for public addresses.

The other change is a flag on the new cluster’s transport node collection, in Step 2.

We put the VNAs on the existing 4-host cluster, not on the GPU host. With the VNAs on two different hosts, one host going down does not take out both, and the GPU host keeps its resources for AI workloads.

What I needed from the network

Item	My lab
VLAN for the distributed connection, trunked to the hosts of both clusters	VLAN 22
Gateway of that VLAN on the router	`10.1.7.129/25`
External IP block, inside that subnet, gateway excluded	`10.1.7.192/26`
Two management IPs for the VNAs, with forward and reverse DNS	`10.1.1.178`, `10.1.1.179`

Holodeck’s router already had VLAN 22 with that gateway. At the customer, this table is what we handed to the network team.

Step 1: Check the transport zone and the span

Two checks before any change. They catch the one mistake that breaks this design: the new cluster ending up on its own transport zone.

The new cluster’s transport node profile must use the same overlay transport zone object as the rest of the domain, not a new zone with the same name. Find the cluster’s transport node collection under GET /policy/api/v1/infra/sites/default/enforcement-points/default/transport-node-collections, read its transport node profile, and compare the transport_zone_id with the existing zone’s path.
The transport node collection must sit inside the span of the default transit gateway (GET /policy/api/v1/orgs/default/projects/default/transit-gateways/default, field span.span_path). A collection with no explicit network_span_paths uses the default span.

In my case both passed, because the cluster spec in Part 2 named the existing zone.

Step 2: Turn on NSX for DVPGs on the new cluster

Set enable_nsx_on_dvpg to true on the new cluster’s transport node collection. Read the whole object, change that one field and send it back:

PUT /policy/api/v1/infra/sites/default/enforcement-points/default/transport-node-collections/<id>

Step 3: Create the distributed VLAN connection

PUT /policy/api/v1/infra/distributed-vlan-connections/distributed-gw-mgmt-a

{
 "resource_type": "DistributedVlanConnection",
 "id": "distributed-gw-mgmt-a",
 "display_name": "distributed-gw-mgmt-a",
 "gateway_addresses": [ "10.1.7.129/25" ],
 "vlan_id": 22
}

Step 4: Attach it to the default transit gateway

PATCH /policy/api/v1/orgs/default/projects/default/transit-gateways/default/attachments/dtgw-att-mgmt-02a

{ "connection_path": "/infra/distributed-vlan-connections/distributed-gw-mgmt-a" }

Step 5: Create the external IP block

PATCH /policy/api/v1/infra/ip-blocks/dtgw-ext-mgmt-a

{ "display_name": "dtgw-ext-mgmt-a", "cidr": "10.1.7.192/26", "visibility": "EXTERNAL" }

Step 6: Deploy the VNA cluster

The VNA cluster goes in through a hierarchical PATCH on /policy/api/v1/infra. These are the bodies from VMware’s Holodeck 9.1 toolkit, which is where I learned the shape. Both appliances go in the same PATCH, as two ChildBaseNetworkAppliance entries in the same children array. Below is the body with the first one. The second entry is the same apart from its id, display_name, hostname and IP:

PATCH /policy/api/v1/infra?enforce_revision_check=true

{
 "resource_type": "Infra",
 "children": [ {
 "resource_type": "ChildResourceReference", "id": "default", "target_type": "Site",
 "children": [ {
 "resource_type": "ChildResourceReference", "id": "default", "target_type": "EnforcementPoint",
 "children": [ {
 "resource_type": "ChildVirtualNetworkApplianceCluster",
 "VirtualNetworkApplianceCluster": {
 "resource_type": "VirtualNetworkApplianceCluster",
 "id": "vna-cluster-mgmt-a",
 "display_name": "vna-cluster-mgmt-a",
 "appliance_form_factor": "MEDIUM",
 "children": [ {
 "resource_type": "ChildBaseNetworkAppliance",
 "BaseNetworkAppliance": {
 "resource_type": "VirtualNetworkAppliance",
 "id": "vna-mgmt-01a",
 "display_name": "vna-mgmt-01a",
 "hostname": "vna-mgmt-01a.site-a.vcf.lab",
 "credentials": {},
 "vm_deployment_config": {
 "compute_manager_id": "<compute manager id>",
 "cluster_or_resource_pool_id": "<cluster moref, e.g. domain-c9>",
 "datastore_id": "<datastore moref>"
 },
 "management_interface": {
 "network_id": "<management portgroup moref>",
 "ip_assignment_specs": [ {
 "ip_assignment_type": "StaticIpv4",
 "management_port_subnets": [ { "ip_addresses": [ "10.1.1.178" ], "prefix_length": "24" } ],
 "default_gateway": [ "10.1.1.1" ]
 } ]
 }
 }
 } ]
 }
 } ]
 } ]
 } ]
}

The compute manager ID comes from GET /api/v1/fabric/compute-managers. The cluster, datastore and portgroup IDs are vCenter morefs of the cluster the VNAs go on (for me, cluster-mgmt-01a), its vSAN datastore and its management portgroup.

I left credentials empty, so NSX generates the appliance passwords. I have not checked yet where to read them back, so find that out before you need a VNA console.

Watch the status:

GET /policy/api/v1/infra/sites/default/enforcement-points/default/virtual-network-appliance-clusters/vna-cluster-mgmt-a/status

In my case it went from UNKNOWN, to DEGRADED with both members ADMIN_DOWN, to UP with both members UP, about 10 minutes after the PATCH.

Then keep the two VNAs on different hosts with a DRS rule:

$cl = Get-Cluster -Name 'cluster-mgmt-01a'
New-DrsRule -Cluster $cl -Name 'vna-mgmt-a-separate' -KeepTogether $false -Enabled $true -VM (Get-VM -Location $cl -Name 'vna-mgmt-01a', 'vna-mgmt-02a')

Mine ended up on esx-01a and esx-03a.

Step 7: Point the VPC connectivity profile at the VNAs

Read the project’s default VPC connectivity profile, add the service gateway and the external block, and PATCH it back:

GET /policy/api/v1/orgs/default/projects/default/vpc-connectivity-profiles/default
PATCH /policy/api/v1/orgs/default/projects/default/vpc-connectivity-profiles/default

The two fields that change:

{
 "service_gateway": {
 "enable": true,
 "nat_config": { "enable_default_snat": true },
 "edge_cluster_paths": [ "/infra/sites/default/enforcement-points/default/virtual-network-appliance-clusters/vna-cluster-mgmt-a" ]
 },
 "external_ip_blocks": [ "/infra/ip-blocks/dtgw-ext-mgmt-a" ]
}

Yes, the VNA cluster path goes in a field called edge_cluster_paths.

This profile is shared. Every VPC in the project that uses it gets the new service gateway and external block too. My lab had no VPCs yet, so that was safe. On a site that already runs VPCs, list them first and consider a dedicated profile for the Supervisor.

NSX is ready. In the next post we enable the Supervisor with a single control plane VM, and deal with the two things that stalled it.

One-Host Supervisor on VCF 9.1, Part 2: Building a One-Host vSAN ESA Cluster

Cosmin Trif — Wed, 21 Oct 2026 08:10:00 +0000

In Part 1 we set up SDDC Manager to accept a single-host cluster. Now we add the GPU host to the domain as its own one-host cluster. In my lab that host is esx-05a, a spare nested host, and the new cluster is cluster-mgmt-02a in the management domain mgmt-a.

This is where the series hit its first failure, and the fix is the part I would have liked to read about before I started.

Before you begin

The host has to pass SDDC Manager’s commissioning checks:

ESXi at the same build as the vLCM image you will use for the cluster (mine: 9.1.0, build 25370933)
Forward and reverse DNS for the host FQDN, and NTP configured
A host certificate whose CN is the FQDN (regenerate it after you set the hostname)
Free vmnics for the new VDS, and unpartitioned local disks for vSAN
A network pool with free vMotion and vSAN addresses

Step 1: Commission the host

In my lab I had already commissioned esx-05a with a short script, so it showed up in SDDC Manager as UNASSIGNED_USEABLE. Validate first, then commission, with the same body:

POST /v1/hosts/validations
POST /v1/hosts

[
 {
 "fqdn": "esx-05a.site-a.vcf.lab",
 "username": "root",
 "password": "<root password>",
 "storageType": "VSAN_ESA",
 "networkPoolId": "<network pool id>",
 "networkPoolName": "mgmt-a-np01"
 }
]

Use VSAN_ESA for an ESA host and VSAN for OSA. Holodeck’s own commissioning template uses VSAN, and on an ESA host that fails validation with “not enough capacity disks”. When the task finishes, the host should be UNASSIGNED_USEABLE.

Step 2: Collect the IDs

The cluster spec needs three IDs, the same ones William’s vcf-9.1.1-single-host-cluster-deployment.ps1 (in lamw/vmware-scripts) looks up:

the domain: GET /v1/domains
the host: GET /v1/hosts?status=UNASSIGNED_USEABLE
the vLCM image: GET /v1/personalities (I used Management-Domain-ESXi-Personality)

Step 3: Build the cluster spec

This is the spec that worked in my lab, with the IDs replaced. The host has four vmnics: two for a VDS that carries management, vMotion and the NSX overlay, and two for a vSAN VDS.

{
 "domainId": "<domain id>",
 "computeSpec": {
 "clusterSpecs": [
 {
 "name": "cluster-mgmt-02a",
 "datacenterName": "dc-mgmt-a",
 "clusterImageId": "<personality id>",
 "hostSpecs": [
 {
 "id": "<host id>",
 "hostNetworkSpec": {
 "vmNics": [
 { "id": "vmnic0", "vdsName": "vds01-mgmt-02a" },
 { "id": "vmnic1", "vdsName": "vds01-mgmt-02a" },
 { "id": "vmnic2", "vdsName": "vds02-mgmt-02a" },
 { "id": "vmnic3", "vdsName": "vds02-mgmt-02a" }
 ]
 }
 }
 ],
 "datastoreSpec": {
 "vsanDatastoreSpec": {
 "datastoreName": "vsan-mgmt-02a",
 "esaConfig": { "enabled": true, "skipHclAutoDiskClaim": true }
 }
 },
 "networkSpec": {
 "vdsSpecs": [
 {
 "name": "vds01-mgmt-02a",
 "mtu": 8000,
 "portGroupSpecs": [
 { "name": "mgmt-vds01-mgmt-02a", "transportType": "MANAGEMENT" },
 { "name": "vmotion-vds01-mgmt-02a", "transportType": "VMOTION" }
 ],
 "nsxtSwitchConfig": {
 "transportZones": [ { "name": "overlay-vds01-mgmt-01a", "transportType": "OVERLAY" } ]
 }
 },
 {
 "name": "vds02-mgmt-02a",
 "mtu": 8000,
 "portGroupSpecs": [ { "name": "vsan-vds02-mgmt-02a", "transportType": "VSAN" } ]
 }
 ],
 "nsxClusterSpec": { "nsxTClusterSpec": { "geneveVlanId": 14 } }
 }
 }
 ]
 },
 "deployWithoutLicenseKeys": true
}

Three things matter here:

The overlay transport zone is the existing one. overlay-vds01-mgmt-01a is the zone the rest of the domain already uses. The VPC networking in Part 3 only spans both clusters if they share the same transport zone.
No uplink names on the vmnics. Without them, the vmnics map to uplink1, uplink2 and so on, in the order listed.
skipHclAutoDiskClaim: true lets vSAN ESA claim disks that are not on the HCL. My nested hosts need it. On HCL hardware, leave it off.

The TEPs use DHCP on VLAN 14, which Holodeck provides. Without DHCP on the TEP VLAN, use a static IP pool instead.

Step 4: Validate

POST /v1/clusters/validations
GET /v1/clusters/validations/<id>

Poll until executionStatus is COMPLETED, then read resultStatus. In my case validation reported every spec error in one pass, so one round of fixes was enough.

My first spec had an uplink name on each vmnic. Validation came back with three errors for each VDS. For the first one:

[
 { "errorCode": "PORTGROUP_UPLINK_CONFIGURATION_UNDEFINED",
 "message": "The uplink configuration must be defined for the portgroups [mgmt-vds01-mgmt-02a, vmotion-vds01-mgmt-02a] when custom uplinks are specified in the input." },
 { "errorCode": "PORTGROUP_UPLINK_CONFIGURATION_INSUFFICIENT_VMNICS",
 "message": "The portgroup specifications [mgmt-vds01-mgmt-02a, vmotion-vds01-mgmt-02a] contain insufficient uplinks. Atleast two uplinks must be specified." },
 { "errorCode": "PORTGROUP_TEAMING_POLICY_UNDEFINED",
 "message": "The teaming policy has to be specified for the portgroup specs [mgmt-vds01-mgmt-02a, vmotion-vds01-mgmt-02a] when uplinks are defined in the input specification." }
]

Once you name uplinks, SDDC Manager wants the uplink configuration, at least two uplinks and a teaming policy on every portgroup. Dropping the uplink names was the simpler fix. The same pass also reported INVALID_NUMBER_OF_HOSTS, which Part 1 takes care of.

Step 5: Create the cluster

If the host is a nested ESXi VM, you may want a snapshot of it first. Remember that you took it. It comes back in Part 4.

POST /v1/clusters

The task created the cluster and its two VDSs, moved the host’s vmknics over, claimed the ESA disks, took the host out of maintenance mode and turned on DRS. About five minutes in, it failed.

The failure: FAILED_TO_CREATE_FILE_INSIDE_DATASTORE

The failed subtask was Validate Datastore Availability:

{ "errorCode": "FAILED_TO_CREATE_FILE_INSIDE_DATASTORE", "message": "Failed to create file inside datastore datastore-55 of datacenter datacenter-3" }

SDDC Manager had already created the vSAN datastore. The problem was its default storage policy. vSAN ESA gave the new datastore an auto-generated default, cluster-mgmt-02a - Optimal Datastore Default Policy - AutoRAID. On the host, esxcli vsan policy getdefault showed hostFailuresToTolerate at 1 for every object class. One host cannot place an object that needs a second copy, so SDDC Manager could not even create a folder.

The task is retryable, so the fix is to correct the datastore and retry the same task. Do not submit a second cluster.

Step 6: Make an FTT=0 policy the datastore default

In PowerCLI, connected to the domain vCenter:

$rule = New-SpbmRule -Capability (Get-SpbmCapability -Name 'VSAN.hostFailuresToTolerate') -Value 0
$pol = New-SpbmStoragePolicy -Name 'sup-single-host-ftt0' -Description 'Single-host vSAN, FTT=0' -AnyOfRuleSets (New-SpbmRuleSet -AllOfRules $rule)

$ds = Get-Datastore -Name 'vsan-mgmt-02a'
Set-SpbmEntityConfiguration -Configuration (Get-SpbmEntityConfiguration -Datastore $ds) -StoragePolicy $pol

Before retrying, I proved the fix by creating and deleting a test folder on the datastore, which is what the failed check does:

$fm = Get-View (Get-View ServiceInstance).Content.FileManager
$dc = (Get-Datacenter -Name 'dc-mgmt-a').ExtensionData.MoRef
$fm.MakeDirectory('[vsan-mgmt-02a] probe', $dc, $false)
$fm.DeleteDatastoreFile_Task('[vsan-mgmt-02a] probe', $dc) | Out-Null

Step 7: Retry the task

# $sddc and $h as in Part 1; $taskId is the id that POST /v1/clusters returned
Invoke-RestMethod -Method Patch -Uri "https://$sddc/v1/tasks/$taskId" -Headers $h -SkipCertificateCheck

The retry passed Validate Datastore Availability, remediated the host to the cluster image, created the NSX transport node collection, and finished about 11 minutes later. The cluster was ACTIVE, the host ASSIGNED, and its NSX transport node was up on the shared overlay transport zone.

If you took a snapshot of a nested host VM in Step 5, delete it now. Part 4 shows what happens if you don’t.

Step 8: Get the cluster ready for a Supervisor

The Supervisor needs vSphere HA and DRS turned on. SDDC Manager created the cluster with both on, and with HA admission control on as well. On one host, admission control with a failover level of 1 reserves the whole host for failover, which leaves no room to power on the control plane VM. So HA and DRS stay on, and admission control goes off before the enable:

Set-Cluster -Cluster 'cluster-mgmt-02a' -HAAdmissionControlEnabled:$false -Confirm:$false

One more gotcha with the policy. My first version of sup-single-host-ftt0 also had VSAN.autoManagedRAID=False. It worked as the datastore default and the folder test passed, but the SPBM compatibility check showed it compatible with no datastore at all. The Supervisor needs a policy that is compatible with the vSAN datastore, so I dropped that rule and kept FTT=0 as the only one. Check it before you move on:

Get-SpbmCompatibleStorage -StoragePolicy (Get-SpbmStoragePolicy -Name 'sup-single-host-ftt0')

It should list vsan-mgmt-02a.

The cluster is ready. In the next post we build the NSX side: VPC networking for the Supervisor, with no edge nodes.

One-Host Supervisor on VCF 9.1, Part 1: Allowing Single-Host Clusters in SDDC Manager

Cosmin Trif — Wed, 21 Oct 2026 08:00:00 +0000

I was working with a customer, and we were going through William Lam’s post on running VCF 9.1.1 on a single ESX host. They wanted a vSphere Supervisor so they could run VCF Private AI Services later, but only one of their hosts had GPUs in it. Buying two more GPU hosts just to meet a cluster minimum was not an option. So the question was simple: can we add a one-host cluster to the existing VCF domain and run a Supervisor on it?

The answer is yes, with a few workarounds. Before we touched their environment, I built the whole thing end to end in my Holodeck lab. This series is what I did, in the order I did it, including the parts that failed.

Lab and POC only. A one-host cluster has no redundancy, and the SDDC Manager properties below are lab workarounds, not a supported production design. Host maintenance means shutting down everything on that host.

What we are building

The existing management domain mgmt-a stays as it is: vCenter vc-mgmt-a, NSX Manager nsx-mgmt-a and the 4-host cluster cluster-mgmt-01a.
A new cluster, cluster-mgmt-02a, with one host (esx-05a) on vSAN ESA, added to that same domain.
A Supervisor with a single control plane VM on the new cluster.
NSX VPC networking with no edges: a distributed transit gateway connected to one VLAN, plus two Virtual Network Appliances (VNAs) on the existing cluster for NAT and load balancing.

No new vCenter and no new NSX Manager. Everything lands in the management domain we already had.

The Supervisor’s VPC traffic leaves through the distributed transit gateway and VLAN 22, with the two VNAs on the existing cluster as its service gateway.

Versions in my lab

Component	Version
SDDC Manager	9.1.0.0.25371088
vCenter	9.1.0, build 25370922
NSX	9.1.0.0.25318225
ESXi	9.1.0, build 25370933

William’s post targets 9.1.1, and my lab was on 9.1.0. The gap is smaller than I expected. His 9.1.1 cluster script still lists the single-host feature flag and the minimum cluster size as manual prerequisites on SDDC Manager, so Step 2 below applies to both versions. Only the three ESA disk-claim keys are specific to 9.1.0. Either way this is API work: per his post, the 9.1.1 UI only supports a single host for a new VCF fleet, and adding a single-host cluster to an existing domain goes through the SDDC Manager API on both versions.

The series

Part 1 (this post): allowing single-host clusters in SDDC Manager
Part 2: building the one-host vSAN ESA cluster
Part 3: NSX VPC networking without edges
Part 4: enabling a Supervisor with one control plane VM

Before you begin

Take a memory-less snapshot of the SDDC Manager VM. Take NSX Manager and vCenter backups too; later posts change both.
Make sure SDDC Manager has no running tasks. The service restart below kills whatever workflow is in flight.
You need SSH as vcf and the root password of the SDDC Manager appliance.

Step 1: See what SDDC Manager says today

If we send a one-host cluster spec to POST /v1/clusters/validations now, validation fails right away:

{ "errorCode": "INVALID_NUMBER_OF_HOSTS", "message": "Number of hosts is less than expected. Actual 3, Provided 1" }

SDDC Manager wants three hosts in a vSAN cluster. The properties below lower that to one.

Step 2: Set the properties

These come from William’s VCF 9.1 Installer and SDDC Manager workarounds post. The first two allow a single host, on 9.1.0 and 9.1.1 alike. The last three let vSAN ESA claim disks that are not on the HCL, which nested hosts need on 9.1.0. William’s 09/03 update says the ESA disk-claim workaround is no longer required on 9.1.1.

File	Key	Why
`/home/vcf/feature.properties`	`feature.vcf.vgl-29121.single.host.domain=true`	allows a single-host domain or cluster
`/etc/vmware/vcf/domainmanager/application-prod.properties`	`bringup.mgmt.cluster.minimum.size=1`	lowers the cluster minimum
`/home/vcf/feature.properties`	`feature.vcf.vgl-43370.vsan.esa.sddc.managed.disk.claim=true`	ESA disk claim without the HCL (9.1.0)
`/etc/vmware/vcf/domainmanager/application-prod.properties`	`vsan.esa.sddc.managed.disk.claim=true`	pairs with the line above
`/etc/vmware/vcf/operationsmanager/application-prod.properties`	`vsan.esa.sddc.managed.disk.claim=true`	pairs with the line above

Two things I ran into:

On 9.1.0, both application-prod.properties files are symlinks to application.properties in the same folder. Edit the real file, and back it up first.
In my case, the Holodeck-built SDDC Manager already had the three ESA disk-claim keys. Only the two single-host keys were missing. Check with grep before you append anything, so you don’t end up with duplicate keys.

SSH to SDDC Manager as vcf, then su -:

f=$(readlink -f /etc/vmware/vcf/domainmanager/application-prod.properties)
cp -p "$f" "$f.bak"
cp -p /home/vcf/feature.properties /home/vcf/feature.properties.bak

# add a key only if it is not there yet (the leading line break covers a file that does not end with one)
addkey() { grep -q "^$2=" "$1" || printf '\n%s=%s\n' "$2" "$3" >> "$1"; }

addkey /home/vcf/feature.properties feature.vcf.vgl-29121.single.host.domain true
addkey "$f" bringup.mgmt.cluster.minimum.size 1

# 9.1.0 with disks that are not on the HCL: the three ESA disk-claim keys as well
# o=$(readlink -f /etc/vmware/vcf/operationsmanager/application-prod.properties); cp -p "$o" "$o.bak"
# addkey /home/vcf/feature.properties feature.vcf.vgl-43370.vsan.esa.sddc.managed.disk.claim true
# addkey "$f" vsan.esa.sddc.managed.disk.claim true
# addkey "$o" vsan.esa.sddc.managed.disk.claim true

# check the result
grep -nE 'vgl-29121|vgl-43370' /home/vcf/feature.properties
grep -nE 'bringup.mgmt.cluster.minimum.size|vsan.esa.sddc.managed.disk.claim' "$f"

If a key is already there with a different value, addkey leaves it alone. Fix that line by hand.

Step 3: Restart the SDDC Manager services

echo 'y' | /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

The services restart one after another. In my case the API was answering again in under two minutes, while the restart was still going. An API that answers is not the same as a restart that is done, so wait until the script has exited and every service is back:

systemctl is-active commonsvcs domainmanager lcm operationsmanager sddc-manager-ui-app

Step 4: Prove the flag is live

The /v1/features endpoint lists every feature flag with its state. The single-host flag should show enabled: true. This needs PowerShell 7 for -SkipCertificateCheck:

$sddc = 'sddcmanager-a.site-a.vcf.lab'
$cred = Get-Credential administrator@vsphere.local
$body = @{ username = $cred.UserName; password = $cred.GetNetworkCredential().Password } | ConvertTo-Json
$token = (Invoke-RestMethod -Method Post -Uri "https://$sddc/v1/tokens" -Body $body -ContentType 'application/json' -SkipCertificateCheck).accessToken
$h = @{ Authorization = "Bearer $token" }

# the parentheses matter: the endpoint returns one JSON array, and Invoke-RestMethod passes it down the pipeline as a single object
(Invoke-RestMethod -Uri "https://$sddc/v1/features" -Headers $h -SkipCertificateCheck) |
 Where-Object name -like '*vgl-29121*'

Then run the same cluster validation again. This time it came back SUCCEEDED, and SDDC Manager was ready for a one-host cluster.

Keep the SDDC Manager snapshot until the whole series is done and verified, then delete it.

In the next post we commission the host and build the one-host vSAN ESA cluster, including the first failure of the series.