In Part 1 we set up SDDC Manager to accept a single-host cluster. Now we add the GPU host to the domain as its own one-host cluster. In my lab that host is esx-05a, a spare nested host, and the new cluster is cluster-mgmt-02a in the management domain mgmt-a.

This is where the series hit its first failure, and the fix is the part I would have liked to read about before I started.

Before you begin

The host has to pass SDDC Manager’s commissioning checks:

  • ESXi at the same build as the vLCM image you will use for the cluster (mine: 9.1.0, build 25370933)
  • Forward and reverse DNS for the host FQDN, and NTP configured
  • A host certificate whose CN is the FQDN (regenerate it after you set the hostname)
  • Free vmnics for the new VDS, and unpartitioned local disks for vSAN
  • A network pool with free vMotion and vSAN addresses

Step 1: Commission the host

In my lab I had already commissioned esx-05a with a short script, so it showed up in SDDC Manager as UNASSIGNED_USEABLE. Validate first, then commission, with the same body:

POST /v1/hosts/validations
POST /v1/hosts

[
  {
    "fqdn": "esx-05a.site-a.vcf.lab",
    "username": "root",
    "password": "<root password>",
    "storageType": "VSAN_ESA",
    "networkPoolId": "<network pool id>",
    "networkPoolName": "mgmt-a-np01"
  }
]

Use VSAN_ESA for an ESA host and VSAN for OSA. Holodeck’s own commissioning template uses VSAN, and on an ESA host that fails validation with “not enough capacity disks”. When the task finishes, the host should be UNASSIGNED_USEABLE.

Step 2: Collect the IDs

The cluster spec needs three IDs, the same ones William’s vcf-9.1.1-single-host-cluster-deployment.ps1 (in lamw/vmware-scripts) looks up:

  • the domain: GET /v1/domains
  • the host: GET /v1/hosts?status=UNASSIGNED_USEABLE
  • the vLCM image: GET /v1/personalities (I used Management-Domain-ESXi-Personality)

Step 3: Build the cluster spec

This is the spec that worked in my lab, with the IDs replaced. The host has four vmnics: two for a VDS that carries management, vMotion and the NSX overlay, and two for a vSAN VDS.

{
  "domainId": "<domain id>",
  "computeSpec": {
    "clusterSpecs": [
      {
        "name": "cluster-mgmt-02a",
        "datacenterName": "dc-mgmt-a",
        "clusterImageId": "<personality id>",
        "hostSpecs": [
          {
            "id": "<host id>",
            "hostNetworkSpec": {
              "vmNics": [
                { "id": "vmnic0", "vdsName": "vds01-mgmt-02a" },
                { "id": "vmnic1", "vdsName": "vds01-mgmt-02a" },
                { "id": "vmnic2", "vdsName": "vds02-mgmt-02a" },
                { "id": "vmnic3", "vdsName": "vds02-mgmt-02a" }
              ]
            }
          }
        ],
        "datastoreSpec": {
          "vsanDatastoreSpec": {
            "datastoreName": "vsan-mgmt-02a",
            "esaConfig": { "enabled": true, "skipHclAutoDiskClaim": true }
          }
        },
        "networkSpec": {
          "vdsSpecs": [
            {
              "name": "vds01-mgmt-02a",
              "mtu": 8000,
              "portGroupSpecs": [
                { "name": "mgmt-vds01-mgmt-02a", "transportType": "MANAGEMENT" },
                { "name": "vmotion-vds01-mgmt-02a", "transportType": "VMOTION" }
              ],
              "nsxtSwitchConfig": {
                "transportZones": [ { "name": "overlay-vds01-mgmt-01a", "transportType": "OVERLAY" } ]
              }
            },
            {
              "name": "vds02-mgmt-02a",
              "mtu": 8000,
              "portGroupSpecs": [ { "name": "vsan-vds02-mgmt-02a", "transportType": "VSAN" } ]
            }
          ],
          "nsxClusterSpec": { "nsxTClusterSpec": { "geneveVlanId": 14 } }
        }
      }
    ]
  },
  "deployWithoutLicenseKeys": true
}

Three things matter here:

  • The overlay transport zone is the existing one. overlay-vds01-mgmt-01a is the zone the rest of the domain already uses. The VPC networking in Part 3 only spans both clusters if they share the same transport zone.
  • No uplink names on the vmnics. Without them, the vmnics map to uplink1, uplink2 and so on, in the order listed.
  • skipHclAutoDiskClaim: true lets vSAN ESA claim disks that are not on the HCL. My nested hosts need it. On HCL hardware, leave it off.

The TEPs use DHCP on VLAN 14, which Holodeck provides. Without DHCP on the TEP VLAN, use a static IP pool instead.

Step 4: Validate

POST /v1/clusters/validations
GET  /v1/clusters/validations/<id>

Poll until executionStatus is COMPLETED, then read resultStatus. In my case validation reported every spec error in one pass, so one round of fixes was enough.

My first spec had an uplink name on each vmnic. Validation came back with three errors for each VDS. For the first one:

[
  { "errorCode": "PORTGROUP_UPLINK_CONFIGURATION_UNDEFINED",
    "message": "The uplink configuration must be defined for the portgroups [mgmt-vds01-mgmt-02a, vmotion-vds01-mgmt-02a] when custom uplinks are specified in the input." },
  { "errorCode": "PORTGROUP_UPLINK_CONFIGURATION_INSUFFICIENT_VMNICS",
    "message": "The portgroup specifications [mgmt-vds01-mgmt-02a, vmotion-vds01-mgmt-02a] contain insufficient uplinks. Atleast two uplinks must be specified." },
  { "errorCode": "PORTGROUP_TEAMING_POLICY_UNDEFINED",
    "message": "The teaming policy has to be specified for the portgroup specs [mgmt-vds01-mgmt-02a, vmotion-vds01-mgmt-02a] when uplinks are defined in the input specification." }
]

Once you name uplinks, SDDC Manager wants the uplink configuration, at least two uplinks and a teaming policy on every portgroup. Dropping the uplink names was the simpler fix. The same pass also reported INVALID_NUMBER_OF_HOSTS, which Part 1 takes care of.

Step 5: Create the cluster

If the host is a nested ESXi VM, you may want a snapshot of it first. Remember that you took it. It comes back in Part 4.

POST /v1/clusters

The task created the cluster and its two VDSs, moved the host’s vmknics over, claimed the ESA disks, took the host out of maintenance mode and turned on DRS. About five minutes in, it failed.

The failure: FAILED_TO_CREATE_FILE_INSIDE_DATASTORE

The failed subtask was Validate Datastore Availability:

{ "errorCode": "FAILED_TO_CREATE_FILE_INSIDE_DATASTORE", "message": "Failed to create file inside datastore datastore-55 of datacenter datacenter-3" }

SDDC Manager had already created the vSAN datastore. The problem was its default storage policy. vSAN ESA gave the new datastore an auto-generated default, cluster-mgmt-02a - Optimal Datastore Default Policy - AutoRAID. On the host, esxcli vsan policy getdefault showed hostFailuresToTolerate at 1 for every object class. One host cannot place an object that needs a second copy, so SDDC Manager could not even create a folder.

The task is retryable, so the fix is to correct the datastore and retry the same task. Do not submit a second cluster.

Step 6: Make an FTT=0 policy the datastore default

In PowerCLI, connected to the domain vCenter:

$rule = New-SpbmRule -Capability (Get-SpbmCapability -Name 'VSAN.hostFailuresToTolerate') -Value 0
$pol  = New-SpbmStoragePolicy -Name 'sup-single-host-ftt0' -Description 'Single-host vSAN, FTT=0' -AnyOfRuleSets (New-SpbmRuleSet -AllOfRules $rule)

$ds = Get-Datastore -Name 'vsan-mgmt-02a'
Set-SpbmEntityConfiguration -Configuration (Get-SpbmEntityConfiguration -Datastore $ds) -StoragePolicy $pol

Before retrying, I proved the fix by creating and deleting a test folder on the datastore, which is what the failed check does:

$fm = Get-View (Get-View ServiceInstance).Content.FileManager
$dc = (Get-Datacenter -Name 'dc-mgmt-a').ExtensionData.MoRef
$fm.MakeDirectory('[vsan-mgmt-02a] probe', $dc, $false)
$fm.DeleteDatastoreFile_Task('[vsan-mgmt-02a] probe', $dc) | Out-Null

Step 7: Retry the task

# $sddc and $h as in Part 1; $taskId is the id that POST /v1/clusters returned
Invoke-RestMethod -Method Patch -Uri "https://$sddc/v1/tasks/$taskId" -Headers $h -SkipCertificateCheck

The retry passed Validate Datastore Availability, remediated the host to the cluster image, created the NSX transport node collection, and finished about 11 minutes later. The cluster was ACTIVE, the host ASSIGNED, and its NSX transport node was up on the shared overlay transport zone.

If you took a snapshot of a nested host VM in Step 5, delete it now. Part 4 shows what happens if you don’t.

Step 8: Get the cluster ready for a Supervisor

The Supervisor needs vSphere HA and DRS turned on. SDDC Manager created the cluster with both on, and with HA admission control on as well. On one host, admission control with a failover level of 1 reserves the whole host for failover, which leaves no room to power on the control plane VM. So HA and DRS stay on, and admission control goes off before the enable:

Set-Cluster -Cluster 'cluster-mgmt-02a' -HAAdmissionControlEnabled:$false -Confirm:$false

One more gotcha with the policy. My first version of sup-single-host-ftt0 also had VSAN.autoManagedRAID=False. It worked as the datastore default and the folder test passed, but the SPBM compatibility check showed it compatible with no datastore at all. The Supervisor needs a policy that is compatible with the vSAN datastore, so I dropped that rule and kept FTT=0 as the only one. Check it before you move on:

Get-SpbmCompatibleStorage -StoragePolicy (Get-SpbmStoragePolicy -Name 'sup-single-host-ftt0')

It should list vsan-mgmt-02a.

The cluster is ready. In the next post we build the NSX side: VPC networking for the Supervisor, with no edge nodes.