One-Host Supervisor on VCF 9.1, Part 3: NSX VPC Networking Without Edges

In Part 2 we built the one-host cluster. Before we can enable a Supervisor on it, NSX needs a way to get VPC traffic in and out. My management domain has NSX with an overlay transport zone, but no edge cluster, no Tier-0 and no Tier-1. In VCF 9.1 that is fine: a Supervisor on VPC networking can use a distributed transit gateway and a small cluster of Virtual Network Appliances (VNAs) instead of edges.

This post covers four new NSX objects and two changes to existing ones, in the order I made them.

How it fits together

• Distributed VLAN connection: connects NSX to one VLAN. The gateway of that VLAN lives on your router.
• Transit gateway attachment: attaches that VLAN connection to the project’s default transit gateway.
• External IP block: the addresses NSX hands out for the Supervisor API VIP, SNAT and LoadBalancer services. I used a range inside the VLAN’s subnet, so the router reaches them directly with no routes to add.
• VNA cluster: two appliances that give the VPCs NAT and load balancing. The VNA rides on the TEP network of the host it runs on, so it needs no network of its own beyond a management address. Two is the minimum (vStellar’s VNA post covers the VNA in depth).
• VPC connectivity profile (a change to an existing object): tells every VPC in the project to use the VNA cluster as its service gateway and the external block for public addresses.

The other change is a flag on the new cluster’s transport node collection, in Step 2.

We put the VNAs on the existing 4-host cluster, not on the GPU host. With the VNAs on two different hosts, one host going down does not take out both, and the GPU host keeps its resources for AI workloads.

What I needed from the network

Holodeck’s router already had VLAN 22 with that gateway. At the customer, this table is what we handed to the network team.

Step 1: Check the transport zone and the span

Two checks before any change. They catch the one mistake that breaks this design: the new cluster ending up on its own transport zone.

• The new cluster’s transport node profile must use the same overlay transport zone object as the rest of the domain, not a new zone with the same name. Find the cluster’s transport node collection under GET /policy/api/v1/infra/sites/default/enforcement-points/default/transport-node-collections, read its transport node profile, and compare the transport_zone_id with the existing zone’s path.
• The transport node collection must sit inside the span of the default transit gateway (GET /policy/api/v1/orgs/default/projects/default/transit-gateways/default, field span.span_path). A collection with no explicit network_span_paths uses the default span.

In my case both passed, because the cluster spec in Part 2 named the existing zone.

Step 2: Turn on NSX for DVPGs on the new cluster

Set enable_nsx_on_dvpg to true on the new cluster’s transport node collection. Read the whole object, change that one field and send it back:

PUT /policy/api/v1/infra/sites/default/enforcement-points/default/transport-node-collections/<id>

Step 3: Create the distributed VLAN connection

PUT /policy/api/v1/infra/distributed-vlan-connections/distributed-gw-mgmt-a

{
  "resource_type": "DistributedVlanConnection",
  "id": "distributed-gw-mgmt-a",
  "display_name": "distributed-gw-mgmt-a",
  "gateway_addresses": [ "10.1.7.129/25" ],
  "vlan_id": 22
}

Step 4: Attach it to the default transit gateway

PATCH /policy/api/v1/orgs/default/projects/default/transit-gateways/default/attachments/dtgw-att-mgmt-02a

{ "connection_path": "/infra/distributed-vlan-connections/distributed-gw-mgmt-a" }

Step 5: Create the external IP block

PATCH /policy/api/v1/infra/ip-blocks/dtgw-ext-mgmt-a

{ "display_name": "dtgw-ext-mgmt-a", "cidr": "10.1.7.192/26", "visibility": "EXTERNAL" }

Step 6: Deploy the VNA cluster

The VNA cluster goes in through a hierarchical PATCH on /policy/api/v1/infra. These are the bodies from VMware’s Holodeck 9.1 toolkit, which is where I learned the shape. Both appliances go in the same PATCH, as two ChildBaseNetworkAppliance entries in the same children array. Below is the body with the first one. The second entry is the same apart from its id, display_name, hostname and IP:

PATCH /policy/api/v1/infra?enforce_revision_check=true

{
  "resource_type": "Infra",
  "children": [ {
    "resource_type": "ChildResourceReference", "id": "default", "target_type": "Site",
    "children": [ {
      "resource_type": "ChildResourceReference", "id": "default", "target_type": "EnforcementPoint",
      "children": [ {
        "resource_type": "ChildVirtualNetworkApplianceCluster",
        "VirtualNetworkApplianceCluster": {
          "resource_type": "VirtualNetworkApplianceCluster",
          "id": "vna-cluster-mgmt-a",
          "display_name": "vna-cluster-mgmt-a",
          "appliance_form_factor": "MEDIUM",
          "children": [ {
            "resource_type": "ChildBaseNetworkAppliance",
            "BaseNetworkAppliance": {
              "resource_type": "VirtualNetworkAppliance",
              "id": "vna-mgmt-01a",
              "display_name": "vna-mgmt-01a",
              "hostname": "vna-mgmt-01a.site-a.vcf.lab",
              "credentials": {},
              "vm_deployment_config": {
                "compute_manager_id": "<compute manager id>",
                "cluster_or_resource_pool_id": "<cluster moref, e.g. domain-c9>",
                "datastore_id": "<datastore moref>"
              },
              "management_interface": {
                "network_id": "<management portgroup moref>",
                "ip_assignment_specs": [ {
                  "ip_assignment_type": "StaticIpv4",
                  "management_port_subnets": [ { "ip_addresses": [ "10.1.1.178" ], "prefix_length": "24" } ],
                  "default_gateway": [ "10.1.1.1" ]
                } ]
              }
            }
          } ]
        }
      } ]
    } ]
  } ]
}

The compute manager ID comes from GET /api/v1/fabric/compute-managers. The cluster, datastore and portgroup IDs are vCenter morefs of the cluster the VNAs go on (for me, cluster-mgmt-01a), its vSAN datastore and its management portgroup.

I left credentials empty, so NSX generates the appliance passwords. I have not checked yet where to read them back, so find that out before you need a VNA console.

Watch the status:

GET /policy/api/v1/infra/sites/default/enforcement-points/default/virtual-network-appliance-clusters/vna-cluster-mgmt-a/status

In my case it went from UNKNOWN, to DEGRADED with both members ADMIN_DOWN, to UP with both members UP, about 10 minutes after the PATCH.

Then keep the two VNAs on different hosts with a DRS rule:

$cl = Get-Cluster -Name 'cluster-mgmt-01a'
New-DrsRule -Cluster $cl -Name 'vna-mgmt-a-separate' -KeepTogether $false -Enabled $true -VM (Get-VM -Location $cl -Name 'vna-mgmt-01a', 'vna-mgmt-02a')

Mine ended up on esx-01a and esx-03a.

Step 7: Point the VPC connectivity profile at the VNAs

Read the project’s default VPC connectivity profile, add the service gateway and the external block, and PATCH it back:

GET   /policy/api/v1/orgs/default/projects/default/vpc-connectivity-profiles/default
PATCH /policy/api/v1/orgs/default/projects/default/vpc-connectivity-profiles/default

The two fields that change:

{
  "service_gateway": {
    "enable": true,
    "nat_config": { "enable_default_snat": true },
    "edge_cluster_paths": [ "/infra/sites/default/enforcement-points/default/virtual-network-appliance-clusters/vna-cluster-mgmt-a" ]
  },
  "external_ip_blocks": [ "/infra/ip-blocks/dtgw-ext-mgmt-a" ]
}

Yes, the VNA cluster path goes in a field called edge_cluster_paths.

This profile is shared. Every VPC in the project that uses it gets the new service gateway and external block too. My lab had no VPCs yet, so that was safe. On a site that already runs VPCs, list them first and consider a dedicated profile for the Supervisor.

NSX is ready. In the next post we enable the Supervisor with a single control plane VM, and deal with the two things that stalled it.