As I am sure many out there (including myself) are eagerly awaiting GA release of VMWare PKS. In late December 2017 VMWare released NSX-T 2.1 which will provide the Networking virtualization in PKS. NSX-T is the multi-cloud and multi-hypervisor environment version of NSX for vSphere. NSX-T can coexist with NSX for vSphere as NSX-T has a separate management plane which does not interact with vCenter which enables you to build NSX-T compute clusters under the same vCenter as clusters managed by NSX for vSphere. This post will walk through the initial installation of the Management and Control Planes and hopefully will be the first in a series about designing and configuring NSX-T for container workloads.
Before you begin some important points for NSX-T:
- Read the VMWare NSX-T 2.1 Installation Guide and plan the deployment
- I highly recommend that you check out Hany Michaels Kubernetes in the Enterprise: The Design Guide; this is an absolutely awesome write up on deploying Kubernetes on vRA leveraging NSX-T full of design guidance for NSX-T
- IPv6 is not supported
- When deploying on the ESXi hypervisor, NSX-T does not support the Host Profiles and Auto Deploy features
- If performing an installation on nested ESXi please note that VMXNET 3 vNIC is only supported for VM NSX Edge
- If an encryption rule is applied on a hypervisor, the virtual tunnel endpoint (VTEP) interface minimum MTU size must be 1700. MTU size 2000 or later is preferred.
- You must enable SSH to access on all hypervisors that will be running NSX-T.
- An NSX Manager must have a static IP address. You cannot change the IP address after installation.
- NSX Manager VM Configuration: The OVA contains three deployment options: Small, Medium and Large; Small is for Lab/PoC and Medium and Large for Production. I have not found any official guidance on sizing guidance for “Medium” vs “Large” however the only difference is resources which can be changed as required; start with a Medium deployment and monitor load on the NSX Manager and increase as required.
- Control Plane clusters can contain only one or three members; no other configuration is possible (eg. Cannot have 5 node Controller Cluster)
- The OVF Template deployment does not work from the vSphere Client (H5) and fails with “Invalid value ‘true/false’ specified for property nsx_IsSSHEnabled : I had this issue deploying on vCenter 6.5 (Build 7312210) this appears to be an issue with the case issue with the translation of the Boolean value that the H5 client passes. Solution: The deployment must be made using the vSphere Web Client (Flash) or using the OVF tool
Step 1. Download the NSX Manager and NSX Controller OVA’s from My VMWare
Step 2. Deploy the NSX Manager VM
- Open the vSphere Web Client and deploy the OVF Template (nsx-unified-appliance-188.8.131.52.0.7395503.ova)
- Provide the standard metrics (VM Name, Location, Storage, Networking
- At the Select Configuration screen select Medium and click Next
- Enter valid values for the appliance at the Customize template screen ensuring the Role Name is set to nsx-manager click Finish and Power On the appliance
NOTE: It is recommended that SSH is enabled on the appliance in order to ease configuration and troubleshooting if your security policy allows it. It can be disabled once the configuration is complete
- After the installation completes browse to https://<address of appliance>/ and logon using the admin credentials provided during template customization and agree to the EULA.
- Select System > Configuration > License and install your NSX License
Step 3. Configuring Automatic Backups
The following outlines the process for setting up an Automated backup of the NSX-T Manager. In addition to backing up the Virtual Machines/appliances it is recommended that application backups are taken on a regular interval as they consume very little space but can enable rollback in the event of an unauthorized/accidental misconfiguration. NSX-T provides an automated backup method to an SFTP server on a configurable schedule to allow for restore points to be created. Before we begin a couple of points:
- SFTP is the only configurable protocol in the GUI
- The target SFTP server must be configured to serve an ECDSA Key in order for the supported SSH thumbprint validation to work (requires a SHA-256 fingerprint generated from the ECDSA Key)
- The schedule allows you to set how oftern backups are taken but not when; i.e. can’t set run at 1:00pm only run every 5 minutes
1. Logon to your SSH/SFTP server and determine the thumbprint for the server using ssh-keygen -lf <keyfilename>
2. Logon the NSX-T Manager and select System > Utilities > Backup from the menu and click Edit
3. Check Enabled next to Automatic Backup and enter the details for the SFTP server and set the Passphrase to encrypt the backups; select the schedule tab and adjust the frequency as required and click Save
4. A backup will complete after the settings are saved; review the Output and ensure that the backups completed successfully.
Now when you or someone else breaks your Lab/Production you might have a way back to a known good state quickly.
Step 4. Install the NSX Controller Plane
NSX Controller is an advanced distributed state management system that provides control plane functions for NSX-T logical switching and routing functions. Provides control plane to distribute network information to the hypervisors, enables VXLAN dependency on multicast routing/PIM in the physical network to be removed and provides suppression of ARP broadcast traffic in VXLAN networks. The Control Plane can be installed as a single VM or as a cluster of three Controller Appliances.
NOTE: The Controller Appliances are deployed by default as 4vCPU, 16GB RAM (Fully Reserved) by default; for Lab Deployments if you don’t have the resources the following resourcing should be sufficient to get you going: 1vCPU/4GB Memory
1. Deploy the nsx-controller-184.108.40.206.0.7395493.ova OVA
2. If you are deploying a three node Control Cluster repeat this again for another two appliances
3. Logon to the NSX Manager via SSH/CLI and execute get certificate api thumbprint
4. Now logon to the Controller Nodes from the CLI/SSH console and execute the following to join the node to the NSX-T Manager: join management-plane NSX-Manager username admin thumbprint <thumbprint>
Verify that the Controller Cluster nodes are Up and showing Manager connectivity from the NSX-T Manager (System > Overview)
5. Logon to your first Controller Appliance via SSH/CLI and set the cluster shared secret by executing set control-cluster security-model shared-secret and then initialize the cluster by executing initialize control-cluster
You can verify the Cluster by executing get control-cluster status verbose and ensure that the node is reporting as master, is in majority, has a status of active, and the Zookeeper Server IP is reachable, ok.
6 (Optional – only if you are deploying a multi-node cluster). Logon to the secondary Controller Appliances via SSH/CLI and set the cluster shared secret by executing set control-cluster security-model shared-secret and record the output of get control-cluster certificate thumbprint
Next, log on the master NSX Controller configured in Step 5 and execute join control-cluster <Cluster Node X IP> thumbprint <Cluster Node X Thumbprint> for each of the controllers
Verify that the control nodes are all showing as active by executing get control-cluster status
Finally logon to each of the non-primary Controller Appliances via SSH/CLI and execute activate control-cluster (ensure that each controller has been activated successfully before moving to the next)
7. Verify that the cluster is online by executing get control-cluster status verbose on any node or from the NSX-T Manager (System > Overview)
8. Setup Anti-Affinity rules to ensure that the Control Plane cluster nodes do not run on the same hosts
Next I will look at configuring an NSX-T Fabric and configuring Edge Clusters and Hosts. Happy New Year.