Load Balancing VMWare Photon Platform 1.2 with Citrix Netscaler

Updated 22/06/2017 : Fixed HSTS Policy configuration on the Netscaler (was bound as a Request policy not a Response Policy) and updated quorum discussion after testing with three Photon Controller Nodes

Greetings; this post is the forth in a series for June where I have been focusing on the Photon Platform 1.2. This post outlines how to publish the application through a Citrix Netscaler and my findings along the way. The Photon Platform deployment appliance will deploy a HAProxy Load Balancer as part of the deployment. The HAProxy deployment has the following limitations when deployed using the photon-setup installer;

  1. Only one Load Balancer VM is deployed (requires HAProxy skills to scale the LB platform)
  2. The installer does not allow for an “external URI” to be defined for Lightwave  or the Photon Controller which means that the Cookies issued and the Auth Redirects behave incorrectly if using DNS names (as of yet I have not been able to figure out how to change these settings through API/config files)
  3. As of yet The SSL certificates are all signed by the Lightwave infrastructure and are not trusted
  4. The way the Deployment appliance deploys the solution the response that returns from https://PhotonController:9000/v1/system/auth only returns a single IP for Lightwave which is not dynamically updated; if this endpoint goes offline it does not get updated (Issue 134 has been logged regarding this)
  5. When the primary Lightwave server goes offline users can’t authenticate as the Open ID Connect Clients are not registered on additional servers and further the endpoint registration does not occur on the Photon Controller Platform
  6. The Photon Controllers are deployed in a cluster with (by default) a Majority Quorum; the Quorum setting can be adjusted via the API.

So out of the box there are a few issues with the scalability and availability of the Management Platform however you can still replace the Load Balancing and improve the availability (a little bit), several issues (Issue 133 and Issue 134) discovered during the prototyping have been logged on Github. I will continue to look for ways to address these issues however be aware of the following points of failure in the Photon Platform 1.2 platform I have found during my testing:

  1. Authentication (Lightwave) : All configuration is on a single instance
  2. The Management User Interface : Will not work if the Load Balancer is offline (the API still functions on Port 9000 but 443 stops working)

Hopefully as these issues are addressed this document will evolve and the Platform will become more resilient to failures however I would recommend deploying the Netscalers in HA mode to address HA Issue #2 listed above.


Step 1. Allocate VIP, create some DNS records and Generate some SSL Certificates

The first step is to allocate a Virtual IP addresses (VIP) that will be used for providing Public access to the backend services (Lightwave and Photon Controller) and perform any NAT/Firewalling to make these accessible to clients. Once this has been done create DNS (A) records for the two services (Lightwave for Authentication and Photon Platform for the Management/API access) eg. lightwave.pigeonnuggets.com & photonplatform.pigeonnuggets.com

Next generate some Web Server certificates against your Enterprise PKI/Public Certificate authority with the Subject Name (and Subject Alternative Names) set to the DNS records created.

Step 2. Prepare Lightwave

Lightwave is used to generate the access tokens (in the form of a cookie) that is used by Photon Controller to authenticate client requests. When a client contacts the Photon Controller without an access token the Photon Controller constructs a 302 Redirect for the client which contains;

  1. The domain to authenticate
  2. The Claims that the token should contain
  3. Where the clients should be redirected back to after authentication
  4. A client id and;
  5. The hostname of the originating request

Lightwave will then generate a cookie with the Domain Name that matches the provided host in the Redirect URI and pass this to the requester for use with Photon. In order for Lightwave to service the request to validates if the “Redirect_URI” provided is allowed for the Client with the Client Id provided and as such we need to add the new domain names to the list of allowed URI’s. To do this;

  1. Navigate to the LightWave Domain Controller administration page (https://lightwavefqdn/lightwaveui/) and enter the LightWave domain and when prompted enter the LightWave administrator account
  1. Select Service Providers from the side menu and select Open ID Connect Client and locate the Client ID for the Photon Controller Management UI; you can locate this by examining the 302 Redirect URL you get when navigating to the Photon Controller via the IP or by looking for the entry with “https://<IP of load balancer>:4343/logout_callback” and clicking Edit
  1. Amend the properties of the Logout URI: https://<fqdn of DNS record for Photon Controller LB>:4343/logout_callback and enter the following URI’s and click Add followed by Save
  1. Clean-up/remove any references to the default load balancer if you intend to decommission it/not allow connections via IP also by clicking the X next to the relevant objects

Step 3. Configure Netscaler Backend Objects and Request Rewrite Policies

The diagram (click here for a PDF version) outlines the configuration on the Load Balancer to deliver the Lightwave and Photon Controller Platform management plane to users. It is important to ensure that Photon Platform continues to work that the VIP for the Load Balancer is the IP that was assigned to the HAProxy machine. The Photon Controller uses the External URI (the IP) when there is more than one Photon Platform deployed to make backend web requests and if you don’t reuse the VIP it breaks.  I don’t as yet have a better solution but during testing I wasn’t able to amend these external_URI values successfully…watch this space.

As mentioned in Step 2 I was not able to determine how to set/change the “External URI” post deployment for the Lightwave and the Photon Controller platforms and as a result some of the responses to/from the client and the Photon controller must be rewritten with the correct external URI’s (instead of internal IP addresses).

Health Monitors and Load Balancing Objects

The following configuration are the basic Load Balancing objects that need to be defined and the health sets; for the health of the Photon Controllers an API call is made to https://server:9000/v1/available which returns if the service is available

Rewrite and Responder Policy

The following rewrite and responder policies are used to;

  • Insert the Required Headers
  • Replace the Internal IP addresses on the headers with Load Balancer VIP DNS names;
  • Replace the API Authentication endpoint URI;
  • A redirect to Port 4343 if a browser client hits the root of the API service

Also find my complete ns.config for the solution above with the passwords removed for the Certificates and the nsroot password however this should be pretty easy to implement.

The Result

Hopefully you should now have the solution Load Balanced through the Netscaler via DNS names and not directly via IP. I hope you find this information helpful and saves you some pain. Hopefully some of the issues with the Lightwave authentication configuration so that the solution can be deployed in a highly available, scalable manner. Enjoy.

Deploying VMWare Photon Platform 1.2

I am late to the party in the Container platform space but my exploration is way overdue so I am aiming at spending some overdue R&D time assessing VMWare Photon Platform from a “vSphere/vCloud” administrators point of view. My goal is to explore how these could be deployed at scale, challenges and advantages of the platform and to develop content on design considerations, deployment and Day 1 and Day 2 considerations. These will be long posts my apologies :S

VMware Photon Controller is an open-source multi-tenant host controller that you can use to manage hardware, containers, VMs, and host clusters. It provides a distributed, high-scale control plane optimized for cloudnative applications, which include containers and developer frameworks, such as Pivotal Cloud Foundry (PCF).

Quick Summary of Architecture

VMware Photon Platform is an open source “container-optimized cloud platform” for Cloud-Native Applications.

  • No vCenter Server; Photon Controller is a decentralized Control Plane
  • ESXi Hosts are used as the Virtualization Platform its designed to scale !
  • No scaling limitations for hosts; can have 100′s or 1000s of ESXi hosts
  • vCloud Director type resource management
  • Has the concept of fault domains, group of underlying hypervisors etc.
  • One-click provision of Kubernetes  as a service; fully supported by VMWare
  • NSX-T and vSAN integrations

Advantages for vCloud/vSphere Admin

  • Familiar management and troubleshooting for underlying Hypervisor; if you’re a vSphere admin than troubleshooting the hypervisor (e.g. performance issues) is going to be a familiar process
  • Trusted platform : ESXi is a solid hypervisor
  • Cost : You can deploy a pretty capable solution for low/zero cost
  • Interoperability and API : Interoperability with existing toolsets
  • Open Source : If you don’t like something, change it, if something breaks, fix it or at least have a look at why
  • Multi-tenant and scalability is massive


  • Deployment effort: Its moderate; the configuration is not to forgiving at typos’ etc. playing around with the configuration/troubleshooting installation issues took a moderate effort – budget some time to Lab
  • Documentation and community content on components (e.g. Project Lightwave) is light (no pun intended) and is often hidden throughout the source tree; be prepared to troubleshoot problems to resolution yourself
  • vSphere Integrated Container deployment is a lot easier:) and has many advantages (but also disadvantages); they both have there place; Photon is for scaling.

Lessons Learnt/Important Notes

I am going to put all this at the front because this is all the little things that caught me; the following are some of the lessons learnt through my Lab deployment which hopefully will save you some time;

Before you begin 

  1. Review the Issues registers for Photon and Lightwave (https://github.com/vmware/photon-controller/issues) before you begin; there are a number of bugs that might catch you
  2. Read the Release Notes


  1. If you plan to deploy NSX-T you must deploy this before you deploy the Photon Platform
  2. Lightwave Appliance (the equivalent of the Platform Services Controller in vSphere) is basically a Domain Controller and has its own DNS service which MUST be used by all of the Photon Platform components for everything to work – this is important as when you are installing/configuring the Photon Platform appliances the “dns” configuration needs to point to the IP of the Lightwave appliances and the DNS of the Lightwave appliances to your actual DNS infrastructure.
  3. The Appliances deployed from the Photon Installer are deployed Thick Provisioned so you require a minimum of 40GB free on your LUN per appliance; in a small deployment there is 3 appliances (Lightwave, Photon Controller and Load Balancer) however if deploying with HA make sure you check disk requirements and ensure you have enough for the vswap files etc. when the machines are Powered On etc.
  4. Lightwave password has a password policy; the Lightwave Administrator and root password must be at have at least one uppercase, one lowercase, one digit, and one punctuation character.
  5. If you intend to use ESXi 6.0 “Free”;
    1. Leave the node in evaluation mode while your building – the ESXi 6.0 Free license disables the vSphere 6.0 API which I used at least to configure the host; once you install the license you loose some PowerCLI capability
    2. ESXi 6.5 “free” has only a restriction on guests which are restricted to 8 vCPU’s; for container host platforms this may be sufficient but assess for your needs
  6. SSH must be enabled on the ESXi hosts to deploy the Photon Agents
  7. You can deploy many Lightwave and Photon Controller Appliances but only one HAProxy Load Balancer through the deployment appliance
  8. You can deploy your management plane components on the same hosts as your “Cloud” hosts (Hosts that will host customer containers) however you may wish to separate your management components on their own hosts/storage to prevent “noisy” containers impacting your management plane and to simplify your patching/maintenance strategies


  1. The Photon Controller 1.2 documentation wiki is not a complete resource; as an open-source offering it has a decent start at documentation on the wiki however be prepared to troubleshoot issues; the log locations are documented on the wiki
  2. There is documentation hidden in text files throughout the source on Github; I recommend you go hunting through the Github repository for text files s as there is tools and other tidbits “hidden” away

Post Deployment

  1. When connecting to the Web Interface you MUST connect via the IP address and not the DNS name of the Load Balancer (https://<ip-address-of-load-balancer>:4343) if you attempt to connect using the FQDN after the deployment you will just end up in a redirect loop. I have not investigated fully but it appears to be due to the token/header forwarded to the Platform Controller.
  2. Installer Appliance
    1. Change the password (root:changeme) to something secure; it holds configuration data with passwords in plain text so secure it
    2. Turn it off; you don’t need it after the build is complete (If you labbing don’t delete it because you’re probably going to redeploy the environment a few times)

Step 1. Designing/Building Hardware/ESXi Platform

Without a vCenter and access to the vSphere API (if using Free ESXi license) there are a few challenges for managing the infrastructure platform; no Update Manager, no Auto Deploy or Host Profiles, no agent upgrade management and limited host/platform monitoring out of the box. In my environment I have the following  host deployment methodology for a “light touch” deployment of new hosts/patching;

  1. A PXE Network Installation Service which performs a zero touch base install of ESXi 6.5d (Refer to this document for configuring this up)
  2. A script that configures a host for Photon Platform and deploys the agent;
    1. Configures NTP, SSHD, vSwitches, iSCSI and installs the license
    2. Optionally deploys the Photon Agents and adds to the Cloud
  3. A script to place Photon Platform Cloud ESXi hosts into “Maintenance” and to patch and deploy the Agents

You could do this a number of ways but for my environment all I need to do is create a DHCP reservation for the new host, add a DNS entry and then PXE Boot into ESXi installer and a Kickstart script deploys the host with a basic deployment. I then customise the host with the following basic script:

Next you will need to design allocate some Network address space for the deployment. The following addressing is required;

  • Networking for ESXi Hosts (Management/iSCSI/etc.)
  • Static IP address for each of the Photon Management Controllers; you need at least one but for HA and load you should have several in different availability zones
  • Static IP for for each of the Lightwave Appliance; these are like Windows Domain Controllers so for HA you should have at least one in each site
  • Static IP for the Load Balancer Appliance
  • Static IP for the vSAN Management Appliance if deploying
  • An IP address for the Deployment Appliance

Step 2. Download the Deployment Appliance and Deploy

An appliance has been developed which you can deploy onto a host and basically roll the platform out via a YAML configuration file. The process is relatively straight forward however the YAML file needs some attention.

  1. Download the latest OVA from here and
  2. Connect to the ESXi host prepared and deploy the OVA (installer-ova-nv-1.2-dd9d360.ova), select storage, Agree to the EULA and enter the some networking details for the appliance and click Finish
  3. Once the deployment completes Power On the appliance and SSH to the device using the default credentials (root:changeme) and change the password 
Step 3. Create a YAML Deployment Configuration File 

The installer script uses a YAML configuration file which describes the environment to deploy the components. In my deployment I am not using VSAN (basically only using “free” components in this lab). For details on deploying VSAN please refer to this document.

The YAML file is made up of 6 distinct parts; three are mandatory (Compute, Lightwave, Photon) for the deployment and the others are optional (Load Balancer, vSAN, NSX):

  • Compute : ESXi hosts are defined
    • Pretty self-explanatory however the dns address is the IP of the Lightwave appliance and the allowed-datastores is the datastores that are presented through to Photon Platform
  • Lightwave: An VMWare open-source directory service : Lightwave furnishes a directory service, a certificate authority, a certificate store, and an authentication service. (Basically PSC for Photon). The options are mostly straightforward however; there are a few that need some explanation;
    • domain : The domain is the “SSO Authentication Domain”; it can be anything however for simplicity align it with a DNS namespace in your Organisational DNS hierarchy and I don’t recommend making it the same as your Active Directory Forest/Domain if you plan on using this as an authentication provider
    • password : Must meet a default password policy (1 upper case, 1 lower case, 1 digit, 1 special character)
    • hostref: This is the host (as defined in the compute) to deploy the appliance
    • datastore: The datastore to install the appliance onto on the host
    • network : This is the Port Group on the host the NIC will be connected
    • dns : In this section (and this section only) this is the actual DNS servers for your organisation
  • Photon Controller : The Management Appliance
    • img-store datastore: The datastore on an ESXi hypervisor for images for VMs and Kubernetes clusters, etc.
    • cloud hostref : ESXi hypervisor to manage/host the containers
    • hostref: This is the host (as defined in the compute) to deploy the appliance
    • datastore: The datastore to install the appliance onto on the host
    • network : This is the Port Group on the host the NIC will be connected
    • dns : Address is the IP of the Lightwave appliance(s)
  • Load Balancer :  A HAProxy Appliance; this is optional however provides a basic load balancing service for the Photon Controller
    • hostref: This is the host (as defined in the compute) to deploy the appliance
    • datastore: The datastore to install the appliance onto on the host
    • network : This is the Port Group on the host the NIC will be connected
    • dns : Address is the IP of the Lightwave appliance(s)

 The following is an example YAML for a basic single node deployment:

The following is a slightly more available solution with 2 Lightwave servers and Photon Controllers to give you an idea of scaling the Management Plane:


Once you have created a configuration file check it is valid with http://www.yamllint.com/

Step 4. Deploy the Platform

Finally; once you have a validated YAML configuration file logon to the device using SSH and create the YAML configuration file from Step 3 using vi/upload the file created to /root and execute photon-setup platform install -config config.yaml where config.yaml is your configuration file and watch the magic happen.

Once the installation is complete if it is successful you should be able to connect to the Web Interface by navigating to the IP Address of the Load Balancer (https://<ip-address-of-load-balancer>:4343). You MUST connect via the IP address and not the DNS name; if you attempt to connect using the FQDN after the deployment you will just end up in a redirect loop.

Next Steps

  1. Configuring Active Directory Integration with Lightwave
  2. Customizing the Lightwave SSO Sign-On Page

I will hopefully have a lot more content to come on this topic in the next few weeks; learning more and more everyday about the product so hopefully I can develop some content; currently looking at deploying Kubernetes-as-a-service, replacing the certificates with Enterprise PKI certificates, alternative load balancing strategies and automation of maintenance via the API/PowerShell, backup and restore and upgrades…so a lot of things to come :) Watch this space …

Further Reading and Troubleshooting Resources

  1. Photon Platform Wiki – Troubleshooting
  2. Running Containers at Scale with Photon Platform
  3. Photon Platform Getting Started Guide
  4. Cormac Hogan’s Photon Controller Blog Posts – Excellent detailed resources
  5. VMWare Comminity DevOps Forums

Customizing the Lightwave SSO Sign-On Page

The following is a brief post to outline how to customise the SSO Sign-On page for Project Lightwave. The Project Lightwave Single Sign-On Service is used as the authentication engine for the Photon Platform. Users will be displayed this page whenever they logon to the service and parts of the page can be customized using the Lightwave Administration UI. The following will walk through the configurable options and the anatomy of the page.

Name : HTML to Display in the Header of the SSO Page
Display Banner Checkbox : If selected this requires users to check a box “I agree to XXXXXXX” before they are able to login or they are displayed with a error “In order to use our services, you must agree to XXXXXXXX” if this is not enabled (and Hide Banner and Title is disabled) the link is shown to the Banner however users are not required to agree.
Hide Banner and Title: If this is enabled the Banner is not shown to the users
Content: HTML to display when users click the “I agree to XXXXXXX”

To change the settings and customize these options select Policies & Configuration from the side menu and select Login Banner and click Edit you can then Edit each of the settings discussed above.

Configuring Active Directory Integration with Lightwave on the VMware Photon Platform 1.2

So for the month of June I have decided to lab the Photon Platform 1.2 by VMWare and will be posting a bunch of content related to the product. The Photon Platform leverages Project Lightwave as its directory service.

Project Lightwave is an open source project comprised of enterprise-grade, identity and access management services targeting critical security, governance, and compliance challenges for Cloud-Native Apps within the enterprise. For vSphere Admins Lightwave performs many of the same functions as the Platform Services Controller;

  • Lightwave Directory Service – standards based, multi-tenant, multi-master, highly scalable LDAP v3 directory service
  • Lightwave Certificate Authority – directory integrated certificate authority helps to simplify certificate-based operations and key management across the infrastructure.
  • Lightwave Certificate Store – endpoint certificate store to store certificate credentials.
  • Lightwave Authentication Services – cloud authentication services with support for Kerberos, OAuth 2.0/OpenID Connect, SAML and WSTrust
  • Lightwave Domain Name Services – directory integrated domain name service to ensure Kerberos Authentication to the Directory Service and Authentication Service (STS)
The following outlines how to configure the Lightwave Domain Controllers to use Active Directory Domain Services as an identity provider which will enable users to leverage their Active Directory credentials to access and administer the Platform.

Before you begin you will need the following:

  1. A service account (just a Domain User account with ability to read the Active Directory Domain)
  2. The domain LDAPS certificate for the domain controller; to obtain this open the Local Computer\Personal\Certificates store on the domain controller and export the Certificate (without the private key) for the Certificate using the Certificate Template Domain Controller in Base64 Format

Step 1. Navigate to the LightWave Domain Controller administration page (https://lightwavefqdn/lightwaveui/) and enter the LightWave domain and when prompted enter the LightWave administrator account

Step 2. Select Identify Sources from the side menu and click Add

Step 3. Select Active Directory as LDAP and click Next

Please Note: At the time of writing the option Active Directory (Integrated Windows Authentication) does not appear to function/there is no UI options to add the machine to the domain; I will investigate further at a later time but I imagine that the Lightwave machines need to be added to the domain via the CLI first.

Step 4. Enter the details for the LDAPS service and the Base DN for the Users and Groups; I have just used the root of the domain however you can scope these to Containers further down your tree as per your requirements

Step 5. Select Choose File and select the certificate for the Domain Controllers LDAPS service and click Next

Step 6. Enter the Service Account credentials (Username in the UPN format) and click Test Connection if the connection is successful click Next

Step 7. Finally review and click Save to complete the configuration

Step 8.
Next select Users & Groups from the side-menu and under Groups select Administrators and click Membership

Step 9. Select the domain from the drop-down menu and locate the User or Group to grant the permissions (in the below example the group R-Photon-Admins), check the checkbox next to the object and select Add Member followed by Save

Finally; sign out of the Platform and Sign back in with the Active Directory account entering the Domain Account Username and Password and clicking Login

NOTE: Do not use the “Use Windows session authentication” it doesn’t work during testing (throws “Internal Processing error”). And voila your Active Directory environment can be leveraged for identity. 

Copy-VMGuestFile “The request was aborted” Exception – Mystery Solved !

Today I thought I would examine an issue that has come up from time to time over the past  6 or 7 years or so which I have never sat down and examined properly; when running the PowerCLI Copy-VMGuestFile cmdlet for larger files an exception is thrown “The request was aborted: The request was cancelled”.

The fix is simple: The WebOperationTimeoutSeconds parameter for PowerCLI needs to be increased from the Default of 300 seconds to a larger value to allow enough time for the transfer to complete or set to infinite.  A big thanks to @vmkdaily on the VMWare {Code} Slack for the info for the solution.

To set to infinite simply execute Set-PowerCLIConfiguration -WebOperationTimeoutSeconds -1 as an Administrator, restart PowerCLI and then rerun the cmdlet.

Some further information
Below is just a summary of the mechanism of how the GuestFileManager and other Guest Processes works for transferring files between the Guest and the local machine via the API. The GuestOperationsManager is a really important and versatile library as it allows for two-way execution and information passing between guests and Orchestration platforms WITHOUT network connectivity between the management plain and the virtual machine which becomes particularly for operations of scale. I use the Invoke-VMScript and Copy-VMGuestFile quite a bit for everything to automating VM Hardware Upgrades and guest reconfiguration/customisation to shipping and executing patches or build automation for customers in a Service Provider environment where I have no network inter-connectivity to their environments.

So the VIX API operations were moved into the vSphere API in vSphere 5 and the process flow for a call is as follows;

  1. A vSphere Web services client program calls a function in the vSphere API.
  2. The client sends a SOAP command over https (port 443) to vCenter Server.
  3. The vCenter Server passes the command to the host agent process hostd, which sends it to VMX.
  4. VMX relays the command to VMware Tools in the guest.
  5. VMware Tools has the Guest OS execute the guest operation.

In PowerCLI this is exposed via the GuestOperationsManager. For the Copy-VMGuestFile cmdlet a call is made to the $GuestOperationsManager.FileManager.InitateFileTransferXXXX methods with the Virtual Machine object, the Guest Authentication details, the destination where the file will be placed on the guest (and filename and attributes) and the size of the file. The returned value is a unique URI with a Token/Session ID which is used via a HTTP PUT to upload the file. The below is an example implementation if you wish to have a play around:

Further information

PowerCLI : Get/Set cmdlets for DNS Configuration of the vCenter Server Appliance

This week after responding to a post on the VMWare Community forums I started playing around with the vSphere REST API. I have quite a bit of experience with the vCloud REST API which is the basis for all of the vCloud PowerCLI code but not a lot with JSON or the vSphere API. So I have thrown together some methods with an aim for reuse for all vSphere APIs that I thought I would share. The module is named Module-vCSA-Administration.psm1 and contains at present three main methods;

  • Connect-VIServerREST : Establishes a connection to the vSphere REST API on a vCenter Server
  • Get-VCSANetworkConfigDNS : Get the DNS Networking configuration for the vCSA
  • Set-VCSANetworkConfigDNS : Amend the DNS Networking configuration for the vCSA

The support methods in the module (Get-vSphereRESTResponseJSON, Update-vSphereRESTDataJSON, Publish-vSphereRESTDataJSON) provide a very simple wrapper that can be used to make any REST API call and extend the functionality. I hope someone finds these useful; I will probably extend these modules further but was a good introduction into playing with JSON based REST API for vSphere.

I generally try to document as much as possible so you can use the get-help cmdlet command to review the help for each of the cmdlets (e.g. Get-help Connect-VIServerREST).

In order to use the module you must load the module using the Import-Module <Path to Module> cmdlet

Use the Connect-VIServerREST to connect to the REST API. The REST API uses the vSphere SSO Service to authenticate sessions so use the credentials (in the format you using use) to logon to the REST API. The method will connect to the REST API and stores the session information in a Global Variable $DefaultVIServerREST for use by other methods.

Once the connection is established you can use the other methods to retrieve or set the DNS configuration either setting to DHCP or setting static servers using the Get-VCSANetworkConfigDNS or Set-VCSANetworkConfigDNS cmdlets.

Module is available from: https://raw.githubusercontent.com/AdrianBegg/vSphere/master/Module-vCSA-Administration.psm1


vCloud Director released

It’s been a big week of announcements and probably the most exciting for me is the release of vCloud yesterday not because of any massive set of new features but because the maintenance release addresses a number of known bugs that have existed in the product for some time which I have previously written on this blog about. One notable new feature is support for VVol (Virtual Volume) datastores which is another great inclusion. The release of vCloud Director 8.20 increasing capability and a maintenance release within 3 months addressing a swagger of bugs is a very positive sign for the product allowing providers to remain competitive and hopefully greater integration between the vCloud product and the rest of the VMWare product suite. A full list of release notes is available here.


  • Download vCloud Director for Service Providers
  • Review the release notes and upgrade guide
  • Review the VMware Product Interoperability Matrices for your environment (if your running 8.20 already there are no changes)
  • Take a backup of your vCloud Director database and cells
  • Change the permissions on the binary to allow execution (chmod +x vmware-vcloud-director-distribution-8.20.0-5515092.bin)
  • Stop user access to the cells by executing the /opt/vmware/vcloud-director/bin/vmware-vcd-cell maintenance command
  • Execute the installer to upgrade the cells
  • Stop the vCloud Director cells if they are running and run the database schema upgrade tool (/opt/vmware-vcloud-director/bin/upgrade) and walk through the wizard
  • Restart cells and monitor the cell start (tail -f /opt/vmware/vcloud-director/logs/cell.log) and once the load is complete log onto vCloud Director and check the version


PowerCLI module to manage Organisation Rights in vCloud Director 8.20

UPDATE: 16/09/2017 – Updated the Module and added new cmdlets and improved error checking. Tested on vCloud Director 8.20.01 and available from GitHub - see this post for new cmdlets.

Good day ! Today I wanted to quickly write up a post about some modules I have been working on for PowerCLI to expose/automate/simplify manipulation of the vCloud Organisation Rights. A really cool addition to vCloud Director 8.20 is the NSX feature parity and the HTML5 User Interface for Edge Gateways that comes with it and a change to how the vCloud Organisation roles work which is now granular for each Organisation (use to be rights enabled for all or none).

So presently if you wish to turn on Distributed Firewall and Advanced Networking Services Rights you have to do this via the API for each organisation in vCloud. (Presently no GUI access to turn these features On) which is described in VMWare KB2149016. These modules (available on GitHub) are designed to make this process a bit easier.

Some quick notes about enabling these Services:

  1. Check your licensing  – the Advanced Networking capabilities might be there but you might not be able to provide them to customers depending on your Service Provider entitlements
  2. The new features require an Edge to be upgraded to an Advanced Edge – you may for many reasons not want customers to be able to do this however by default customers have the ability to Convert Edges (Org Admin)

This code need work and I will be making a revision and extending these as I explore the new APIs and the gaps in the builtin cmdlets however please enjoy any bugs/feedback is appreciated. Load via “Import-Module Module-vCloud-RightsManagement.psm1″

Below are a quick summary of the main user functions;

The cmdlet is designed to allow the Org Rights to be exported to a CSV for manipulation by other tools which can be later imported back into vCloud. This could be used for third party compliance/reporting or just for loading into Excel to enable/disable selected services in a nicer interface then direct API calls. The CSV is just in the format of the Role Name and if it is enabled (true) or disabled (False) for the Org.

The cmdlet will replace the OrgRights with those set to enabled in the provided CSV (which should be generated from an Export-CIOrgRights cmdlet.

Returns a collection of the available rights in the cloud and if they are enabled for the provided Organisation

Returns a collection of the available rights in the Global cloud infrastructure.

Adds a single vCloud Director right to an Organisation

Removes a single vCloud Director right from an Organisation


Increasing VMware vCloud Director for Service Providers REST API XML element limit

vCloud Director for Service Providers has a pre-configured limit for REST API calls that restricts POST calls to 4,096 XML elements. The behaviour is designed as a security precaution and is not a technical/addressing limitation and as such it can be increased if required. Most API calls will not exceed this limit however if you are modifying Edge Gateway services or similar and have large firewall rulesets you will quickly exceed the limit and be presented with HTTP/1.1 500 Server Error with an exception com.vmware.vcloud.api.rest.toolkit.exceptions.InternalServerErrorRestApiException: com.vmware.vcloud.security.filters.ValidityException: The provided XML has too many elements: XXXX. The maximum is 4,096.

The setting is not exposed in the UI  and is not documented anywhere to my knowledge but the process is pretty straightforward to amend; to increase the limit the following query is required to be run against the vCloud DB.

Step 1. Take a backup of the vCloud Director database
Step 2. Logon to your DBMS and execute the following query against your vCloud Database to determine if a custom value has been set; if no value is returned the default will be used (4,096)

Step 3a. If no value is returned execute the following query to set a higher limit replacing the value with the desired element limit; in the below example I will set the allowed XML elements for a REST API call to 24,384:

Step 3b. If a value was returned by the query in set 2 then the setting has been previously set; to increase or decrease the value use the following query; in the below example I will set the allowed XML elements for a REST API call to 6,096:

Once the setting has been added/amended the change will take effect immediately; there is no requirement to restart the cells just re-run the REST API call and it should succeed.

Identify VMs affected by VMware Tools and RSS Incompatibility Issue

Last month VMWare published a blog post (https://blogs.vmware.com/apps/2017/03/rush-post-vmware-tools-rss-incompatibility-issues.html) about an issue affecting Windows Server 2012/2012 R2 and newer Guest OS’s running VMware Tools versions 9.10.0 up to 10.1.5, the paravirtual VMXNet3 driver. Basically Windows Receive Side Scaling (RSS) is not functional. RSS enables network adapters to distribute the kernel-mode network processing load across multiple processor cores in multi-core computers. The distribution of this processing makes it possible to support higher network traffic loads than would be possible if only a single core were to be used. This still hasn’t been fixed but when it does you may need to determine which machines are still impacted.

In order to determine which machines are exposed/potentially impacted in your environment by this bug (and also check if RSS is flagged as enabled in guest) the following script can be used. Once this bug is fixed; if you have machines that don’t have RSS enabled that have multicore you should consider enabling it !

For the latest version please refer to https://github.com/AdrianBegg/vSphere/