Creating a Rancher cluster with Windows worker nodes - Keyva

In this guide we will deal with building a Rancher cluster with windows worker nodes. The cluster will still need a Linux master and worker node as well. As with our last Rancher blog post we will be using CentOS 7. Please see our last blog post about setting up a Rancher management node if you do not already have one. That part of the process is the same. We are going to assume you are starting at the point that you have a Rancher management interface up and accessible to log in to.

In order to allow us to use Windows worker nodes we will need to create a custom cluster in Rancher. This means we will not be able to use Rancher’s ability to automatically boot nodes for us and we will need to create the nodes by hand before we bring up our Rancher cluster.

We are going to use VMware vSphere 6.7 for our VM deployments. The windows node must run Windows Server 2019, version 1809 or 1903. Kubernetes may fail to run if you are using an older image and do not have the latest updates from Microsoft. In our testing we used version 1809, build 17763.1339 and did not need to install and additional KBs manually. Builds prior to 17763.379 are known to be missing required updates. It is also critical that you have VMware Tools 11.1.x or later installed on the Windows guest VM. See here for additional details on version information.
https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info

1. Provision two CentOS 7 nodes in VMware with 2CPUs and 4GB of RAM or greater.
2. After they have booted, log in to the nodes and prepare them to be added to Rancher. We have created the following script to help with this. Please add any steps your org needs as well. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-centos7-node-prep.sh
3. Provision the windows server worker node in vSphere, note that 1.5 CPUs and 2.5GB of RAM are reserved for windows. You may want to over-provision this node by a bit. I used 6CPUs and 8GB ram so there was some overhead in my lab.
4. Modify the windows node CPU settings and enable “Hardware virtualization”, then make any other changes you need and boot the node.
5. You can confirm the windows node version by running ‘winver’ at the powershell prompt.
6. Check to make sure the VMware Tools version you are running is 11.1.0 or later.
7. After you boot the windows node open an admin powershell prompt and run the commands in this powershell script to set up the system, install docker and open the proper firewall ports. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-windows-node-prep.ps1
8. After you run the script you can then set the hostname, make any other changes for your org and reboot.
9. Once the reboot is complete open a powershell prompt as admin and run ‘docker ps‘, then run ‘docker run hello-world‘ to test the install.

There are more details here on the docker install method we used:
https://github.com/OneGet/MicrosoftDockerProvider

This page contains documentation on an alternate install method for docker on windows:
https://docs.mirantis.com/docker-enterprise/current/dockeree-products/docker-engine-enterprise/dee-windows.html

For some windows containers it is important your base images matches your windows version. Check your Windows version with ‘winver’ on the command prompt.
If you are running 1809 this is the command to pull the current microsoft nanoserver image:

docker image pull mcr.microsoft.com/windows/nanoserver:1809

Now that we have our nodes provisioned in VMware with docker installer we are ready to create a cluster in Rancher.

1. Log in to the rancher management web interface, select the global cluster screen and click “add cluster”.
2. Choose “From existing nodes (custom)” this is the only option where windows is supported currently.
3. Set a cluster name, choose your kubernetes version, for Network Provider select “Flannel” from the dropdown.
4. Flannel is the only network type to support windows, the windows support option should now allow you to select “Enabled“. Leave the Flannel Backend set to VXLAN.
5. You can now review the other settings, but you likely don’t need to make any other changes. Click “Next” at the bottom of the page.
6. You are now presented with the screen showing docker commands to add nodes. You will need to copy these commands and run them by hand on each node. Be sure to run the windows command in an admin powershell prompt.
   1. For the master node select Linux with etcd and Control Plane.
   2. For the linux worker select Linux with only Worker.
   3. For the windows worker node select windows, worker is the only option.
7. This cluster will now provision itself and come up. This may take 5-10 mins.
8. After the cluster is up select the cluster name from the main drop down in the upper left, then go to “Projects/Namespaces” and click on “Project: System”. Be sure you are on the Resources > Workloads page. All services should say “Active”. If there are any issues here you may need to troubleshoot further.

Troubleshooting

Every environment is different, so you may need to go through some additional steps to set up Windows nodes with Rancher. This guide may help you get past the initial setup challenges. A majority of the issues we have seen getting started were caused by DNS, firewalls, selinux being set to “enforcing”, and automatic certs that were generated using “.local” domains or short hostnames.

If you need to wipe Rancher from any nodes and start over see this page:
https://rancher.com/docs/rancher/v2.x/en/cluster-admin/cleaning-cluster-nodes/

You can use these commands in windows to check on the docker service status and restart it.

sc.exe qc docker
sc.exe stop docker
sc.exe start docker