Blog & Insights

By Brad Johnson, Lead DevOps Engineer

In this guide we will deal with building a Rancher cluster with windows worker nodes. The cluster will still need a Linux master and worker node as well. As with our last Rancher blog post we will be using CentOS 7. Please see our last blog post about setting up a Rancher management node if you do not already have one. That part of the process is the same. We are going to assume you are starting at the point that you have a Rancher management interface up and accessible to log in to.

In order to allow us to use Windows worker nodes we will need to create a custom cluster in Rancher. This means we will not be able to use Rancher’s ability to automatically boot nodes for us and we will need to create the nodes by hand before we bring up our Rancher cluster.

We are going to use VMware vSphere 6.7 for our VM deployments. The windows node must run Windows Server 2019, version 1809 or 1903. Kubernetes may fail to run if you are using an older image and do not have the latest updates from Microsoft. In our testing we used version 1809, build 17763.1339 and did not need to install and additional KBs manually. Builds prior to 17763.379 are known to be missing required updates. It is also critical that you have VMware Tools 11.1.x or later installed on the Windows guest VM. See here for additional details on version information.
https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info

1. Provision two CentOS 7 nodes in VMware with 2CPUs and 4GB of RAM or greater.
2. After they have booted, log in to the nodes and prepare them to be added to Rancher. We have created the following script to help with this. Please add any steps your org needs as well. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-centos7-node-prep.sh
3. Provision the windows server worker node in vSphere, note that 1.5 CPUs and 2.5GB of RAM are reserved for windows. You may want to over-provision this node by a bit. I used 6CPUs and 8GB ram so there was some overhead in my lab.
4. Modify the windows node CPU settings and enable “Hardware virtualization”, then make any other changes you need and boot the node.
5. You can confirm the windows node version by running ‘winver’ at the powershell prompt.
6. Check to make sure the VMware Tools version you are running is 11.1.0 or later.
7. After you boot the windows node open an admin powershell prompt and run the commands in this powershell script to set up the system, install docker and open the proper firewall ports. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-windows-node-prep.ps1
8. After you run the script you can then set the hostname, make any other changes for your org and reboot.
9. Once the reboot is complete open a powershell prompt as admin and run ‘docker ps‘, then run ‘docker run hello-world‘ to test the install.

There are more details here on the docker install method we used:
https://github.com/OneGet/MicrosoftDockerProvider

This page contains documentation on an alternate install method for docker on windows:
https://docs.mirantis.com/docker-enterprise/current/dockeree-products/docker-engine-enterprise/dee-windows.html

For some windows containers it is important your base images matches your windows version. Check your Windows version with ‘winver’ on the command prompt.
If you are running 1809 this is the command to pull the current microsoft nanoserver image:

docker image pull mcr.microsoft.com/windows/nanoserver:1809

Now that we have our nodes provisioned in VMware with docker installer we are ready to create a cluster in Rancher.

1. Log in to the rancher management web interface, select the global cluster screen and click “add cluster”.
2. Choose “From existing nodes (custom)” this is the only option where windows is supported currently.
3. Set a cluster name, choose your kubernetes version, for Network Provider select “Flannel” from the dropdown.
4. Flannel is the only network type to support windows, the windows support option should now allow you to select “Enabled“. Leave the Flannel Backend set to VXLAN.
5. You can now review the other settings, but you likely don’t need to make any other changes. Click “Next” at the bottom of the page.
6. You are now presented with the screen showing docker commands to add nodes. You will need to copy these commands and run them by hand on each node. Be sure to run the windows command in an admin powershell prompt.
   1. For the master node select Linux with etcd and Control Plane.
   2. For the linux worker select Linux with only Worker.
   3. For the windows worker node select windows, worker is the only option.
7. This cluster will now provision itself and come up. This may take 5-10 mins.
8. After the cluster is up select the cluster name from the main drop down in the upper left, then go to “Projects/Namespaces” and click on “Project: System”. Be sure you are on the Resources > Workloads page. All services should say “Active”. If there are any issues here you may need to troubleshoot further.

Troubleshooting

Every environment is different, so you may need to go through some additional steps to set up Windows nodes with Rancher. This guide may help you get past the initial setup challenges. A majority of the issues we have seen getting started were caused by DNS, firewalls, selinux being set to “enforcing”, and automatic certs that were generated using “.local” domains or short hostnames.

If you need to wipe Rancher from any nodes and start over see this page:
https://rancher.com/docs/rancher/v2.x/en/cluster-admin/cleaning-cluster-nodes/

You can use these commands in windows to check on the docker service status and restart it.

sc.exe qc docker
sc.exe stop docker
sc.exe start docker

At Keyva we often meet with clients that need just a little help, something to get them over the hump and continue on their way building out new and exciting IT capabilities. This seems to happen most often when organizations adopt new and emerging technologies. Often teams haven't built up their internal skills and capabilities around tech like Kubernetes or automation platforms such as Red Hat Ansible. Or, perhaps the team is already very skilled, but want someone to help with their OpenShift 3.x -> 4.x upgrade path, or need someone to write a new Ansible module so that they can expand their ability to offer automation capabilities via their playbooks.  

There hasn't been a good way to get this kind of incremental help – no granular consumption model for technical expertise – it's not a function of your vendor's L1 support. Your vendor will tell you to buy a TAM or their own expensive consulting services. It's also not something readily available in the community at large. There are user forums and networks for days, but will you get a response to your questions? Will the responses be correct?  

Keyva created Guru Services to address this exact issue. It's more than L1 support, not as heavy as a consulting engagement, it's enterprise grade and far more reliable than crowdsourcing the community for answers and assistance. 

Guru Service is just as easy to use: choose from 3 different service levels and you're on your way. You'll have access to our client portal from which you can schedule your On Demand Guru. We'll send you a meeting invite with web conference information and you'll be over the hump and on your way in no time. We currently provide On Demand Gurus for Red Hat Ansible, OpenShift, and Kong and are actively adding technologies to our suite of Guru Services. To learn more about these offerings, check out our vendor pages for Kong (https://keyvatech.com/kong-enterprise/) and Red Hat (https://keyvatech.com/red-hat/). Reach out to our Keyva team at: info@keyvatech.com to request additional information or a quote on our Guru Services .  

With most of us in IT still working from home and unable to attend in-person events there's been an absolute explosion of online events. In our opinion here are some of the best technology events you can attend while we wait for the all-clear to resume attending our local events. The following list are a selection of events being put on by some of our vendor partners and are arranged by vendor.  

HashiCorp:  

We are a go-to-market and services delivery partner of HashiCorp. By now you've probably heard of them and their portfolio. Terraform is one of their flagship products, an excellent solution to deliver infrastructure as code and platform provisioning. They have 3 interesting events. The first of which is a customer roundtable (you need to register an account, but do not need to be a customer). These events are usually very good for seeing how other organizations out there are using Terraform. Additionally, there are two workshops later in the month where you can get some hands-on with Terraform in AWS or Azure. Visit the links below to register. Note all times are in CDT. 

Customer Roundtable – 7/16 – 12-2:30: https://events.hashicorp.com/strategydays/july16 

Terraform on AWS Workshop – 7/14 – 3-7: https://events.hashicorp.com/workshops/terraform-july14 

Terraform on Azure Workshop – 7/28 – 11-2:30: https://events.hashicorp.com/workshops/terraform-july28 

ServiceNow:  

We are an ISV of ServiceNow and have been building NOW certified integrations between ServiceNow and other major vendor solutions for years. This event should be interesting to all of us: Optimizing Agile Enterprises post COVID. Visit the link below to register for the event.  

Building, sustaining, and optimizing an agile enterprise post COVID-19 – 7/21 11-12: https://go.servicenow.com/LP=14515 

Snowflake:  

Snowflake is a next gen cloud-based data warehousing solution that is a truly unique and transformational technology. If you have an old, crusty data warehouse on prem gathering dust while it bleeds your team dry you should check out these events. They have a live demo every Thursday and you can hop into their well-attended users group taking place later this month. Visit the links below to register.  

Live Demo Every Thursday: https://www.snowflake.com/live-demo/?utm_cta=events-page-featured-live-demo (click Americas) and select the link for the session 

Snowflake users group – 7/15 – 11 AM: https://usergroups.snowflake.com/events/details/snowflake-group-by-presents-group-by-data-heroes-a-virtual-symposium-led-by-snowflake-users/?_ga=2.135575154.299628294.1594135230-1207227298.1594135230#/ 

curl https://releases.rancher.com/install-docker/19.03.sh | sh

mkdir /opt/rancher

docker run -d --restart=unless-stopped -p 80:80 -p 443:443 -v /opt/rancher:/var/lib/rancher rancher/rancher:latest

https://raw.githubusercontent.com/keyvatech/blog_files/master/centos7_cloudinit_vmtemplate.sh

Evolving Solutions and Keyva Named Red Hat Apex Partner

MINNEAPOLIS, JULY 1, 2020 – Evolving Solutions and its affiliated company, Keyva, jointly announced today that they have been named as a Red Hat Apex Partner in North America. As part of Red Hat’s partner ecosystem, Evolving Solutions and Keyva have shown that they have deep expertise in emerging technologies and hybrid cloud infrastructure platforms. Red Hat Apex Partners have made investments in Red Hat’s portfolio of application development, delivery, and integration resources, and bring industry expertise to Red Hat regulated industries. They are well-trained and very committed to working with Red Hat on business opportunities.

An invitation-only program, Red Hat Apex Partners are able to support implementations of Red Hat’s emerging technologies including Red Hat OpenShift, Red Hat Ansible Automation, Red Hat OpenStack Platform, Red Hat Middleware, Red Hat CloudForms, Red Hat OpenShift Container Storage, Red Hat Ceph Storage, and Red Hat Gluster Storage, to position customers for success. Partners such as Evolving Solutions and Keyva offer the technical expertise and working practices to deliver a high degree of client satisfaction across a range of deployment scenarios and projects.

Evolving Solutions and Keyva develop and support certified integrations between Red Hat emerging technologies such as OpenShift and Ansible and other leading technologies such as ServiceNow. End to end automation and integration is at the core of how the companies help clients drive business value and technical capabilities from their technology investments. “Our family of companies have a deep level of expertise in the Red Hat solutions and the broader technology marketplace,” says Jaime Gmach, President/CEO of Evolving Solutions and Keyva. “As highly skilled integrators, this collaboration allows us to offer clients greater benefit and deliver value to their organizations.”

According to Ernest Jones, Vice President of North American Partner Sales at Red Hat, “Red Hat’s partner ecosystem is a vital component in delivering powerful, flexible, and open solutions to global enterprises. We’re pleased to have Evolving Solutions and Keyva as Apex Partners and look forward to delivering open innovation to our joint clients with them.”

“The Apex partnership signifies Red Hat’s acknowledgment of Evolving Solutions’ and Keyva’s investment and success in delivering value for our clients using the Red Hat portfolio. As one of a small group of Apex partners in North America, we are well-positioned to help our clients succeed with their digital transformation utilizing best of breed technologies such as Ansible Tower and OpenShift,” states Gmach.

###

Media Contact

About Evolving Solutions

Evolving Solutions is a technology solutions provider that helps clients modernize and automate their mission-critical infrastructure to support digital transformation. Our business is client-centric, providing consulting and delivering technical solutions to enable modern operations in the hybrid cloud.

Evolving Solutions has deep partner relationships with IBM, HPE, Cisco, AppDynamics, Dynatrace, NetApp, Nutanix, Azure, Red Hat, AWS and over 70 other vendors that help us deliver exceptional results to service our clients’ technology needs. Learn more at www.evolvingsol.com.

About Keyva

Keyva is a consulting firm focused on delivering innovative technology solutions. Keyva simplifies IT to free up time and allow businesses to focus on their core offering and on client value. Keyva consultants help enterprises automate multi-clouds, multi-vendors, processes, applications and infrastructure within their environment, while leading transformation initiatives to allow companies to take the next step on their business journey.  Learn more at www.keyvatech.com.

Additional Resources

• Learn more about Evolving Solutions and Keyva
• Read more about Red Hat Connect for Business Partners

###

Red Hat, Red Hat Enterprise Linux, the Red Hat logo, Ansible and OpenShift are trademarks or registered trademarks of Red Hat, Inc. or its subsidiaries in the U.S. and other countries. Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries. The OpenStack Word Mark is either a registered trademark/service mark or trademark/service mark of the OpenStack Foundation, in the United States and other countries, and is used with the OpenStack Foundation's permission. Red Hat is not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

This guide will walk through how to set up Red Hat Ansible Tower in a highly-available configuration. In this example, we will set up 4 different systems – 1 for PostgreSQL database (towerdb), and 3 web nodes for Tower (tower1, tower2, tower3). 

We will be using Ansible Tower v3.6 and PostgreSQL 10, on RHEL 7 systems running in VMware for this technical guide. The commands for setting up the same configuration on RHEL 8 will be different for some cases.  This guide does not account for clustering of the PostgreSQL database. If you are setting up Tower in HA capacity for Production environments, it is recommended to follow best practices for PostgreSQL clustering, to avoid a single point of failure. 

First, we will need to prep all the RHEL instances by enabling the Red Hat repos. All the commands below are to be run on all 4 systems – towerdb, tower1, tower2, tower3 

subscription-manager register 

subscription-manager refresh 

subscription-manager attach –-auto 

subscription-manager repos –-list  

subscription-manager repos --enable rhel-7-server-rh-common-beta-rpms 

subscription-manager repos --enable rhel-7-server-rpms 

subscription-manager repos --enable rhel-7-server-source-rpms 

subscription-manager repos --enable rhel-7-server-rh-common-source-rpms 

subscription-manager repos --enable rhel-7-server-rh-common-debug-rpms 

subscription-manager repos --enable rhel-7-server-optional-source-rpms 

subscription-manager repos --enable rhel-7-server-extras-rpms 

sudo yum update 

sudo yum install wget 

sudo yum install python36 

sudo pip3 install httpie 

Also: 

1. a) Update the /etc/hosts file on all 4 hosts with entries for all systems
2. b) Add and copy thesshkeys on all systems 

On the Database system (towerdb), we will now set up PostgreSQL 10 

sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm 

sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm 

sudo yum install postgresql10 postgresql10-server 

Initialize the database 

/usr/pgsql-10/bin/postgresql-10-setup initdb 

systemctl enable postgresql-10 

systemctl start postgresql-10 

Verify you can log in to the database 

:~
nbsp;sudo su – postgres 
:~
nbsp;Psql  
# \list 
This command will show you the existing (default) database list. 
Next, we will configure the database to make sure it can talk to all the Tower web nodes: 
sudo vi /var/lib/pgsql/10/data/pg_hba.conf 

Add/update the line with 'md5' entry to allow all hosts:  
host    all             all             0.0.0.0/0            md5 
Update the postgresql.conf file 
sudo vi /var/lib/pgsql/10/data/postgresql.conf

Add/update the entry to listen to all incoming requests:  
listen_addresses = '*' 

Restart the database services, to pick up the changes made: 
sudo systemctl restart postgresql-10 

sudo systemctl status postgresql-10 
 
On each of the Tower web nodes (tower1, tower2, tower3), we will set up the Ansible Tower binaries: 
mkdir ansible-tower 

cd ansible-tower/ 

wget https://releases.ansible.com/ansible-tower/setup-bundle/ansible-tower-setup-bundle-3.6.2-1.el7.tar.gz 

tar xvzf ansible-tower-setup-bundle-3.6.2-1.el7.tar.gz  

cd ansible-tower-setup-bundle-3.6.2-1 

python -c 'from hashlib import md5; print("md5" + md5("password" + "awx").hexdigest())' 

md5f58b4d5d85dbde46651335d78bb56b8c 
Where password will be the password that you will be using a hash of, when authenticating against the database 
Back on the database server (towerdb), we will go ahead and set up the database schema pre-requisites for Tower install:  
:~
nbsp;sudo su – postgres 
:~
nbsp;Psql  
postgres=# CREATE USER awx; CREATE DATABASE awx OWNER awx; ALTER USER awx WITH password 'password'; 
On tower1, tower2, tower3, update the inventory file and run the setup. Make sure your script contents match on all tower web tier systems. 
You will need to update at least the following values and customize them for your environment: 
admin_password='password' 

pg_password='password' 

rabbit_mq = 'password' 
Under the [tower] section, you will have to add entries for all your tower web hosts. The first entry will typically serve as the primary node when the cluster is run.  
We will now run the setup script: 
./setup.sh 
You can either copy this inventory file on the other 2 tower systems (tower2 and tower3), or replicate the content to match the content in the file on tower1, and run the setup script on the other 2 tower systems as well.  
Once the setup script is run on all hosts, and it finishes successfully, you will be able to test your cluster instance. You can do so by going to one of the tower hosts URL, initiating a job template, and see which specific tower node it runs on – based on the tower node that is designated to be the primary node at that time. You will also be able to view the same console details, and logs of job runs, regardless of which tower web URL you go to.  
If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to info@keyvatech.com.

Let us look at setting up a Kubernetes cluster with 1 master node (kubemaster.bpic.local) and 2 worker nodes (kubenode1.bpic.local, kubenode2.bpic.local) on VMware based RHEL 7 instances. 

We have set up an additional user (other than root) on these machines, as we will be running kubectl (client) commands as the non-root user.

First, we will need to prep all the RHEL instances by enabling the Red Hat repos. All the commands below are to be run on all 3 components – kubemaster, kubenode1, kubenode2

subscription-manager register
subscription-manager refresh
subscription-manager attach –-auto
subscription-manager repos –-list
subscription-manager repos --enable rhel-7-server-rh-common-beta-rpms
subscription-manager repos --enable rhel-7-server-rpms
subscription-manager repos --enable rhel-7-server-source-rpms
subscription-manager repos --enable rhel-7-server-rh-common-source-rpms
subscription-manager repos --enable rhel-7-server-rh-common-debug-rpms
subscription-manager repos --enable rhel-7-server-optional-source-rpms
subscription-manager repos --enable rhel-7-server-extras-rpms

The rhel-7-server-extras-rpms repo contains docker and other utilities. 

Since this is our lab environment, we will be disabling firewalls. If it is a production environment, you can open up specific ports for communication of your applications, and for Kubernetes components instead of disabling the firewall completely.

systemctl disable firewalld 
systemctl stop firewalld

Since we are using VMware VMs, it is recommended to set up VMware-tools 

yum install perl 
mkdir /mnt/cdrom 
Mount /dev/cdrom /mnt/cdrom 
cp /mnt/cdrom/VMwareTools-version.tar.gz /tmp/ 
tar -zxvf VMwareTools-version.tar.gz 
/tmp/vmware-tools-distrib/./vmware-install.pl  
umount /mnt/cdrom 

Update the yum repositories

yum –y update yum 
install yum-utils

Configure additional settings

swapoff –a

Also, comment out the swap line in

etc/fstab 
#/dev/mapper/rhel-swap   swap                    swap    defaults        0 0 

Install and enable docker

yum –y install docker 
systemctl enable docker 
systemctl start docker 

Set up repo for Kubernetes

cat <<EOF > /etc/yum.repos.d/kubernetes.repo 
[kubernetes] 
name=Kubernetes 
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 
enabled=1 
gpgcheck=1 
repo_gpgcheck=1 
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg 
EOF 

Additional enforcement settings

setenforce 0

Update the config file to change the selinux settings

vi /etc/selinux/config 

Change the settings from 

selinux=enforcing to selinux=permissive 

Install and enable kubelet service

yum -y install kubelet kubeadm kubectl 
systemctl enable kubelet 

start kubelet 

Enable sysctl settings

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl –system

Alternatively, you can update the /etc/sysctl.conf file 

vi /etc/sysctl.conf 

Add/update the following lines

net/bridge/bridge-nf-call-iptables = 1 

net/ipv4/ip_forward = 1 

On the Kubernetes master node only, we will set up the flannel networking component using fat manifest:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 


kubeadm init --pod-network-cidr=10.244.0.0/16 


kubeadm token create --print-join-command 

Capture the results of the above command, specifically the part describing how to add nodes to this cluster

You can now join any number of machines by running the following on each node as root:

kubeadm join kubemaster.bpic.local:6443 --token cll0gw.50jagb64e80uw0da \ 

    --discovery-token-ca-cert-hash sha256:4d699e7f06ce0e7e80b78eadc47453e465358021aee52d956dceed1dfbc0ee34

And then after changing to a non-root user, run the following commands

su – nonrootuser 
mkdir -p $HOME/.kube 
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 
sudo chown $(id -u):$(id -g) $HOME/.kube/config 

On the Kubernetes nodes (kubenode1 and kubenode2), we will run the join command, to add those nodes to the cluster:

kubeadm join kubemaster.bpic.local:6443 --token cll0gw.50jagb64e80uw0da \ 

    --discovery-token-ca-cert-hash sha256:4d699e7f06ce0e7e80b78eadc47453e465358021aee52d956dceed1dfbc0ee34 
You can now test the cluster by running the below command on either of the nodes, or the master as the non-root user:

kubectl get nodes

You should see results like this (changed for your system names) showing the cluster configuration:

 
NAME                    STATUS   ROLES    AGE   VERSION 
kubemaster.bpic.local   Ready    master   15h   v1.17.3 
kubenode1.bpic.local    Ready    <none>   14h   v1.17.3 
kubenode2.bpic.local    Ready    <none>   14h   v1.17.3

If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to info@keyvatech.com.

By Anuj Tuli, Chief Technology Officer

Typically when you hear about containers and Kubernetes, it is in the context of Linux or Unix platforms. But there are a large number of organizations that use Windows and .NET based applications, and they are still trying to determine the best way forward for containerization of their Windows based business critical applications.  

Kubernetes added support for Windows based components (worker nodes) starting with release v1.14.  

In the example below, we will join a Windows worker node (v1.16.x) with a Kubernetes cluster v1.17.x. 

As of this moment, Windows worker nodes are supported on Windows 2019 operating system only. In this example, we will leverage the flannel network set up on our master node on RHEL (see instructions above).  

Step 1: Download the sig-windows-tools repository from https://github.com/kubernetes-sigs/sig-windows-tools , and extract the files 

Step 2: Navigate to, and update the Kubernetes configuration file at C:\<Download-Path>\kubernetes\kubeadm\v1.16.0\Kubeclustervxlan 

In our instance, we will update the following values: 

• Interface Name 
• Control Plane details – IP address, username, KubeadmToken, KubeadmCAHash. The values for these come from the output of the kubectl join command, when it was run during set up of the master node

Step 3: Open up PowerShell console in Admin mode and install kubernetes via the downloaded script. This step requires reboot of the server 

PS C:\Users\Administrator> cd C:\<Download-Path>\kubernetes\kubeadm 

PS C:\<Download-Path>\kubernetes\kubeadm> .\KubeCluster.ps1 -ConfigFile C:\<Download-Path>\kubernetes\kubeadm\v1.16.0\Kubeclustervxlan.json -install 

Step 4: Once K8s is installed, join it to the existing kubernetes cluster. This step takes the values you entered in the modified Kubeclustervxlan file 

PS C:\<Download-Path>\kubernetes\kubeadm> .\KubeCluster.ps1 -ConfigFile C:\<Download-Path>\kubernetes\kubeadm\v1.16.0\Kubeclustervxlan.json -join 

Step 5: Verify that the Windows worker node was successfully added to the cluster. You can do this by running the kubectl command from any client (Windows or Linux nodes on the cluster)  

PS C:\<Download-Path>\kubernetes\kubeadm> kubectl get nodes 

NAME                    STATUS   ROLES    AGE   VERSION 
kubemaster.bpic.local   Ready    master   15h   v1.17.3 
kubenode1.bpic.local    Ready    <none>   14h   v1.17.3 
kubenode2.bpic.local    Ready    <none>   14h   v1.17.3 
win-eo5rgh4493r         Ready    <none>   12h   v1.16.2 

If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to info@keyvatech.com 

By Brad Johnson, Lead DevOps Engineer

In this guide we will deal with building a Rancher cluster with windows worker nodes. The cluster will still need a Linux master and worker node as well. As with our last Rancher blog post we will be using CentOS 7. Please see our last blog post about setting up a Rancher management node if you do not already have one. That part of the process is the same. We are going to assume you are starting at the point that you have a Rancher management interface up and accessible to log in to.

In order to allow us to use Windows worker nodes we will need to create a custom cluster in Rancher. This means we will not be able to use Rancher’s ability to automatically boot nodes for us and we will need to create the nodes by hand before we bring up our Rancher cluster.

We are going to use VMware vSphere 6.7 for our VM deployments. The windows node must run Windows Server 2019, version 1809 or 1903. Kubernetes may fail to run if you are using an older image and do not have the latest updates from Microsoft. In our testing we used version 1809, build 17763.1339 and did not need to install and additional KBs manually. Builds prior to 17763.379 are known to be missing required updates. It is also critical that you have VMware Tools 11.1.x or later installed on the Windows guest VM. See here for additional details on version information.
https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info

1. Provision two CentOS 7 nodes in VMware with 2CPUs and 4GB of RAM or greater.
2. After they have booted, log in to the nodes and prepare them to be added to Rancher. We have created the following script to help with this. Please add any steps your org needs as well. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-centos7-node-prep.sh
3. Provision the windows server worker node in vSphere, note that 1.5 CPUs and 2.5GB of RAM are reserved for windows. You may want to over-provision this node by a bit. I used 6CPUs and 8GB ram so there was some overhead in my lab.
4. Modify the windows node CPU settings and enable “Hardware virtualization”, then make any other changes you need and boot the node.
5. You can confirm the windows node version by running ‘winver’ at the powershell prompt.
6. Check to make sure the VMware Tools version you are running is 11.1.0 or later.
7. After you boot the windows node open an admin powershell prompt and run the commands in this powershell script to set up the system, install docker and open the proper firewall ports. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-windows-node-prep.ps1
8. After you run the script you can then set the hostname, make any other changes for your org and reboot.
9. Once the reboot is complete open a powershell prompt as admin and run ‘docker ps‘, then run ‘docker run hello-world‘ to test the install.

There are more details here on the docker install method we used:
https://github.com/OneGet/MicrosoftDockerProvider

This page contains documentation on an alternate install method for docker on windows:
https://docs.mirantis.com/docker-enterprise/current/dockeree-products/docker-engine-enterprise/dee-windows.html

For some windows containers it is important your base images matches your windows version. Check your Windows version with ‘winver’ on the command prompt.
If you are running 1809 this is the command to pull the current microsoft nanoserver image:

docker image pull mcr.microsoft.com/windows/nanoserver:1809

Now that we have our nodes provisioned in VMware with docker installer we are ready to create a cluster in Rancher.

1. Log in to the rancher management web interface, select the global cluster screen and click “add cluster”.
2. Choose “From existing nodes (custom)” this is the only option where windows is supported currently.
3. Set a cluster name, choose your kubernetes version, for Network Provider select “Flannel” from the dropdown.
4. Flannel is the only network type to support windows, the windows support option should now allow you to select “Enabled“. Leave the Flannel Backend set to VXLAN.
5. You can now review the other settings, but you likely don’t need to make any other changes. Click “Next” at the bottom of the page.
6. You are now presented with the screen showing docker commands to add nodes. You will need to copy these commands and run them by hand on each node. Be sure to run the windows command in an admin powershell prompt.
   1. For the master node select Linux with etcd and Control Plane.
   2. For the linux worker select Linux with only Worker.
   3. For the windows worker node select windows, worker is the only option.
7. This cluster will now provision itself and come up. This may take 5-10 mins.
8. After the cluster is up select the cluster name from the main drop down in the upper left, then go to “Projects/Namespaces” and click on “Project: System”. Be sure you are on the Resources > Workloads page. All services should say “Active”. If there are any issues here you may need to troubleshoot further.

Troubleshooting

Every environment is different, so you may need to go through some additional steps to set up Windows nodes with Rancher. This guide may help you get past the initial setup challenges. A majority of the issues we have seen getting started were caused by DNS, firewalls, selinux being set to “enforcing”, and automatic certs that were generated using “.local” domains or short hostnames.

If you need to wipe Rancher from any nodes and start over see this page:
https://rancher.com/docs/rancher/v2.x/en/cluster-admin/cleaning-cluster-nodes/

You can use these commands in windows to check on the docker service status and restart it.

sc.exe qc docker
sc.exe stop docker
sc.exe start docker