Blog & Insights

By Anuj Tuli, CTO

Keyva announces the certification of their ServiceNow App for Red Hat Ansible Tower against the Paris release (latest release) of ServiceNow. ServiceNow announced its early availability of Paris, which is the newest version in the long line of software updates since the company's creation.  

Upon general availability of the Paris release, customers will be able to upgrade their ServiceNow App for Red Hat Ansible Tower from previous ServiceNow Releases – Madrid, New York, Orlando – to Paris release seamlessly. 

You can find out more about the App, and view all the ServiceNow releases it is certified against, on the ServiceNow store here - https://bit.ly/3jMkbPn 

Red Hat OpenShift Container Platform is an Enterprise Kubernetes offering by Red Hat, that allows users to deploy cloud native applications as well as manage lifecycle of microservices deployed in containers. OpenShift Online is a SaaS offering for OpenShift provided by Red Hat. OpenShift Online takes away the effort required to set up your OpenShift clusters on-prem and allows organizations to quickly leverage all that OpenShift offers, including Developer console, without worry about managing the underlying infrastructure.  

OpenShift Online provides REST based APIs for all functions that can be carried out via the console, and the oc command line. Therefore, teams can build automated functionality that leverages Kubernetes cluster using OpenShift management plane. Today, we will look at one such function – to create a Project. Any user that wants to create a project using APIs is required to have appropriate role bindings in the specific namespace that they want to create or manage projects in. By default, OpenShift Online provides you the ability to create Projects via the console, using the ProjectRequest API call.  

Assuming you have the oc command line setup, the command to create a project is: 

$ oc new-project <project_name>  
 --description="<description>" --display-name="<display_name>" 

We will take a look at how to create a Project in OpenShift Online using the REST API. We will be using Postman to trigger our API call. This sample was run against OpenShift v3.11, Postman v7.30.1.  

1) The first thing we will do is log into our OpenShift Online console, and on the top right section in the drop-down that shows up under your name, select 'Copy Login Command'. Paste the copied contents into Notepad and capture the 'token' value. 

2) Download and import the Postman collection for this sample API call here  

3) Paste the copied token value under 'Authorization' section of the request 

4) Update the sections in bold for an appropriate name you want your Project to have 

{ 

    "kind": "ProjectRequest", 

    "apiVersion": "v1", 

    "displayName": "82520759", 

    "description": "test project from postman", 

    "metadata": { 

        "labels": { 

            "name": "82520759", 

            "namespace": "82520759" 

        }, 

        "name": "82520759" 

    } 

} 

5) Execute the Postman call. You should now see a new project created under your OpenShift Online instance.  

You can adjust the Body of the sample call to pass in more values associated with the ProjectRequest object. For reference, the object schema includes the below  

https://docs.openshift.com/container-platform/3.11/rest_api/oapi/v1.ProjectRequest.html 

apiVersion: 
description: 
displayName: 
kind: 
metadata: 
  annotations: 
    clusterName: 
      creationTimestamp: 
      deletionGracePeriodSeconds: 
      deletionTimestamp: 
   finalizers: 
     generateName: 
     generation: 
   initializers: 
   labels: 
     name: 
     namespace: 
   ownerReferences: 
     resourceVersion: 
     selfLink: 
     uid: 
 

Once you've unit tested the REST call with Postman for your OpenShift Online environment, you can very easily port this over to using one of the existing modules in Ansible, and making it a step within your playbook.  

If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to info@keyvatech.com 

In this guide we will deal with building a Rancher cluster with windows worker nodes. The cluster will still need a Linux master and worker node as well. As with our last Rancher blog post we will be using CentOS 7. Please see our last blog post about setting up a Rancher management node if you do not already have one. That part of the process is the same. We are going to assume you are starting at the point that you have a Rancher management interface up and accessible to log in to.

In order to allow us to use Windows worker nodes we will need to create a custom cluster in Rancher. This means we will not be able to use Rancher’s ability to automatically boot nodes for us and we will need to create the nodes by hand before we bring up our Rancher cluster.

We are going to use VMware vSphere 6.7 for our VM deployments. The windows node must run Windows Server 2019, version 1809 or 1903. Kubernetes may fail to run if you are using an older image and do not have the latest updates from Microsoft. In our testing we used version 1809, build 17763.1339 and did not need to install and additional KBs manually. Builds prior to 17763.379 are known to be missing required updates. It is also critical that you have VMware Tools 11.1.x or later installed on the Windows guest VM. See here for additional details on version information.
https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info

1. Provision two CentOS 7 nodes in VMware with 2CPUs and 4GB of RAM or greater.
2. After they have booted, log in to the nodes and prepare them to be added to Rancher. We have created the following script to help with this. Please add any steps your org needs as well. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-centos7-node-prep.sh
3. Provision the windows server worker node in vSphere, note that 1.5 CPUs and 2.5GB of RAM are reserved for windows. You may want to over-provision this node by a bit. I used 6CPUs and 8GB ram so there was some overhead in my lab.
4. Modify the windows node CPU settings and enable “Hardware virtualization”, then make any other changes you need and boot the node.
5. You can confirm the windows node version by running ‘winver’ at the powershell prompt.
6. Check to make sure the VMware Tools version you are running is 11.1.0 or later.
7. After you boot the windows node open an admin powershell prompt and run the commands in this powershell script to set up the system, install docker and open the proper firewall ports. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-windows-node-prep.ps1
8. After you run the script you can then set the hostname, make any other changes for your org and reboot.
9. Once the reboot is complete open a powershell prompt as admin and run ‘docker ps‘, then run ‘docker run hello-world‘ to test the install.

There are more details here on the docker install method we used:
https://github.com/OneGet/MicrosoftDockerProvider

This page contains documentation on an alternate install method for docker on windows:
https://docs.mirantis.com/docker-enterprise/current/dockeree-products/docker-engine-enterprise/dee-windows.html

For some windows containers it is important your base images matches your windows version. Check your Windows version with ‘winver’ on the command prompt.
If you are running 1809 this is the command to pull the current microsoft nanoserver image:

docker image pull mcr.microsoft.com/windows/nanoserver:1809

Now that we have our nodes provisioned in VMware with docker installer we are ready to create a cluster in Rancher.

1. Log in to the rancher management web interface, select the global cluster screen and click “add cluster”.
2. Choose “From existing nodes (custom)” this is the only option where windows is supported currently.
3. Set a cluster name, choose your kubernetes version, for Network Provider select “Flannel” from the dropdown.
4. Flannel is the only network type to support windows, the windows support option should now allow you to select “Enabled“. Leave the Flannel Backend set to VXLAN.
5. You can now review the other settings, but you likely don’t need to make any other changes. Click “Next” at the bottom of the page.
6. You are now presented with the screen showing docker commands to add nodes. You will need to copy these commands and run them by hand on each node. Be sure to run the windows command in an admin powershell prompt.
   1. For the master node select Linux with etcd and Control Plane.
   2. For the linux worker select Linux with only Worker.
   3. For the windows worker node select windows, worker is the only option.
7. This cluster will now provision itself and come up. This may take 5-10 mins.
8. After the cluster is up select the cluster name from the main drop down in the upper left, then go to “Projects/Namespaces” and click on “Project: System”. Be sure you are on the Resources > Workloads page. All services should say “Active”. If there are any issues here you may need to troubleshoot further.

Troubleshooting

Every environment is different, so you may need to go through some additional steps to set up Windows nodes with Rancher. This guide may help you get past the initial setup challenges. A majority of the issues we have seen getting started were caused by DNS, firewalls, selinux being set to “enforcing”, and automatic certs that were generated using “.local” domains or short hostnames.

If you need to wipe Rancher from any nodes and start over see this page:
https://rancher.com/docs/rancher/v2.x/en/cluster-admin/cleaning-cluster-nodes/

You can use these commands in windows to check on the docker service status and restart it.

sc.exe qc docker
sc.exe stop docker
sc.exe start docker

At Keyva we often meet with clients that need just a little help, something to get them over the hump and continue on their way building out new and exciting IT capabilities. This seems to happen most often when organizations adopt new and emerging technologies. Often teams haven't built up their internal skills and capabilities around tech like Kubernetes or automation platforms such as Red Hat Ansible. Or, perhaps the team is already very skilled, but want someone to help with their OpenShift 3.x -> 4.x upgrade path, or need someone to write a new Ansible module so that they can expand their ability to offer automation capabilities via their playbooks.  

There hasn't been a good way to get this kind of incremental help – no granular consumption model for technical expertise – it's not a function of your vendor's L1 support. Your vendor will tell you to buy a TAM or their own expensive consulting services. It's also not something readily available in the community at large. There are user forums and networks for days, but will you get a response to your questions? Will the responses be correct?  

Keyva created Guru Services to address this exact issue. It's more than L1 support, not as heavy as a consulting engagement, it's enterprise grade and far more reliable than crowdsourcing the community for answers and assistance. 

Guru Service is just as easy to use: choose from 3 different service levels and you're on your way. You'll have access to our client portal from which you can schedule your On Demand Guru. We'll send you a meeting invite with web conference information and you'll be over the hump and on your way in no time. We currently provide On Demand Gurus for Red Hat Ansible, OpenShift, and Kong and are actively adding technologies to our suite of Guru Services. To learn more about these offerings, check out our vendor pages for Kong (https://keyvatech.com/kong-enterprise/) and Red Hat (https://keyvatech.com/red-hat/). Reach out to our Keyva team at: info@keyvatech.com to request additional information or a quote on our Guru Services .  

curl https://releases.rancher.com/install-docker/19.03.sh | sh

mkdir /opt/rancher

docker run -d --restart=unless-stopped -p 80:80 -p 443:443 -v /opt/rancher:/var/lib/rancher rancher/rancher:latest

https://raw.githubusercontent.com/keyvatech/blog_files/master/centos7_cloudinit_vmtemplate.sh

This guide will walk through how to set up Red Hat Ansible Tower in a highly-available configuration. In this example, we will set up 4 different systems – 1 for PostgreSQL database (towerdb), and 3 web nodes for Tower (tower1, tower2, tower3). 

We will be using Ansible Tower v3.6 and PostgreSQL 10, on RHEL 7 systems running in VMware for this technical guide. The commands for setting up the same configuration on RHEL 8 will be different for some cases.  This guide does not account for clustering of the PostgreSQL database. If you are setting up Tower in HA capacity for Production environments, it is recommended to follow best practices for PostgreSQL clustering, to avoid a single point of failure. 

First, we will need to prep all the RHEL instances by enabling the Red Hat repos. All the commands below are to be run on all 4 systems – towerdb, tower1, tower2, tower3 

subscription-manager register 

subscription-manager refresh 

subscription-manager attach –-auto 

subscription-manager repos –-list  

subscription-manager repos --enable rhel-7-server-rh-common-beta-rpms 

subscription-manager repos --enable rhel-7-server-rpms 

subscription-manager repos --enable rhel-7-server-source-rpms 

subscription-manager repos --enable rhel-7-server-rh-common-source-rpms 

subscription-manager repos --enable rhel-7-server-rh-common-debug-rpms 

subscription-manager repos --enable rhel-7-server-optional-source-rpms 

subscription-manager repos --enable rhel-7-server-extras-rpms 

sudo yum update 

sudo yum install wget 

sudo yum install python36 

sudo pip3 install httpie 

Also: 

1. a) Update the /etc/hosts file on all 4 hosts with entries for all systems
2. b) Add and copy thesshkeys on all systems 

On the Database system (towerdb), we will now set up PostgreSQL 10 

sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm 

sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm 

sudo yum install postgresql10 postgresql10-server 

Initialize the database 

/usr/pgsql-10/bin/postgresql-10-setup initdb 

systemctl enable postgresql-10 

systemctl start postgresql-10 

Verify you can log in to the database 

:~
nbsp;sudo su – postgres 
:~
nbsp;Psql  
# \list 
This command will show you the existing (default) database list. 
Next, we will configure the database to make sure it can talk to all the Tower web nodes: 
sudo vi /var/lib/pgsql/10/data/pg_hba.conf 

Add/update the line with 'md5' entry to allow all hosts:  
host    all             all             0.0.0.0/0            md5 
Update the postgresql.conf file 
sudo vi /var/lib/pgsql/10/data/postgresql.conf

Add/update the entry to listen to all incoming requests:  
listen_addresses = '*' 

Restart the database services, to pick up the changes made: 
sudo systemctl restart postgresql-10 

sudo systemctl status postgresql-10 
 
On each of the Tower web nodes (tower1, tower2, tower3), we will set up the Ansible Tower binaries: 
mkdir ansible-tower 

cd ansible-tower/ 

wget https://releases.ansible.com/ansible-tower/setup-bundle/ansible-tower-setup-bundle-3.6.2-1.el7.tar.gz 

tar xvzf ansible-tower-setup-bundle-3.6.2-1.el7.tar.gz  

cd ansible-tower-setup-bundle-3.6.2-1 

python -c 'from hashlib import md5; print("md5" + md5("password" + "awx").hexdigest())' 

md5f58b4d5d85dbde46651335d78bb56b8c 
Where password will be the password that you will be using a hash of, when authenticating against the database 
Back on the database server (towerdb), we will go ahead and set up the database schema pre-requisites for Tower install:  
:~
nbsp;sudo su – postgres 
:~
nbsp;Psql  
postgres=# CREATE USER awx; CREATE DATABASE awx OWNER awx; ALTER USER awx WITH password 'password'; 
On tower1, tower2, tower3, update the inventory file and run the setup. Make sure your script contents match on all tower web tier systems. 
You will need to update at least the following values and customize them for your environment: 
admin_password='password' 

pg_password='password' 

rabbit_mq = 'password' 
Under the [tower] section, you will have to add entries for all your tower web hosts. The first entry will typically serve as the primary node when the cluster is run.  
We will now run the setup script: 
./setup.sh 
You can either copy this inventory file on the other 2 tower systems (tower2 and tower3), or replicate the content to match the content in the file on tower1, and run the setup script on the other 2 tower systems as well.  
Once the setup script is run on all hosts, and it finishes successfully, you will be able to test your cluster instance. You can do so by going to one of the tower hosts URL, initiating a job template, and see which specific tower node it runs on – based on the tower node that is designated to be the primary node at that time. You will also be able to view the same console details, and logs of job runs, regardless of which tower web URL you go to.  
If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to info@keyvatech.com.

Let us look at setting up a Kubernetes cluster with 1 master node (kubemaster.bpic.local) and 2 worker nodes (kubenode1.bpic.local, kubenode2.bpic.local) on VMware based RHEL 7 instances. 

We have set up an additional user (other than root) on these machines, as we will be running kubectl (client) commands as the non-root user.

First, we will need to prep all the RHEL instances by enabling the Red Hat repos. All the commands below are to be run on all 3 components – kubemaster, kubenode1, kubenode2

subscription-manager register
subscription-manager refresh
subscription-manager attach –-auto
subscription-manager repos –-list
subscription-manager repos --enable rhel-7-server-rh-common-beta-rpms
subscription-manager repos --enable rhel-7-server-rpms
subscription-manager repos --enable rhel-7-server-source-rpms
subscription-manager repos --enable rhel-7-server-rh-common-source-rpms
subscription-manager repos --enable rhel-7-server-rh-common-debug-rpms
subscription-manager repos --enable rhel-7-server-optional-source-rpms
subscription-manager repos --enable rhel-7-server-extras-rpms

The rhel-7-server-extras-rpms repo contains docker and other utilities. 

Since this is our lab environment, we will be disabling firewalls. If it is a production environment, you can open up specific ports for communication of your applications, and for Kubernetes components instead of disabling the firewall completely.

systemctl disable firewalld 
systemctl stop firewalld

Since we are using VMware VMs, it is recommended to set up VMware-tools 

yum install perl 
mkdir /mnt/cdrom 
Mount /dev/cdrom /mnt/cdrom 
cp /mnt/cdrom/VMwareTools-version.tar.gz /tmp/ 
tar -zxvf VMwareTools-version.tar.gz 
/tmp/vmware-tools-distrib/./vmware-install.pl  
umount /mnt/cdrom 

Update the yum repositories

yum –y update yum 
install yum-utils

Configure additional settings

swapoff –a

Also, comment out the swap line in

etc/fstab 
#/dev/mapper/rhel-swap   swap                    swap    defaults        0 0 

Install and enable docker

yum –y install docker 
systemctl enable docker 
systemctl start docker 

Set up repo for Kubernetes

cat <<EOF > /etc/yum.repos.d/kubernetes.repo 
[kubernetes] 
name=Kubernetes 
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 
enabled=1 
gpgcheck=1 
repo_gpgcheck=1 
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg 
EOF 

Additional enforcement settings

setenforce 0

Update the config file to change the selinux settings

vi /etc/selinux/config 

Change the settings from 

selinux=enforcing to selinux=permissive 

Install and enable kubelet service

yum -y install kubelet kubeadm kubectl 
systemctl enable kubelet 

start kubelet 

Enable sysctl settings

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl –system

Alternatively, you can update the /etc/sysctl.conf file 

vi /etc/sysctl.conf 

Add/update the following lines

net/bridge/bridge-nf-call-iptables = 1 

net/ipv4/ip_forward = 1 

On the Kubernetes master node only, we will set up the flannel networking component using fat manifest:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 


kubeadm init --pod-network-cidr=10.244.0.0/16 


kubeadm token create --print-join-command 

Capture the results of the above command, specifically the part describing how to add nodes to this cluster

You can now join any number of machines by running the following on each node as root:

kubeadm join kubemaster.bpic.local:6443 --token cll0gw.50jagb64e80uw0da \ 

    --discovery-token-ca-cert-hash sha256:4d699e7f06ce0e7e80b78eadc47453e465358021aee52d956dceed1dfbc0ee34

And then after changing to a non-root user, run the following commands

su – nonrootuser 
mkdir -p $HOME/.kube 
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 
sudo chown $(id -u):$(id -g) $HOME/.kube/config 

On the Kubernetes nodes (kubenode1 and kubenode2), we will run the join command, to add those nodes to the cluster:

kubeadm join kubemaster.bpic.local:6443 --token cll0gw.50jagb64e80uw0da \ 

    --discovery-token-ca-cert-hash sha256:4d699e7f06ce0e7e80b78eadc47453e465358021aee52d956dceed1dfbc0ee34 
You can now test the cluster by running the below command on either of the nodes, or the master as the non-root user:

kubectl get nodes

You should see results like this (changed for your system names) showing the cluster configuration:

 
NAME                    STATUS   ROLES    AGE   VERSION 
kubemaster.bpic.local   Ready    master   15h   v1.17.3 
kubenode1.bpic.local    Ready    <none>   14h   v1.17.3 
kubenode2.bpic.local    Ready    <none>   14h   v1.17.3

If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to info@keyvatech.com.

By Anuj Tuli, Chief Technology Officer

Typically when you hear about containers and Kubernetes, it is in the context of Linux or Unix platforms. But there are a large number of organizations that use Windows and .NET based applications, and they are still trying to determine the best way forward for containerization of their Windows based business critical applications.  

Kubernetes added support for Windows based components (worker nodes) starting with release v1.14.  

In the example below, we will join a Windows worker node (v1.16.x) with a Kubernetes cluster v1.17.x. 

As of this moment, Windows worker nodes are supported on Windows 2019 operating system only. In this example, we will leverage the flannel network set up on our master node on RHEL (see instructions above).  

Step 1: Download the sig-windows-tools repository from https://github.com/kubernetes-sigs/sig-windows-tools , and extract the files 

Step 2: Navigate to, and update the Kubernetes configuration file at C:\<Download-Path>\kubernetes\kubeadm\v1.16.0\Kubeclustervxlan 

In our instance, we will update the following values: 

• Interface Name 
• Control Plane details – IP address, username, KubeadmToken, KubeadmCAHash. The values for these come from the output of the kubectl join command, when it was run during set up of the master node

Step 3: Open up PowerShell console in Admin mode and install kubernetes via the downloaded script. This step requires reboot of the server 

PS C:\Users\Administrator> cd C:\<Download-Path>\kubernetes\kubeadm 

PS C:\<Download-Path>\kubernetes\kubeadm> .\KubeCluster.ps1 -ConfigFile C:\<Download-Path>\kubernetes\kubeadm\v1.16.0\Kubeclustervxlan.json -install 

Step 4: Once K8s is installed, join it to the existing kubernetes cluster. This step takes the values you entered in the modified Kubeclustervxlan file 

PS C:\<Download-Path>\kubernetes\kubeadm> .\KubeCluster.ps1 -ConfigFile C:\<Download-Path>\kubernetes\kubeadm\v1.16.0\Kubeclustervxlan.json -join 

Step 5: Verify that the Windows worker node was successfully added to the cluster. You can do this by running the kubectl command from any client (Windows or Linux nodes on the cluster)  

PS C:\<Download-Path>\kubernetes\kubeadm> kubectl get nodes 

NAME                    STATUS   ROLES    AGE   VERSION 
kubemaster.bpic.local   Ready    master   15h   v1.17.3 
kubenode1.bpic.local    Ready    <none>   14h   v1.17.3 
kubenode2.bpic.local    Ready    <none>   14h   v1.17.3 
win-eo5rgh4493r         Ready    <none>   12h   v1.16.2 

If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to info@keyvatech.com 

By Anuj Tuli, CTO

Keyva announces the certification of their ServiceNow App for Red Hat Ansible Tower against the Paris release (latest release) of ServiceNow. ServiceNow announced its early availability of Paris, which is the newest version in the long line of software updates since the company's creation.  

Upon general availability of the Paris release, customers will be able to upgrade their ServiceNow App for Red Hat Ansible Tower from previous ServiceNow Releases – Madrid, New York, Orlando – to Paris release seamlessly. 

You can find out more about the App, and view all the ServiceNow releases it is certified against, on the ServiceNow store here - https://bit.ly/3jMkbPn