Blog & Insights

Red Hat OpenShift Container Platform is an Enterprise Kubernetes offering by Red Hat, that allows users to deploy cloud native applications as well as manage lifecycle of microservices deployed in containers. OpenShift Online is a SaaS offering for OpenShift provided by Red Hat. OpenShift Online takes away the effort required to set up your OpenShift clusters on-prem and allows organizations to quickly leverage all that OpenShift offers, including Developer console, without worry about managing the underlying infrastructure.  

OpenShift Online provides REST based APIs for all functions that can be carried out via the console, and the oc command line. Therefore, teams can build automated functionality that leverages Kubernetes cluster using OpenShift management plane. Today, we will look at one such function – to create a Project. Any user that wants to create a project using APIs is required to have appropriate role bindings in the specific namespace that they want to create or manage projects in. By default, OpenShift Online provides you the ability to create Projects via the console, using the ProjectRequest API call.  

Assuming you have the oc command line setup, the command to create a project is: 

$ oc new-project <project_name>  
 --description="<description>" --display-name="<display_name>" 

We will take a look at how to create a Project in OpenShift Online using the REST API. We will be using Postman to trigger our API call. This sample was run against OpenShift v3.11, Postman v7.30.1.  

1) The first thing we will do is log into our OpenShift Online console, and on the top right section in the drop-down that shows up under your name, select 'Copy Login Command'. Paste the copied contents into Notepad and capture the 'token' value. 

2) Download and import the Postman collection for this sample API call here  

3) Paste the copied token value under 'Authorization' section of the request 

4) Update the sections in bold for an appropriate name you want your Project to have 

{ 

    "kind": "ProjectRequest", 

    "apiVersion": "v1", 

    "displayName": "82520759", 

    "description": "test project from postman", 

    "metadata": { 

        "labels": { 

            "name": "82520759", 

            "namespace": "82520759" 

        }, 

        "name": "82520759" 

    } 

} 

5) Execute the Postman call. You should now see a new project created under your OpenShift Online instance.  

You can adjust the Body of the sample call to pass in more values associated with the ProjectRequest object. For reference, the object schema includes the below  

https://docs.openshift.com/container-platform/3.11/rest_api/oapi/v1.ProjectRequest.html 

apiVersion: 
description: 
displayName: 
kind: 
metadata: 
  annotations: 
    clusterName: 
      creationTimestamp: 
      deletionGracePeriodSeconds: 
      deletionTimestamp: 
   finalizers: 
     generateName: 
     generation: 
   initializers: 
   labels: 
     name: 
     namespace: 
   ownerReferences: 
     resourceVersion: 
     selfLink: 
     uid: 
 

Once you've unit tested the REST call with Postman for your OpenShift Online environment, you can very easily port this over to using one of the existing modules in Ansible, and making it a step within your playbook.  

If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to [email protected] 

In this guide we will deal with building a Rancher cluster with windows worker nodes. The cluster will still need a Linux master and worker node as well. As with our last Rancher blog post we will be using CentOS 7. Please see our last blog post about setting up a Rancher management node if you do not already have one. That part of the process is the same. We are going to assume you are starting at the point that you have a Rancher management interface up and accessible to log in to.

In order to allow us to use Windows worker nodes we will need to create a custom cluster in Rancher. This means we will not be able to use Rancher’s ability to automatically boot nodes for us and we will need to create the nodes by hand before we bring up our Rancher cluster.

We are going to use VMware vSphere 6.7 for our VM deployments. The windows node must run Windows Server 2019, version 1809 or 1903. Kubernetes may fail to run if you are using an older image and do not have the latest updates from Microsoft. In our testing we used version 1809, build 17763.1339 and did not need to install and additional KBs manually. Builds prior to 17763.379 are known to be missing required updates. It is also critical that you have VMware Tools 11.1.x or later installed on the Windows guest VM. See here for additional details on version information.
https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info

1. Provision two CentOS 7 nodes in VMware with 2CPUs and 4GB of RAM or greater.
2. After they have booted, log in to the nodes and prepare them to be added to Rancher. We have created the following script to help with this. Please add any steps your org needs as well. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-centos7-node-prep.sh
3. Provision the windows server worker node in vSphere, note that 1.5 CPUs and 2.5GB of RAM are reserved for windows. You may want to over-provision this node by a bit. I used 6CPUs and 8GB ram so there was some overhead in my lab.
4. Modify the windows node CPU settings and enable “Hardware virtualization”, then make any other changes you need and boot the node.
5. You can confirm the windows node version by running ‘winver’ at the powershell prompt.
6. Check to make sure the VMware Tools version you are running is 11.1.0 or later.
7. After you boot the windows node open an admin powershell prompt and run the commands in this powershell script to set up the system, install docker and open the proper firewall ports. https://raw.githubusercontent.com/keyvatech/blog_files/master/rancher-windows-node-prep.ps1
8. After you run the script you can then set the hostname, make any other changes for your org and reboot.
9. Once the reboot is complete open a powershell prompt as admin and run ‘docker ps‘, then run ‘docker run hello-world‘ to test the install.

There are more details here on the docker install method we used:
https://github.com/OneGet/MicrosoftDockerProvider

This page contains documentation on an alternate install method for docker on windows:
https://docs.mirantis.com/docker-enterprise/current/dockeree-products/docker-engine-enterprise/dee-windows.html

For some windows containers it is important your base images matches your windows version. Check your Windows version with ‘winver’ on the command prompt.
If you are running 1809 this is the command to pull the current microsoft nanoserver image:

docker image pull mcr.microsoft.com/windows/nanoserver:1809

Now that we have our nodes provisioned in VMware with docker installer we are ready to create a cluster in Rancher.

1. Log in to the rancher management web interface, select the global cluster screen and click “add cluster”.
2. Choose “From existing nodes (custom)” this is the only option where windows is supported currently.
3. Set a cluster name, choose your kubernetes version, for Network Provider select “Flannel” from the dropdown.
4. Flannel is the only network type to support windows, the windows support option should now allow you to select “Enabled“. Leave the Flannel Backend set to VXLAN.
5. You can now review the other settings, but you likely don’t need to make any other changes. Click “Next” at the bottom of the page.
6. You are now presented with the screen showing docker commands to add nodes. You will need to copy these commands and run them by hand on each node. Be sure to run the windows command in an admin powershell prompt.
   1. For the master node select Linux with etcd and Control Plane.
   2. For the linux worker select Linux with only Worker.
   3. For the windows worker node select windows, worker is the only option.
7. This cluster will now provision itself and come up. This may take 5-10 mins.
8. After the cluster is up select the cluster name from the main drop down in the upper left, then go to “Projects/Namespaces” and click on “Project: System”. Be sure you are on the Resources > Workloads page. All services should say “Active”. If there are any issues here you may need to troubleshoot further.

Troubleshooting

Every environment is different, so you may need to go through some additional steps to set up Windows nodes with Rancher. This guide may help you get past the initial setup challenges. A majority of the issues we have seen getting started were caused by DNS, firewalls, selinux being set to “enforcing”, and automatic certs that were generated using “.local” domains or short hostnames.

If you need to wipe Rancher from any nodes and start over see this page:
https://rancher.com/docs/rancher/v2.x/en/cluster-admin/cleaning-cluster-nodes/

You can use these commands in windows to check on the docker service status and restart it.

sc.exe qc docker
sc.exe stop docker
sc.exe start docker

At Keyva we often meet with clients that need just a little help, something to get them over the hump and continue on their way building out new and exciting IT capabilities. This seems to happen most often when organizations adopt new and emerging technologies. Often teams haven't built up their internal skills and capabilities around tech like Kubernetes or automation platforms such as Red Hat Ansible. Or, perhaps the team is already very skilled, but want someone to help with their OpenShift 3.x -> 4.x upgrade path, or need someone to write a new Ansible module so that they can expand their ability to offer automation capabilities via their playbooks.  

There hasn't been a good way to get this kind of incremental help – no granular consumption model for technical expertise – it's not a function of your vendor's L1 support. Your vendor will tell you to buy a TAM or their own expensive consulting services. It's also not something readily available in the community at large. There are user forums and networks for days, but will you get a response to your questions? Will the responses be correct?  

Keyva created Guru Services to address this exact issue. It's more than L1 support, not as heavy as a consulting engagement, it's enterprise grade and far more reliable than crowdsourcing the community for answers and assistance. 

Guru Service is just as easy to use: choose from 3 different service levels and you're on your way. You'll have access to our client portal from which you can schedule your On Demand Guru. We'll send you a meeting invite with web conference information and you'll be over the hump and on your way in no time. We currently provide On Demand Gurus for Red Hat Ansible, OpenShift, and Kong and are actively adding technologies to our suite of Guru Services. To learn more about these offerings, check out our vendor pages for Kong (https://keyvatech.com/kong-enterprise/) and Red Hat (https://keyvatech.com/red-hat/). Reach out to our Keyva team at: [email protected] to request additional information or a quote on our Guru Services .  

curl https://releases.rancher.com/install-docker/19.03.sh | sh

mkdir /opt/rancher

docker run -d --restart=unless-stopped -p 80:80 -p 443:443 -v /opt/rancher:/var/lib/rancher rancher/rancher:latest

https://raw.githubusercontent.com/keyvatech/blog_files/master/centos7_cloudinit_vmtemplate.sh

This guide will walk through how to set up Red Hat Ansible Tower in a highly-available configuration. In this example, we will set up 4 different systems – 1 for PostgreSQL database (towerdb), and 3 web nodes for Tower (tower1, tower2, tower3). 

We will be using Ansible Tower v3.6 and PostgreSQL 10, on RHEL 7 systems running in VMware for this technical guide. The commands for setting up the same configuration on RHEL 8 will be different for some cases.  This guide does not account for clustering of the PostgreSQL database. If you are setting up Tower in HA capacity for Production environments, it is recommended to follow best practices for PostgreSQL clustering, to avoid a single point of failure. 

First, we will need to prep all the RHEL instances by enabling the Red Hat repos. All the commands below are to be run on all 4 systems – towerdb, tower1, tower2, tower3 

subscription-manager register 

subscription-manager refresh 

subscription-manager attach –-auto 

subscription-manager repos –-list  

subscription-manager repos --enable rhel-7-server-rh-common-beta-rpms 

subscription-manager repos --enable rhel-7-server-rpms 

subscription-manager repos --enable rhel-7-server-source-rpms 

subscription-manager repos --enable rhel-7-server-rh-common-source-rpms 

subscription-manager repos --enable rhel-7-server-rh-common-debug-rpms 

subscription-manager repos --enable rhel-7-server-optional-source-rpms 

subscription-manager repos --enable rhel-7-server-extras-rpms 

sudo yum update 

sudo yum install wget 

sudo yum install python36 

sudo pip3 install httpie 

Also: 

1. a) Update the /etc/hosts file on all 4 hosts with entries for all systems
2. b) Add and copy thesshkeys on all systems 

On the Database system (towerdb), we will now set up PostgreSQL 10 

sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm 

sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm 

sudo yum install postgresql10 postgresql10-server 

Initialize the database 

/usr/pgsql-10/bin/postgresql-10-setup initdb 

systemctl enable postgresql-10 

systemctl start postgresql-10 

Verify you can log in to the database 

:~
nbsp;sudo su – postgres 
:~
nbsp;Psql  
# \list 
This command will show you the existing (default) database list. 
Next, we will configure the database to make sure it can talk to all the Tower web nodes: 
sudo vi /var/lib/pgsql/10/data/pg_hba.conf 

Add/update the line with 'md5' entry to allow all hosts:  
host    all             all             0.0.0.0/0            md5 
Update the postgresql.conf file 
sudo vi /var/lib/pgsql/10/data/postgresql.conf

Add/update the entry to listen to all incoming requests:  
listen_addresses = '*' 

Restart the database services, to pick up the changes made: 
sudo systemctl restart postgresql-10 

sudo systemctl status postgresql-10 
 
On each of the Tower web nodes (tower1, tower2, tower3), we will set up the Ansible Tower binaries: 
mkdir ansible-tower 

cd ansible-tower/ 

wget https://releases.ansible.com/ansible-tower/setup-bundle/ansible-tower-setup-bundle-3.6.2-1.el7.tar.gz 

tar xvzf ansible-tower-setup-bundle-3.6.2-1.el7.tar.gz  

cd ansible-tower-setup-bundle-3.6.2-1 

python -c 'from hashlib import md5; print("md5" + md5("password" + "awx").hexdigest())' 

md5f58b4d5d85dbde46651335d78bb56b8c 
Where password will be the password that you will be using a hash of, when authenticating against the database 
Back on the database server (towerdb), we will go ahead and set up the database schema pre-requisites for Tower install:  
:~
nbsp;sudo su – postgres 
:~
nbsp;Psql  
postgres=# CREATE USER awx; CREATE DATABASE awx OWNER awx; ALTER USER awx WITH password 'password'; 
On tower1, tower2, tower3, update the inventory file and run the setup. Make sure your script contents match on all tower web tier systems. 
You will need to update at least the following values and customize them for your environment: 
admin_password='password' 

pg_password='password' 

rabbit_mq = 'password' 
Under the [tower] section, you will have to add entries for all your tower web hosts. The first entry will typically serve as the primary node when the cluster is run.  
We will now run the setup script: 
./setup.sh 
You can either copy this inventory file on the other 2 tower systems (tower2 and tower3), or replicate the content to match the content in the file on tower1, and run the setup script on the other 2 tower systems as well.  
Once the setup script is run on all hosts, and it finishes successfully, you will be able to test your cluster instance. You can do so by going to one of the tower hosts URL, initiating a job template, and see which specific tower node it runs on – based on the tower node that is designated to be the primary node at that time. You will also be able to view the same console details, and logs of job runs, regardless of which tower web URL you go to.  
If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to [email protected].

By Anuj Tuli, Chief Technology Officer

Typically when you hear about containers and Kubernetes, it is in the context of Linux or Unix platforms. But there are a large number of organizations that use Windows and .NET based applications, and they are still trying to determine the best way forward for containerization of their Windows based business critical applications.  

Kubernetes added support for Windows based components (worker nodes) starting with release v1.14.  

In the example below, we will join a Windows worker node (v1.16.x) with a Kubernetes cluster v1.17.x. 

As of this moment, Windows worker nodes are supported on Windows 2019 operating system only. In this example, we will leverage the flannel network set up on our master node on RHEL (see instructions above).  

Step 1: Download the sig-windows-tools repository from https://github.com/kubernetes-sigs/sig-windows-tools , and extract the files 

Step 2: Navigate to, and update the Kubernetes configuration file at C:\<Download-Path>\kubernetes\kubeadm\v1.16.0\Kubeclustervxlan 

In our instance, we will update the following values: 

• Interface Name 
• Control Plane details – IP address, username, KubeadmToken, KubeadmCAHash. The values for these come from the output of the kubectl join command, when it was run during set up of the master node

Step 3: Open up PowerShell console in Admin mode and install kubernetes via the downloaded script. This step requires reboot of the server 

PS C:\Users\Administrator> cd C:\<Download-Path>\kubernetes\kubeadm 

PS C:\<Download-Path>\kubernetes\kubeadm> .\KubeCluster.ps1 -ConfigFile C:\<Download-Path>\kubernetes\kubeadm\v1.16.0\Kubeclustervxlan.json -install 

Step 4: Once K8s is installed, join it to the existing kubernetes cluster. This step takes the values you entered in the modified Kubeclustervxlan file 

PS C:\<Download-Path>\kubernetes\kubeadm> .\KubeCluster.ps1 -ConfigFile C:\<Download-Path>\kubernetes\kubeadm\v1.16.0\Kubeclustervxlan.json -join 

Step 5: Verify that the Windows worker node was successfully added to the cluster. You can do this by running the kubectl command from any client (Windows or Linux nodes on the cluster)  

PS C:\<Download-Path>\kubernetes\kubeadm> kubectl get nodes 

NAME                    STATUS   ROLES    AGE   VERSION 
kubemaster.bpic.local   Ready    master   15h   v1.17.3 
kubenode1.bpic.local    Ready    <none>   14h   v1.17.3 
kubenode2.bpic.local    Ready    <none>   14h   v1.17.3 
win-eo5rgh4493r         Ready    <none>   12h   v1.16.2 

If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to [email protected] 

By Anuj Tuli, CTO

Keyva announces the certification of their ServiceNow App for Red Hat Ansible Tower against the Orlando release (latest release) of ServiceNow. ServiceNow announced its release of Orlando on January 23rd, 2020, which is the newest version in the long line of software updates since the company's creation.  

Customers can now upgrade their ServiceNow App for Ansible Tower from previous ServiceNow Releases – London, Madrid, New York – to Orlando release seamlessly. 

You can find out more about the App, and view all the ServiceNow releases it is certified against, on the ServiceNow store here: http://bit.ly/2W5tYHv

By Brad Johnson, Lead DevOps Engineer

When developing automation you may be faced with challenges that are simply too complicated or tedious to accomplish with Ansible alone. There may even be cases where you are told that “it can’t be automated”. However, when you combine the abilities of Ansible and custom python using the pexpect  module, then you are able to automate practically anything you can do on the command line. In this post we will discuss the basics of creating a custom Ansible module in python.  

Here are a few examples of cases where you might need to create a custom module: 

• Running command line programs that drop into a new shell or interactive interface. 
• Processing complex data and returning a subset of specific data in a new format. 
• Interacting with a database and returning data in a specific format for further Ansible processing.
  

For the purposes of this article we will focus on the first case. When writing a traditional linux shell or bash script it simply isn’t possibly to continue your script when a command you run drops you into a new shell or new interactive interface. If these tools also provided a non-interactive mode or config/script input we would not need to do this. To overcome this situation we need to use python with pexpect. The native Ansible “expect” module provides a simple interface to this functionality and should be evaluated before writing a custom module. However, when you need more complex interactions, want specific data returned or want to provide a re-usable and simpler interface to an underlying program for to others to consume, then custom development if warranted.  

In this guide I will talk about the requirements and steps needed to create your own library module. The source code with our example is located here and contains notes in the code as well. The pexpect code is intentionally complex to demonstrate some use cases.

Module Code (Python)

#!/usr/bin/env python
import os
import getpass

DOCUMENTATION = '''
---
module: my_module

short_description: This is a custom module using pexpect to run commands in myscript.sh

description:
    - "This module runs commands inside a script in a shell. When run without commands it returns current settings only."

options:
    commands:
        description:
            - The commands to run inside myscript in order
        required: false
    options:
        description:
            - options to pass the script
        required: false
    timeout:
        description:
            - Timeout for finding the success string or running the program
        required: false
        default: 300
    password:
        description:
            - Password needed to run myscript
        required: true

author:
    - Brad Johnson - Keyva
'''

EXAMPLES = '''
- name: "Run myscript to set up myprogram"
  my_module:
    options: "-o myoption"
    password: "{{ myscript_password }}"
    commands:
      - "set minheap 1024m"
      - "set maxheap 5120m"
      - "set port 7000"
      - "set webport 80"
    timeout: 300
'''

RETURN = '''
current_settings: String containing current settings after last command was run and settings saved
    type: str
    returned: On success
logfile: String containing logfile location on the remote host from our script
    type: str
    returned: On success
'''


def main():
    # This is the import required to make this code an Ansible module
    from ansible.module_utils.basic import AnsibleModule
    # This instantiates the module class and provides Ansible with
    # input argument information, it also enforces input types
    module = AnsibleModule(
        argument_spec=dict(
            commands=dict(required=False, type='list', default=[]),
            options=dict(required=False, type='str', default=""),
            password=dict(required=True, type='str', no_log=True),
            timeout=dict(required=False, type='int', default='300')
        )
    )
    commands = module.params['commands']
    options = module.params['options']
    password = module.params['password']
    timeout = module.params['timeout']

    try:
        # Importing the modules here allows us to catch them not being installed on remote hosts
        #   and pass back a failure via ansible instead of a stack trace.
        import pexpect
    except ImportError:
        module.fail_json(msg="You must have the pexpect python module installed to use this Ansible module.")

    try:
        # Run our pexpect function
        current_settings, changed, logfile = run_pexpect(commands, options, password, timeout)
        # Exit on success and pass back objects to ansible, which are available as registered vars
        module.exit_json(changed=changed, current_settings=current_settings, logfile=logfile)
    # Use python exception handling to keep all our failure handling in our main function
    except pexpect.TIMEOUT as err:
        module.fail_json(msg="pexpect.TIMEOUT: Unexpected timeout waiting for prompt or command: {0}".format(err))
    except pexpect.EOF as err:
        module.fail_json(msg="pexpect.EOF: Unexpected program termination: {0}".format(err))
    except pexpect.exceptions.ExceptionPexpect as err:
        # This catches any pexpect exceptions that are not EOF or TIMEOUT
        # This is the base exception class
        module.fail_json(msg="pexpect.exceptions.{0}: {1}".format(type(err).__name__, err))
    except RuntimeError as err:
        module.fail_json(msg="{0}".format(err))


def run_pexpect(commands, options, password, timeout=300):
    import pexpect
    changed = True
    script_path = '/path/to/myscript.sh'
    if not os.path.exists(script_path):
        raise RuntimeError("Error: the script '{0}' does not exist!".format(script_path))
    if script_path == '/path/to/myscript.sh':
        raise RuntimeError("This module example is based on a hypothetical command line interactive program and "
                           "can not run. Please use this as a basis for your own development and testing.")
    # Set prompt to expect with username embedded in it
    # YOU MAY NEED TO CHANGE THIS PROMPT FOR YOUR SYSTEM
    # My default RHEL prompt regex
    prompt = r'\[{0}\@.+?\]\$'.format(getpass.getuser())
    output = ""

    child = pexpect.spawn('/bin/bash')
    try:
        # Look for initial bash prompt
        child.expect(prompt)
        # Start our program
        child.sendline("{0} {1}".format(script_path, options))
        # look for our scripts logfile prompt
        # Example text seen in output: 'Logfile: /path/to/mylog.log'
        child.expect(r'Logfile\:.+?/.+?\.log')
        # Note that child.after contains the text of the matching regex
        logfile = child.after.split()[1]
        # Look for password prompt
        i = child.expect([r"Enter password\:", '>'])
        if i == 0:
            # Send password
            child.sendline(password)
            child.expect('>')
        # Increase timeout for longer running interactions after quick initial ones
        child.timeout = timeout
        try:
            # Look for program internal prompt or new config dialog
            i = child.expect([r'Initialize New Config\?', '>'])
            # pexpect will return the index of the regex it found first
            if i == 0:
                # Answer 'y' to initialize new config prompt
                child.sendline('y')
                child.expect('>')
            # If any commands were passed in loop over them and run them one by one.
            for command in commands:
                child.sendline(command)
                i = child.expect([r'ERROR.+?does not exist', r'ERROR.+?$', '>'])
                if i == 0:
                    # Attempt to intelligently add items that may have multiple instances and are missing
                    # e.g. "socket.2" may need "add socket" run before it.
                    # Try to allow the user just to use the set command and run add as needed
                    try:
                        new_item = child.after.split('"')[1].split('.')[0]
                    except IndexError:
                        raise RuntimeError("ERROR: unable to automatically add new item in myscript,"
                                           " file a bug\n  {0}".format(child.after))
                    child.sendline('add {0}'.format(new_item))
                    i = child.expect([r'ERROR.+?$', '>'])
                    if i == 0:
                        raise RuntimeError("ERROR: unable to automatically add new item in myscript,"
                                           " file a bug\n  {0}".format(child.after.strip()))
                    # Retry the failed original command after the add
                    child.sendline(command)
                    i = child.expect([r'ERROR.+?$', '>'])
                    if i == 0:
                        raise RuntimeError("ERROR: unable to automatically add new item in myscript,"
                                           " file a bug\n  {0}".format(child.after.strip()))
                elif i == 1:
                    raise RuntimeError("ERROR: unspecified error running a myscript command\n"
                                       "  {0}".format(child.after.strip()))
            # Set timeout shorter for final commands
            child.timeout = 15
            # If we processed any commands run the save function last
            if commands:
                child.sendline('save')
                # Using true loops with expect statements allow us to process multiple items in a block until
                #    some kind of done or exit condition is met where we then call a break.
                while True:
                    i = child.expect([r'No changes made', r'ERROR.+?$', '>'])
                    if i == 0:
                        changed = False
                    elif i == 1:
                        raise RuntimeError("ERROR: unexpected error saving configuration\n"
                                           "  {0}".format(child.after.strip()))
                    elif i == 2:
                        break
            # Always print out the config data from out script and return it to the user
            child.sendline('print config')
            child.expect('>')
            # Note that child.before contains the output from the last expected item and this expect
            current_settings = child.before.strip()
            # Run the 'exit' command that is inside myscript
            child.sendline('exit')
            # Look for a linux prompt to see if we quit
            child.expect(prompt)
        except pexpect.TIMEOUT:
            raise RuntimeError("ERROR: timed out waiting for a prompt in myscript")
        # Get shell/bash return code of myscript
        child.sendline("echo $?")
        child.expect(prompt)
        # process the output into a variable and remove any whitespace
        exit_status = child.before.split('\r\n')[1].strip()
        if exit_status != "0":
            raise RuntimeError("ERROR: The command returned a non-zero exit code! '{0}'\n"
                               "Additional info:\n{1}".format(exit_status, output))
        child.sendline('exit 0')
        # run exit as many times as needed to exit the shell or subshells
        # This might be useful if you ran a script that put you into a new shell where you then ran some other scripts
        # This is also a good example of
        while True:
            i = child.expect([prompt, pexpect.EOF])
            if i == 0:
                child.sendline('exit 0')
            elif i == 1:
                break
    finally:
        # Always try to close the pexpect process
        child.close()
    return current_settings, changed, logfile


if __name__ == '__main__':
    main()

In order to create a module you need to put your new “mymodule.py” file somewhere in the Ansible module library path, typically the “library” directory next to your playbook or library inside your role. It’s also important to note that Ansible library modules run on the target ansible host, so if you want to use the ansible “expect” module or make a custom module with pexpect in it then you will need to install the python pexpect module on the remote host before running module. (Note: the pexpect version provided in RHEL/CentOS repos is old and will not support the Ansible “expect” module, install via pip instead for the latest version.) 

 Information on the library path is located here: 

https://docs.ansible.com/ansible/latest/dev_guide/developing_locally.html 

 Your example.py file needs to be a standard file with a python shebang header and also import the ansible module. Here is a bare minimum amount of code needed for an ansible module. 

#!/usr/bin/env python 
from ansible.module_utils.basic import AnsibleModule 
module = AnsibleModule(argument_spec=dict(mysetting=dict(required=False, type='str'))) 
try: 
    return_value = "mysetting value is: {0}".format(module.params['mysetting']) 
except: 
    module.fail_json(msg="Unable to process input variable into string") 
module.exit_json(changed=True, my_output=return_value) 

With this example you can see how variables are passed into and out of the module. This also includes a basic exception handle for dealing with errors and allowing ansible to deal with the failure. This exception clause is too broad for normal use as it will catch and hide all errors that could happen in the try block. When you create your module you should only except error types that you anticipate to avoid hiding stack traces of unexpected errors from your logs. 

Now we can add in some custom pexpect processing code. This is again a very basic example. The example code linked in this blog post has a complicated and in-depth example. This function would then be added into our try-except block in the code above. 

def run_pexpect(password): 
    import pexpect 
    child = pexpect.spawn('/path/to/myscript.sh') 
    child.timeout = 60 
    child.expect(r"Enter password\:") 
    child.sendline(password) 
    child.expect('Thank you') 
    child.sendline('exit') 
    child.expect(pexpect.EOF) 
    exit_dialog = child.before.strip() 

    return exit_dialog

There are some important things to note here when dealing with pexpect and Ansible. 

• If the program hits a timeout it will raise “pexpect.TIMEOUT” and if it terminates unexpectedly it will raise “pexpect.EOF”. These exceptions will need to be either expected, with child.expect or excepted using pythons exception handling. Any other exceptions don’t really need to be handled as then are likely real errors that should cause failure and raise a stack trace.  
• Always use a timeout! Be careful never to set the timeout to None as an unattended process will hang forever waiting on any new/unexpected prompt. It’s always better to set a very generous timeout over none at all. You can change the timeout multiple times in code based on how long you expect each prompt to take to come back. 
• If you do not set a timeout value at all the default for the spawn class is 30 seconds. This is the timeout looking for the text in an expect method. Even if your program is outputting text to stdout, when the timeout is hit before the string is found then the program is killed and pexpect.TIMEOUT is raised. 

• Don’t use print functions in python to try to send information back to Ansible. Printing to stdout with your module will cause ansible to register a failure. All output and information should be passed back through an Ansible method like “module.exit_json”.  
• For debugging you may want to use the “child.logfile” facility to create log files on the remote system. 
• The “child.expect” method takes regular expressions as input. If you want an explicit string you can always “import re” and use the “re.escape” method on a string to escape it. 

When creating custom modules I would encourage you to give thought to making the simplest, most maintainable and modular modules possible. It can be easy to create one module/script to rule them all, but the linux concept of having one tool to do one thing well will save you rewriting chunks of code that do the same thing and also help future maintainers of the automation you create. 

Helpful links: 

https://docs.ansible.com/ansible/latest/modules/expect_module.html 

https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html 

https://pexpect.readthedocs.io/en/stable/overview.html 

If you have any questions about the steps documented here, would like more information on the custom development process, or have any feedback or requests, please let us know at [email protected].

Red Hat OpenShift Container Platform is an Enterprise Kubernetes offering by Red Hat, that allows users to deploy cloud native applications as well as manage lifecycle of microservices deployed in containers. OpenShift Online is a SaaS offering for OpenShift provided by Red Hat. OpenShift Online takes away the effort required to set up your OpenShift clusters on-prem and allows organizations to quickly leverage all that OpenShift offers, including Developer console, without worry about managing the underlying infrastructure.  

OpenShift Online provides REST based APIs for all functions that can be carried out via the console, and the oc command line. Therefore, teams can build automated functionality that leverages Kubernetes cluster using OpenShift management plane. Today, we will look at one such function – to create a Project. Any user that wants to create a project using APIs is required to have appropriate role bindings in the specific namespace that they want to create or manage projects in. By default, OpenShift Online provides you the ability to create Projects via the console, using the ProjectRequest API call.  

Assuming you have the oc command line setup, the command to create a project is: 

$ oc new-project <project_name>  
 --description="<description>" --display-name="<display_name>" 

We will take a look at how to create a Project in OpenShift Online using the REST API. We will be using Postman to trigger our API call. This sample was run against OpenShift v3.11, Postman v7.30.1.  

1) The first thing we will do is log into our OpenShift Online console, and on the top right section in the drop-down that shows up under your name, select 'Copy Login Command'. Paste the copied contents into Notepad and capture the 'token' value. 

2) Download and import the Postman collection for this sample API call here  

3) Paste the copied token value under 'Authorization' section of the request 

4) Update the sections in bold for an appropriate name you want your Project to have 

{ 

    "kind": "ProjectRequest", 

    "apiVersion": "v1", 

    "displayName": "82520759", 

    "description": "test project from postman", 

    "metadata": { 

        "labels": { 

            "name": "82520759", 

            "namespace": "82520759" 

        }, 

        "name": "82520759" 

    } 

} 

5) Execute the Postman call. You should now see a new project created under your OpenShift Online instance.  

You can adjust the Body of the sample call to pass in more values associated with the ProjectRequest object. For reference, the object schema includes the below  

https://docs.openshift.com/container-platform/3.11/rest_api/oapi/v1.ProjectRequest.html 

apiVersion: 
description: 
displayName: 
kind: 
metadata: 
  annotations: 
    clusterName: 
      creationTimestamp: 
      deletionGracePeriodSeconds: 
      deletionTimestamp: 
   finalizers: 
     generateName: 
     generation: 
   initializers: 
   labels: 
     name: 
     namespace: 
   ownerReferences: 
     resourceVersion: 
     selfLink: 
     uid: 
 

Once you've unit tested the REST call with Postman for your OpenShift Online environment, you can very easily port this over to using one of the existing modules in Ansible, and making it a step within your playbook.  

If you have any questions or comments on the tutorial content above, or run in to specific errors not covered here, please feel free to reach out to [email protected]