“Config” is factor 3 of The Twelve-Factor App. Embedded in that config is the data an application uses to understand and work with the environment it is deployed. Config may contain hostnames, URLs, tags, contacts, and importantly, secrets: Passwords, API keys, certificates, client tokens, cryptographic keys, utilized by the application to secure or access resources and data.
Configuration is kept separate from an app to make it easy to configure per-environment, but also to rotate secrets to ensure compliance or protect data in case of exposure. Deploying applications with Red Hat Ansible provides several ways to easily and securely inject secrets into the configuration, so that deployments are unique to the environment they’re running in.
Managing secrets with Ansible Vaults
Ansible vault is a tool embedded within the Ansible package that encrypts and decrypts variables and files for use within the Ansible ecosystem.
Ansible vaults are:
- Variables, embedded within a playbook or variable file as a string of encrypted values,
- Variable files, containing multiple variables and values encrypted individually,
- Files, containing the encrypted data encrypted and managed by Ansible vault.
Introduction to Ansible vaults
The key benefit of Ansible vaults are they are the Ansible-native way to manage secrets within Red Hat Ansible infrastructure and integrate neatly with the rest of the Ansible Automation Platform, playbooks, and templates.
Being the Ansible-native way of managing secrets, Ansible vault is probably already installed on the hosts and systems developing roles and deploying playbooks. Vault data can be included in playbooks with minor changes to deployment scripts within the CLI or to job templates in Ansible Automation Platform.
Using Vault IDs
Vault IDs associate a password source with an encrypted object. To Ansible, they provide the root key that is used to encrypt the
- When encrypting, the vault ID tells Ansible vault which source should be used to encrypt data. That source can be a password (root key or “cryptovariable”) from stdin, prompt, password from a file, or a script, which is used to encrypt the data.
- When encrypted and at rest, the ID is a label appended to the ciphertext and specifies the source of the password. Ansible vault uses the label as a hint for the password source (though, not strictly; more on that).
- When decrypting, Ansible will attempt to decrypt the data using all of the provided passwords, from all vault IDs, starting with the password from the vault ID first.
Encrypting secrets with Ansible vaults
Encrypting and decrypting data with Ansible vault likely doesn’t require installing any additional packages.
1. Create and install a new virtual environment
$ python3 -m venv env $ source env/bin/activate
2. Install Ansible
$ pip install ansible Collecting ansible
3. Create file vars.yml with variables
$ <> vars.yml api_key: My API Key db_password: MyD8Pa55word EOT
4. Encrypt with Ansible vault (enter a password when prompted)
$ ansible-vault encrypt vars.yml New Vault password: Confirm New Vault password: Encryption successful
To decrypt and edit the contents, run ansible-vault decrypt
$ ansible-vault decrypt vars.yml Vault password: Decryption successful
Working with vault data in Ansible
More likely, though, is that you’ll be using those variables within an Ansible Playbook execution.
1. The vars.yml needs to be re-encrypted:
$ ansible-vault encrypt vars.yml New Vault password: Confirm New Vault password: Encryption successful
2. Update the playbook to use the newly created vars file
# playbook.yml
---
- name: Use API key
hosts: localhost
vars_files:
- vars.yml
vars:
api_key: "{{ undef(hint=Specify your API key') }}"
db_password: "{{ undef(hint='Provide a DB password') }}"
tasks:
- ansible.builtin.debug:
msg: "my super secret API key: {{ api_key }}"3. Specify –ask-vault-pass to prompt for password when playbook is executed.
$ ansible-playbook playbook.yml --ask-vault-pass
Vault password:
PLAY [Use API Key] *******************************************************************
TASK [Gathering Facts] *******************************************************************
ok: [localhost]
TASK [ansible.builtin.debug] *******************************************************************
ok: [localhost] => {
"msg": "my super secret API key: My API Key"
}When Ansible encounters the encrypted variables, it will see the header defining it as a vault, and then automatically use the provided IDs to decrypt the data.
Credentials can be added within the Ansible Automation platform like any other credentials, and then included in the Job template to be made available within the play.
Working with external secrets
Not all secrets will be stored within Ansible Automation Platform. Organizations may have security or audit policies that require secrets be stored in a central platform or service.
For that, there are two more commonly used methods for accessing secrets: Lookup plugins and modules.
Accessing external secrets with Ansible Lookup plugins
Lookup plugins add additional functionality to Jinja2, the templating framework utilized by Ansible, to return data using the configured provider. Many plugins are included in the default Ansible distribution, and authors can create and include their own plugins within collections, roles, and playbooks. A list of available plugins can be viewed by calling ansible-doc -t lookup -l in the CLI.
With lookup plugins, playbooks can template in dynamic data directly within the configuration, eliminating a lot of additional config and code that may otherwise be needed to provide the values to a play.
For example, the community.dns.lookup_as_dict plugin will query DNS and return a dictionary of DNS entries for the provided domain.
$ ansible localhost \
-m 'ansible.builtin.debug' \
-a "msg={{ lookup('community.dns.lookup_as_dict', 'example.org') }}"
localhost | SUCCESS => {
"msg": [
{
"address": "96.7.128.186"
},
{
"address": "23.215.0.132"
},
{
"address": "96.7.128.192"
},
{
"address": "23.215.0.133"
}
]
}
The lookup function can just as easily be used with external secret managers, like Hashicorp Vault, AWS Secrets Manager, and Azure Key Vault.
The lookup function is configured similarly to other Jinja functions and can be used throughout playbooks or templates to pull in secrets. The following example playbook and template file utilize the lookup plugin to grab secrets from Hashicorp Vault and inject them into the task or template.
# playbook.yml
---
- name: Query KV from Hashi Vault using Lookup plugin
hosts: localhost
vars:
# Alternatively, export the VAULT_ADDR env to the Ansible runtime
vault_address: https://my-vault-url.dev:8201
tasks:
- name: Using lookup within a playbook
ansible.builtin.debug:
msg: "{{ lookup('community.hashi_vault.vault_kv2_get', 'my_secret', url=vault_address) }}"
- name: Using lookup with a template
ansible.builtin.template:
src: service_config.yml.j2
dest: /etc/myservice/config.yml
owner: user
group: group
mode: '0644'
{# service_config.yml.j2 #}
---
connnection:
hostname: my_url
username: my_user
password: {{ lookup('community.hashi_vault.vault_kv2_get', 'database/password', url=vault_address) }}Using a lookup plugin reduces the potential for exposure of a secret by only accessing it when required, and writing it directly to the task or template.
Using Modules to manage secrets
Lookup plugins are probably the easiest way of reading secrets, but if a playbook needs to manage the full lifecycle of a secret, then it may be better to use Ansible modules.
Modules make up the tasks that are performed by Ansible during a run. Here is an example of pulling a secret from Hashicorp Vault using a module.
# playbook.yml --- - name: Create secret in Hashi Vault using Module hosts: localhost module_defaults: group/community.hashi_vault.vault: url: https://my-vault-url.dev:8201 tasks: - name: Write secret to Vault community.hashi_vault.vault_kv2_write: path: application/secret_value data: mysecretvalue
Protecting secrets in Ansible
There are a few things to be aware of, though, when writing playbooks that utilize Ansible vault and/or external secrets.
Keeping secrets at rest
Ansible vault uses AES-256 to encrypt vault data at-rest, and encrypted HMACs to ensure the integrity of that data.
To encrypt, Vault uses the provided password and a unique salt to generate data encryption keys for the data and HMAC each time it performs encryption, and uses that same password and salt to verify and decrypt the data when it reads.
The security of data stored by Ansible vault is protected by controlling access to the password(s) used to encrypt, rather than needing to control access to the data itself.
Even though data is encrypted with AES-256, it’s still important to keep secrets, even encrypted secrets, secure from unauthorized access by storing them within an artifact repository, object store, or secured filesystem
Protecting secrets during Ansible Plays
When Ansible accesses secrets during a play, the content of a secret, whether from Ansible vault or from an external provider, can potentially be output to logs. Printing to logs within Ansible Automation Platform isn’t immediately an issue, but, many organizations configure AAP to forward logs to a central repository for compliance and management.
Again, this may be expected. But, when that data is forwarded, it increases the risk that secrets can be seen or accessed by unauthorized individuals.
Within Ansible Automation Platform, job template authors can change the configure to limit logging verbosity, reducing the amount of information published to the output during a play. But that setting can still be overridden, and the risk of accident exposure from unaware authors is still there.
The best way to control secrets during Ansible plays is to enable the no_log: True for the task.
By enabling no_log, Ansible will not print any information from the task, even with verbosity at level 4.
$ ansible-playbook playbook.yml --ask-vault-pass -vvvv
ansible-playbook [core 2.18.2]
...
Vault password:
...
PLAYBOOK: playbook.yml *******************************************************************
...
1 plays in playbook.yml
Trying secret <ansible.parsing.vault.PromptVaultSecret object at 0x1025ba510> for vault_id=default
...
PLAY [localhost] *******************************************************************
TASK [Gathering Facts] *******************************************************************
ok: [localhost]
TASK [ansible.builtin.debug] *******************************************************************
task path: .../playbook.yml:9
ok: [localhost] => {
"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"
}Even with connection debug enabled, in the truncated logs above, the task output is hidden.
That secret can still be exposed to output, though, if it’s utilized by other tasks. Any task that includes that secret within it’s config or output would need to enable no_log to prevent exposure. This can be frustrating because it makes it difficult to debug issues during a play because several tasks from the playbook might need to be hidden.
Ansible Automation Platform provides many tools to secure secrets and data within job templates and plays, and available plugins and modules provide multiple methods of utilizing those secrets within plays. But, authors need to be careful how those secrets are utilized within the play, and utilize the correct features of Ansible to ensure secrets are not exposed or stored insecurely at rest.
If you would like to learn more or have a conversation about how Ansible Automation Platform can provide value in your organization, contact us.
 | Christopher Gerber, Solutions Architect Chris brings extensive experience in building developer and data platforms on AWS and GCP. As part of our technical pre-sales team, he supports sales efforts and leads projects focused on AWS, GCP, Ansible, CloudFormation, and container implementations. With a passion for enhancing developer experiences and creating resilient platforms, Chris is dedicated to helping clients achieve their goals through innovative solutions. Like what you read? Follow Christopher on LinkedIn |