Testing

With many people contributing to the automation, it is crucial to test the automation content in-depth. So when you’re developing new Ansible Content like playbooks, roles and collections, it’s a good idea to test the content in a test environment before using it to automate production infrastructure. Testing ensures the automation works as designed and avoids unpleasant surprises down the road.
Testing automation content is often a challenge, since it requires the deployment of specific testing infrastructure as well as setting up the testing conditions to ensure the tests are relevant.

Consider the following list for testing your Ansible content, with increasing complexity:

1. yamllint
2. ansible-playbook --syntax-check
3. ansible-lint
4. molecule test
5. ansible-playbook --check (against production)
6. Parallel infrastructure

Syntax check

The whole playbook (and all roles and tasks) need to, minimally, pass a basic ansible-playbook syntax check run.

ansible-playbook main.yml --syntax-check

Running this as a step in a CI Pipeline is advisable.

Linting

Take a look at the Linting section for further information.

Molecule

The Molecule project is designed to aid in the development and testing of Ansible roles, provides support for testing with multiple instances, operating systems and distributions, virtualization providers and testing scenarios. Test scenarios can target any system or service reachable from Ansible, from containers and virtual machines to cloud infrastructure, hyperscaler services, APIs, databases, and network devices. Molecule can also validate inventory configurations and dynamic inventory sources.
Molecule is mostly used to test roles in isolation (although it is possible to test multiple roles or playbooks at once).

To test against a fresh system, molecule uses a container runtime to provision virtualized/containerized test hosts, runs commands on them, asserts the success and destroys them afterwards.
By default, Containers don't allow services to be installed, started and stopped as in a virtual machine. We will be using custom systemd-enabled images, which are designed to run an init system as PID 1 for running multi-services inside the container. Also, some additional configuration is needed in the Molecule configuration file as shown below.

Take a look at the Molecule documentation for a full overview.

Installation

The described configuration below expects the Podman container runtime on the Ansible Controller (other drivers like Docker are available). You can install Podman with the following command:

sudo apt install podman

The Molecule binary and dependencies are installed through the Python package manager, you'll need a fairly new Python version (Python >= 3.10 with ansible-core >= 2.12).
Use a Python Virtual environment (requires the python3-venv package) to encapsulate the installation from the rest of your Controller.

python3 -m venv molecule-venv

Activate the VE:

source molecule-venv/bin/activate

Install dependencies, after upgrading pip:

pip3 install --upgrade pip setuptools

pip3 install ansible-core molecule molecule-plugins[podman]

Molecule plugins contains the following provider:

• azure
• containers
• docker
• ec2
• gce
• openstack
• podman
• vagrant

ansible-galaxy collection install containers.podman

If you are done with Molecule testing, use deactivate to leave your VE.

Configuration

Create the directory molecule/default and at least the molecule.yml and converge.yml.
Depending on your project setup (classic role structure or collection), the Molecule configuration files need to be stored at different locations.

roles/
└── webserver_demo
    ├── defaults
    │   └── main.yml
    ├── molecule
    │   └── default
    │       ├── converge.yml
    │       └── molecule.yml
    ├── tasks
    │   └── main.yml
    └── templates
        └── index.html

.
├── README.md
├── extensions
│   └── molecule
│       └── default
│           ├── converge.yml
│           └── molecule.yml
├── galaxy.yml
├── meta
│   └── runtime.yml
└── roles
    └── webserver_demo
        ├── defaults
        │   └── main.yml
        ├── tasks
        │   └── main.yml
        └── templates
            └── welcome.html.j2

You may use this (minimal) example configuration as a starting point.

---
driver:
  name: podman
platforms: # (1)!
  - name: rhel9-instance1 # (2)!
    image: ghcr.io/timgrt/rhel9-molecule-test-image:main # (3)!
    volumes: # (4)!
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    command: "/usr/sbin/init"
    published_ports: # (5)!
      - 8080:80/tcp
    groups: # (6)!
      - molecule
ansible:
  executor:
    args:
      ansible_playbook:
        - --inventory=../../../../inventory/ # (7)!
  cfg:
    defaults:
      interpreter_python: auto_silent
      remote_user: ansible # (8)!
      callbacks_enabled: ansible.posix.timer, ansible.posix.profile_tasks # (9)!
      callback_result_format: yaml  # (10)!
      roles_path: "${MOLECULE_PROJECT_DIRECTORY}/.." # (11)!
    diff: # (12)!
      always: true

---
- name: Converge
  hosts: molecule # (1)!
  become: true
  roles:
    - webserver_demo # (2)!

---
- name: Prepare
  hosts: molecule
  become: true
  tasks:
    - name: Install OpenSSH for ssh-keygen
      ansible.builtin.package:
        name: openssh
        state: present

---
- name: Verify
  hosts: molecule
  become: true
  tasks:
    - name: Get service facts
      ansible.builtin.service_facts:

    # Service may have started, returning 'OK' in the service module, but may have failed later.
    - name: Ensure that MariaDB is in running state
      assert:
        that:
          - ansible_facts['services']['mariadb.service']['state'] == 'running'

Molecule variables

The configuration options may contain environment variables, either Molecule-specific or default environment variables, e.g. USER. Some example variables are the following:

The variables can be used to create custom instance names:

---
platforms:
  - name: rhel9-$MOLECULE_SCENARIO_NAME-$USER
    image: ghcr.io/timgrt/rhel9-molecule-test-image:main

(ve-molecule) timgrt@wsl-ubuntu:demo$ molecule list -f yaml
INFO     Collection 'cc_ansible_community.demo' detected.
INFO     Scenarios will be used from 'extensions/molecule'
WARNING  Driver podman does not provide a schema.
INFO     default ➜ list: Executing
INFO     default ➜ list: Executed: Successful
---
- Converged: 'false'
  Created: 'true'
  Driver Name: podman
  Instance Name: rhel9-default-timgrt
  Provisioner Name: ansible
  Scenario Name: default
- Converged: 'false'  
  Created: 'false'  
  Driver Name: podman  
  Instance Name: rhel9-hardening-timgrt  
  Provisioner Name: ansible  
  Scenario Name: hardening

Scenario definition

A scenario is a self-contained directory containing everything necessary for testing the content in a particular way.
The default scenario is named default, but you can define additional ones. The scenario name will be the directory name hosting the files.
For example, you can have a default scenario which uses Podman containers as infrastructure and another scenario which uses the libvirt driver.

roles/
└── webserver_demo
    ├── defaults
    │   └── main.yml
    ├── molecule
    │   ├── default
    │   |   ├── converge.yml
    │   |   └── molecule.yml
    │   └── libvirt
    │       ├── converge.yml
    │       └── molecule.yml
    ├── tasks
    │   └── main.yml
    └── templates
        └── index.html

Usage

Activating your Python VE with molecule:

source molecule-venv/bin/activate

In a collection project, you can execute Molecule directly from the project root directory.
If your are using Molecule in a classic project, it is executed from within the role you want to test. Change directory:

cd roles/webserver_demo

molecule test -s libvirt

To only create the defined containers, but not run the Ansible tasks:

molecule create

To run the Ansible tasks of the role (if the container does not exist, it will be created):

molecule converge

To destroy the provisioned infrastructure.

molecule destroy

To execute a full test circle (existing containers are deleted, re-created and Ansible tasks are executed and containers are deleted(!) afterwards):

molecule test

If you want to login to a running container instance:

molecule login

$ molecule login -h rhel9-instance1
[root@rhel9-instance1 /]# grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Red Hat Enterprise Linux 9.7 (Plow)"

Temporary files

Molecule writes a couple of temporary files to indicate which steps of a sequence were already performed. For example, if a test instance was already created and prepared, this state is written to a state.yml file. All temporary files are written to ~/.ansible/tmp/molecule.<hash>.<scenario-name>/.

$ tree ~/.ansible/tmp/molecule.KpdR.default/
/home/timgrt/.ansible/tmp/molecule.KpdR.default/
├── ansible.cfg
├── inventory
│   └── ansible_inventory.yml
├── molecule.yml
└── state.yml

Instance unreachable

In some cases, you may encounter the following error:

TASK [Gathering Facts] *********************************************************
fatal: [rhel9-instance]: UNREACHABLE! =>
    changed: false
    msg: 'Failed to create temporary directory. In some cases, you may have been able
        to authenticate and did not have permissions on the target directory. Consider
        changing the remote tmp path in ansible.cfg to a path rooted in "/tmp", for more
        error information use -vvv. Failed command was: ( umask 77 && mkdir -p "` echo
        ~/.ansible/tmp `"&& mkdir "` echo ~/.ansible/tmp/ansible-tmp-1768236855.049918-34546-5922621697359
        `" && echo ansible-tmp-1768236855.049918-34546-5922621697359="` echo ~/.ansible/tmp/ansible-tmp-1768236855.049918-34546-5922621697359
        `" ), exited with result 125'
    unreachable: true

This can happen if the container was removed, but the temporary files were not cleaned up correctly.
Run molecule reset or molecule destroy to cleanup the potentially still existing resources. Afterwards, run a new test sequence.

Minimal testing environment

You'll miss out on the convenient and frankly easy to use possibilities of Molecule, but, if you just need a small environment for testing your Ansible content without impacting your Ansible Control Node, the following setup spins up a small one in (Podman) containers. You will need Podman and Ansible (naturally), but nothing else.

Installation

You can install Podman with the following command:

sudo apt install podman

The playbook to create the testing instances uses the containers.podman collection, if you only installed ansible-core, you'll need to install the collection separately:

ansible-galaxy collection install containers.podman

Configuration

Copy the three files in the separate tabs, a playbook for creating the testing environment, an inventory file defining the testing instances and a small demo playbook which can be used to test your Ansible content.

---
- name: Create or delete demo environment for local testing
  hosts: localhost
  connection: local
  vars:
    testing_image: docker.io/timgrt/rockylinux9-ansible:latest
  tasks:
    - name: "{{ (delete | default(false)) | ternary('Delete', 'Create') }} demo instance"
      containers.podman.podman_container:
        name: "{{ item }}"
        hostname: "{{ item }}"
        image: "{{ testing_image }}"
        volumes:
          - /sys/fs/cgroup:/sys/fs/cgroup:ro
        command: "/usr/sbin/init"
        state: "{{ (delete | default(false)) | ternary('absent', 'started') }}"
      loop: "{{ groups['test'] }}"

[test]
instance1

[test:vars]
ansible_user=ansible
ansible_connection=podman

---
- name: Testing playbook
  hosts: test
  tasks:
    - name: Output distribution
      ansible.builtin.debug:
        msg: "{{ ansible_distribution }}"

Usage

First, create the testing instances by executing the testing_environment.yml playbook:

ansible-playbook -i testing_inventory.ini testing_environment.yml

Add your tasks to the testing_playbook.yml (or use your existing playbook, target the test group) and execute:

ansible-playbook -i testing_inventory.ini testing_playbook.yml

After finishing your tests remove the instances by running the testing_environment.yml playbook and provide the extra-var delete:

ansible-playbook -i testing_inventory.ini testing_environment.yml -e delete=true