{{ vExpose }}.Blog

Author: Alberto

  • Keep a container alive to troubleshoot it

    Containers are a great way to package code and dependencies in a portable package. This is fantastic in a world where dependencies keep getting bigger and more complex. But we are not here to discuss their benefits. If Google brought you here, more than likely you know already what a container is and you are trying to troubleshoot one. If that’s the case, I hope you find useful the tip I am about to tell you.

    To understand the solution we must first remember that containers are designed to run a task as soon as they start. You define such task with either CMD or ENTRYPOINT. The problem is that if the task dies the container dies too and you can’t log in into it to find out what happened. Sometimes “docker logs” will give you a hint of what’s happening but other times you wish you could simply log in to the container and check things out, but of course you can’t because the container is dead

    A solution I have become fond of is to override the ENTRYPOINT with a process that keeps the container alive and then open a terminal session into it. Let me give you an example.

    IMPORTANT: I am using Podman but all of this will work perfectly fine with Docker as well. This is the “Dockerfile“. Notice how the main task for this container is “python app.py“

    FROM python:3.10.5
WORKDIR /app
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 7860
CMD ["python", "app.py"]

    This is the code contained in “app.py“. It is a simple Gradio chatbot. It gets the users prompt and sends an API call to to get a response.

    import gradio as gr
import os, requests, import urllib3
urllib3.disable_warnings()

appurl = os.environ["APP_URL"]

def give_response(query, history):
    payload = {"query": query}
    response = requests.post(appurl, json=payload)
    return response.json()["response"]

demo = gr.ChatInterface(
              give_response,
              type = "messages",
              title="My first Chatbot",
              description="Ask me a question, don't be shy")

demo.launch(server_name="0.0.0.0")

    As you can see the code requires us to define an environment variable “APP_URL” with the URL to send the request to. Let’s say that we don’t define the variable and attempt to run the container.

    pi@piper1:~$ podman run -d -p 7860:7860 localhost/blog:v1
19dc02db0b2e06c51756c7f55a3eb861c11f831a21574efdcd2e00081c9191a1
pi@piper1:~$ podman ps -a
CONTAINER ID  IMAGE              COMMAND        CREATED        STATUS      NAMES
19dc02db0b2e  localhost/blog:v1  python app.py  3 seconds ago  Exited (1)  dry_rice

    As expected the container fails but here is the trick … we run it again but we use the “–entrypoint” argument to run “tail -f /dev/null“. The container is not changing we are simply overriding “python app.py” with this “tail” command which stays running and in doing so keeps the container alive. Notice the “single quote” around the square brackets.

    pi@piper1:~$ podman run -d --entrypoint='["tail", "-f", "/dev/null"]' -p 7860:7860 localhost/blog:v1
e1031d4c2ecbf3d4ec244aef262e71fb2f4e227154b6e22eabf634e4e053f4a0
pi@piper1:~$ podman ps
CONTAINER ID  IMAGE              STATUS        PORTS                   NAMES
e1031d4c2ecb  localhost/blog:v1  Up 6 seconds  0.0.0.0:7860->7860/tcp  sour_soup

    Now we can login into the container and do whatever checks we need to do. We can even run “python app.py” and see what’s going on live.

    pi@piper1:~$ podman exec -it e1031d4c2ecb /bin/bash
root@e1031d4c2ecb:/app# ls -l
total 12
-rw-r--r-- 1 root root 518 Mar 24 04:35 Dockerfile
-rw-r--r-- 1 root root 566 Mar 24 04:35 app.py
-rw-r--r-- 1 root root 897 Mar 24 04:35 requirements.txt
root@e1031d4c2ecb:/app# python3 app.py
Traceback (most recent call last):
  File "/app/app.py", line 7, in <module>
    appurl = os.environ["APP_URL"]
  File "/usr/local/lib/python3.10/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'APP_URL'

    This was a simplistic example but you get point. There could be a file or a path missing, a typo, a permissions issue … by using this trick you can troubleshoot interactively inside the container. Once you know what the issue is you can fix your files and rebuild the container image.

    The next question is, can you do this in Kubernetes as well? Yes, you can add the “command” to the “Deployment”, not to the “Pod”. This is a list of strings as you see in the last line in the simplified manifest below. 

    apiVersion: v1
kind: Deployment
metadata:
  name: blog
spec:
  containers:
  - name: blog-container
    image: debian
    command: ["tail", "-f", "/dev/null"]

    As soon as I save the changes to the deployment manifest, Openshift creates a new pod and kills the old one. Then I can terminal into the new pod and browse around. Notice I can even launch the application from the terminal session. 

    $ pwd         
/app
$ ls -l
total 12
-rw-r--r--. 1 root root 518 Mar 24 04:35 Dockerfile
-rw-r--r--. 1 root root 566 Mar 24 04:35 app.py
-rw-r--r--. 1 root root 897 Mar 24 04:35 requirements.txt
$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
1000770+       1       0  0 00:00 ?        00:00:00 tail -f /dev/null
1000770+       7       0  0 00:00 pts/0    00:00:00 sh -i -c TERM=xterm sh
1000770+      13       7  0 00:00 pts/0    00:00:00 sh
1000770+      90      13  0 00:02 pts/0    00:00:00 ps -ef

$ python3 app.py
* Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.

    Of course, this is only intended for troubleshooting. Once you find out what’s wrong you can fix the image and manifest and deploy them again.

  • Podman build fails with /etc/passwd: permission denied

    A short post today to share an issue that troubled me for a while. Hopefully it can help you too.

    Basically I was trying to build a new container image using Podman in Ubuntu. These are my versions:

    • Ubuntu 24.04.2 LTS
    • podman version 4.9.3

    My Dockerfile is straight forward

    FROM python:3.10.5
WORKDIR /app
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt
CMD ["python", "app.py"]

    When I tried to create the container image I get the following output

    (env) user1@mytpl:~/Desktop/myapp$ podman build -t myapp:v1 .
STEP 1/6: FROM python:3.10.5
Resolved "python" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/python:3.10.5...
Getting image source signatures
Copying blob 588423e31bcf done   |
Copying blob 001c52e26ad5 done   |
Copying blob 2068746827ec done   |
Copying blob d9d4b9b6e964 done   |
Copying blob 8a335986117b done   |
Copying blob 9daef329d350 done   |
Copying blob ecb6a3f01c0d done   |
Copying blob 00d40f20f0cf done   |
Copying blob 5d588b4f3b55 done   |
Error: creating build container: copying system image from manifest list: writing blob: adding layer with blob "sha256:001c52e26ad57e3b25b439ee0052f66           92e5c0f2d5d982a00a8819ace5e521452": processing tar file(open /etc/passwd: permission denied): exit status 1

    In the end I tracked it down to a service called “mfetpd.service” which appears to be some Trellix endpoint security software. 

    (env) user1@mytpl:~/Desktop/myapp$ sudo systemctl status mfetpd.service
â—Ź mfetpd.service - Trellix Endpoint Security for Linux Threat Prevention
     Loaded: loaded (/usr/lib/systemd/system/mfetpd.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-03-24 14:35:44 +08; 5min ago
       Docs: man:mfetpd(8)
    Process: 30546 ExecStartPre=/opt/McAfee/ens/tp/scripts/aac-control-wrapper.sh systemd (code=exited, status=0/SUCCESS

    The solution was to temporarily stop this service to build the container image

    sudo systemctl stop mfetpd.service
podman build ...
sudo systemctl start mfetpd.service

    Once the image “python:3.10.5” layer is downloaded I am able to build more containers using the same version of Python without having to pause the “mfetpd” service which makes my security team happy.

    I hope this helps you same some of your precious time!

  • Ubuntu VMs get same IP from DCHP in vSphere

    The problem

    This is a quick post to present a solution to this problem that bugged me for a while. If you are using Ubuntu as the guest OS for vSphere virtual machines you might encounter this problem. I was running Ubuntu 22.04 but I think it might be the same problem for other versions.

    Every new VM I created was getting the same IP address from the DCHP server even though the MAC address was different.

    The explanation

    I had a Ubuntu VM template configured for DHCP as follows:

    root@albmtest:~# cat /etc/netplan/99-netcfg-vmware.yaml
# Generated by VMWare customization engine.
network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      dhcp4: yes
      dhcp4-overrides:
        use-dns: false
      nameservers:
        addresses:
          - 172.24.1.10

    The explanation is that by default Ubuntu’s Netplan doesn’t use the MAC address for DHCP. By default it uses the “machine-id”. This is explained in this paragraph of the Netplan documentation.

    So, unless you set “dhcp-identifier” to “mac“, by default netplan uses “machine-id“. You can find this in “/etc/machine-id” inside the VM. But as I found out, it transpires VMware doesn’t change regenerate “/etc/machine-id” in the guest by default. When you put those 2 default behaviors together you get a problem.

    The Solution

    You can solve this by either:

    • erasing the machine-id before creating a template by running “echo > /etc/machine-id“
    • editing the netplan yaml file to force it to use “mac” addresses for DHCP
    root@albmtest:~# cat /etc/netplan/99-netcfg-vmware.yaml
# Generated by VMWare customization engine.
network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      dhcp4: yes
      dhcp4-overrides:
        use-dns: false
      dhcp-identifier: mac
      nameservers:
        addresses:
          - 172.24.1.10

    If you modify “netplan” don’t forget to “apply” to make sure any changes you make are activated.

    root@albmtest:~# netplan apply

    Final TIP: when you are testing your changes, you can force the DHCP client to renew the IP like this

    root@albmtest:~# dhclient -r

    I hope this post saved you some trouble. Good luck!

  • Using Ansible to run Terraform plans

    Yes, your eyes are not deceiving you 🙂 Usually, when the words Ansible and Terraform appear in the same sentence is for the opposite scenario. Many articles out there provide examples of using Terraform to deploy infrastructure followed by running Ansible to configure it using the local-exec provisioner. This is a good thing because, if you need some post-provisioning configuration, it is better to rely on another declarative and idempotent tool like Ansible, rather than invoking shell scripts.

    But in this post we are not going to do any of that. What we are going to do is the other way around. We are going to have an Ansible playbook that runs your Terraform plan.

    This article is a follow up from a series of posts to introduce infrastructure admins and engineers to Terraform and to explain how to use it to manage on-prem physical infrastructure in a datacenter. You can find access the full series here.

    Reasons to do it

    But why? Well, if you have an physical infrastructure background and are running a traditional datacenter more than likely you have been playing with Ansible. Chances are you started with Ansible running in the command line and hard-coding your infrastructure’s credentials inside the playbook. Then you evolved into using an external credentials file and encrypting it using “ansible-vault”. Eventually you might have deployed Ansible AWX or even RedHat AAP. At this point, you are enjoying RBAC (who can run what playbook and on what infrastructure) and amazing (there is no other way to put it) credential management.

    Then, you talk to your developers over coffee and they tell you about Terraform. But when you get familiar with it, you discover that unless you pay for a SaaS offering you are stuck in the “command-line with hard-coded credentials” scenario. Wouldn’t it be great if you could use AWX to run Terraform? You could enjoy RBAC and credential management and all the other goodness that it provides. This is possible thanks to Ansible’s Terraform module.

    First steps

    Let’s say we have a very basic Terraform project to create a storage group in Dell PowerMax. The “main.tf” shown below starts the PowerMax provider and then creates a storage group “terraform_sg”. I don’t think it is strictly necessary, but if you are interested in learning more about provisioning storage in Dell PowerMax you can check out this previous post. For simplicity we haven’t created or referenced any variables. All the information is contained into a single file, the “main.tf” shown below.

    terraform {
  required_providers {
    powermax = {
      source = "dell/powermax"
    }
  }
}

provider "powermax" {
  username      = "smc"
  password      = "TFp2ssw0rd"
  endpoint      = "https://10.1.2.3:8443"
  serial_number = "000123456789"
  pmax_version  = 100
  insecure      = true
}

resource "powermax_storagegroup" "tf_sg" {
  name          = "terraform_sg"
  srp_id         = "SRP_1"
}

    In the same machine we have both Terraform and Ansible installed. In order to run the above Terraform plan with Ansible, we need at a minimum a playbook that looks like follows. It needs to include two mandatory parameters: “project_path” and “state”. The first parameter specifies the directory that contains the plan, ie the “main.tf” and typically any other files like “variables.tf”. My Ansible playbook is called “runtf.yml”.

    ---
- hosts: localhost
  gather_facts: no

  vars:
    project_dir: "/root/powermax" # folder that contains main.tf, ...

  tasks:
    - name: Run terraform
      community.general.terraform:
        project_path: '{{ project_dir }}'
        state: present
      register: output

    - debug:
        var: output

    As you can see I have registered the output of the terraform task and then created a second task to print it out on the terminal so that we can get more details. All is left now is to run it using the “ansible-playbook” command

    root@alb-terraform:~# ansible-playbook runtf.yml

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
changed: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": true,
        "command": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmpjaojemdl.tfplan",
        "failed": false,
        "outputs": {},
        "state": "present",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Creating...\npowermax_storagegroup.tf_sg: Creation complete after 0s [id=terraform_sg]\n\nApply complete! Resources: 1 added, 0 changed, 0 destroyed.\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Creating...",
            "powermax_storagegroup.tf_sg: Creation complete after 0s [id=terraform_sg]",
            "",
            "Apply complete! Resources: 1 added, 0 changed, 0 destroyed."
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    ignored=0

    Ansible run Terraform, and Terraform managed to make the changes so as expected, Ansible reports the task as “changed”. In the “debug” task, notice the “command” parameter. This is how Ansible is invoking Terraform. This invocation includes the “-auto-approve” flag so that Terraform doesn’t look for confirmation. The “stdout” and stdout_lines” parameters show the output that has been captured from Terraform about the creation of the different resources .

    A very important feature of both Ansible and Terraform is idempotency, ie the ability to run the same code repeatedly without making any damage. Let’s run it again and observe what happens.

    root@alb-terraform:~# ansible-playbook runtf.yml

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
ok: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": false,
        "command": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmp718kkwbo.tfplan",
        "failed": false,
        "outputs": {},
        "state": "present",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Refreshing state... [id=terraform_sg]\n\nNo changes. Your infrastructure matches the configuration.\n\nTerraform has compared your real infrastructure against your configuration\nand found no differences, so no changes are needed.\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Refreshing state... [id=terraform_sg]",
            "",
            "No changes. Your infrastructure matches the configuration.",
            "",
            "Terraform has compared your real infrastructure against your configuration",
            "and found no differences, so no changes are needed."
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost : ok=2    changed=0    unreachable=0    failed=0    skipped=0     ignored=0

    As expected, Ansible is reporting “ok”, which means the current state matched the desired state and therefore no changes were made. It has also captured the relevant from Terraform stating the same thing.

    Using variables

    This is a good start but ideally we would like to have the ability to make the plan more flexible by using variables for information such as the storage group name in this case. We can modify the “main.tf” as follows. Notice how the “default” value of the variable is commented out. It doesn’t matter if it is commented or not because when we invoke Terraform with external variables we overwrite the default value defined in the variable blocks.

    terraform {
  required_providers {
    powermax = {
      source = "dell/powermax"
    }
  }
}

provider "powermax" {
  username      = "smc"
  password      = var.password
  endpoint      = "https://10.1.2.3:8443"
  serial_number = "000123456789"
  pmax_version  = 100
  insecure      = true
}

variable "sg_name" {
 description = "Name of volume to create"
 type        = string
 #default     = "terraform_sg"
}

variable "password" {
  type        = string
  description = "Stores the password of Unisphere."
  #default = ""
}

resource "powermax_storagegroup" "tf_sg" {
  name          = var.sg_name
  srp_id         = "SRP_1"
}

    Additionally, notice how in the “provider” block we have also referenced another variable for the “password”. For security reasons we might want to have, sensitive information like that, stored in a separate file and protect it. In this case we haven’t even declared a default value because we can leverage ansible-vault or AWX to encrypt it and to pass it to Terraform at run time.

    Now, the corresponding Ansible playbook needs to use the “variables” parameter. This parameter essentially converts each of the variables into a “-var” flag when invoking Terraform. 

    ---
- hosts: localhost
  gather_facts: no

  vars:
    project_dir: "/root/powerstore" # folder that contains main.tf, ...

  tasks:
    - name: Run terraform
      community.general.terraform:
        project_path: '{{ project_dir }}'
        state: present
        variables:
          sg_name: "{{ vol_name }}"
          password: "{{ password }}"
      register: output

    - debug:
        var: output

    Now, at runtime we are going to select a different storage group name. The storage group is created as expected and the relevant Terraform output is captured. 

    root:~# ansible-playbook runtf.yml --extra-vars "sg_name=tf-sg2 password=TFp2ssw0rd"

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
changed: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": true,
        "command": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmpimdwzdu1.tfplan",
        "failed": false,
        "outputs": {},
        "state": "present",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Creating...\npowermax_storagegroup.tf_sg: Creation complete after 5s [id=vol1]\n\nApply complete! Resources: 1 added, 0 changed, 0 destroyed.\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Creating...",
            "powermax_storagegroup.tf_sg: Creation complete after 5s [id=vol1]",
            "",
            "Apply complete! Resources: 1 added, 0 changed, 0 destroyed."
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    ignored=0

    Terraform plan

    We went straight to “terraform apply” but most Terraform workflows also include running the plan command first. This is done with Ansible by running the playbook in “check mode”. Check mode invokes “terraform plan” for us.

    root:~# ansible-playbook runtf.yml --check --extra-vars "sg_name=tf-sg3 password=TFpass"

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
ok: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": false,
        "command": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmp6f2o83k5.tfplan",
        "failed": false,
        "outputs": {},
        "state": "present",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Refreshing state... [id=tf-sg2]\n\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  ~ update in-place\n\nTerraform will perform the following actions:\n\n  # powermax_storagegroup.tf_sg will be updated in-place\n  ~ resource \"powermax_storagegroup\" \"tf_sg\" {\n      ~ cap_gb                   = 0 -> (known after apply)\n      + child_storage_group      = (known after apply)\n      ~ compression              = true -> (known after apply)\n      + compression_ratio        = (known after apply)\n      ~ compression_ratio_to_one = 0 -> (known after apply)\n      + device_emulation         = (known after apply)\n      ~ id                       = \"tf-sg2\" -> (known after apply)\n      + maskingview              = (known after apply)\n      ~ name                     = \"tf-sg2\" -> \"tf-sg3\"\n      ~ num_of_child_sgs         = 0 -> (known after apply)\n      ~ num_of_masking_views     = 0 -> (known after apply)\n      ~ num_of_parent_sgs        = 0 -> (known after apply)\n      + num_of_snapshot_policies = (known after apply)\n      ~ num_of_snapshots         = 0 -> (known after apply)\n      ~ num_of_vols              = 0 -> (known after apply)\n      + parent_storage_group     = (known after apply)\n      + service_level            = (known after apply)\n      ~ slo                      = \"NONE\" -> (known after apply)\n      ~ slo_compliance           = \"NONE\" -> (known after apply)\n      + snapshot_policies        = (known after apply)\n      + tags                     = (known after apply)\n      ~ type                     = \"Standalone\" -> (known after apply)\n      ~ unprotected              = true -> (known after apply)\n      ~ unreducible_data_gb      = 0 -> (known after apply)\n      + uuid                     = (known after apply)\n      ~ volume_ids               = [] -> (known after apply)\n      + vp_saved_percent         = (known after apply)\n      + workload                 = (known after apply)\n        # (2 unchanged attributes hidden)\n    }\n\nPlan: 0 to add, 1 to change, 0 to destroy.\n\n─────────────────────────────────────────────────────────────────────────────\n\nSaved the plan to: /tmp/tmp6f2o83k5.tfplan\n\nTo perform exactly these actions, run the following command to apply:\n    terraform apply \"/tmp/tmp6f2o83k5.tfplan\"\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Refreshing state... [id=tf-sg2]",
            "",
            "Terraform used the selected providers to generate the following execution",
            "plan. Resource actions are indicated with the following symbols:",
            "  ~ update in-place",
            "",
            "Terraform will perform the following actions:",
            "",
            "  # powermax_storagegroup.tf_sg will be updated in-place",
            "  ~ resource \"powermax_storagegroup\" \"tf_sg\" {",
            "      ~ cap_gb                   = 0 -> (known after apply)",
            "      + child_storage_group      = (known after apply)",
            "      ~ compression              = true -> (known after apply)",
            "      + compression_ratio        = (known after apply)",
            "      ~ compression_ratio_to_one = 0 -> (known after apply)",
            "      + device_emulation         = (known after apply)",
            "      ~ id                       = \"tf-sg2\" -> (known after apply)",
            "      + maskingview              = (known after apply)",
            "      ~ name                     = \"tf-sg2\" -> \"tf-sg3\"",
            "      ~ num_of_child_sgs         = 0 -> (known after apply)",
            "      ~ num_of_masking_views     = 0 -> (known after apply)",
            "      ~ num_of_parent_sgs        = 0 -> (known after apply)",
            "      + num_of_snapshot_policies = (known after apply)",
            "      ~ num_of_snapshots         = 0 -> (known after apply)",
            "      ~ num_of_vols              = 0 -> (known after apply)",
            "      + parent_storage_group     = (known after apply)",
            "      + service_level            = (known after apply)",
            "      ~ slo                      = \"NONE\" -> (known after apply)",
            "      ~ slo_compliance           = \"NONE\" -> (known after apply)",
            "      + snapshot_policies        = (known after apply)",
            "      + tags                     = (known after apply)",
            "      ~ type                     = \"Standalone\" -> (known after apply)",
            "      ~ unprotected              = true -> (known after apply)",
            "      ~ unreducible_data_gb      = 0 -> (known after apply)",
            "      + uuid                     = (known after apply)",
            "      ~ volume_ids               = [] -> (known after apply)",
            "      + vp_saved_percent         = (known after apply)",
            "      + workload                 = (known after apply)",
            "        # (2 unchanged attributes hidden)",
            "    }",
            "",
            "Plan: 0 to add, 1 to change, 0 to destroy.",
            "",
            "─────────────────────────────────────────────────────────────────────────────",
            "",
            "Saved the plan to: /tmp/tmp6f2o83k5.tfplan",
            "",
            "To perform exactly these actions, run the following command to apply:",
            "    terraform apply \"/tmp/tmp6f2o83k5.tfplan\""
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    ignored=0

    No changes are made and the Ansible module has captured the details of the changes that will be made if the plan is applied.

    Terraform destroy

    Finally we need to learn how to use “terraform destroy”. You might have guessed it by now. The native way of destroying something in Ansible is by setting the state to “absent”.

    ---
- hosts: localhost
  gather_facts: no

  vars:
    project_dir: "/root/powermax" # dir that contains main.tf, ...

  tasks:
    - name: Run terraform
      community.general.terraform:
        project_path: '{{ project_dir }}'
        state: absent
        variables:
          sg_name: "{{ sg_name }}"
          password: "{{ password }}"
      register: output

    - debug:
        var: output

    Let’s run it and see what it does.

    root:~# ansible-playbook runtf.yml --extra-vars "sg_name=tf-sg2 password=TFp2ssw0rd"

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
changed: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": true,
        "command": "/usr/bin/terraform destroy -no-color -auto-approve -lock=true -var sg_name=tf-sg2 -var password=smc",
        "failed": false,
        "outputs": {},
        "state": "absent",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Refreshing state... [id=tf-sg2]\n\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  - destroy\n\nTerraform will perform the following actions:\n\n  # powermax_storagegroup.tf_sg will be destroyed\n  - resource \"powermax_storagegroup\" \"tf_sg\" {\n      - cap_gb                   = 0 -> null\n      - compression              = true -> null\n      - compression_ratio_to_one = 0 -> null\n      - host_io_limit            = {} -> null\n      - id                       = \"tf-sg2\" -> null\n      - name                     = \"tf-sg2\" -> null\n      - num_of_child_sgs         = 0 -> null\n      - num_of_masking_views     = 0 -> null\n      - num_of_parent_sgs        = 0 -> null\n      - num_of_snapshots         = 0 -> null\n      - num_of_vols              = 0 -> null\n      - slo                      = \"NONE\" -> null\n      - slo_compliance           = \"NONE\" -> null\n      - srp_id                   = \"SRP_1\" -> null\n      - type                     = \"Standalone\" -> null\n      - unprotected              = true -> null\n      - unreducible_data_gb      = 0 -> null\n      - volume_ids               = [] -> null\n    }\n\nPlan: 0 to add, 0 to change, 1 to destroy.\npowermax_storagegroup.tf_sg: Destroying... [id=tf-sg2]\npowermax_storagegroup.tf_sg: Destruction complete after 0s\n\nDestroy complete! Resources: 1 destroyed.\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Refreshing state... [id=tf-sg2]",
            "",
            "Terraform used the selected providers to generate the following execution",
            "plan. Resource actions are indicated with the following symbols:",
            "  - destroy",
            "",
            "Terraform will perform the following actions:",
            "",
            "  # powermax_storagegroup.tf_sg will be destroyed",
            "  - resource \"powermax_storagegroup\" \"tf_sg\" {",
            "      - cap_gb                   = 0 -> null",
            "      - compression              = true -> null",
            "      - compression_ratio_to_one = 0 -> null",
            "      - host_io_limit            = {} -> null",
            "      - id                       = \"tf-sg2\" -> null",
            "      - name                     = \"tf-sg2\" -> null",
            "      - num_of_child_sgs         = 0 -> null",
            "      - num_of_masking_views     = 0 -> null",
            "      - num_of_parent_sgs        = 0 -> null",
            "      - num_of_snapshots         = 0 -> null",
            "      - num_of_vols              = 0 -> null",
            "      - slo                      = \"NONE\" -> null",
            "      - slo_compliance           = \"NONE\" -> null",
            "      - srp_id                   = \"SRP_1\" -> null",
            "      - type                     = \"Standalone\" -> null",
            "      - unprotected              = true -> null",
            "      - unreducible_data_gb      = 0 -> null",
            "      - volume_ids               = [] -> null",
            "    }",
            "",
            "Plan: 0 to add, 0 to change, 1 to destroy.",
            "powermax_storagegroup.tf_sg: Destroying... [id=tf-sg2]",
            "powermax_storagegroup.tf_sg: Destruction complete after 0s",
            "",
            "Destroy complete! Resources: 1 destroyed."
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost  : ok=2    changed=1    unreachable=0    failed=0    skipped=0     ignored=0

    The storage group resource has been destroyed and that shows in Ansible as a task with a status of “changed”. The “stdout” and “stdout_lines” parameters provide a record of what was destroyed.

    In summary, in this article we have seen how to use the terraform Ansible module to plan, apply and destroy Terraform projects. In doing that, organizations can leverage a tool like AWX or RedHat AAP to provide RBAC, strong credential management and a powerful upstream REST API and integrate with other tools like ITSM etc. AWX and AAP can also became a polyglot automation platform capable of running the two most popular automation tools out there to satisfy both traditional infrastructure teams and developers.

  • Ansible Dynamic Inventory Tutorial

    This is the first of a 3 part series on Ansible dynamic inventories. I was looking for some information about dynamic inventories as part of a project and I realized that even though there are a few articles out there, there is little detailed content specially around building your own dynamic inventory script and how to use it in Tower/AWX. So I am planning to share all the lessons I learned to help you in your own journey. This series is structured as follows:

    • Introduction to dynamic inventory
    • Develop your own dynamic inventory
    • Dynamic inventory in AWX/AAP

    The code for the examples shown in this series is written in Python and is available in this GitHub repo: https://github.com/cermegno/ansible-dynamic-inventory. And now, without further ado let’s start with the introduction.

    Static Inventory 1 min recap

    If you are reading this article more than likely you are not new to Ansible and you know what a static inventory is, but we need to spend a minute laying out the example we will use in this tutorial. Inventories can be formatted in several ways but INI is perhaps the most common. In an inventory you can define groups and children groups. The following is an example that shows 2 groups: “webprod” and “webdev”, each one with 2 target systems on it.

    [webprod]
web1 http_port=123
web2 http_port=456

[webdev]
dev1
dev2

[webdev:vars]
user=admin
pass=password

    You also have the ability to define variables for a group and for individual hosts. In this example we have defined a host variable for each host in the “webprod” group. However, the 2 variables we need for the “webdev” group are the same for all systems in that group so it makes sense to use group variables as shown.

    Another way you can format inventory files is JSON, and as we will see next, the JSON format is specially useful when creating dynamic inventories

    Understanding Dynamic Inventories

    Static files like these are the most common type of Ansible inventories and as long as the information contained in them doesn’t change often this is the way to go. However there are many environments that are very dynamic in nature and a static inventory file like this becomes obsolete very quickly and it is hard to maintain. So the solution is to use a dynamic inventory.

    Another consideration is that sometimes the targets themselves don’t change much but the variables (for hosts or groups) do. The use case we will play with in the second part of this tutorial is a great example of this.

    So, what is a dynamic inventory? It is a script that returns the inventory information in a specific format that Ansible expects. What format is that? JSON. The following is the JSON equivalent of the static inventory we used earlier

    {
    "webprod": {
        "hosts": [
            "web1",
            "web2"
        ]
    },
    "webdev": {
        "hosts": [
            "dev1",
            "dev2"
        ],
        "vars": {
            "user": "admin",
            "pass": "password"
        }
    },
    "_meta": {
        "hostvars": {
            "web1": {
                "http_port": "123"
            },
            "web2": {
                "http_port": "456"
            }
        }
    }
}

    Notice how there is a top level key for each group. Each of these “group” keys can have the following keys:

    • hosts. This key contains a list of targets
    • vars. This is a dictionary that contains the variables for the group. The “webprod” didn’t have any group variables, so the “vars” key can be omitted
    • children. This key contains a list of children groups of this group. If it is not present Ansible will assume the group doesn’t have any children groups

    Also notice how there is another top level key called “_meta” which includes the host variables under a key called “hostvars”.

    Ansible expects the “dynamic inventory” script to implement 2 flags. Only one of these will be invoked at once:

    • “–list”. When the script is run with this flag it has to return the entire JSON structure as shown above
    • “–host”. This flag is followed by the name of a specific host in the inventory and when used your script has to return the variables for that host.

    IMPORTANT: In the past the “–host” flag was the only way of getting host variables but it is very inefficient to do things when Ansible needs to get the “hostvars” from many hosts, one at a time. So in modern versions of Ansible, the preferred way of implementing this functionality with the “–list” flag is by using the “_meta” key shown above. Also, to ensure Ansible pays attention only to the “_meta” section, the “–host” flag needs to be functional but needs to return only an empty dictionary.

    NOTE: If your preference is to use the “–host” flag instead your “_meta” still needs to include an empty “hostvars” dictionary as described in the documentation.

    Using the ansible-inventory tool

    Ansible provides a very handy tool to display and troubleshoot the inventory. The tool is called “ansible-inventory”. We can use this tool to observe how Ansible sees the inventory. You can see the full help page of the command by typing “ansible-inventory -h” but let’s play with some of the options here. Firstly we have “–list” that shows the full inventory as Ansible sees it.

    
[root@alb-ansible3]# ansible-inventory -i staticinv.ini --list
{
    "_meta": {
        "hostvars": {
            "dev1": {
                "pass": "password",
                "user": "admin"
            },
            "dev2": {
                "pass": "password",
                "user": "admin"
            },
            "web1": {
                "http_port": 123
            },
            "web2": {
                "http_port": 456
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "webdev",
            "webprod"
        ]
    },
    "webdev": {
        "hosts": [
            "dev1",
            "dev2"
        ]
    },
    "webprod": {
        "hosts": [
            "web1",
            "web2"
        ]
    }
}

    There are a few things to note from the output above:

    • You need to specify the inventory file with the “-i” option. It doesn’t matter whether it is a script or a static file, you need to use the “-i” option. In this case I am using the INI inventory file we showed first. I have called it “staticinv.ini”
    • Notice how Ansible has taken the group variables of the “webdev” group and has turned them into host variables under the “_meta” section. So for us it is more efficient to use group variables but internally Ansible unfolds those into individual hosts variables. Consequently, the group keys show only the list of hosts in the group
    • Ansible has created the “all” group and has add the other groups as “children” of “all”. It has also added the “ungrouped” group. Now we understand how Ansible handles your plays when you specify “hosts: all”

    So the “–list” option helps us understand many things. But it would be good if it could provide the optimal JSON version of a static INI inventory file that we can use as a template to write the code of our dynamic inventory script. Luckily for us, that option is also available. It is just a matter of adding the “–export” flag to the previous command. Please note how this is in addition to “–list” not as a replacement.

    [root@alb-ansible3]# ansible-inventory -i staticinv.ini --list --export
{
    "_meta": {
        "hostvars": {
            "web1": {
                "http_port": 123
            },
            "web2": {
                "http_port": 456
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "webdev",
            "webprod"
        ]
    },
    "webdev": {
        "hosts": [
            "dev1",
            "dev2"
        ],
        "vars": {
            "pass": "password",
            "user": "admin"
        }
    },
    "webprod": {
        "hosts": [
            "web1",
            "web2"
        ]
    }
}

    Now the variables “user” and “pass” have been turned into group variables of the “webdev”. So that’s it, if we want to create a dynamic inventory script we can start by creating a sample INI version of the inventory and use the “–export” flag to create a JSON equivalent. We then write our code to produce that on demand

    Another handy option available in the “ansible-inventory” tool is “–graph”. When invoked it provides a graph representation of the inventory. Group names are prepended with “@”. Hosts in the group are shown indented under the group name

    [root@alb-ansible3]# ansible-inventory -i staticinv.ini --graph
@all:
  |--@ungrouped:
  |--@webdev:
  |  |--dev1
  |  |--dev2
  |--@webprod:
  |  |--web1
  |  |--web2

    Built-in inventory plugins

    Ansible comes with some built-in inventory plugins to help you with some of the most common use cases. You can see what inventory plugins were installed by using the “ansible-doc” command as follows. My version of Ansible is 2.9.21 and this is what it came by default

    [root@alb-ansible3]# ansible-doc -t inventory -l
advanced_host_list  Parses a 'host list' with ranges
auto                Loads and executes an inventory plugin specified in a YAML config
aws_ec2             EC2 inventory source
aws_rds             rds instance source
azure_rm            Azure Resource Manager inventory plugin
cloudscale          cloudscale.ch inventory source
constructed         Uses Jinja2 to construct vars and groups based on existing inventory
docker_machine      Docker Machine inventory source
docker_swarm        Ansible dynamic inventory plugin for Docker swarm nodes
foreman             foreman inventory source
gcp_compute         Google Cloud Compute Engine inventory source
generator           Uses Jinja2 to construct hosts and groups from patterns
gitlab_runners      Ansible dynamic inventory plugin for GitLab runners
hcloud              Ansible dynamic inventory plugin for the Hetzner Cloud
host_list           Parses a 'host list' string
ini                 Uses an Ansible INI file as inventory source
k8s                 Kubernetes (K8s) inventory source
kubevirt            KubeVirt inventory source
linode              Ansible dynamic inventory plugin for Linode
netbox              NetBox inventory source
nmap                Uses nmap to find hosts to target
online              Online inventory source
openshift           OpenShift inventory source
openstack           OpenStack inventory source
scaleway            Scaleway inventory source
script              Executes an inventory script that returns JSON
toml                Uses a specific TOML file as an inventory source
tower               Ansible dynamic inventory plugin for Ansible Tower
virtualbox          virtualbox inventory source
vmware_vm_inventory VMware Guest inventory source
vultr               Vultr inventory source
yaml                Uses a specific YAML file as an inventory source

    Create your first dynamic inventory

    So far we know that a dynamic inventory script needs to return a well-known JSON structure. So how do you obtain the information to populate the JSON? It depends on your use case. Ultimately, you will have a source of truth for your inventory information and you will have to interact with that source programmatically. For example, if the inventory information is on a database you will have to write a script that queries the database to find out what to put in the JSON output. Other examples could be Excel or even a text file. Nowadays it is very common to deal with systems that expose a REST API interface and all programming languages have web client libraries that allow you to talk to such API’s. In the next post we will show you an example of how to create a dynamic inventory for your Dell infrastructure by using CloudIQ’s REST API

    To wrap up this post let’s show how to create the most basic dynamic inventory script how to use it . For simplicity we are going to drop the “webdev” group and create a script that returns the details of the “webprod” group only, including the “_meta” with their host variables. In my system I have created a python script called “basicinv.py” that looks as follows:

    [root@alb-ansible3]# cat basicinv.py
#!/usr/bin/env python3
import json
output = {
    "_meta": {
        "hostvars": {
            "web1": {
                "http_port": 123
            },
            "web2": {
                "http_port": 456
            }
        }
    },
    "webprod": {
        "hosts": [
            "web1",
            "web2"
        ]
    }
}
print(json.dumps(output))

    It is creating a dictionary with the static information and using the “json.dumps” function to dump it on the terminal. Notice how for simplicity I am not checking for “–list” or “–host” flag. So the script dumps everything every time which is the default behavior of “–list”. Hence running tasks for “all” hosts will fine but not for specific hosts. The objective was to create the script as simple as possible so we will leave the “–host” flag out in this post. In the next post of this series we will use the “argparse” library to implement both flags properly.

    [root@alb-ansible3]# ansible all -i basic.py -m debug -a "var=http_port"
web2 | SUCCESS => {
    "http_port": 456
}
web1 | SUCCESS => {
    "http_port": 123
}
[root@alb-ansible3]# ansible web1 -i basic.py -m debug -a "var=http_port"
[WARNING]: Unable to parse /root/basic.py as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

    The previous screenshot shows the ad-hoc “debug” running successfully against “all” hosts but not against host “web1” as expected as this would require a functional “–host” flag as explained earlier. Notice how we use the “-i” option to specify the script.

    Now let’s see if it works with a playbook too. I have created the following “ping.yml” playbook

    - name: Check that our targets are reachable
  hosts: webprod
  gather_facts: false

  tasks:
  - ping:

    Now we run it like this. Notice above how we are targeting the “webprod” group. Using “all” works as well. If we try with an individual host like “web1” it also works, but this is only because our simple “ping.yml” doesn’t require Ansible to look for the “hostvars”. In that case we would need to make our simple dynamic inventory script award of the “–host” flag and return an empty dictionary as we discussed earlier

    [root@alb-ansible3]# ansible-playbook -i basicinv.py ping.yml

PLAY [Check that our targets are reachable] *********************************************************************

TASK [ping] *******************************************************************************
ok: [web2]
ok: [web1]

PLAY RECAP *******************************************************************************
web1 : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0
web2 : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0

    Things to watch out for – Possible errors

    There are 2 important things to watch out for when you create your own script:

    • Make sure your script has execute permission, otherwise you will get the error “Unable to parse /root/basic.py as an inventory source”
    • Make sure you add the shebang that matches your Pyhon version, which in my case is “!/usr/bin/env python3“. Otherwise you get the error “Failed to parse with script plugin” below. Notice how it tries to parse it as a script first and then it tries to parse it as an INI before it gives up
    [root@alb-ansible3]# ansible-playbook -i basic.py ping.yml
[WARNING]:  * Failed to parse /root/basic.py with script plugin: problem running
/root/basic.py --list ([Errno 8] Exec format error: '/root/basic.py')
[WARNING]:  * Failed to parse /root/basic.py with ini plugin:
/root/basic.py:1: Expected key=value host variable assignment, got: json
[WARNING]: Unable to parse /root/basic.py as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not
match 'all'

    I hope you found this introduction valuable. On the second part we will create the full-blown solution for a dynamic inventory script where we extract the data from a REST API at runtime. See you soon!

  • Grafana variables – Easy step-by-step tutorial

    This is a very important and powerful feature in Grafana. Unfortunately, when one searches in Google the top hits tend to be Grafana official documentation, which more often than not is very formal. There are many people out there that learn by seeing examples. That’s the purpose of the tutorial, to show a clear step-by-step tutorial that is easy to understand and follow. I am running Grafana v7.5.15. I am going to include a good amount of screenshots to make it as easy to follow as possible. No doubt the interface will change in the future (that’s the nature of software) but hopefully it won’t have changed much by the time you read it.

    Variables allow you to update your dashboard dynamically by selecting interactively what you want to see. For example, I have a dashboard here that provides information about storage arrays in a datacenter, such as capacity and performance metrics. The datacenter in question has multiple storage arrays of different types. By default the panels in the dashboard show information about “all” of the arrays. But often, I would like to focus only on arrays of a certain type. The way of enabling this is to have a drop-down menu that allows me select the array type. When I select one type all the other arrays are hidden. As we will see the behavior is configurable and allows us to enable multiple selections.

    Grafana shows a separate drop-down for each variable you configure in your dashboard. You can have multiple drop-down menus. When you make a selection in any of them, the changes are applied to all panels in the dashboard.

    The way you obtain the labels for the drop-down menu varies. In our example we will use the “Query” type which means we perform a query to the data source to extract them. The actual syntax depends on the data source. Why? Grafana uses the query language native to the data source and as you know Grafana can work with a wide range of data sources: InfluxDB, Prometheus, MySQL … . In this post I am going to focus on Prometheus which seems to be one of the most prevalent out there nowadays.

    Everything starts by creating the variable. For this you need to open the “Dashboard settings” by click the “gear” symbol.

    Once in “Dashboard settings” click on “Variables”. Let’s click “New” to create our first variable.

    The “General” section has some very important fields. We are going to use “Query” type but you get other interesting possibilities. Two other important fields:

    • Name: This field needs to match what you use in the “query” field in the panel configuration
    • Label: This is what gets displayed in the drop-down menu for users to select

    Since we selected “Query” in the “General” section, we get a “Query Options” section. Here we need to select our data source and the query that will extract the labels that will be shown in the drop-down menu. As we mentioned, the query is written in a specific way to the data source type. For example, for a MySQL query type we would need to write an actual SQL query, but for Prometheus we use “PromQL”, ie Prometheus Query Language. In the screenshot above you can see I am using “label_values(name)”. This query will retrieve all the labels that were defined for the metric. So it is very important that what you put here matches exactly the labels you used. For example, the following is the code in my exporter:

    STORAGE_FREE_PERCENT = Gauge('storage_free_percent','Percentage of free storage',['name','system_type'])

    Notice how ‘name’ is one of the two labels defined for the metric. There could be multiple ones, hence they are a “list” enclosed by square brackets.

    I you enable “sort”, the labels in the drop-down menu will be sorted alphabetically. Selecting “multi-value” will allow you to select multiple labels at the same time. The “Include All option” will also show an “All” label that selects all labels. If all goes well, at the bottom of the “Edit” page you can see a “preview of values”. It helps you see if your query is working.

    Once you are finished configuring the variable, click “Update”. This brings you back to the “Variables” top level menu. You will see a green mark if the variable is correctly being referenced by your dashboard or by other variables. In the screenshot below you can see I have created a variable “system_type” and another one for “name”

    Before we move on to the final step let’s see graphically how the settings.

    As you can see the final step is to build a query in your panel that includes the name of the metric and the reference to your variables in between curly brackets. The variable itself needs to be prepended with the “$” symbol.

  • Multi-cloud agility with governance in ServiceNow

    I have been talking a lot lately about using ServiceNow as a tool to deliver unified end-user experience and governance for multi-cloud architectures. Most organizations today have multiple clouds public and private. This hopefully helps provide the best suitable landing zone for every workload.

    However, the mainstream approach to consuming public cloud is to give developers a credit card and say “here you go”, which is not a good governance best practice. So it doesn’t come as a surprise when an outrageous bills come at the end of the month. The other big issue with this approach is that every cloud becomes a silo. On the positive side, developers get more productive and deliver value to the business faster.

    In videos like this one you can see how it is possible to deliver a unified multi-cloud experience across multiple public and private clouds. The example uses ServiceNow but it will work for other tools like it. The principle is to offer the same type of item with comparable T-shirt sizes from a single catalog in ServiceNow. You will notice in the video how the user can see what the same VM costs on each cloud. When you pair this with chargeback, it helps developers to make the right decision as to where to run a workload. You can also throw into the mix approvals based on the characteristics of the item (ex: is the storage capacity larger than a certain amount? does the VM require too much RAM or a GPU? does the VM cost more than $100 a month?) as well as CMDB integration for resources in all clouds, etc.

    The governance benefits are enormous but developers are not equally impressed. They like to use tools like Terraform or Ansible to provision their environments at the speed of light (almost) and they will argue that going to a GUI like a ServiceNow catalog makes them slower. And of course, when developers use “Infrastructure as Code” directly on the public cloud they bypass all the governance.

    By now, probably you can guess where I am heading: is there a way of reconciling both views of the world? Ultimately the key to delivering agility via automation is having an API … and ServiceNow has a great REST API. In particular, you can order items from a service catalog using the REST API, which means that developers can order catalog items using a tool like Terraform or Ansible but still be subjected to the guardrails that the governance policies have put in place

    With this approach:

    • the organization creates a unified governance and end-user experience for all public and private cloud resources
    • developers can use Infrastructure as Code tools and processes that make them agile
    • the same REST API provides programmatic access to resources in all clouds
    • approvals and other control processes remain in place and it is easier to keep costs under control

    If you are interested in the details of how to do this, you can visit my previous post about how to create requests from a ServiceNow catalog using Ansible and Postman (what about Terraform?)

    Let’s see an example of how it works in practice. This is our infrastructure catalog. It allows the user to consume infrastructure from private and public cloud. We are going to use a STaaS (Storage-as-a-Service) as an example because the approvals are very intuitive. This catalog item allows you to provision file storage to an existing virtual machine. This item includes a capacity-based approval. The idea is that any request for more than a certain threshold is going to require an approval. You could have additional thresholds that require 2 or potentially VP approval. You can see how this constraint is going to encourage reasonable consumption by the end users. No one likes to wait for anything these days. In this demo environment the threshold is set very low, to just 50GB.

    The objective is to use code to order storage for a VM and demonstrate that:

    • if the capacity requested is lower than the threshold the request goes straight through
    • if the capacity requested is above the threshold the approval process kicks in

    We are going to use Postman to do this demonstration. You can find the Postman collection I am using in this GitHub repo. The process has essentially 2 steps: add an item to the cart and finally submit the cart. Here you can see we have sent a REST API call to add the File STaaS to the cart

    URL needs the sys_id of the catalog (get it from URL) or programmatically (with this API call). The payload needs to include quantity and all the variables required. You can find out the variables required using this API call. The status code will be 200. At this stage you can see there is 1 item in the cart

    The second step then is to submit the cart. We can do so with this REST API call. Note how it is also a POST call but it doesn’t require a Body parameter. It feels intuitive that you might have to at least specify the cart ID, but this is not necessary because a user only has one cart.

    You can see request “REQ0011028” was created. If I navigate to “Requests” for the user we can see the status of the request. The approval process is triggered because we requested 60GB which is more than the threshold. This is good because we can use Infrastructure-as-Code tools to provision infrastructure while maintaining governance.

    A key ingredient to make this both agile and useful for the business is to be wise with the thresholds. Your organization should consider what threshold provides the 80/20 or even perhaps the 90/10 rule so that the majority of provisioning activity can happen without delays.

    To finish off let’s repeat the request but this time the capacity required will be 40GB which is under the threshold.

    This time, when we go to “Requests” we can see that the request is already complete. It didn’t need an approval.

    If I log in to the Virtual Machine straight away I can see that the storage has been created, mounted and configured. As you can see I can even create a file on it.

    Conclusion

    Many organizations are implementing private clouds to that deliver a self-service experience to end-users by implementing automation and exposing it to users via an ITSM tool like ServiceNow. This architecture can be extended to consume also public cloud resources in such a way that a consistent end-user experience is delivered regardless of where the resources are created. This also brings an opportunity to break down silos and create a unified governance framework for all clouds along with integrated CMDB and incident management. This is very important especially now that cost control has become paramount. On the other hand developers demand accessing resources via API so that they can iterate faster and deliver more value to the business. In this blog post we have demonstrated how developers can leverage the ServiceNow REST API to automate the provisioning of resources in any cloud at the speed they need without jeopardizing governance.

  • ServiceNow catalog request with Postman

    In this blog post we will explore how to order items from a ServiceNow catalog using the ServiceNow REST API. There are a few examples out there that use “REST API Explorer” inside ServiceNow. I will focus here on how to do it from an external tool. In particular I am going to provide instructions for Postman. This post is the stepping stone for this follow up post where I try to solve an interesting use case of providing multi-cloud with governance through ServiceNow. You can get the Postman collection I am using from this GitHub repo.

    This is the latest addition to a series of popular ServiceNow related posts:

    There is also a follow up post that uses what I am covering here to allow developers to provision multi-cloud IT resources while observing any governance that might have been implemented in ServiceNow.

    In the post about creating Incidents we covered the “REST API Explorer” tool in more detail. Please refer to that one if you are new to it. In this post we will use the “Service Catalog API“. In this screenshot you can see it is a very extensive API. It provides 47 API calls as of time of this writing.

    Postman

    Now let’s move to Postman. Feel free to use any other REST API developer tool you are used to. If you want to follow along you can retrieve my Postman collection from this GitHub repo. The Postman collection provides 3 folders. In this post we will use the “Catalog Request” folder. The other 2 folders contain the API calls that were used in 2 previous posts in this series mentioned above.

    Requesting items from catalogs can be done in two ways that mimic the behavior of the user in the GUI:

    • order now
    • add to cart and submit the cart

    Add to Cart and Submit order

    This is a 2-step process but it might be more convenient if you are planning to order multiple items in the same request.

    • add an item to the cart
    • submit the cart

    Adding an item to the cart is done with the following API call. Please replace {instance} with your particular ServiceNow instance.

    POST https://{instance}.service-now.com/api/sn_sc/servicecatalog/items/{sys_id}/add_to_cart

    There are 3 things you need to include:

    • “sys_id“. This is the “sys_id” of the item itself. As you can see it needs to be included in the URL
    • “sysparm_quantity“. This is the quantity of this item you want to add to the cart. It is included in the “Body” parameter and it is required
    • “variables“. This is also in the “Body” and needs to include all the information that you would normally provide through the GUI when creating the request. In my case this catalog entry allows me to request the installation of an application on a virtual machine. At a minimum you will need to include all variables that are defined as “mandatory” in your catalog. In my example below I am going to request the installation of an application on an existing virtual machine. You can see I am requesting the installation of Apache web server on a VM called “albtest3_01” which lives in my “sin” datacenter

    If all goes well, you should get a status code of 200 and the details of the cart will be provided in the response.

    You can find out the “sys_id” of the catalog item by browsing to the item itself in your catalog. It will be included in the URL itself as seen here

    Alternatively you can use other calls in the “Service Catalog API” to get it programmatically. One example could be executing this call and looking for the “name” of the item you want:

    GET https://{instance}.service-now.com/api/sn_sc/servicecatalog/items

    You can also find out programmatically what variables you need to include with the following API call. It returns information about every parameter the specific item defined by “sys_id”. Use the “name” fields as the keys in your “Body” parameter. Also pay attention to whether variables show as “mandatory: true“.

    GET https://{instance}.service-now.com/api/sn_sc/servicecatalog/items/{sys_id}/variables

    Notice how the response is providing even the valid “choices” for a variable that has been configured as a “select box”.

    At this point you have an item in the cart. You could potentially add other items to the cart but if you need things to be ordered in a certain sequence order 1 item at a time and orchestrate them all in your Infrastructure as Code tool of choice. The final step is to “submit the order”. This is done with the following API call

    POST https://{instance}.service-now.com/api/sn_sc/servicecatalog/cart/submit_order

    This API call is also part of the Postman collection you downloaded. As you can see in the screenshot below, even though this is a POST call, it doesn’t require a “Body” payload. If successful you should get a status code of 200 and the “Body” of the response will contain the “requests number”.

    Order now

    This is the most straight forward process as it consists of a single REST API call. It uses the “Buy Item (POST)” call. This is the call as shown in REST API Explorer.

    This call needs the same payload that we used to add an item to the cart, ie the quantity and the specific variables that are typically provided in the form in the GUI.

    As you can see, this API call returned a request number instead of cart details because the request was created in a single step. In the following screenshot we can see the request in the “Requests” application in the GUI.

    And in the details of the “Request Item” we can see that the information we specified in the body of the API call was used to create the request.

    Next steps

    Postman allows you to generate code in your language of choice, ex: Python, Golang, etc. You can use this to include what we have covered into a larger script that runs additional tasks.

    Alternatively, you might want to use configuration management or provisioning tools like Ansible or Terraform to order items from the ServiceNow catalog.

  • Grafana “invalid query: unexpected character”

    While working on a dashboard in Grafana I came across an error that puzzled me for a while. So I am writing this in the hope that it will help other people who value their time 🙂

    My setup is quite straight forward:

    • Prometheus v2.46.0
    • Grafana v7.5.15

    However, don’t get too hung up on the versions because I have seen other people complaining about the same behavior with other versions. The key is that the problem might lie with the browser. For completeness, I am using Google Chrome v116.0.5845.141.

    The symptoms are:

    • I edit a new panel in a dashboard and I start typing the Prometheus query in the “Metrics” box. When I put a metric only nothing bad happens, but as soon as I add a “curly bracket” ie “{” the Metrics box gets a life of its own. It starts duplicating letters and the cursors don’t seem to be responding. When using the cursors it feels like I need to use the cursor 3 times to advance a single character.
    • The panel gets a red exclamation box in the top-left corner. If I hover over the exclamation the following message pops up “invalid parameter \”query\”: 1:21: parse error: unexpected character: ‘\\ufeff’”

    This is what it looks like in my case

    Everything points to some strange characters being inserted

    If it is true that the problem is the browser, you might have to use a different browser. But as a workaround what I figured is that you can edit the Metric query in the panel’s JSON source directly.

    You can get there by either:

    • click on the “red exclamation” and then the “JSON” tab
    • Or click on the panel’s name drop-down arrow, then “Inspect” and then “JSON”

    At this point if you scroll down to “targets” you can see the “expr” field with all the garbage around it

    Now you can click on the “expr” line and edit the query cleanly without introducing those weird characters.

    IMPORTANT: One thing to watch out for is that you need to escape the double quotes in your query with a backslash. Otherwise they will get confused with the JSON’s own double quotes. This is what it looks like in my example:

    storage_free_percent{system_type=~\"$system_type\",name=~\"$name\"}

    I hope it helps!

  • Notifications in Ansible Tower, AAP and AWX

    We have published a few videos in the IaC Avengers YouTube channel about automation using Ansible and how this can be integrated into an ITSM tool like ServiceNow to provide a cloud-like experience for private infrastructure. In the last two videos we have even demonstrated how to treat ServiceNow as the single pane of glass to consume both private and public clouds. This approach provides a much needed unified governance and cost control in a multicloud environment.

    However, while showing the demos and having conversations with customers, I can see a question coming up more often. How do we cope with errors? It makes sense that this question is coming up now. We are taking the automation conversation out of the realm of the datacenter and elevating it all the way to the end-user in the ITSM world. This means “Enterprise” requirements, which in turn means less room for failure. Also, if we expose it to the end-user we are no longer talking about dozens of engineers, now we have potentially thousands of possible consumers.

    A sample architecture like shown in the videos is as follows:

    The requirements are:

    • let the user know that the workflow didn’t complete so that they are not sitting there waiting. Depending on the error they might want to retry
    • inform the engineers that a specific workflow is failing and they need to look into it

    This can and should be done both at the ServiceNow level and the Ansible level. In this post we are going to focus on the Ansible side of things. Most mature organizations use RedHat Ansible Automation Platform. I have also included the old name Ansible Tower because somehow is still stuck in people’s heads … it is certainly shorter and easier to pronounce. Of course this is also applicable to AWX, the community support edition

    From an Ansible syntax perspective you can do error handling with things like “blocks and rescue” or other techniques. However, our guiding principle here is not so much to make sure the playbook continues despite errors and ends gracefully. What we want in this case is to make sure that both the engineer and/or user gets notified. For this purpose I find the “Notifications” functionality does the job nicely. You can find “Notifications” on the left bar under the Administration menu. If you click the “Add” button you get a menu like this

    After providing a name you need to select the notification “Type”. Depending on your selection a number of relevant configuration options are shown. For example if email is selected it will ask for IP and port of the SMTP server and so on. Once you fill those details scroll to the very bottom and slide the “Customize messages” button. This will reveal the syntax of the notification messages. The tool supports sending notification on 7 different types of events including start, error, time out and even the outcome of an approval. Notice how the prepopulated messages use variables with the double curly bracket syntax.

    In my example I have created a notification to send emails to a Zimbra SMTP server we have in the lab. As you saw in the previous image is called “Zimbra email”. For testing purposes I have created a job template that runs a playbook called “wrong.yml”. This is single task playbook that uses the “uri” module to access a webpage. I have fed the task with an IP address that doesn’t exist, so the playbook will fail

    From the template we click in the “Notifications” tab. It will show you all the notifications you have configured. In my example “Zimbra email” is the only one. On the right side you will have the opportunity to enable any of the available notifications when the template starts, succeeds or fails. If you do the same thing for a “workflow template” it will show an additional slide button named “Approval”

    All is left to do is to run the template. When I run it fails as expected and I get an email in my Inbox with the following message. Notice how the body of the email maps to the syntax we saw in the “Customize messages” menu

    These messages could be sent to a group of engineers that look after the platform. I particularly like the fact that there is a “Webhook” type. This opens the possibility of sending a notification to a Teams channel which is a more popular choice than email these days. You can see in this previous post how to send notifications to a Teams channel. Additionally it would make sense to create an incident automatically in your ITSM tool. Some time ago we also published a tutorial to show you how to create incidents in ServiceNow programmatically.