{{ vExpose }}.Blog

Tag: automation

  • Integrating Generative AI in ITSM Workflows

    Integrating Generative AI in ITSM Workflows

    IT Service Management (ITSM) teams are increasingly turning to generative AI (GenAI) to streamline workflows, reduce resolution times, and improve user satisfaction. In this post, we’ll explore two architectures for integrating GenAI into platforms like ServiceNow: man-in-the-middle (human-in-the-loop) and agentic automation (autonomous AI agents).

    Why GenAI for ITSM?

    • Ticket overload: Teams handle hundreds of tickets daily, leading to burnout and delays.
    • Knowledge gaps: Agents struggle to find relevant solutions quickly.
    • Repetitive tasks: Manual triage and updates consume valuable time.

    GenAI can automate classification, suggest solutions, and even resolve incidents-but how you integrate it matters. Let’s compare two approaches.

    Architecture 1: Man-in-the-Middle (Human-in-the-Loop)

    In this approach, GenAI acts as an assistant to human agents. It analyzes tickets, suggests actions, and automates tasks but requires human approval before execution.

    Workflow Diagram:

    Use Cases:

    • High-risk scenarios (e.g., critical infrastructure changes).
    • Compliance-heavy environments (e.g., healthcare, finance).

    Implementation Steps:

    1. Set Up ServiceNow Outbound REST Integration (javascript)

    // ServiceNow Scripted REST API (Outbound)
    var request = new sn_ws.RESTMessageV2();
    request.setEndpoint('https://your-middleware.com/process-ticket');
    request.setHttpMethod('POST');
    request.setRequestBody(JSON.stringify(current));
    var response = request.execute();

    2. Build Middleware (Python/Flask Example)

    from flask import Flask, request, jsonify
    import openai

    app = Flask(__name__)
    openai.api_key = "your-api-key"

    @app.route('/process-ticket', methods=['POST'])
    def handle_ticket():
        ticket_data = request.json

        # GenAI analysis
        prompt = f"""
        Classify this ITSM ticket and suggest priority (Critical/High/Medium/Low):
        Title: {ticket_data['short_description']}
        Description: {ticket_data['description']}
        """

        response = openai.ChatCompletion.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}]
        )

        suggestion = response.choices[0].message['content']
        return jsonify({"suggestion": suggestion, "ticket_id": ticket_data['number']})

    3. Human Review & Action
    Agents review GenAI’s suggestions in a dashboard and approve/reject them.

    Architecture 2: GenAI Agentic Automation

    Here, GenAI agents act autonomously within guardrails. They analyze tickets, execute actions (e.g., resolving incidents, updating KBs), and only escalate exceptions to humans.

    Workflow Diagram:

    Use Cases:

    • Low-risk, repetitive tasks (e.g., password resets, FAQ responses).
    • High-volume environments needing 24/7 support.

    Implementation Steps:

    1. Autonomous Ticket Resolution with Python

    import requests
    import openai

    def resolve_ticket_automatically(ticket_id):
        # Fetch ticket from ServiceNow
        snow_url = f"https://instance.service-now.com/api/now/table/incident/{ticket_id}"
        headers = {"Accept": "application/json"}
        auth = ("admin", "password")
        response = requests.get(snow_url, headers=headers, auth=auth)
        ticket = response.json()['result']

        # GenAI analysis
        prompt = f"""
        Resolve this ticket autonomously if possible. Provide a solution and mark as closed.
        Ticket: {ticket['description']}
        """

        ai_response = openai.ChatCompletion.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}]
        )

        # Auto-resolve if confidence is high
        if "reset password" in ai_response.choices[0].message['content'].lower():
            update_data = {
                "state": "6",  # Resolved
                "close_notes": "Automatically resolved: Password reset instructions sent."
            }
            requests.patch(snow_url, json=update_data, headers=headers, auth=auth)

    2. Guardrails for Safety

    • Limit permissions (e.g., agents can’t modify user roles).
    • Log all actions for auditing.
    • Escalate tickets containing keywords like “outage” or “data breach” to humans.

    Key Considerations

    Combined Architecture Diagram:

    Best Practices for Both Architectures

    • Data Privacy: Mask PII/PHI in tickets before sending to GenAI APIs.
    • Feedback Loops: Let agents rate AI suggestions to improve models.
    • Tooling: Use frameworks like LangChain for complex workflows.

    Conclusion

    Whether you choose man-in-the-middle or agentic automation depends on your risk tolerance and use case. Start with a hybrid approach: use autonomous agents for simple tasks (e.g., FAQs) and human-in-the-loop for critical workflows. As trust in the system grows, expand automation cautiously.

    GenAI is set to redefine ITSM, propelling organizations from automation to true intelligence. The journey is just beginning-those who embrace this shift will not only optimize IT operations but also unlock new levels of agility, resilience, and innovation. The future of ITSM is not just faster or cheaper-it’s smarter, more adaptive, and profoundly more human.

  • Infrastructure as Code (IaC) Security Best Practices

    Infrastructure as Code (IaC) Security Best Practices

    Infrastructure as Code (IaC) has revolutionized how we provision and manage infrastructure, enabling speed, repeatability, and scalability. However, as with any code, IaC introduces new security considerations. Misconfigurations, exposed secrets, and lack of visibility can lead to significant risks in production environments.

    This guide explores actionable best practices for securing your Terraform, Ansible, and other IaC pipelines-helping you build robust, compliant, and resilient infrastructure. Sample code and how-to notes included!

    1. Treat IaC Like Application Code

    Just as you would with application code, store your IaC in version control systems (e.g., Git). This enables:

    • Change tracking: Who changed what, when, and why.
    • Peer reviews: Enforce code reviews and approvals before merging.
    • Rollback: Revert to previous known-good states if issues arise.

    Tip:
    Use branch protection rules and require pull request reviews for all changes.

    2. Secrets Management

    Never hard-code secrets or sensitive data (API keys, passwords, certificates) in your IaC files. Instead:

    • Use secret management tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.
    • Integrate secrets into your pipelines at runtime, not in code.
    • Leverage environment variables or encrypted files for sensitive values.

    Sample: Fetching Secrets from Vault in Terraform

    provider "vault" {
  address = "https://vault.example.com"
}

resource "vault_generic_secret" "example" {
  path = "secret/data/myapp"
}

output "db_password" {
  value = vault_generic_secret.example.data["password"]
}

    How-to:

    • Replace https://vault.example.com with your Vault server address.
    • The vault_generic_secret resource fetches secrets dynamically.
    • Output blocks can reference secrets securely-never hardcode them!

    Sample: Encrypting Secrets in Ansible with ansible-vault

    - hosts: all
  vars_files:
    - secrets.yml
  tasks:
    - name: Use secret password
      debug:
        msg: "The password is {{ secret_password }}"

    How-to:

    • Create and encrypt the secrets file: ansible-vault create secrets.yml
    • Reference the encrypted file in your playbook under vars_files.
    • Access secrets in tasks using Jinja2 templating, e.g., {{ secret_password }}.

    3. Automated Security Scanning

    Integrate security tools into your CI/CD pipeline to catch misconfigurations and vulnerabilities early:

    Sample: Terraform Security Scanning with tfsec in GitHub Actions

    name: Terraform Security Scan

on: [push]

jobs:
  tfsec:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run tfsec
        uses: aquasecurity/tfsec-action@v1
        with:
          tfsec_version: 'latest'

    How-to:

    • Add this workflow to your .github/workflows directory.
    • The scan will fail the build if critical security issues are found, helping you catch problems early.

    4. Principle of Least Privilege

    Ensure that your IaC tools, pipelines, and the resources they provision follow the principle of least privilege:

    • Limit IAM permissions for automation accounts.
    • Avoid using overly broad roles or root accounts.
    • Regularly audit permissions and remove unnecessary access.

    5. Policy as Code

    Define and enforce security and compliance policies programmatically:

    • Use tools like Open Policy Agent (OPA), Sentinel (for Terraform Cloud), or Conftest.
    • Enforce rules such as “no public S3 buckets,” “encryption enabled,” or “no hardcoded credentials.”

    Example:
    OPA policies can block deployments that violate security standards before they reach production.

    6. Continuous Monitoring and Drift Detection

    Even after deployment, infrastructure can drift from the desired state:

    • Use tools like Terraform Cloud, AWS Config, or Driftctl to detect and remediate drift.
    • Set up alerts for unauthorized changes and automate remediation where possible.

    7. Regular Reviews and Updates

    IaC modules and dependencies evolve-so should your security practices:

    • Schedule regular reviews of your IaC codebase and third-party modules.
    • Update modules to patch vulnerabilities and leverage new security features.
    • Document your security practices and train your team.

    Conclusion

    Securing Infrastructure as Code is not a one-time task, but a continuous process woven into your development and deployment lifecycle. By treating IaC like application code, managing secrets securely, automating scanning, enforcing least privilege, and using policy as code, you can drastically reduce risk and build trust in your automation.

  • Using Ansible to run Terraform plans

    Yes, your eyes are not deceiving you 🙂 Usually, when the words Ansible and Terraform appear in the same sentence is for the opposite scenario. Many articles out there provide examples of using Terraform to deploy infrastructure followed by running Ansible to configure it using the local-exec provisioner. This is a good thing because, if you need some post-provisioning configuration, it is better to rely on another declarative and idempotent tool like Ansible, rather than invoking shell scripts.

    But in this post we are not going to do any of that. What we are going to do is the other way around. We are going to have an Ansible playbook that runs your Terraform plan.

    This article is a follow up from a series of posts to introduce infrastructure admins and engineers to Terraform and to explain how to use it to manage on-prem physical infrastructure in a datacenter. You can find access the full series here.

    Reasons to do it

    But why? Well, if you have an physical infrastructure background and are running a traditional datacenter more than likely you have been playing with Ansible. Chances are you started with Ansible running in the command line and hard-coding your infrastructure’s credentials inside the playbook. Then you evolved into using an external credentials file and encrypting it using “ansible-vault”. Eventually you might have deployed Ansible AWX or even RedHat AAP. At this point, you are enjoying RBAC (who can run what playbook and on what infrastructure) and amazing (there is no other way to put it) credential management.

    Then, you talk to your developers over coffee and they tell you about Terraform. But when you get familiar with it, you discover that unless you pay for a SaaS offering you are stuck in the “command-line with hard-coded credentials” scenario. Wouldn’t it be great if you could use AWX to run Terraform? You could enjoy RBAC and credential management and all the other goodness that it provides. This is possible thanks to Ansible’s Terraform module.

    First steps

    Let’s say we have a very basic Terraform project to create a storage group in Dell PowerMax. The “main.tf” shown below starts the PowerMax provider and then creates a storage group “terraform_sg”. I don’t think it is strictly necessary, but if you are interested in learning more about provisioning storage in Dell PowerMax you can check out this previous post. For simplicity we haven’t created or referenced any variables. All the information is contained into a single file, the “main.tf” shown below.

    terraform {
  required_providers {
    powermax = {
      source = "dell/powermax"
    }
  }
}

provider "powermax" {
  username      = "smc"
  password      = "TFp2ssw0rd"
  endpoint      = "https://10.1.2.3:8443"
  serial_number = "000123456789"
  pmax_version  = 100
  insecure      = true
}

resource "powermax_storagegroup" "tf_sg" {
  name          = "terraform_sg"
  srp_id         = "SRP_1"
}

    In the same machine we have both Terraform and Ansible installed. In order to run the above Terraform plan with Ansible, we need at a minimum a playbook that looks like follows. It needs to include two mandatory parameters: “project_path” and “state”. The first parameter specifies the directory that contains the plan, ie the “main.tf” and typically any other files like “variables.tf”. My Ansible playbook is called “runtf.yml”.

    ---
- hosts: localhost
  gather_facts: no

  vars:
    project_dir: "/root/powermax" # folder that contains main.tf, ...

  tasks:
    - name: Run terraform
      community.general.terraform:
        project_path: '{{ project_dir }}'
        state: present
      register: output

    - debug:
        var: output

    As you can see I have registered the output of the terraform task and then created a second task to print it out on the terminal so that we can get more details. All is left now is to run it using the “ansible-playbook” command

    root@alb-terraform:~# ansible-playbook runtf.yml

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
changed: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": true,
        "command": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmpjaojemdl.tfplan",
        "failed": false,
        "outputs": {},
        "state": "present",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Creating...\npowermax_storagegroup.tf_sg: Creation complete after 0s [id=terraform_sg]\n\nApply complete! Resources: 1 added, 0 changed, 0 destroyed.\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Creating...",
            "powermax_storagegroup.tf_sg: Creation complete after 0s [id=terraform_sg]",
            "",
            "Apply complete! Resources: 1 added, 0 changed, 0 destroyed."
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    ignored=0

    Ansible run Terraform, and Terraform managed to make the changes so as expected, Ansible reports the task as “changed”. In the “debug” task, notice the “command” parameter. This is how Ansible is invoking Terraform. This invocation includes the “-auto-approve” flag so that Terraform doesn’t look for confirmation. The “stdout” and stdout_lines” parameters show the output that has been captured from Terraform about the creation of the different resources .

    A very important feature of both Ansible and Terraform is idempotency, ie the ability to run the same code repeatedly without making any damage. Let’s run it again and observe what happens.

    root@alb-terraform:~# ansible-playbook runtf.yml

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
ok: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": false,
        "command": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmp718kkwbo.tfplan",
        "failed": false,
        "outputs": {},
        "state": "present",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Refreshing state... [id=terraform_sg]\n\nNo changes. Your infrastructure matches the configuration.\n\nTerraform has compared your real infrastructure against your configuration\nand found no differences, so no changes are needed.\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Refreshing state... [id=terraform_sg]",
            "",
            "No changes. Your infrastructure matches the configuration.",
            "",
            "Terraform has compared your real infrastructure against your configuration",
            "and found no differences, so no changes are needed."
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost : ok=2    changed=0    unreachable=0    failed=0    skipped=0     ignored=0

    As expected, Ansible is reporting “ok”, which means the current state matched the desired state and therefore no changes were made. It has also captured the relevant from Terraform stating the same thing.

    Using variables

    This is a good start but ideally we would like to have the ability to make the plan more flexible by using variables for information such as the storage group name in this case. We can modify the “main.tf” as follows. Notice how the “default” value of the variable is commented out. It doesn’t matter if it is commented or not because when we invoke Terraform with external variables we overwrite the default value defined in the variable blocks.

    terraform {
  required_providers {
    powermax = {
      source = "dell/powermax"
    }
  }
}

provider "powermax" {
  username      = "smc"
  password      = var.password
  endpoint      = "https://10.1.2.3:8443"
  serial_number = "000123456789"
  pmax_version  = 100
  insecure      = true
}

variable "sg_name" {
 description = "Name of volume to create"
 type        = string
 #default     = "terraform_sg"
}

variable "password" {
  type        = string
  description = "Stores the password of Unisphere."
  #default = ""
}

resource "powermax_storagegroup" "tf_sg" {
  name          = var.sg_name
  srp_id         = "SRP_1"
}

    Additionally, notice how in the “provider” block we have also referenced another variable for the “password”. For security reasons we might want to have, sensitive information like that, stored in a separate file and protect it. In this case we haven’t even declared a default value because we can leverage ansible-vault or AWX to encrypt it and to pass it to Terraform at run time.

    Now, the corresponding Ansible playbook needs to use the “variables” parameter. This parameter essentially converts each of the variables into a “-var” flag when invoking Terraform. 

    ---
- hosts: localhost
  gather_facts: no

  vars:
    project_dir: "/root/powerstore" # folder that contains main.tf, ...

  tasks:
    - name: Run terraform
      community.general.terraform:
        project_path: '{{ project_dir }}'
        state: present
        variables:
          sg_name: "{{ vol_name }}"
          password: "{{ password }}"
      register: output

    - debug:
        var: output

    Now, at runtime we are going to select a different storage group name. The storage group is created as expected and the relevant Terraform output is captured. 

    root:~# ansible-playbook runtf.yml --extra-vars "sg_name=tf-sg2 password=TFp2ssw0rd"

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
changed: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": true,
        "command": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmpimdwzdu1.tfplan",
        "failed": false,
        "outputs": {},
        "state": "present",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Creating...\npowermax_storagegroup.tf_sg: Creation complete after 5s [id=vol1]\n\nApply complete! Resources: 1 added, 0 changed, 0 destroyed.\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Creating...",
            "powermax_storagegroup.tf_sg: Creation complete after 5s [id=vol1]",
            "",
            "Apply complete! Resources: 1 added, 0 changed, 0 destroyed."
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    ignored=0

    Terraform plan

    We went straight to “terraform apply” but most Terraform workflows also include running the plan command first. This is done with Ansible by running the playbook in “check mode”. Check mode invokes “terraform plan” for us.

    root:~# ansible-playbook runtf.yml --check --extra-vars "sg_name=tf-sg3 password=TFpass"

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
ok: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": false,
        "command": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmp6f2o83k5.tfplan",
        "failed": false,
        "outputs": {},
        "state": "present",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Refreshing state... [id=tf-sg2]\n\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  ~ update in-place\n\nTerraform will perform the following actions:\n\n  # powermax_storagegroup.tf_sg will be updated in-place\n  ~ resource \"powermax_storagegroup\" \"tf_sg\" {\n      ~ cap_gb                   = 0 -> (known after apply)\n      + child_storage_group      = (known after apply)\n      ~ compression              = true -> (known after apply)\n      + compression_ratio        = (known after apply)\n      ~ compression_ratio_to_one = 0 -> (known after apply)\n      + device_emulation         = (known after apply)\n      ~ id                       = \"tf-sg2\" -> (known after apply)\n      + maskingview              = (known after apply)\n      ~ name                     = \"tf-sg2\" -> \"tf-sg3\"\n      ~ num_of_child_sgs         = 0 -> (known after apply)\n      ~ num_of_masking_views     = 0 -> (known after apply)\n      ~ num_of_parent_sgs        = 0 -> (known after apply)\n      + num_of_snapshot_policies = (known after apply)\n      ~ num_of_snapshots         = 0 -> (known after apply)\n      ~ num_of_vols              = 0 -> (known after apply)\n      + parent_storage_group     = (known after apply)\n      + service_level            = (known after apply)\n      ~ slo                      = \"NONE\" -> (known after apply)\n      ~ slo_compliance           = \"NONE\" -> (known after apply)\n      + snapshot_policies        = (known after apply)\n      + tags                     = (known after apply)\n      ~ type                     = \"Standalone\" -> (known after apply)\n      ~ unprotected              = true -> (known after apply)\n      ~ unreducible_data_gb      = 0 -> (known after apply)\n      + uuid                     = (known after apply)\n      ~ volume_ids               = [] -> (known after apply)\n      + vp_saved_percent         = (known after apply)\n      + workload                 = (known after apply)\n        # (2 unchanged attributes hidden)\n    }\n\nPlan: 0 to add, 1 to change, 0 to destroy.\n\n─────────────────────────────────────────────────────────────────────────────\n\nSaved the plan to: /tmp/tmp6f2o83k5.tfplan\n\nTo perform exactly these actions, run the following command to apply:\n    terraform apply \"/tmp/tmp6f2o83k5.tfplan\"\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Refreshing state... [id=tf-sg2]",
            "",
            "Terraform used the selected providers to generate the following execution",
            "plan. Resource actions are indicated with the following symbols:",
            "  ~ update in-place",
            "",
            "Terraform will perform the following actions:",
            "",
            "  # powermax_storagegroup.tf_sg will be updated in-place",
            "  ~ resource \"powermax_storagegroup\" \"tf_sg\" {",
            "      ~ cap_gb                   = 0 -> (known after apply)",
            "      + child_storage_group      = (known after apply)",
            "      ~ compression              = true -> (known after apply)",
            "      + compression_ratio        = (known after apply)",
            "      ~ compression_ratio_to_one = 0 -> (known after apply)",
            "      + device_emulation         = (known after apply)",
            "      ~ id                       = \"tf-sg2\" -> (known after apply)",
            "      + maskingview              = (known after apply)",
            "      ~ name                     = \"tf-sg2\" -> \"tf-sg3\"",
            "      ~ num_of_child_sgs         = 0 -> (known after apply)",
            "      ~ num_of_masking_views     = 0 -> (known after apply)",
            "      ~ num_of_parent_sgs        = 0 -> (known after apply)",
            "      + num_of_snapshot_policies = (known after apply)",
            "      ~ num_of_snapshots         = 0 -> (known after apply)",
            "      ~ num_of_vols              = 0 -> (known after apply)",
            "      + parent_storage_group     = (known after apply)",
            "      + service_level            = (known after apply)",
            "      ~ slo                      = \"NONE\" -> (known after apply)",
            "      ~ slo_compliance           = \"NONE\" -> (known after apply)",
            "      + snapshot_policies        = (known after apply)",
            "      + tags                     = (known after apply)",
            "      ~ type                     = \"Standalone\" -> (known after apply)",
            "      ~ unprotected              = true -> (known after apply)",
            "      ~ unreducible_data_gb      = 0 -> (known after apply)",
            "      + uuid                     = (known after apply)",
            "      ~ volume_ids               = [] -> (known after apply)",
            "      + vp_saved_percent         = (known after apply)",
            "      + workload                 = (known after apply)",
            "        # (2 unchanged attributes hidden)",
            "    }",
            "",
            "Plan: 0 to add, 1 to change, 0 to destroy.",
            "",
            "─────────────────────────────────────────────────────────────────────────────",
            "",
            "Saved the plan to: /tmp/tmp6f2o83k5.tfplan",
            "",
            "To perform exactly these actions, run the following command to apply:",
            "    terraform apply \"/tmp/tmp6f2o83k5.tfplan\""
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    ignored=0

    No changes are made and the Ansible module has captured the details of the changes that will be made if the plan is applied.

    Terraform destroy

    Finally we need to learn how to use “terraform destroy”. You might have guessed it by now. The native way of destroying something in Ansible is by setting the state to “absent”.

    ---
- hosts: localhost
  gather_facts: no

  vars:
    project_dir: "/root/powermax" # dir that contains main.tf, ...

  tasks:
    - name: Run terraform
      community.general.terraform:
        project_path: '{{ project_dir }}'
        state: absent
        variables:
          sg_name: "{{ sg_name }}"
          password: "{{ password }}"
      register: output

    - debug:
        var: output

    Let’s run it and see what it does.

    root:~# ansible-playbook runtf.yml --extra-vars "sg_name=tf-sg2 password=TFp2ssw0rd"

PLAY [localhost] ***************************************************************************************

TASK [Run terraform] ***************************************************************************************
changed: [localhost]

TASK [debug] ***************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": true,
        "command": "/usr/bin/terraform destroy -no-color -auto-approve -lock=true -var sg_name=tf-sg2 -var password=smc",
        "failed": false,
        "outputs": {},
        "state": "absent",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "powermax_storagegroup.tf_sg: Refreshing state... [id=tf-sg2]\n\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  - destroy\n\nTerraform will perform the following actions:\n\n  # powermax_storagegroup.tf_sg will be destroyed\n  - resource \"powermax_storagegroup\" \"tf_sg\" {\n      - cap_gb                   = 0 -> null\n      - compression              = true -> null\n      - compression_ratio_to_one = 0 -> null\n      - host_io_limit            = {} -> null\n      - id                       = \"tf-sg2\" -> null\n      - name                     = \"tf-sg2\" -> null\n      - num_of_child_sgs         = 0 -> null\n      - num_of_masking_views     = 0 -> null\n      - num_of_parent_sgs        = 0 -> null\n      - num_of_snapshots         = 0 -> null\n      - num_of_vols              = 0 -> null\n      - slo                      = \"NONE\" -> null\n      - slo_compliance           = \"NONE\" -> null\n      - srp_id                   = \"SRP_1\" -> null\n      - type                     = \"Standalone\" -> null\n      - unprotected              = true -> null\n      - unreducible_data_gb      = 0 -> null\n      - volume_ids               = [] -> null\n    }\n\nPlan: 0 to add, 0 to change, 1 to destroy.\npowermax_storagegroup.tf_sg: Destroying... [id=tf-sg2]\npowermax_storagegroup.tf_sg: Destruction complete after 0s\n\nDestroy complete! Resources: 1 destroyed.\n",
        "stdout_lines": [
            "powermax_storagegroup.tf_sg: Refreshing state... [id=tf-sg2]",
            "",
            "Terraform used the selected providers to generate the following execution",
            "plan. Resource actions are indicated with the following symbols:",
            "  - destroy",
            "",
            "Terraform will perform the following actions:",
            "",
            "  # powermax_storagegroup.tf_sg will be destroyed",
            "  - resource \"powermax_storagegroup\" \"tf_sg\" {",
            "      - cap_gb                   = 0 -> null",
            "      - compression              = true -> null",
            "      - compression_ratio_to_one = 0 -> null",
            "      - host_io_limit            = {} -> null",
            "      - id                       = \"tf-sg2\" -> null",
            "      - name                     = \"tf-sg2\" -> null",
            "      - num_of_child_sgs         = 0 -> null",
            "      - num_of_masking_views     = 0 -> null",
            "      - num_of_parent_sgs        = 0 -> null",
            "      - num_of_snapshots         = 0 -> null",
            "      - num_of_vols              = 0 -> null",
            "      - slo                      = \"NONE\" -> null",
            "      - slo_compliance           = \"NONE\" -> null",
            "      - srp_id                   = \"SRP_1\" -> null",
            "      - type                     = \"Standalone\" -> null",
            "      - unprotected              = true -> null",
            "      - unreducible_data_gb      = 0 -> null",
            "      - volume_ids               = [] -> null",
            "    }",
            "",
            "Plan: 0 to add, 0 to change, 1 to destroy.",
            "powermax_storagegroup.tf_sg: Destroying... [id=tf-sg2]",
            "powermax_storagegroup.tf_sg: Destruction complete after 0s",
            "",
            "Destroy complete! Resources: 1 destroyed."
        ],
        "workspace": "default"
    }
}

PLAY RECAP ***************************************************************************************
localhost  : ok=2    changed=1    unreachable=0    failed=0    skipped=0     ignored=0

    The storage group resource has been destroyed and that shows in Ansible as a task with a status of “changed”. The “stdout” and “stdout_lines” parameters provide a record of what was destroyed.

    In summary, in this article we have seen how to use the terraform Ansible module to plan, apply and destroy Terraform projects. In doing that, organizations can leverage a tool like AWX or RedHat AAP to provide RBAC, strong credential management and a powerful upstream REST API and integrate with other tools like ITSM etc. AWX and AAP can also became a polyglot automation platform capable of running the two most popular automation tools out there to satisfy both traditional infrastructure teams and developers.

  • Ansible Dynamic Inventory Tutorial

    This is the first of a 3 part series on Ansible dynamic inventories. I was looking for some information about dynamic inventories as part of a project and I realized that even though there are a few articles out there, there is little detailed content specially around building your own dynamic inventory script and how to use it in Tower/AWX. So I am planning to share all the lessons I learned to help you in your own journey. This series is structured as follows:

    • Introduction to dynamic inventory
    • Develop your own dynamic inventory
    • Dynamic inventory in AWX/AAP

    The code for the examples shown in this series is written in Python and is available in this GitHub repo: https://github.com/cermegno/ansible-dynamic-inventory. And now, without further ado let’s start with the introduction.

    Static Inventory 1 min recap

    If you are reading this article more than likely you are not new to Ansible and you know what a static inventory is, but we need to spend a minute laying out the example we will use in this tutorial. Inventories can be formatted in several ways but INI is perhaps the most common. In an inventory you can define groups and children groups. The following is an example that shows 2 groups: “webprod” and “webdev”, each one with 2 target systems on it.

    [webprod]
web1 http_port=123
web2 http_port=456

[webdev]
dev1
dev2

[webdev:vars]
user=admin
pass=password

    You also have the ability to define variables for a group and for individual hosts. In this example we have defined a host variable for each host in the “webprod” group. However, the 2 variables we need for the “webdev” group are the same for all systems in that group so it makes sense to use group variables as shown.

    Another way you can format inventory files is JSON, and as we will see next, the JSON format is specially useful when creating dynamic inventories

    Understanding Dynamic Inventories

    Static files like these are the most common type of Ansible inventories and as long as the information contained in them doesn’t change often this is the way to go. However there are many environments that are very dynamic in nature and a static inventory file like this becomes obsolete very quickly and it is hard to maintain. So the solution is to use a dynamic inventory.

    Another consideration is that sometimes the targets themselves don’t change much but the variables (for hosts or groups) do. The use case we will play with in the second part of this tutorial is a great example of this.

    So, what is a dynamic inventory? It is a script that returns the inventory information in a specific format that Ansible expects. What format is that? JSON. The following is the JSON equivalent of the static inventory we used earlier

    {
    "webprod": {
        "hosts": [
            "web1",
            "web2"
        ]
    },
    "webdev": {
        "hosts": [
            "dev1",
            "dev2"
        ],
        "vars": {
            "user": "admin",
            "pass": "password"
        }
    },
    "_meta": {
        "hostvars": {
            "web1": {
                "http_port": "123"
            },
            "web2": {
                "http_port": "456"
            }
        }
    }
}

    Notice how there is a top level key for each group. Each of these “group” keys can have the following keys:

    • hosts. This key contains a list of targets
    • vars. This is a dictionary that contains the variables for the group. The “webprod” didn’t have any group variables, so the “vars” key can be omitted
    • children. This key contains a list of children groups of this group. If it is not present Ansible will assume the group doesn’t have any children groups

    Also notice how there is another top level key called “_meta” which includes the host variables under a key called “hostvars”.

    Ansible expects the “dynamic inventory” script to implement 2 flags. Only one of these will be invoked at once:

    • “–list”. When the script is run with this flag it has to return the entire JSON structure as shown above
    • “–host”. This flag is followed by the name of a specific host in the inventory and when used your script has to return the variables for that host.

    IMPORTANT: In the past the “–host” flag was the only way of getting host variables but it is very inefficient to do things when Ansible needs to get the “hostvars” from many hosts, one at a time. So in modern versions of Ansible, the preferred way of implementing this functionality with the “–list” flag is by using the “_meta” key shown above. Also, to ensure Ansible pays attention only to the “_meta” section, the “–host” flag needs to be functional but needs to return only an empty dictionary.

    NOTE: If your preference is to use the “–host” flag instead your “_meta” still needs to include an empty “hostvars” dictionary as described in the documentation.

    Using the ansible-inventory tool

    Ansible provides a very handy tool to display and troubleshoot the inventory. The tool is called “ansible-inventory”. We can use this tool to observe how Ansible sees the inventory. You can see the full help page of the command by typing “ansible-inventory -h” but let’s play with some of the options here. Firstly we have “–list” that shows the full inventory as Ansible sees it.

    
[root@alb-ansible3]# ansible-inventory -i staticinv.ini --list
{
    "_meta": {
        "hostvars": {
            "dev1": {
                "pass": "password",
                "user": "admin"
            },
            "dev2": {
                "pass": "password",
                "user": "admin"
            },
            "web1": {
                "http_port": 123
            },
            "web2": {
                "http_port": 456
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "webdev",
            "webprod"
        ]
    },
    "webdev": {
        "hosts": [
            "dev1",
            "dev2"
        ]
    },
    "webprod": {
        "hosts": [
            "web1",
            "web2"
        ]
    }
}

    There are a few things to note from the output above:

    • You need to specify the inventory file with the “-i” option. It doesn’t matter whether it is a script or a static file, you need to use the “-i” option. In this case I am using the INI inventory file we showed first. I have called it “staticinv.ini”
    • Notice how Ansible has taken the group variables of the “webdev” group and has turned them into host variables under the “_meta” section. So for us it is more efficient to use group variables but internally Ansible unfolds those into individual hosts variables. Consequently, the group keys show only the list of hosts in the group
    • Ansible has created the “all” group and has add the other groups as “children” of “all”. It has also added the “ungrouped” group. Now we understand how Ansible handles your plays when you specify “hosts: all”

    So the “–list” option helps us understand many things. But it would be good if it could provide the optimal JSON version of a static INI inventory file that we can use as a template to write the code of our dynamic inventory script. Luckily for us, that option is also available. It is just a matter of adding the “–export” flag to the previous command. Please note how this is in addition to “–list” not as a replacement.

    [root@alb-ansible3]# ansible-inventory -i staticinv.ini --list --export
{
    "_meta": {
        "hostvars": {
            "web1": {
                "http_port": 123
            },
            "web2": {
                "http_port": 456
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "webdev",
            "webprod"
        ]
    },
    "webdev": {
        "hosts": [
            "dev1",
            "dev2"
        ],
        "vars": {
            "pass": "password",
            "user": "admin"
        }
    },
    "webprod": {
        "hosts": [
            "web1",
            "web2"
        ]
    }
}

    Now the variables “user” and “pass” have been turned into group variables of the “webdev”. So that’s it, if we want to create a dynamic inventory script we can start by creating a sample INI version of the inventory and use the “–export” flag to create a JSON equivalent. We then write our code to produce that on demand

    Another handy option available in the “ansible-inventory” tool is “–graph”. When invoked it provides a graph representation of the inventory. Group names are prepended with “@”. Hosts in the group are shown indented under the group name

    [root@alb-ansible3]# ansible-inventory -i staticinv.ini --graph
@all:
  |--@ungrouped:
  |--@webdev:
  |  |--dev1
  |  |--dev2
  |--@webprod:
  |  |--web1
  |  |--web2

    Built-in inventory plugins

    Ansible comes with some built-in inventory plugins to help you with some of the most common use cases. You can see what inventory plugins were installed by using the “ansible-doc” command as follows. My version of Ansible is 2.9.21 and this is what it came by default

    [root@alb-ansible3]# ansible-doc -t inventory -l
advanced_host_list  Parses a 'host list' with ranges
auto                Loads and executes an inventory plugin specified in a YAML config
aws_ec2             EC2 inventory source
aws_rds             rds instance source
azure_rm            Azure Resource Manager inventory plugin
cloudscale          cloudscale.ch inventory source
constructed         Uses Jinja2 to construct vars and groups based on existing inventory
docker_machine      Docker Machine inventory source
docker_swarm        Ansible dynamic inventory plugin for Docker swarm nodes
foreman             foreman inventory source
gcp_compute         Google Cloud Compute Engine inventory source
generator           Uses Jinja2 to construct hosts and groups from patterns
gitlab_runners      Ansible dynamic inventory plugin for GitLab runners
hcloud              Ansible dynamic inventory plugin for the Hetzner Cloud
host_list           Parses a 'host list' string
ini                 Uses an Ansible INI file as inventory source
k8s                 Kubernetes (K8s) inventory source
kubevirt            KubeVirt inventory source
linode              Ansible dynamic inventory plugin for Linode
netbox              NetBox inventory source
nmap                Uses nmap to find hosts to target
online              Online inventory source
openshift           OpenShift inventory source
openstack           OpenStack inventory source
scaleway            Scaleway inventory source
script              Executes an inventory script that returns JSON
toml                Uses a specific TOML file as an inventory source
tower               Ansible dynamic inventory plugin for Ansible Tower
virtualbox          virtualbox inventory source
vmware_vm_inventory VMware Guest inventory source
vultr               Vultr inventory source
yaml                Uses a specific YAML file as an inventory source

    Create your first dynamic inventory

    So far we know that a dynamic inventory script needs to return a well-known JSON structure. So how do you obtain the information to populate the JSON? It depends on your use case. Ultimately, you will have a source of truth for your inventory information and you will have to interact with that source programmatically. For example, if the inventory information is on a database you will have to write a script that queries the database to find out what to put in the JSON output. Other examples could be Excel or even a text file. Nowadays it is very common to deal with systems that expose a REST API interface and all programming languages have web client libraries that allow you to talk to such API’s. In the next post we will show you an example of how to create a dynamic inventory for your Dell infrastructure by using CloudIQ’s REST API

    To wrap up this post let’s show how to create the most basic dynamic inventory script how to use it . For simplicity we are going to drop the “webdev” group and create a script that returns the details of the “webprod” group only, including the “_meta” with their host variables. In my system I have created a python script called “basicinv.py” that looks as follows:

    [root@alb-ansible3]# cat basicinv.py
#!/usr/bin/env python3
import json
output = {
    "_meta": {
        "hostvars": {
            "web1": {
                "http_port": 123
            },
            "web2": {
                "http_port": 456
            }
        }
    },
    "webprod": {
        "hosts": [
            "web1",
            "web2"
        ]
    }
}
print(json.dumps(output))

    It is creating a dictionary with the static information and using the “json.dumps” function to dump it on the terminal. Notice how for simplicity I am not checking for “–list” or “–host” flag. So the script dumps everything every time which is the default behavior of “–list”. Hence running tasks for “all” hosts will fine but not for specific hosts. The objective was to create the script as simple as possible so we will leave the “–host” flag out in this post. In the next post of this series we will use the “argparse” library to implement both flags properly.

    [root@alb-ansible3]# ansible all -i basic.py -m debug -a "var=http_port"
web2 | SUCCESS => {
    "http_port": 456
}
web1 | SUCCESS => {
    "http_port": 123
}
[root@alb-ansible3]# ansible web1 -i basic.py -m debug -a "var=http_port"
[WARNING]: Unable to parse /root/basic.py as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

    The previous screenshot shows the ad-hoc “debug” running successfully against “all” hosts but not against host “web1” as expected as this would require a functional “–host” flag as explained earlier. Notice how we use the “-i” option to specify the script.

    Now let’s see if it works with a playbook too. I have created the following “ping.yml” playbook

    - name: Check that our targets are reachable
  hosts: webprod
  gather_facts: false

  tasks:
  - ping:

    Now we run it like this. Notice above how we are targeting the “webprod” group. Using “all” works as well. If we try with an individual host like “web1” it also works, but this is only because our simple “ping.yml” doesn’t require Ansible to look for the “hostvars”. In that case we would need to make our simple dynamic inventory script award of the “–host” flag and return an empty dictionary as we discussed earlier

    [root@alb-ansible3]# ansible-playbook -i basicinv.py ping.yml

PLAY [Check that our targets are reachable] *********************************************************************

TASK [ping] *******************************************************************************
ok: [web2]
ok: [web1]

PLAY RECAP *******************************************************************************
web1 : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0
web2 : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0

    Things to watch out for – Possible errors

    There are 2 important things to watch out for when you create your own script:

    • Make sure your script has execute permission, otherwise you will get the error “Unable to parse /root/basic.py as an inventory source”
    • Make sure you add the shebang that matches your Pyhon version, which in my case is “!/usr/bin/env python3“. Otherwise you get the error “Failed to parse with script plugin” below. Notice how it tries to parse it as a script first and then it tries to parse it as an INI before it gives up
    [root@alb-ansible3]# ansible-playbook -i basic.py ping.yml
[WARNING]:  * Failed to parse /root/basic.py with script plugin: problem running
/root/basic.py --list ([Errno 8] Exec format error: '/root/basic.py')
[WARNING]:  * Failed to parse /root/basic.py with ini plugin:
/root/basic.py:1: Expected key=value host variable assignment, got: json
[WARNING]: Unable to parse /root/basic.py as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not
match 'all'

    I hope you found this introduction valuable. On the second part we will create the full-blown solution for a dynamic inventory script where we extract the data from a REST API at runtime. See you soon!

  • Multi-cloud agility with governance in ServiceNow

    I have been talking a lot lately about using ServiceNow as a tool to deliver unified end-user experience and governance for multi-cloud architectures. Most organizations today have multiple clouds public and private. This hopefully helps provide the best suitable landing zone for every workload.

    However, the mainstream approach to consuming public cloud is to give developers a credit card and say “here you go”, which is not a good governance best practice. So it doesn’t come as a surprise when an outrageous bills come at the end of the month. The other big issue with this approach is that every cloud becomes a silo. On the positive side, developers get more productive and deliver value to the business faster.

    In videos like this one you can see how it is possible to deliver a unified multi-cloud experience across multiple public and private clouds. The example uses ServiceNow but it will work for other tools like it. The principle is to offer the same type of item with comparable T-shirt sizes from a single catalog in ServiceNow. You will notice in the video how the user can see what the same VM costs on each cloud. When you pair this with chargeback, it helps developers to make the right decision as to where to run a workload. You can also throw into the mix approvals based on the characteristics of the item (ex: is the storage capacity larger than a certain amount? does the VM require too much RAM or a GPU? does the VM cost more than $100 a month?) as well as CMDB integration for resources in all clouds, etc.

    The governance benefits are enormous but developers are not equally impressed. They like to use tools like Terraform or Ansible to provision their environments at the speed of light (almost) and they will argue that going to a GUI like a ServiceNow catalog makes them slower. And of course, when developers use “Infrastructure as Code” directly on the public cloud they bypass all the governance.

    By now, probably you can guess where I am heading: is there a way of reconciling both views of the world? Ultimately the key to delivering agility via automation is having an API … and ServiceNow has a great REST API. In particular, you can order items from a service catalog using the REST API, which means that developers can order catalog items using a tool like Terraform or Ansible but still be subjected to the guardrails that the governance policies have put in place

    With this approach:

    • the organization creates a unified governance and end-user experience for all public and private cloud resources
    • developers can use Infrastructure as Code tools and processes that make them agile
    • the same REST API provides programmatic access to resources in all clouds
    • approvals and other control processes remain in place and it is easier to keep costs under control

    If you are interested in the details of how to do this, you can visit my previous post about how to create requests from a ServiceNow catalog using Ansible and Postman (what about Terraform?)

    Let’s see an example of how it works in practice. This is our infrastructure catalog. It allows the user to consume infrastructure from private and public cloud. We are going to use a STaaS (Storage-as-a-Service) as an example because the approvals are very intuitive. This catalog item allows you to provision file storage to an existing virtual machine. This item includes a capacity-based approval. The idea is that any request for more than a certain threshold is going to require an approval. You could have additional thresholds that require 2 or potentially VP approval. You can see how this constraint is going to encourage reasonable consumption by the end users. No one likes to wait for anything these days. In this demo environment the threshold is set very low, to just 50GB.

    The objective is to use code to order storage for a VM and demonstrate that:

    • if the capacity requested is lower than the threshold the request goes straight through
    • if the capacity requested is above the threshold the approval process kicks in

    We are going to use Postman to do this demonstration. You can find the Postman collection I am using in this GitHub repo. The process has essentially 2 steps: add an item to the cart and finally submit the cart. Here you can see we have sent a REST API call to add the File STaaS to the cart

    URL needs the sys_id of the catalog (get it from URL) or programmatically (with this API call). The payload needs to include quantity and all the variables required. You can find out the variables required using this API call. The status code will be 200. At this stage you can see there is 1 item in the cart

    The second step then is to submit the cart. We can do so with this REST API call. Note how it is also a POST call but it doesn’t require a Body parameter. It feels intuitive that you might have to at least specify the cart ID, but this is not necessary because a user only has one cart.

    You can see request “REQ0011028” was created. If I navigate to “Requests” for the user we can see the status of the request. The approval process is triggered because we requested 60GB which is more than the threshold. This is good because we can use Infrastructure-as-Code tools to provision infrastructure while maintaining governance.

    A key ingredient to make this both agile and useful for the business is to be wise with the thresholds. Your organization should consider what threshold provides the 80/20 or even perhaps the 90/10 rule so that the majority of provisioning activity can happen without delays.

    To finish off let’s repeat the request but this time the capacity required will be 40GB which is under the threshold.

    This time, when we go to “Requests” we can see that the request is already complete. It didn’t need an approval.

    If I log in to the Virtual Machine straight away I can see that the storage has been created, mounted and configured. As you can see I can even create a file on it.

    Conclusion

    Many organizations are implementing private clouds to that deliver a self-service experience to end-users by implementing automation and exposing it to users via an ITSM tool like ServiceNow. This architecture can be extended to consume also public cloud resources in such a way that a consistent end-user experience is delivered regardless of where the resources are created. This also brings an opportunity to break down silos and create a unified governance framework for all clouds along with integrated CMDB and incident management. This is very important especially now that cost control has become paramount. On the other hand developers demand accessing resources via API so that they can iterate faster and deliver more value to the business. In this blog post we have demonstrated how developers can leverage the ServiceNow REST API to automate the provisioning of resources in any cloud at the speed they need without jeopardizing governance.

  • ServiceNow catalog request with Postman

    In this blog post we will explore how to order items from a ServiceNow catalog using the ServiceNow REST API. There are a few examples out there that use “REST API Explorer” inside ServiceNow. I will focus here on how to do it from an external tool. In particular I am going to provide instructions for Postman. This post is the stepping stone for this follow up post where I try to solve an interesting use case of providing multi-cloud with governance through ServiceNow. You can get the Postman collection I am using from this GitHub repo.

    This is the latest addition to a series of popular ServiceNow related posts:

    There is also a follow up post that uses what I am covering here to allow developers to provision multi-cloud IT resources while observing any governance that might have been implemented in ServiceNow.

    In the post about creating Incidents we covered the “REST API Explorer” tool in more detail. Please refer to that one if you are new to it. In this post we will use the “Service Catalog API“. In this screenshot you can see it is a very extensive API. It provides 47 API calls as of time of this writing.

    Postman

    Now let’s move to Postman. Feel free to use any other REST API developer tool you are used to. If you want to follow along you can retrieve my Postman collection from this GitHub repo. The Postman collection provides 3 folders. In this post we will use the “Catalog Request” folder. The other 2 folders contain the API calls that were used in 2 previous posts in this series mentioned above.

    Requesting items from catalogs can be done in two ways that mimic the behavior of the user in the GUI:

    • order now
    • add to cart and submit the cart

    Add to Cart and Submit order

    This is a 2-step process but it might be more convenient if you are planning to order multiple items in the same request.

    • add an item to the cart
    • submit the cart

    Adding an item to the cart is done with the following API call. Please replace {instance} with your particular ServiceNow instance.

    POST https://{instance}.service-now.com/api/sn_sc/servicecatalog/items/{sys_id}/add_to_cart

    There are 3 things you need to include:

    • “sys_id“. This is the “sys_id” of the item itself. As you can see it needs to be included in the URL
    • “sysparm_quantity“. This is the quantity of this item you want to add to the cart. It is included in the “Body” parameter and it is required
    • “variables“. This is also in the “Body” and needs to include all the information that you would normally provide through the GUI when creating the request. In my case this catalog entry allows me to request the installation of an application on a virtual machine. At a minimum you will need to include all variables that are defined as “mandatory” in your catalog. In my example below I am going to request the installation of an application on an existing virtual machine. You can see I am requesting the installation of Apache web server on a VM called “albtest3_01” which lives in my “sin” datacenter

    If all goes well, you should get a status code of 200 and the details of the cart will be provided in the response.

    You can find out the “sys_id” of the catalog item by browsing to the item itself in your catalog. It will be included in the URL itself as seen here

    Alternatively you can use other calls in the “Service Catalog API” to get it programmatically. One example could be executing this call and looking for the “name” of the item you want:

    GET https://{instance}.service-now.com/api/sn_sc/servicecatalog/items

    You can also find out programmatically what variables you need to include with the following API call. It returns information about every parameter the specific item defined by “sys_id”. Use the “name” fields as the keys in your “Body” parameter. Also pay attention to whether variables show as “mandatory: true“.

    GET https://{instance}.service-now.com/api/sn_sc/servicecatalog/items/{sys_id}/variables

    Notice how the response is providing even the valid “choices” for a variable that has been configured as a “select box”.

    At this point you have an item in the cart. You could potentially add other items to the cart but if you need things to be ordered in a certain sequence order 1 item at a time and orchestrate them all in your Infrastructure as Code tool of choice. The final step is to “submit the order”. This is done with the following API call

    POST https://{instance}.service-now.com/api/sn_sc/servicecatalog/cart/submit_order

    This API call is also part of the Postman collection you downloaded. As you can see in the screenshot below, even though this is a POST call, it doesn’t require a “Body” payload. If successful you should get a status code of 200 and the “Body” of the response will contain the “requests number”.

    Order now

    This is the most straight forward process as it consists of a single REST API call. It uses the “Buy Item (POST)” call. This is the call as shown in REST API Explorer.

    This call needs the same payload that we used to add an item to the cart, ie the quantity and the specific variables that are typically provided in the form in the GUI.

    As you can see, this API call returned a request number instead of cart details because the request was created in a single step. In the following screenshot we can see the request in the “Requests” application in the GUI.

    And in the details of the “Request Item” we can see that the information we specified in the body of the API call was used to create the request.

    Next steps

    Postman allows you to generate code in your language of choice, ex: Python, Golang, etc. You can use this to include what we have covered into a larger script that runs additional tasks.

    Alternatively, you might want to use configuration management or provisioning tools like Ansible or Terraform to order items from the ServiceNow catalog.

  • Notifications in Ansible Tower, AAP and AWX

    We have published a few videos in the IaC Avengers YouTube channel about automation using Ansible and how this can be integrated into an ITSM tool like ServiceNow to provide a cloud-like experience for private infrastructure. In the last two videos we have even demonstrated how to treat ServiceNow as the single pane of glass to consume both private and public clouds. This approach provides a much needed unified governance and cost control in a multicloud environment.

    However, while showing the demos and having conversations with customers, I can see a question coming up more often. How do we cope with errors? It makes sense that this question is coming up now. We are taking the automation conversation out of the realm of the datacenter and elevating it all the way to the end-user in the ITSM world. This means “Enterprise” requirements, which in turn means less room for failure. Also, if we expose it to the end-user we are no longer talking about dozens of engineers, now we have potentially thousands of possible consumers.

    A sample architecture like shown in the videos is as follows:

    The requirements are:

    • let the user know that the workflow didn’t complete so that they are not sitting there waiting. Depending on the error they might want to retry
    • inform the engineers that a specific workflow is failing and they need to look into it

    This can and should be done both at the ServiceNow level and the Ansible level. In this post we are going to focus on the Ansible side of things. Most mature organizations use RedHat Ansible Automation Platform. I have also included the old name Ansible Tower because somehow is still stuck in people’s heads … it is certainly shorter and easier to pronounce. Of course this is also applicable to AWX, the community support edition

    From an Ansible syntax perspective you can do error handling with things like “blocks and rescue” or other techniques. However, our guiding principle here is not so much to make sure the playbook continues despite errors and ends gracefully. What we want in this case is to make sure that both the engineer and/or user gets notified. For this purpose I find the “Notifications” functionality does the job nicely. You can find “Notifications” on the left bar under the Administration menu. If you click the “Add” button you get a menu like this

    After providing a name you need to select the notification “Type”. Depending on your selection a number of relevant configuration options are shown. For example if email is selected it will ask for IP and port of the SMTP server and so on. Once you fill those details scroll to the very bottom and slide the “Customize messages” button. This will reveal the syntax of the notification messages. The tool supports sending notification on 7 different types of events including start, error, time out and even the outcome of an approval. Notice how the prepopulated messages use variables with the double curly bracket syntax.

    In my example I have created a notification to send emails to a Zimbra SMTP server we have in the lab. As you saw in the previous image is called “Zimbra email”. For testing purposes I have created a job template that runs a playbook called “wrong.yml”. This is single task playbook that uses the “uri” module to access a webpage. I have fed the task with an IP address that doesn’t exist, so the playbook will fail

    From the template we click in the “Notifications” tab. It will show you all the notifications you have configured. In my example “Zimbra email” is the only one. On the right side you will have the opportunity to enable any of the available notifications when the template starts, succeeds or fails. If you do the same thing for a “workflow template” it will show an additional slide button named “Approval”

    All is left to do is to run the template. When I run it fails as expected and I get an email in my Inbox with the following message. Notice how the body of the email maps to the syntax we saw in the “Customize messages” menu

    These messages could be sent to a group of engineers that look after the platform. I particularly like the fact that there is a “Webhook” type. This opens the possibility of sending a notification to a Teams channel which is a more popular choice than email these days. You can see in this previous post how to send notifications to a Teams channel. Additionally it would make sense to create an incident automatically in your ITSM tool. Some time ago we also published a tutorial to show you how to create incidents in ServiceNow programmatically.

  • Fix GCP error Permission ‘storage.buckets.get’ denied

    This will be a quick one. I was recently experimenting with creating an S3 bucket GCP using Ansible and I came across this error:

    {
  "msg": "GCP returned error: {'error': {'code': 403, 'message': \ansible@vexpose.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).\, 'errors': [{'message': \ansible@vexpose.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).\, 'domain': 'global', 'reason': 'forbidden'}]}}",
  "invocation": {
    "module_args": {
      "name": "gcp_s3",
      "project": "vexpose",
      "auth_kind": "serviceaccount",
      "storage_class": "COLDLINE",
... <<< output truncated >>>

    After seeing “Permission Denied” I naturally started to look at the roles that were assigned to the account . I discovered later that the “Storage Admin” role provided already that permission, but in the process I wasted some precious time adding other roles that provided that permission yet again. So I felt compelled to write this quick post to help other people save their time.

    If this is happening to you the resolution could be quite simple. We must remember that GCP (like other public cloud providers) uses a single namespace for all customers. Therefore the bucket name must be “universally” unique. If it isn’t it takes it as you are trying to make changes to an existing bucket that another customer owns and it throws the misleading “permission denied” error. So, simply choose a more complex name and see if that fixes the error.

    You can quickly test if it is an issue with your name not being unique by trying to create a bucket using GCP web interface for example. It if is already taken you will receive a message like this.

  • Murphy’s Law – How to stall your Kubernetes enterprise rollout.

    Murphy’s Law – How to stall your Kubernetes enterprise rollout.

    In the world of software development, Murphy’s law holds an unassailable truth: Anything that can go wrong, will go wrong. As a proud member of this masochistic club, you might be looking for innovative ways to stall your Kubernetes enterprise rollout. Maybe you want to add a little chaos to your routine CI/CD workflow, or perhaps you’re just a thrill-seeker who loves the high stakes game of orchestration roulette. Either way, you’ve come to the right place. Sit back, relax, and let us guide you through the delightful maze of missteps and detours that will ensure your Kubernetes enterprise rollout is anything but a walk in the park.

    Ah, Kubernetes! The open-source platform that’s become the equivalent of a Hollywood blockbuster in the tech world. It’s like the Iron Man of container orchestration, bringing together an array of superpowers including automation, scaling, and management of container deployment. Enterprises are lining up to get their tickets, excited by the promises of streamlined application deployment. But before you go head over heels for Kubernetes, remember, even Iron Man had his quirks. Navigating the CI/CD waterfall can sometimes feel more like a rollercoaster ride without a seatbelt. So before you charge headfirst into your enterprise rollout, take a moment to consider Murphy’s Law – anything that can go wrong, will go wrong. So buckle up, my friends, it’s going to be a wild ride.

    This Photo by Unknown Author is licensed under CC BY-NC

    Indeed, Kubernetes is a boon for developers, cloud-native architects, and business owners alike. Its versatility and flexibility can make you feel like a superhero orchestrating seamless deployments. But when it comes to deploying and scaling in Enterprise data centers, Kubernetes might just swap its Iron Man suit for a Godzilla costume, spawning fresh challenges born out of its cloud-native architecture. This transition can lead to excessive mental gymnastics as you grapple with these new beasts of burden. So, if you’re feeling like a deer caught in the headlights, staring down the Kubernetes-python poised to gobble up your Enterprise applications, fear not! This blog is your sanctuary, your guide, your ‘how-to-tame-your-dragon’ manual. Stay with us, as we venture into the labyrinth of Kubernetes deployment and come out the other side grinning. 🙂

    Let’s dive into the 10 ways you might unintentionally stall your enterprise Kubernetes rollout, and inadvertently send your organization spiraling back to the ‘golden age’ of monolithic architecture:

    1. Do Not Plan Your Deployment: Some may argue that the beauty of Kubernetes lies in its simplicity, and indeed, the internet is abundant with blogs and videos promoting the notion that the deployment process is a walk in the park. Following such advice without investing time in understanding your unique use case could be a significant pitfall. Kubernetes deployments require thoughtful planning, taking into account the intricacies of workload requirements, resource allocation, and network architecture. The idea of “Kubernetes is the easy button for everything” is a perilous assumption that can easily derail your enterprise rollout. Always remember, while Kubernetes does a spectacular job in many aspects, it’s not a one-size-fits-all solution for every enterprise problem.

    This Photo by Unknown Author is licensed under CC BY-SA

    2. Avoid Using Certified Kubernetes Distributions: Now this one’s a head-scratcher, isn’t it? Here’s the thing, though: Kubernetes is powerful, flexible, and can be customized to a dizzying degree. However, this does not mean you should do everything from scratch. Consider this – why would you build your car when you can buy a perfectly good one off the lot? Certified Kubernetes distributions, like Red Hat OpenShift or VMware Tanzu, come with the assurance of being properly configured, tested, and meeting industry standards. They’re like your ready-to-drive cars, offering robust features and world-class support. By choosing to bypass these options, you’re essentially signing up for unnecessary headaches that could easily stall your enterprise rollout. Remember, Kubernetes is a tool, not an ideology. There’s no virtue in unnecessary complexities.

    Image from CNCF landscape

    3. Do NOT Implement Security Best Practices: This one’s a classic misstep in the tech world. Yes, Kubernetes is inherently secure, but that doesn’t mean it’s invincible. If you’re looking to stall your enterprise rollout, then by all means, ignore security best practices. However, if you’re keen on a smoothly functioning system, pay close attention to security measures like access control, network segmentation, and encryption. It’s akin to leaving your car unlocked in a crowded parking lot — sure, it might have an immobilizer and alarm system, but why invite trouble? Access control ensures only authorized personnel can interact with your Kubernetes clusters, network segmentation limits the blast radius of potential breaches, and encryption keeps your sensitive data safe in transit and at rest. Failing to implement these measures is like leaving the keys in your car with the engine running – a surefire way to invite mischief. Remember, in the world of Kubernetes deployments, security is not an afterthought, it’s a primary driver of successful CI/CD pipelines and enterprise rollouts.

    Image courtesy - https://www.google.com/url?sa=i&url=https%3A%2F%2Fsnyk.io%2Flearn%2Fcloud-application-security%2F&psig=AOvVaw0cLWWp0MtGNoPo1idw-iL7&ust=1691737538095000&source=images&cd=vfe&opi=89978449&ved=0CBIQjhxqFwoTCPCrh8vD0YADFQAAAAAdAAAAABAE

    4. Forget about Using CI/CD Pipelines: There’s a certain masochistic charm in choosing not to use CI/CD pipelines in your enterprise rollout. After all, who needs automation when you can manually deploy your applications, right? CI/CD pipelines, or Continuous Integration/Continuous Deployment pipelines, are like that studious classmate who always double-checks their work before submitting it – they automate the deployment process and ensure that all changes are thoroughly examined and validated before entering the production environment. If you’re a fan of chaos and unpredictability (and potentially stalling your Kubernetes enterprise rollout), then by all means, go ahead and give CI/CD pipelines a pass. However, if you value efficiency, reliability, and sanity, incorporating CI/CD pipelines into your operations could be a game-changer. They ensure a streamlined, error-free process that keeps your applications updated and secure, allowing your team to focus on what truly matters – building and improving your products. But hey, if you’re in the market for a bit of pandemonium, feel free to ignore this advice!

    image courtesy - https://devrant.com/rants/1535091/ci-cd-in-a-nutshell

    5. Visibility/Observability, what’s that?: There’s something rather intriguing about stumbling in the dark, isn’t there? For those of you who enjoy a good surprise, why not apply this approach to your Kubernetes deployment? Think about it, with no comprehensive monitoring solution in place, every day is like a thrilling game of hide-and-seek with your application’s performance, capacity, and availability. However, if you (like most sane people) prefer to know what’s happening under the hood, it’s time to incorporate a robust monitoring solution into your Kubernetes enterprise rollout. Think of it as a reliable co-pilot that keeps an eye on the road while you’re busy steering the ship. It helps you identify potential roadblocks or speed bumps, ensuring your journey toward a successful enterprise rollout is as smooth as possible. So go ahead, embrace the unknown, or better yet, ensure your unknowns are known with a comprehensive monitoring solution. But remember, no pressure; after all, it’s only your Kubernetes deployment we’re talking about here!

    image courtesy - https://linkedin.github.io/school-of-sre/level101/metrics_and_monitoring/observability/

    6. Don’t care about Config Management Tools: If you’re fond of unpredictability and enjoy the thrill of variance, throwing caution to the wind when it comes to config management could be your next adrenaline spike! Who needs tools like Ansible or Puppet that automate the configuration of your Kubernetes deployment and ensure consistent settings across your environment? Why make life easier and your enterprise rollout smoother when you can indulge in the chaotic symphony of inconsistency? Sure, these tools can simplify the management of your Kubernetes configuration, reduce errors, and ensure uniformity across your deployment, but where’s the fun in that? So sit back, relax, and let the inconsistencies rollick through your deployment, because who wouldn’t love a good configuration surprise?

    image courtesy - https://www.atlassian.com/microservices/microservices-architecture/configuration-management

    7. Forget about Disaster Recovery: If you’re the type who loves to live on the edge, why not take a leap of faith with your Kubernetes rollout too? After all, implementing disaster recovery measures like backup and recovery procedures is like carrying an umbrella all the time just because it might rain. Sure, these measures could prevent your enterprise from figuratively getting drenched in the event of an unexpected outage or data loss, but what’s a little water, right? Having a disaster recovery plan could mean the difference between a minor hiccup and a full-fledged organizational crisis during a catastrophe, but let’s face it, who doesn’t love a little game of Russian Roulette with their Kubernetes deployment? So, go ahead and roll the dice. After all, disaster recovery is just for those who aren’t fans of suspense, right?

    image courtesy - https://www.sungardas.com/en-us/blog/how-to-create-a-dr-plan-you-can-be-confident-in/

    8. Train Your Staff (or Don’t): Now, here’s a real knee-slapper: education. Nothing quite like seeing your team scramble around like a bunch of cats on a hot tin roof because they don’t know their Pods from their Nodes. Who needs well-trained staff, conversant with Kubernetes best practices, when you can bask in the glorious pandemonium of mismanaged deployments instead? Sure, giving your employees the necessary skills to effectively manage and operate your Kubernetes deployment might lead to fewer issues, greater efficiency, and a more successful enterprise rollout. But let’s be real, why stifle the potential theatre of the absurd that could result from untrained staff wrestling a mammoth like Kubernetes? Life is a stage, after all, and in your Kubernetes drama, training is just too mainstream a script. So, sit back, grab some popcorn, and enjoy the show!

    image courtesy - https://www.makemebetter.net/learning-to-go-with-the-flow/

    9. Live in the Past: Here’s a revolutionary idea – rolling with the times. You could, if you’re feeling particularly adventurous, actually stay up-to-date with the latest Kubernetes releases and security patches. That, of course, would imply that you’re interested in ensuring your deployment operates with the latest features and security updates. But hey, who doesn’t love a little nostalgia? Sure, you could prioritize keeping your enterprise rollout in line with the newest, slickest versions of Kubernetes, ensuring that your CI/CD pipelines are as cutting-edge as they come, but isn’t there a certain charm in running your enterprise on an antiquated version that’s as outdated as a floppy disk in an AI lab? After all, cybersecurity threats, outdated functionalities, and inefficiencies are just minor speed bumps on the road of enterprise rollouts. So, why not kick back, ignore those pesky update notifications, and let your Kubernetes deployment bask in the warm glow of obsolescence? Just remember – living in the past is only fun until the ghosts of security vulnerabilities and outdated features come knocking on your door.

    10. Embrace Impermanence (non-persistence): In the grand scheme of things, isn’t Kubernetes is supposed to be ephemeral? Why should your data be any different? Go ahead, live dangerously. Don’t bother with planning for data persistence in your Kubernetes rollout. Imagine the thrill of living on the edge, knowing that you could lose all your data the moment a pod goes down or the system crashes. Sure, you could use the Kubernetes Persistent Volume (PV) and Persistent Volume Claim (PVC) architecture to ensure your data survives even when your pods don’t, but where’s the fun in that? Data persistence is so pedestrian. Remember, the goal here is to stall your Kubernetes enterprise rollout, not to make it robust, resilient, and reliable. So, go ahead, and throw caution (and your data) to the wind. It’s only important business information after all, right?

    image courtesy - https://cloudtweaks.com/2016/11/4-cloud-tools-help-business-save-money/

    But hey, here’s a novel idea – what if you actually wanted to succeed in your cloud-native strategy? I know, I know, it sounds a bit radical given our prior conversation. But bear with me. For those of you who enjoy sailing smoothly on the seas of enterprise IT, without the thrill of hitting every possible iceberg, Dell has created a glorious solution. A tool, that’s as much a life preserver as it is a nautical chart, guiding you safely through the treacherous waters of Kubernetes enterprise rollouts. This magic wand is called the Container Storage Modules (CSM). 

    https://dell.github.io/csm-docs/docs/

    This isn’t just any tool – it’s your co-pilot on the journey to a seamless Kubernetes implementation. It’s like having a Swiss army knife for enterprise data management. The CSM ensures that your data persistence strategies are as solid as a rock, ensuring that no pod crash or system failure can sweep your data into the abyss. With CSM, you can laugh in the face of data loss, secure in the knowledge that your enterprise information is safe and sound. So, for the daredevils who actually like to succeed in their endeavors, the Dell Technologies CSM is the perfect tool to ensure your Kubernetes enterprise rollout is as smooth and trouble-free as a hot knife through butter.

    I hope this post proves helpful, regardless of which direction you choose for your enterprise cloud journey. If you’re inclined to thrill and enjoy the odd game of Russian roulette with your data, you now have some innovative strategies to stall your Kubernetes rollout. However, if your preference is smooth as a jazz tune and your data as secure as Dell’s Project Fort Zero, then Dell’s Container Storage Modules (CSM) is the tool you need. The CSM is your beacon in the foggy world of Kubernetes enterprise rollout, ensuring that no data loss or system failure can derail your cloud-native strategy. It’s your data’s best friend, your enterprise’s lifeline, and your ticket to a successful Kubernetes implementation.

    Enjoy the journey, and remember – with the right tools and strategies, Murphy’s Law doesn’t stand a chance!

  • Manage Kubernetes with Ansible

    Kubernetes keeps increasing in popularity and not just in public cloud. It keeps making inroads into the on-premises market. This is creating the need for automation. In many Kubernetes environments you tend to find developers using CI/CD pipelines not just for their applications code for the Kubernetes objects that deploy the code in the cluster (ex: deployment, service …). This means that most of the automation needs are covered. However there are several instances where you might want to use automation tools (ex: Ansible) either to replace or to supplement CI/CD tools. By the way, I am not talking about the deployment of the Kubernetes cluster itself, which is a valid use case. I am talking about the things that you would normally do with the “kubectl” tool

    While creating a new video for the IaC Avengers channel in Youtube I came across one such use case and this prompt me to investigate how to manage Kubernetes with Ansible. This article contains my lessons learned.

    My use case is as follows. I wanted to expose the creation of namespaces in any cloud to end-users from ServiceNow. The idea is that rather than giving developers and other personas the right to create their own namespaces an organization would like to keep a central control plane where they can implement the much needed governance and cost transparency. This use case is very important in RedHat OpenShift environments because the general guidance is to share a few clusters as opposed to creating a cluster per tenant as other vendors recommend. Namespaces is the native mechanism to keep tenants separate with this approach

    This “Multi-Cloud Kubernetes as a Service” is the latest in a growing set of demos that we have been creating for a while.

    In this article we are going to cover:

    1. Architecture
    2. Installation in command line Ansible
    3. Installation in AWX/Tower
    4. A practical example

    Architecture

    We will use a single Ansible module for this solution: “kubernetes.core.k8s”, which might surprise many of you. At first when I was thinking about this solution I thought there would be multiple modules to manage all the different objects in the Kubernetes API: pods, deployments, secrets … but no, there is a single one. To put this into perspective let’s bear in mind that there are more than 150 different modules to manage all aspects of vSphere environments.

    So why is there a single module for Kubernetes? At the end of the day Kubernetes and Ansible have much in common. Both frameworks use a declarative syntax where you express your desired state and then the system does whatever is necessary to implement your specified end state. Furthermore, they both use YAML files. So rather than creating multiple modules, you embed your each individual Kubernetes task manifest inside its own Ansible task. You need to watch out for the right indentations but that in essence how it works. We will see some examples in a later section

    Another clever shortcut the creators of the module took is that the module doesn’t include its own Kubernetes client. Instead what the Ansible engine will do is to SSH into a machine that has “kubectl” and the “kubeconfig” installed. You could install “kubectl” in your Ansible system if you wanted (and use “localhost” as the target) but you don’t have to. In my case I have created a separate VM with “kubectl” and all the “kubeconfig” files for all clusters I am managing and the Ansible playbook is targeting that VM which is defined in the inventory. In OpenShift environments your Kubernetes client machine will need to run also the “oc” tool

    In our video we assumed there will be multiple clusters available for different combinations of:

    • Cloud (vSphere based private cloud, AWS, Azure and GCP)
    • Production or development (You might want to have more like UAT …)
    • Different Kubernetes versions (v1.22, v1.23, v1.24)

    The actual selections made by the user determine the target cluster in which to create the “namespace” (a.k.a “project” in RedHat parlance). The playbook takes the 3 parameters selected by the user and builds the name of the “kubeconfig” file to use. The Ansible module allows you to specify a “kubeconfig” file. From that point any tasks are run in the relevant cluster

    The Ansible playbook allows you to specify also a “context”. At the beginning I started using a single “kubeconfig” with multiple contexts but as I kept adding clusters it was getting hard to manage. I think the “kubeconfig” method is easier. Every time you create a new cluster, grab the file, rename it to match the type/location of the cluster (ex: “aws-prod-22.config”) and place it in the directory where the client machine expects to find them and you are done

    Installation in command line Ansible

    The installation requires you to install things in both the Ansible and Kubernetes client system. With other modules you typically install some Python libraries as a prerequisite and then install the Ansible collection. A very important difference with the Kubernetes collection is the libraries are required in the Kubernetes client system, not in the Ansible system. Of course if you have decided to run the Kubernetes client in your Ansible system you will install everything in the same machine.

    Before you start please make sure you are running Python 3.6 or higher in the client. In my case I started installing this in a system with CentOS7 which comes with Python 2.7 by default and I was getting errors until I did

    ln -s /usr/bin/python3 /usr/bin/python

    In terms of libraries you need the following in the Kubernetes client machine:

    • kubernetes >= 12.0.0
    • PyYAML >= 3.11
    • jsonpatch

    In my case I just did “pip install kubernetes” and it installed everything else. OpenShift environments are better managed with the “oc” tool. For that reason you also need an additional library called “openshift”.

    The ‘kubernetes’ library expects the kubeconfig file to be present in .kube/config. However, as we discussed earlier you can specify a different location and kubeconfig file name as part of the task inside the playbook

    Now in the the Ansible machine you need to install the Ansible collection

    ansible-galaxy collection install kubernetes.core

    Finally, you will need to add your Kubernetes client to the inventory in the Ansible machine, This is mine:

    [root@ansible-vm ~] # cat inv.ini
[kubectl01]
172.24.167.53

    You can test that everything works by running a simple playbook

    [root@ansible-vm ~] # cat create-ns.yaml
- name: Create namespaces in kubernetes cluster
  hosts: kubectl01

  tasks:
  - name: Create namespace in default Kubernetes cluster
    kubernetes.core.k8s:
      name: "ansible-ns"
      api_version: v1
      kind: Namespace
      state: present

[root@ansible-vm ~] # ansible-playbook create-ns.yaml

    The above syntax assumes that the kubeconfig is in the default location, i.e. ~/.kube/config in the home directory of the user running the playbook as in the kubernetes client system. Keep reading to see how to store the config in a different location

    Installation in AWX/Tower

    If we need to run the playbook in AWX or Ansible Tower, nothing of we discussed previously for the Kubernetes clients changes. So you still need the following in the client:

    • the Python libraries
    • a supported version of Python in the client
    • the “kubectl” tool (and “oc” if you are managing OpenShift clusters

    However, on the Ansible system you need to:

    • create the inventory entry that points to the Kubernetes client system
    • install the “kubernetes.core” collection in the “task” container
    • create a job template as usual

    This is how I installed the “kubernetes.core” collection in my AWX system. Notice how I install it in the “awx_task” container

    [root@awx17 ~]# docker exec -it awx_task /bin/bash
bash-4.4# ansible-galaxy collection install kubernetes.core

    However, when I went to trigger the job template I got this error

    TASK [Create namespace in target Kubernetes cluster] ***************************
fatal: [172.24.167.53]: FAILED! => {"msg": "Could not find imported module support code for ansiblemodule.  Looked for either AnsibleTurboModule.py or module.py"}

    I fixed it by installing the “cloud.common” collection also inside the “task” container:

    [root@awx17 ~]# docker exec -it awx_task /bin/bash
bash-4.4# ansible-galaxy collection install cloud.common
Process install dependency map
Starting collection install process
Installing 'cloud.common:2.1.2' to '/var/lib/awx/.ansible/collections/ansible_collections/cloud/common'

    A practical example

    The example we are going to use will do 2 things:

    • create a namespace
    • assign permissions to the namespace to the user that requested the namespace

    In this Kubernetes as a Service design the assumption is that developers and other personas they cannot create or join namespaces by themselves. This is achieved by creating a new namespace or joining an existing one. Hence the need to assign the relevant permissions in the playbook. A future blog post show the “join namespace” scenario which includes including the creator of the namespace in a ServiceNow workflow approval.

    The first thing the playbook does is to figure out what kubeconfig file needs to be use. It does so by combining 3 pieces of information. In the video you can see how these details are provided by the user that is requesting the namespace in ServiceNow. They allow us to uniquely identify the Kubernetes cluster we have to use to apply the changes

      - name: Build the kubeconfig file name out of input parameters
    set_fact:
      configname: "{{ cloud }}-{{ envtype }}-{{ version }}"

    So for example if the user selects “aws”, “production” and “1.22” the playbook will look for a file named “aws-prod-22.config” and run the remaining tasks on the cluster that is defined in that kubeconfig file. Note how we decided to drop the “1.” from the Kubernetes version to make the file names more streamlined. With this approach, onboarding a new cluster couldn’t be easier. Let’s say in the future we want to create a new development cluster in GCP that is running v1.25. All we need to do is grab the kubeconfig file and place it in the same directory as the other files in the client and rename it to “gcp-dev-25.config”. No further changes are required

    Let’s take a look at the playbook

    ---
- name: Create a namespace in a kubernetes cluster
  hosts: kubectl01
  gather_facts: false

  vars:
    #nsname: ansible           # needs to be provided by end-user
    #version: 22               # corresponds to k8s version 1.22, 1.23 ...
    #envtype: dev              # type of environment: prod, dev ...
    #cloud: vsphere            # vpshere, gcp, aws ...
    #snow_username: finance1   # this comes also in the API call
    #backup_type: gold         # user needs to choose between gold/silver policies

  tasks:
  - name: Build the kubeconfig file name out of input parameters
    set_fact:
      configname: "{{ cloud }}-{{ envtype }}-{{ version }}"
	  
  - debug:
      msg: "Let's create namespace {{ nsname }} with kubeconfig {{ configname }}.config"

  - name: Create namespace in target Kubernetes cluster
    kubernetes.core.k8s:
      state: present
      kubeconfig: "~/.kube/{{ configname }}.config"
      kind: Namespace
      name: "{{ nsname }}"
      definition:
        metadata:
          labels:
            backuptype: "{{ backup_type }}"
            snowowner: "{{ snow_username }}"

  - name: Create role binding for user {{ snow_username }}
    kubernetes.core.k8s:
      state: present
      kubeconfig: "~/.kube/{{ configname }}.config"
      definition:
        kind: RoleBinding
        apiVersion: rbac.authorization.k8s.io/v1
        metadata:
          name: "{{ nsname }}-owner"
          namespace: "{{ nsname }}"
        subjects:
        - kind: User
          name: "{{ snow_username }}"
        roleRef:
          kind: ClusterRole
          name: admin

    I have commented out all the variables required as they are being passed as parameters but you can remove the comments when you are testing the playbook

    Pay close attention to the “definition” section in the “role binding” task. If you took everything that follows, insert it into a YAML file and use “kubectl apply” it accomplish the same thing. This is what I was referring to about the beauty of how the creators have designed the Ansible module

    Notice how we are adding 2 labels to the namespace. These will be used for the “join namespace” workflow and for automatically adding the namespace to a backup policy in PPDM (PowerProtect Data Manager). We will cover these two features in future posts

    The “snow_username” is the username of the user that places the request in ServiceNow. In our demo we used KeyCloak to create in seamless authentication infrastructure across ServiceNow and the rest of our infrastructure including OpenShift

    Finally, notice how we are binding the default “admin” role to the user, but restricted to the namespace, which is what you would expect from an owner. However, by the rules of least privilege, if you wanted to you could restrict to whatever you need by defining a specific role. You could potentially create this role at the only once at the cluster level. In that case it wouldn’t need to be part of this playbook. We will use this technique for offering various roles in the “join namespace” workflow. The following code is an example for a “deployment manager” role in a specific namespace

      - name: Create a new role for deployment managers
    kubernetes.core.k8s:
      state: present
      kubeconfig: "~/.kube/{{ configname }}.config"
      definition:
        kind: Role
        apiVersion: rbac.authorization.k8s.io/v1beta1  #rbac.authorization.k8s.io/v1
        metadata:
          namespace: office
          name: deployment-manager
        rules:
        - apiGroups: ["", "extensions", "apps"]
          resources: ["deployments", "replicasets", "pods"]
          verbs: ["*"]

    I hope you found this helpful. Keep an eye on the follow up video and the two follow up blog articles