Dell EMC CloudIQ combines proactive monitoring, machine learning, and predictive analytics so you can take quick action and simplify operations of your infrastructure. Earlier we had written a few blog posts on CloudIQ features, you can find them here.

In the latest update, CloudIQ has released the support for the REST API in addition to Webhooks. You can find the entire REST API documentation in the Dell Developer Portal.

In this post, we will explore the CloudIQ REST API through a use case where an automated remediation action needs to run in response to a CloudIQ webhook alert (as seen in this video demo). We will go through the steps on how to leverage the CloudIQ REST API to obtain information about the implicated system to help us automate the remediation.

In this post, we will cover

• Introduction to CloudIQ Webhook functionality
• CloudIQ REST API Authentication
• Using REST API to get the IP address of the array
• Performa the automated remediation
• Consolidated Python code and sample output

Note: The code in this tutorial has been written in Python and tested in Python 3.9.2. You can install the python using this link.

Let’s dive in

Introduction to CloudIQ Webhook functionality

In addition to email notification, CloudIQ can also be configured to send Webhook notifications when a Health Issue is detected. These types of notifications are meant to be used for integration with other applications. Essentially HTTP POST is sent with a JSON payload to a destination of choice every time a Health Issue is detected.

Webhooks are available in the “Integrations” menu under “Admin”

The payload that is sent contains information about the system that generated the event as well as the details of the event. The below sample payload was generated in response to a File System named “albtestFS02” getting full.

{
  "system_display_identifier": "CKM00192702222",
  "system_name": "UnityXT-480F",
  "system_model": "Unity 480F",
  "timestamp": 1618449077305,
  "timestamp_iso8601": "2021-04-15T01:11Z",
  "current_score": 60,
  "new_issues": [
    {
      "id": "5AF47517FA0F032FE415FA85AD3DA6AD91DE2C814761ED8475562E40C6981819",
      "impact": -40,
      "description": "The file system 'albtestFS02' is full.",
      "resolution": "Consider increasing the size of the file system.",
      "rule_id": "FILESYSTEM_FULL_NOW_RULE",
      "category": "CAPACITY",
      "impacted_objects": [
        {
          "object_native_id": "fs_178",
          "object_name": null,
          "object_id": "CKM00192702222__FILESYSTEM__fs_178",
          "object_native_type": "FILESYSTEM"
        }
      ]
    }
  ],
  "resolved_issues": []
}

Webhook’s are HTTP requests that use the POST method, so essentially the webhook needs to be intercepted by an HTTP server. This can be done either through a specialized monitoring tool or programmatically using a language of choice. For this blog post, we are going to assume that

• The listener already exists
• The payload has been extracted from the webhook and inserted into a variable called “payload”. In Python, this variable is a dictionary.

In this sample payload there are some parameters that are worth explaining:

• “new_issues” / “resolved_issues” – Webhooks are sent both when issues occur and when they get resolved. In our case we will pay attention to “new_issues”
• “rule_id” – This parameter tells us the type of event we are dealing with. We might be able to automate the remediation of some types of events but not others, so it is important for our code to check the “rule_id” to determine the next course of action. In this case we see this event is about a File System getting full, so we will attempt to extend it automatically

try:
    incoming_issue = payload["new_issues"][0]["rule_id"]
    print("Initiating automated remediation")
except KeyError:
    print("No automated remediation available for this rule_id. Exiting now ...")
    exit()

• “system_display_identifier” – This tells us the identity of the array serving the implicated file system. However, in order to interact with the array’s REST API to extend the file system, we need its IP address and not its CloudIQ identifier. This is the problem we will solve programmatically by leveraging CloudIQ’s REST API

Now, let’s move to the REST API authentication

CloudIQ REST API Authentication

CloudIQ’s REST API uses the OAuth 2.0 client credentials grant for authentication to obtain a token, and to access the endpoints. So, the first step is to get a token. We can then attach the token to all subsequent REST API calls.

The body of the response is formatted as JSON and will look like this:

{
    "access_token": "951b6b96-89d8-43e1-9d85-6ee3883da84b",
    "token_type": "Bearer",
    "expires_in": 3600,
    "scope": "oob"
}

As you can see the token provider is valid for 1 hour (3600 seconds). Now we need to extract “access_token”. Below is the Python code to send the call to the API and extract it.

import requests

url = "https://cloudiq.apis.dell.com/auth/oauth/v2/token"
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
data = {
    'grant_type': 'client_credentials',
    'client_id': '1112223334445557f9da8e6d79b7c1eee',
    'client_secret': '111222333444555f929ab752a4c89ddd'
}
response = requests.post(url, data=data, headers=headers)

response_body = response.json()
token = response_body["access_token"]
print(token)

The variable “token” now contains the CloudIQ token, which will be used for subsequent REST API calls.

Next, we will use the token to make the REST API calls to CloudIQ.

Use REST API to get the IP address of the array

The CloudIQ REST API exposes a range of calls with a lot of information about the systems being managed. In this example, we are going to use the following call to get the IP address of an array given its identifier. We need to use the “/rest/v1/storage_systems” endpoint

The first thing to note is that we need to include the token we obtained in the previous step as an “Authorization” header, ex:

"Authorization": "Bearer 951b6b96-89d8-43e1-9d85-6ee3883da84b"

One important thing to be aware of is that by default the CloudIQ REST API returns all attributes for all the objects. You may want to specify explicitly the attributes you want to use. This is done with the “select” parameter along with the list of required attributes separated by a comma.

For example, this API call will retrieve, the id, IP address, and model.

GET /rest/v1/storage_systems?select=id,ipv4_address,model

and below is the output of the GET call

{
    "paging": {
        "total_instances": 3
    },
    "results": [
        {
            "id": "000197900513",
            "ipv4_address": "10.246.89.238",
            "model": "PowerMax_2000"
        },
        {
            "id": "CKM00192702222",
            "ipv4_address": "10.245.18.158",
            "model": "Unity 680F"
        },
        {
            "id": "POWERSTORE-PSc00bba5a1c16",
            "ipv4_address": "10.245.16.227",
            "model": "PowerStore 500T"
        }
    ]
}

For our use case, we only need to find out the IP address of the array so we will select explicitly that property. In Python, the first step is to extract the identifier from the variable “payload” that contains the payload of the webhook we received.

system_display_id = payload["system_display_identifier"]
print("Storage system id is " + system_display_id)

We can now use this identifier to build the URL. The following Python code:

• Builds the URL and the headers
• Performs the query to the REST API
• Extracts the IP address

url = "https://cloudiq.apis.dell.com/cloudiq/rest/v1/storage_systems/" + system_display_id + "?select=ipv4_address"
headers = {"Authorization": "Bearer " + token}
response = requests.get(url, headers = headers)
response_body = response.json()
ip = response_body["ipv4_address"]
print("System IP Address is " + ip)

The IP address of the array that the webhook is referring to is now stored in the variable “ip” ready to be used in the next use case demonstrating the automatic remediation.

Performing the Automated Remediation

At this point, you should be able to build the URL of the storage array’s REST API call to attempt the remediation. However, this is going to depend on:

• The use case we are trying to solve.
  • In this example that would be extending a file system. However, there could be many others: changing a performance policy, relocate a volume between appliances …
• The array type is also a factor.
  • For example, extending a file system in Unity is going to be different from extending a file system in PowerStore or PowerScale because their REST API’s are different. The good news is that the payload of the webhook also provides “system_model” which allows us to invoke a specific remediation for that system model

One of the most common ways of changing the configuration of a storage array (and performing automated remediation) is using Ansible. DellEMC provides official Ansible modules for our storage arrays. These modules can be downloaded from the Dell GitHub site. You can also find lots of examples on configuring the various products in the Dell Technologies portfolio in this repository and on the Dell GitHub site.

Now the question is, how do you invoke an Ansible playbook as part of the Python code we have been covering in this tutorial? It is important to understand that most organizations (instead of Ansible CLI) use Ansible Tower (which is part of the RedHat Ansible Automation Platform) as it provides enterprise features such as RBAC, encryption of credentials, auditing, logging, etc. Some organizations use AWX (the upstream open-source community project) which is functionally equivalent to Ansible Tower. AWX is very popular in non-production environments.

Ansible Tower and AWX also expose a RESTful API. This allows you to trigger a remediation action by sending a REST API call. You typically need to include the “extra_vars” parameter as part of the payload with the necessary details to perform the remediation action, ex: you might have to pass the file system name and the new size in order to extend a file system that filled up. In Ansible Tower, playbooks are wrapped in a structure called a job template. The template also includes other things like credentials needed to execute the playbook, permissions. Each job template has a template ID, and that’s what we use to request its execution. For example, the following REST API call will trigger the execution of template 43:

POST  https://10.1.1.1/api/v2/job_templates/43/launch/

This API call uses the POST method and therefore requires a Body with any variables that you want to pass externally, ie the “extra_vars” we mentioned earlier.

The following Python code will trigger the execution of job 43. The underlying Ansible playbook will attempt the extension of a file system with the details provided in the payload (i.e. file system name, and size).

import requests
import json

url = "https://10.1.1.1/api/v2/job_templates/43/launch/"

payload = json.dumps({
  "extra_vars": {
    "fs_name": "oracle01",
    "size": 200
  }
})
headers = {
  'Content-Type': 'application/json',
  'Authorization': 'Basic YWRtaW46cGFzabcdefgh' # This is the base 64 encoding of "username:password"
}

response = requests.request("POST", url, headers=headers, data=payload)
print(response.text) 

Now, let’s wrap this up together and see what consolidated output will look like.

Consolidated Python Code and Sample Output

The following is an example of the output you will get from the final code.

C:\>python CIQ-tutorial.py
Initiating automated remediation
 ... Storage system ID: CKM00192702222
 ... Access Token     : d8e2d990-9037-4489-9fd5-f7f037abf509
 ... System IP Address: 10.245.18.158

The following is the full consolidated code. It assumes the webhook payload has been already extracted. Note that it doesn’t demonstrate the remediation code because it’ll vary based on the use case and storage array.

import requests

#### This is the payload that was received and extracted from the webhook
payload = {
    "system_display_identifier": "CKM00192702222",
    "system_name": "UnityXT-480F",
    "system_model": "Unity 480F",
    "timestamp": 1620476550458,
    "timestamp_iso8601": "2021-05-08T12:22Z",
    "current_score": 70,
    "new_issues": [
        {
            "id": "A247D699297F328D473E9E2E4DD9976270340FB50FD6ADC22B167825FB",
            "impact": -5,
            "description": "The file system 'albtestFS02' is full.",
            "resolution": "Consider increasing the size of the file system.",
            "rule_id": "FILESYSTEM_FULL_NOW_RULE",
            "category": "CAPACITY",
            "impacted_objects": [
                {
                    "object_native_id": "fs_178",
                    "object_name": None,
                    "object_id": "CKM00192702222__FILESYSTEM__fs_178",
                    "object_native_type": "FILESYSTEM"
                }
            ]
        }
    ],
    "resolved_issues": []
}

#### Check that the rule_id corresponds to an event we can remediate automatically
try:
    incoming_issue = payload["new_issues"][0]["rule_id"]
    print("Initiating automated remediation")
except KeyError:
    print("No automated remediation available for this rule_id. Exiting now ...")
    exit()
    
#### Extract the display identifier of the array
system_display_id = payload["system_display_identifier"]
print(" ... Storage system ID: " + system_display_id)

#### Login to CloudIQ to get a token
url = "https://cloudiq.apis.dell.com/auth/oauth/v2/token"
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
data = {
    'grant_type': 'client_credentials',
    'client_id': '11223344556677442e959909b7f65d3777',
    'client_secret': '11223344556677d9b442b478e5d4888'
    }
response = requests.post(url, data=data, headers=headers)
response_body = response.json()
token = response_body["access_token"]
print(" ... Access Token     : " + token)

#### Send API to call to CloudIQ to get the storage system's IP address
url = "https://cloudiq.apis.dell.com/cloudiq/rest/v1/storage_systems/" + system_display_id + "?select=ipv4_address"
headers = {"Authorization": "Bearer " + token}
response = requests.get(url, headers = headers)
response_body = response.json()
ip = response_body["ipv4_address"]
print(" ... System IP Address: " + ip)

#### Not demonstrated - send API call to automate remediation - depends on use case

In this tutorial RedHat Ansible Tower is used to trigger the remediation. To help you get started I have written series of blog posts on Ansible with Dell EMC portfolio.

I hope with this blog post you’ve got a fair understanding on how to interact with CloudIQ REST API and extract the details from the output payload.