Author: Alberto

  • Fix GCP error Permission ‘storage.buckets.get’ denied

    This will be a quick one. I was recently experimenting with creating an S3 bucket GCP using Ansible and I came across this error:

    {
      "msg": "GCP returned error: {'error': {'code': 403, 'message': \ansible@vexpose.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).\, 'errors': [{'message': \ansible@vexpose.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).\, 'domain': 'global', 'reason': 'forbidden'}]}}",
      "invocation": {
        "module_args": {
          "name": "gcp_s3",
          "project": "vexpose",
          "auth_kind": "serviceaccount",
          "storage_class": "COLDLINE",
    ... <<< output truncated >>>
    

    After seeing “Permission Denied” I naturally started to look at the roles that were assigned to the account . I discovered later that the “Storage Admin” role provided already that permission, but in the process I wasted some precious time adding other roles that provided that permission yet again. So I felt compelled to write this quick post to help other people save their time.

    If this is happening to you the resolution could be quite simple. We must remember that GCP (like other public cloud providers) uses a single namespace for all customers. Therefore the bucket name must be “universally” unique. If it isn’t it takes it as you are trying to make changes to an existing bucket that another customer owns and it throws the misleading “permission denied” error. So, simply choose a more complex name and see if that fixes the error.

    You can quickly test if it is an issue with your name not being unique by trying to create a bucket using GCP web interface for example. It if is already taken you will receive a message like this.

  • Manage Kubernetes with Ansible

    Kubernetes keeps increasing in popularity and not just in public cloud. It keeps making inroads into the on-premises market. This is creating the need for automation. In many Kubernetes environments you tend to find developers using CI/CD pipelines not just for their applications code for the Kubernetes objects that deploy the code in the cluster (ex: deployment, service …). This means that most of the automation needs are covered. However there are several instances where you might want to use automation tools (ex: Ansible) either to replace or to supplement CI/CD tools. By the way, I am not talking about the deployment of the Kubernetes cluster itself, which is a valid use case. I am talking about the things that you would normally do with the “kubectl” tool

    While creating a new video for the IaC Avengers channel in Youtube I came across one such use case and this prompt me to investigate how to manage Kubernetes with Ansible. This article contains my lessons learned.

    My use case is as follows. I wanted to expose the creation of namespaces in any cloud to end-users from ServiceNow. The idea is that rather than giving developers and other personas the right to create their own namespaces an organization would like to keep a central control plane where they can implement the much needed governance and cost transparency. This use case is very important in RedHat OpenShift environments because the general guidance is to share a few clusters as opposed to creating a cluster per tenant as other vendors recommend. Namespaces is the native mechanism to keep tenants separate with this approach

    This “Multi-Cloud Kubernetes as a Service” is the latest in a growing set of demos that we have been creating for a while.

    In this article we are going to cover:

    1. Architecture
    2. Installation in command line Ansible
    3. Installation in AWX/Tower
    4. A practical example

    Architecture

    We will use a single Ansible module for this solution: “kubernetes.core.k8s”, which might surprise many of you. At first when I was thinking about this solution I thought there would be multiple modules to manage all the different objects in the Kubernetes API: pods, deployments, secrets … but no, there is a single one. To put this into perspective let’s bear in mind that there are more than 150 different modules to manage all aspects of vSphere environments.

    So why is there a single module for Kubernetes? At the end of the day Kubernetes and Ansible have much in common. Both frameworks use a declarative syntax where you express your desired state and then the system does whatever is necessary to implement your specified end state. Furthermore, they both use YAML files. So rather than creating multiple modules, you embed your each individual Kubernetes task manifest inside its own Ansible task. You need to watch out for the right indentations but that in essence how it works. We will see some examples in a later section

    Another clever shortcut the creators of the module took is that the module doesn’t include its own Kubernetes client. Instead what the Ansible engine will do is to SSH into a machine that has “kubectl” and the “kubeconfig” installed. You could install “kubectl” in your Ansible system if you wanted (and use “localhost” as the target) but you don’t have to. In my case I have created a separate VM with “kubectl” and all the “kubeconfig” files for all clusters I am managing and the Ansible playbook is targeting that VM which is defined in the inventory. In OpenShift environments your Kubernetes client machine will need to run also the “oc” tool

    In our video we assumed there will be multiple clusters available for different combinations of:

    • Cloud (vSphere based private cloud, AWS, Azure and GCP)
    • Production or development (You might want to have more like UAT …)
    • Different Kubernetes versions (v1.22, v1.23, v1.24)

    The actual selections made by the user determine the target cluster in which to create the “namespace” (a.k.a “project” in RedHat parlance). The playbook takes the 3 parameters selected by the user and builds the name of the “kubeconfig” file to use. The Ansible module allows you to specify a “kubeconfig” file. From that point any tasks are run in the relevant cluster

    The Ansible playbook allows you to specify also a “context”. At the beginning I started using a single “kubeconfig” with multiple contexts but as I kept adding clusters it was getting hard to manage. I think the “kubeconfig” method is easier. Every time you create a new cluster, grab the file, rename it to match the type/location of the cluster (ex: “aws-prod-22.config”) and place it in the directory where the client machine expects to find them and you are done

    Installation in command line Ansible

    The installation requires you to install things in both the Ansible and Kubernetes client system. With other modules you typically install some Python libraries as a prerequisite and then install the Ansible collection. A very important difference with the Kubernetes collection is the libraries are required in the Kubernetes client system, not in the Ansible system. Of course if you have decided to run the Kubernetes client in your Ansible system you will install everything in the same machine.

    Before you start please make sure you are running Python 3.6 or higher in the client. In my case I started installing this in a system with CentOS7 which comes with Python 2.7 by default and I was getting errors until I did

    ln -s /usr/bin/python3 /usr/bin/python

    In terms of libraries you need the following in the Kubernetes client machine:

    • kubernetes >= 12.0.0
    • PyYAML >= 3.11
    • jsonpatch

    In my case I just did “pip install kubernetes” and it installed everything else. OpenShift environments are better managed with the “oc” tool. For that reason you also need an additional library called “openshift”.

    The ‘kubernetes’ library expects the kubeconfig file to be present in .kube/config. However, as we discussed earlier you can specify a different location and kubeconfig file name as part of the task inside the playbook

    Now in the the Ansible machine you need to install the Ansible collection

    ansible-galaxy collection install kubernetes.core

    Finally, you will need to add your Kubernetes client to the inventory in the Ansible machine, This is mine:

    [root@ansible-vm ~] # cat inv.ini
    [kubectl01]
    172.24.167.53
    

    You can test that everything works by running a simple playbook

    [root@ansible-vm ~] # cat create-ns.yaml
    - name: Create namespaces in kubernetes cluster
      hosts: kubectl01
    
      tasks:
      - name: Create namespace in default Kubernetes cluster
        kubernetes.core.k8s:
          name: "ansible-ns"
          api_version: v1
          kind: Namespace
          state: present
    
    [root@ansible-vm ~] # ansible-playbook create-ns.yaml

    The above syntax assumes that the kubeconfig is in the default location, i.e. ~/.kube/config in the home directory of the user running the playbook as in the kubernetes client system. Keep reading to see how to store the config in a different location

    Installation in AWX/Tower

    If we need to run the playbook in AWX or Ansible Tower, nothing of we discussed previously for the Kubernetes clients changes. So you still need the following in the client:

    • the Python libraries
    • a supported version of Python in the client
    • the “kubectl” tool (and “oc” if you are managing OpenShift clusters

    However, on the Ansible system you need to:

    • create the inventory entry that points to the Kubernetes client system
    • install the “kubernetes.core” collection in the “task” container
    • create a job template as usual

    This is how I installed the “kubernetes.core” collection in my AWX system. Notice how I install it in the “awx_task” container

    [root@awx17 ~]# docker exec -it awx_task /bin/bash
    bash-4.4# ansible-galaxy collection install kubernetes.core
    

    However, when I went to trigger the job template I got this error

    TASK [Create namespace in target Kubernetes cluster] ***************************
    fatal: [172.24.167.53]: FAILED! => {"msg": "Could not find imported module support code for ansiblemodule.  Looked for either AnsibleTurboModule.py or module.py"}
    

    I fixed it by installing the “cloud.common” collection also inside the “task” container:

    [root@awx17 ~]# docker exec -it awx_task /bin/bash
    bash-4.4# ansible-galaxy collection install cloud.common
    Process install dependency map
    Starting collection install process
    Installing 'cloud.common:2.1.2' to '/var/lib/awx/.ansible/collections/ansible_collections/cloud/common'
    

    A practical example

    The example we are going to use will do 2 things:

    • create a namespace
    • assign permissions to the namespace to the user that requested the namespace

    In this Kubernetes as a Service design the assumption is that developers and other personas they cannot create or join namespaces by themselves. This is achieved by creating a new namespace or joining an existing one. Hence the need to assign the relevant permissions in the playbook. A future blog post show the “join namespace” scenario which includes including the creator of the namespace in a ServiceNow workflow approval.

    The first thing the playbook does is to figure out what kubeconfig file needs to be use. It does so by combining 3 pieces of information. In the video you can see how these details are provided by the user that is requesting the namespace in ServiceNow. They allow us to uniquely identify the Kubernetes cluster we have to use to apply the changes

      - name: Build the kubeconfig file name out of input parameters
        set_fact:
          configname: "{{ cloud }}-{{ envtype }}-{{ version }}"
    

    So for example if the user selects “aws”, “production” and “1.22” the playbook will look for a file named “aws-prod-22.config” and run the remaining tasks on the cluster that is defined in that kubeconfig file. Note how we decided to drop the “1.” from the Kubernetes version to make the file names more streamlined. With this approach, onboarding a new cluster couldn’t be easier. Let’s say in the future we want to create a new development cluster in GCP that is running v1.25. All we need to do is grab the kubeconfig file and place it in the same directory as the other files in the client and rename it to “gcp-dev-25.config”. No further changes are required

    Let’s take a look at the playbook

    ---
    - name: Create a namespace in a kubernetes cluster
      hosts: kubectl01
      gather_facts: false
    
      vars:
        #nsname: ansible           # needs to be provided by end-user
        #version: 22               # corresponds to k8s version 1.22, 1.23 ...
        #envtype: dev              # type of environment: prod, dev ...
        #cloud: vsphere            # vpshere, gcp, aws ...
        #snow_username: finance1   # this comes also in the API call
        #backup_type: gold         # user needs to choose between gold/silver policies
    
      tasks:
      - name: Build the kubeconfig file name out of input parameters
        set_fact:
          configname: "{{ cloud }}-{{ envtype }}-{{ version }}"
    	  
      - debug:
          msg: "Let's create namespace {{ nsname }} with kubeconfig {{ configname }}.config"
    
      - name: Create namespace in target Kubernetes cluster
        kubernetes.core.k8s:
          state: present
          kubeconfig: "~/.kube/{{ configname }}.config"
          kind: Namespace
          name: "{{ nsname }}"
          definition:
            metadata:
              labels:
                backuptype: "{{ backup_type }}"
                snowowner: "{{ snow_username }}"
    
      - name: Create role binding for user {{ snow_username }}
        kubernetes.core.k8s:
          state: present
          kubeconfig: "~/.kube/{{ configname }}.config"
          definition:
            kind: RoleBinding
            apiVersion: rbac.authorization.k8s.io/v1
            metadata:
              name: "{{ nsname }}-owner"
              namespace: "{{ nsname }}"
            subjects:
            - kind: User
              name: "{{ snow_username }}"
            roleRef:
              kind: ClusterRole
              name: admin
    

    I have commented out all the variables required as they are being passed as parameters but you can remove the comments when you are testing the playbook

    Pay close attention to the “definition” section in the “role binding” task. If you took everything that follows, insert it into a YAML file and use “kubectl apply” it accomplish the same thing. This is what I was referring to about the beauty of how the creators have designed the Ansible module

    Notice how we are adding 2 labels to the namespace. These will be used for the “join namespace” workflow and for automatically adding the namespace to a backup policy in PPDM (PowerProtect Data Manager). We will cover these two features in future posts

    The “snow_username” is the username of the user that places the request in ServiceNow. In our demo we used KeyCloak to create in seamless authentication infrastructure across ServiceNow and the rest of our infrastructure including OpenShift

    Finally, notice how we are binding the default “admin” role to the user, but restricted to the namespace, which is what you would expect from an owner. However, by the rules of least privilege, if you wanted to you could restrict to whatever you need by defining a specific role. You could potentially create this role at the only once at the cluster level. In that case it wouldn’t need to be part of this playbook. We will use this technique for offering various roles in the “join namespace” workflow. The following code is an example for a “deployment manager” role in a specific namespace

      - name: Create a new role for deployment managers
        kubernetes.core.k8s:
          state: present
          kubeconfig: "~/.kube/{{ configname }}.config"
          definition:
            kind: Role
            apiVersion: rbac.authorization.k8s.io/v1beta1  #rbac.authorization.k8s.io/v1
            metadata:
              namespace: office
              name: deployment-manager
            rules:
            - apiGroups: ["", "extensions", "apps"]
              resources: ["deployments", "replicasets", "pods"]
              verbs: ["*"]

    I hope you found this helpful. Keep an eye on the follow up video and the two follow up blog articles

  • Send message to Teams channel with REST API or Ansible

    There is no question Microsoft Teams is everywhere these days. Many IT teams use it to collaborate on projects or even day-to-day tasks. On the other hand engineers like to know what their automation solution is doing. Therefore, sending notifications is a very welcome addition to most automation workflows and Teams is the way to go nowadays as opposed to the venerable email. This is a skill we have used in many of our latest infrastructure automation videos in the IaC Avengers YouTube channel and a something that my customers have been wanted to learn. So here we go.

    In this article I am going to cover:

    • prepare your Teams channel to accept messages programmatically
    • use the REST API to create those messages. I will demonstrate it with Postman
    • send a message to Teams using Ansible

    Prepare a Teams channel to receive Webhooks

    The easiest to accomplish this by enabling webhooks for the channel. Webhooks allow one application to notify another of an event. As you will see in the examples later, webhooks are HTTP messages that use the POST method and include a payload that is formatted in a way that makes sense for the recipient application.

    As you can see, I have created a new channel called “Alert Testing”. We can configure webhooks for this channel by easily going into the “Connectors” option

    Once in the Connectors menu you need to find the “Incoming Webhook” option . In my menu it shows up by default at the top. If it doesn’t you can click on the “Developer Tools” category or you can type “webhook” in the search box at the top-left corner

    Now the “Incoming Webhook” configuration page pops up and the only thing we need to do is to provide a name and click “Create”. As soon as you do that it provides you with a “very long” URL as you can see at the bottom on the following screenshot. This is the URL we need to send our HTTP messages to. Now simply copy the URL and keep it somewhere handy. Finally click “Done” and we are ready.

    If you missplace the URL, don’t worry you can always go back to “Connectors” and find it in the “Configured” category.

    Send Teams messages using the REST API

    I like to use Postman when I am discovering a new REST API. It helps me track of what I am discovering and it allows me to package what I have learnt as a “collection” in a format that is easy to share. And this time is no exception. I have created a Postman collection with the 2 examples we are using in this article as well as a sample Ansible playbook. They are available at this repository in GitHub.

    What determines the format and content of the message card in Teams is the payload. You will see this in the “Body” tab. As you can see it is formatted as JSON. Cards in Teams can have a lot of features and the payload can get very complicated but as you can see the basic message only requires 2 fields.

    Don’t forget to paste the webhook URL you got from the previous step so that the message to your channel. To run it simply click “Send”. You should get a “200 OK” status code and your first message will show up in the channel straight away.

    Let’s do now a more sophisticated example. Let’s say that as part of an automation workflow we have created 2 virtual machines and we want to notify a channel in Teams and provide details of the VM’s that were just created. In the example below you can see the “title” includes the actual “id” of the provisioning job. This is something you could extract from the automation workflow and introduce it here for reference.

    In this case the payload this includes a “sections” key which we hadn’t used before. The value of this key is a list, which allows you to define multiple sections in your card. In this case we added only one section and we are populating it with a 2 column table that includes two “facts”. You can have more than 2 facts in your section or you could research how to add other elements like buttons, links …

    When we click “Send” we get our second message in the channel.

    Send Teams messages using Ansible

    Going back to the previous example you might be using Ansible to automate the creation of Virtual Machines and you want to let a team know the details of their details. As you do your provisioning tasks you can register the details of the resources you are creating and then use those to populate payload.

    In Ansible we can use the “uri” module to interact with web services. This includes REST APIs. This module allows us to specify any method we need (“POST” in this case), headers, payload etc. The following screenshot shows an Ansible playbook that creates the same card as the previous example. Notice how the payload has been converted from JSON into its YAML equivalent. You have to pay careful attention indentations and hyphens. You can download the code from the GitHub repo.

    Have fun creating your own messages!

  • ServiceNow CMDB REST API Tutorial

    In this tutorial we are going to use a practical example to show you how to use the ServiceNow REST API. I personally find ServiceNow has great online documentation but I like to see some examples to understand how other people are using it. This is the motivation for this tutorial and for the previous one on ServiceNow incidents with REST API. Let’s get to it!

    Introduction

    The CMDB (Configuration Management Database) in ServiceNow is a key component that underpins multiple services. Most organizations nowadays have a requirement to automate services delivery in order to achieve greater agility and efficiency. If we are going to implement automation in ServiceNow sooner or later we need to deal with the CMDB and the way to do this is to use the REST API.

    In this tutorial we will explore this by using a Postman collection you can find in this GitHub repo. If you need code in a specific language, Postman can help you generate code for any of the API calls in the collection. The collection also contains REST API calls to manage Incidents in ServiceNow. These calls were used in a previous tutorial called “Creating ServiceNow Incidents via REST API” which showed some examples on how to address one of the most common tasks organizations are willing to automate in ServiceNow.

    If you are reading this more than likely you know what a CMDB now, so I won’t provide much detail here. The main thing to know is that the CMDB is organized in a large hierarchy of tables. At the top of the hierarchy we have the “cmdb” table and from that a single child called “cmdb_ci”. This last table is the parent for everything else. (services, applications, servers, networks, databases …).

    All objects in this database are called “Configuration Items” or “CI” for short. You can show all the items by typing “cmdb_ci.list” in the “Navigator”. My developer instance has more than 2800 CI’s. The “class” parameter tells us what type of CI it is and as you will see it is a very important detail when dealing with the REST API.

    The CMDB is accessed in the ServiceNow interface with the “configuration” application. Once you type “configuration” in the “Navigator” you can scroll down to see everything it contains.

    The REST API

    When working with a new REST API, the first step is to learn how to authenticate. In that regard, the ServiceNow REST API is straight forward as you can use basic authentication, ie username and password.

    There are two main tools available to learn what API calls are available: the online product documentation and the REST API Explorer. In the online documentation you have to find the “REST API Reference” and then scroll down to “CMDB Instance API“. This is publicly accessible. In the following screenshot you can see it contains 7 REST API calls that allow you to do a range of CRUD operations.

    Notice how the API calls have the “classname” URL parameter. This corresponds to the “class” we mentioned earlier. Every piece of information one would expect to find (parameters, headers, status codes …) is provided in this documentation

    The second tool is the “REST API Explorer”. This a a great tool that allows you to build your API calls in a graphical manner, including the “body” payload for POST /PUT/PATCH calls. I showed how to use it in the previous tutorial “Creating ServiceNow Incidents via REST API“. However, given the sheer amount of attributes available for all classes of CI’s I am going to suggest a different way of doing this

    Learning with a practical example

    Let’s say we have an automation script that creates and configures a Linux virtual machine and as part of the same script now we want to add an entry for the virtual machine in the CMDB. The following API call is the one we need to use in order to create the CI:

    POST /now/cmdb/instance/cmdb_ci_linux_server

    This POST call requires a request “body” parameter which can have a large number of attributes as well as inbound/outbound relation information, ie how this CI is related to other CI’s. To help us configure the “body” we are going to:

    • create manually a sample resource of the same class (ie. a Linux server in this case) and configure it the way we need it. It is important to configure every attribute we want to use
    • use a GET call for that resource. The response will serve you as a very good reference for the “body” of the POST call we want to automate

    We type “configuration” and scroll down to “Servers” and click on “Linux”. Once in “Linux Servers”, click “New” to create a new Linux server. I have filled in those fields that are relevant to my use case.

    Once ready, you can click submit and the new CI will be created.

    At this point, you might want to add information on how this CI is related to other CI’s. This will be very useful for example to see what services/applications are impacted if there is an incident on this CI. In my example let’s say I want to show that this Linux server depends on a storage volume. Follow these steps:

    • Open the CI you have just created
    • Scroll down to “Related Items”
    • On the right click the “+” symbol next to “Search for CI” to open the “Relationship Editor”

    This opens the “Relationship Editor” as seen below. Now you can select the type of relationship and use the filter tool to search for the CI it depends on, ie a storage volume in our example. Then tick the CI and click the “+” symbol to add the relationship

    In my case I have also created an additional relationship with an application named “Alberto Inventory app”. For the application I chose the “Used by” relationship, meaning that the application is the one that “depends on” the Linux server. In the resulting “Relationships” table below you can see both relationships and the parent/child relationship:

    Back in Linux server record we can see at a glance the resulting relationships in the “Related Items” section. Notice how this is showing an additional relationship “Used by – Alberto Inventory Service”. I didn’t explicitly declared this relationship but I had an existing relationship between this service and the application. So in the end the service also depends on this Linux server

    Now it is time to use Postman to see what the payload looks like. Download the collection and create 2 environment variables “pwd” and “instance” as instructed in the GitHub repo. At this point you can open the “Get CMDB Linux servers” call and click “Send”. This will return a list of Linux servers. But for each of them only the “sys_id” and the “name” are displayed.

    Copy the the “sys_id” of your newly created Linux server CI and open the “GET CMDB Linux server details” call in Postman. In the URL you can the “sys_id” to the end of the URL (see highlighted in yellow) and click “Send”

    This is now showing a very good approximation of the “body” parameter we were after including attributes as well as inbound/outbound relations. Notice though, how it is nested under “result”. Notice also how the JSON payload is showing all the attributes, including attributes were not displayed in the form when we created the resource. Any attribute we didn’t specify during CI creation will be empty. At this point you can make a note of any extra attributes you want to include in your payload.

    Now in Postman let’s open the “select the “POST Create CMDB Linux server” call. This will create a second server called “alblinux02” with the same relationships as the server we created manually. Open the “Body” tab to see the JSON payload.

    The most significant change we need to make to the output of the GET call is to modify those attributes whose value was a “dictionary” with three keys (“display_value”, “link” and “value”). These kind of attributes are in the “attributes” section as well as in the “inbound and outbound relations” sections. What we need to do is to replace the whole dictionary value with just the “sys_id” which happens to be the “value” field in the GET call output

    Now we can click “Send” and the new Linux server CI will be created. You will get the status code “201 Created”. Now in the Linux servers app you should see both Linux servers.

    And if you want to see the whole stack end-to-end you can go to the “application” and click on “Show Dependency views”

    Supplementary API calls

    You might have noticed in the request “Body” of the POST call that I included several “sys_id”. While you can get “sys_id” in the ServiceNow GUI by using right-click, ultimately if you want to automate a process you will have to get that information programmatically. For that purpose the Postman collection includes three API calls to help you get “sys_id” you need:

    • CMDB relationship types
    • CMDB users
    • CMDB CI

    In the screenshot below you can see the three supplementary API calls highlighted in yellow. In the response to the “relationship types” call you would have to locate the item in the list with the “name” of the relationship you need and then extract its “sys_id”.

    The process is very similar if you need to extract the “sys_id” of a “user” or a “CI” with the other two API calls

    Tips and tricks

    It might happen that when you build the payload for a POST call, like the “Create CMDB Linux server” you might accidentally omit a mandatory attribute. You could go to the documentation to see what the mandatory attributes are, but one thing I found really useful is that the ServiceNow REST API let’s you know what you are missing in the error message. In the screenshot below I removed the “discovery_source” attribute and sent the API call. As you can see the error message let’s me know exactly what the problem is.

    One thing to bear in mind is that by default GET api calls return only 1000 records. If you are looking for a certain “CI” in order to extract its “sys_id” it might well happen that you have more than 1000 CI’s in the CMDB and the CI is not in the output. In that case you can use the “sysparm_query” parameters to narrow down the search or use the parameter “sysparm_limit” to retrieve more than 1000 records. You will notice in the Postman collection how the “Get CMDB CI” call is using “sysparm_limit” to retrieve 10000 records.

    That was a long post! Thanks for putting up with me for the last 15 mins 😉 I hope you found this tutorial useful.

  • Create a health score in Grafana

    This is part 2 of a small series we started to show some techniques that allow us to build a hierarchy of dashboards with Grafana. In the first part we learnt how to create links both at the panel and at the dashboard level. In this article we are going to explore how to create health status metrics that we can use in our “Top” level dashboard to get an at-a-glance view of a single system. Our dashboard will likely have one of these for every system being managed

    Producing a health metric is going to require some basic math. The question is “where” do we do these calculations. Even though the calculations are not necessarily complicated the general guidance is not to do them in Grafana. What we are going to show here is how to calculate the health score as we take the measurement. Then we will store the health data along with the sensor data into the time series database as it is produced. If you want to do this in Grafana you might want to explore the “Expressions” functionality, but at a time of writing is a beta feature and you get warned that it might not be there in future versions.

    I will provide some sample scripts in Python. If you don’t use Python in your environment, it doesn’t matter as the main thing now is to focus on the actual logic. In order to work with sample data we are going to create some random numbers for 3 different metrics representing the environment conditions of a certain location (a warehouse, a lab …). These metrics will be fake temperature, humidity and noise readings but you could use different metrics for your use case. Also not that the time series database we are using is InfluxDB, which is very popular these days.

    When we calculate a health metric that summarizes all these readings we have 2 options.

    • Combine all metrics into one
    • Use the health of the worse metric

    Combine all metrics into one

    In this method we start with the maximum health value and then discount health points as you parse the data to find out the current health. Note how I have used 10 as the top health score. Other use cases might benefit from using 100 as the top score in which case you can interpret the health number as a percentage

    import time
    import random
    from influxdb import InfluxDBClient
    inf_db = "iot_database"
    client = InfluxDBClient(host='localhost', port=8086)
    client.switch_database(inf_db)
    
    for x in range(10000):
        i = random.randint(20,45) # temperature reading
        j = random.randint(40,80) # humidity reading
        k = random.randint(40,60) # noise reading
    
        # Let's calculate the health based on the current readings
        health = 10 # Start with max possible health and substract from there
        if i > 30: health -= 1
        if i > 40: health -= 2
        if j > 60: health -= 1
        if j > 70: health -= 1
        if k > 50: health -= 1
    
        data = 'lab Temperature={},Humidity={},Noise={},Health={}'.format(i,j,k,health)
        print str(x).zfill(4) + " : " + data
        client.write([data],{'db':inf_db},204,'line')
    
        time.sleep(5)
    

    Notice how I am taking health points if temperature is high and then take extra points if it is even higher. In my opinion this produces simpler code than doing double conditions such as “if i < 40 and i < 30”

    We can add more penalty to metrics or conditions that are more severe. For example notice how temperatures over 40 take 2 extra points instead of 1.

    If you have many metrics contributing to health and all of them are taking many points away you might end up with negative numbers. You might add another line of code that turns health to 0 if the calculated value is a negative number

    We run the code and we get the following output. As we are using random numbers we get metrics swinging very wildly but it is a good thing in this case because we can see how the health parameter is reacting

    The “Health” metric is numerical so if we want to use a “stat” panel with status such as “OK” or “CRITICAL” we will need to use “Value Mapping” in Grafana. First let’s create a panel in the “Top” level dashboard. Make sure is of the type “stat”. You can configure the “query” as follows. Notice the “FORMAT AS Table”:

    Then go to the “settings” in the right-pane and scroll-down to “Value mappings”. You can configure your value mappings as follows. Don’t forget to set the “Display text” and the “Color”

    We can then get out of panel editing mode and observe how our “stat” panel behaves. Notice how I have added a link to another dashboard that shows the actual time series for all the variables as described in the previous post.

    Since we are representing “Health” by a number, another way of presenting it in our “Top” level dashboard is with a “Gauge” panel. These types of panels are also very visual as they show you the current value in relation with the minimum and maximum values in the range. Let’s add a new “Gauge” panel and configure the query as follows, notice how we are now using “FORMAT AS Time series”

    For a “Gauge” it is important to define the range of possible values. In our case this is 0 and 10 as shown below. If you have defined your health metric as a percentage you can set the range from 0 to 100 and select “Percent(0-100)” in the “Unit” field

    If we get out of edit mode the changes are made right away and our new “Gauge” panel looks like this

    You can also use value mapping to show a health label instead of the health number, which along with the color conveys a very clear message. In the screenshots below I am using the same “Value mappings” we used for the “Stat” panel above

    Use the health of the worse metric

    Another way of calculating a metric would be to pick up the status of the worst metric. This approach is more conservative and it has its merits. The first thing we need to do is to calculate a different health score for each metric and then select the worst one as the overall health of the whole system. You can see some sample Python code below to illustrate the concept

    import time
    import random
    from influxdb import InfluxDBClient
    inf_db = "iot_database"
    client = InfluxDBClient(host='localhost', port=8086)
    client.switch_database(inf_db)
    
    for x in range(10000):
        i = random.randint(20,45) # temperature reading
        j = random.randint(40,80) # humidity reading
        k = random.randint(40,60) # noise reading
    
        # Let's calculate the health based on the current readings
        temp_health  = 10
        humi_health  = 10
        noise_health = 10
    
        if i > 40: temp_health -= 2
        if i > 30: temp_health -= 1
        if j > 70: humi_health -= 1
        if j > 60: humi_health -= 1
        if k > 50: noise_health -= 1
    
        # Now let's pick the metric with the smallest value
        health = min(temp_health, humi_health, noise_health)
    
        data = 'lab Temperature={},Humidity={},Noise={},Health={}'.format(i,j,k,health)
        print str(x).zfill(4) + " : " + data
        client.write([data],{'db':inf_db},204,'line')
    
        time.sleep(5)
    

    As before you could put more weight on a given metric if a bad situation on that subsystem tends to produce more critical situations. In our example you can see how we are discounting more health points in Temperature than the other 2 metrics

    This is a sample output of the script

    Notice how the last 2 intervals produce the same overall health status of 9 based on very different conditions. In interval “0006” the Temperature threshold was exceeded. Whereas in “0007” it was the humidity threshold that determined the “Health” value.

    I hope it helps!

  • Grafana dashboard Hierarchy

    It took a while to decide the title of this post but I am still unsure whether it conveys the purpose of the post. The point is that we all start our Grafana journey by creating some cool graphs in a dashboard, but after a while we typically end up with many many dashboards … so eventually we start looking for an at-a-glance view that summarizes all our dashboards. Think of one of those dashboards that use at the operations centers

    We are going to explore some techniques that enable you to build such a hierarchy in Grafana:

    The objective is to have a top level dashboard that summarizes all the others. We also need a way of bringing up those other second level dashboards if we want more detail about a specific system. In this section we will see how to create links in Grafana.

    In my environment I have 2 dashboards:

    • Top. This is my pretend top level dashboard that will contain the at-a-glance view of all my monitored systems. This dashboard is likely to contain “Stat” panels (and maybe gauges) showing the overall status of multiple systems
    • Second. This is a dashboard that contains details about a specific system and as such is likely to contain time series, chart panels and other sophisticated visualizations

    In Grafana you can create links at the panel level and the dashboard level

    The main idea here is that if I click on one of the stat panels it will take me to the second level dashboard where I can see all the details. Maybe the stat panel is showing a red colored “CRITICAL” message and by clicking on it I can go to the second level and see what subsystem is causing the issue.

    In my “Top” level dashboard I have currently a single Stat panel that shows us the health of a certain location, a “warehouse” in this case. There will be some metrics that we aggregate to produce this “health”. In the next article we will explore how to do that. For now this is how it looks:

    Dashboards in Grafana are displayed in a browser by using their URL. This includes a unique ID as well as the actual name as you can see in the following screenshot. You will need to record the URL of the “second” level dashboard so go ahead, open the “second” level dashboard and record its URL.

    Once you have the URL you can go to the Top level dashboard and define a link on the “Warehouse” health stat panel. Then go to the Options section in the right pane and select “Panel links” and then “Add link”

    Here you can add a title for your link and the URL of the second level dashboard we save in the previous step. Optionally you can choose to open the second dashboard in a new tab. Click “Save” and get out of Edit mode

    Now, in the “Top” level dashboard you can see a little arrow icon on the top-left corner of the Stat panel. If you hover your mouse over it you will see the title of the link you provided. And when you click on it it will open the second level dashboard

    The other alternative is to create dashboard links. These will permanently display at the top-right corner of your dashboard.

    Let’s go ahead and create 2 dashboard links:

    • Link to other dashboards
    • Link to an external site

    Start by opening the dashboard settings. You will find the icon at the top-right corner of the dashboard

    Then click on “Links” on the left and then “New Link”. Let’s create a link called “Dashboards”. The “Title” is the string that will be clickable. For “Type” we select “Dashboards”. If you only have a handful of dashboards they will all display in the same line. However if you are planning to have many, it’s better to tick the “Show as dropdown” checkbox to things neater.

    Considering this “Top” level dashboard is likely to show in an operations center, another use case would be to have handy some links related to support, such as an internal ticketing system, a vendor support site or a list of emergency contacts. Now let’s go ahead and create another link that links to the support page of a vendor so that we can open a service request. This time we will select the “Link” type and provide the URL to open when clicked. You can customize the link by selecting an “icon” that is meaningful for your use case. Notice how I have also force it to open the linked page in a “new tab” as we want our dashboard to remain open after we are done with the service request

    This is what the “Links” menu in dashboard “Settings” looks like with both links configured

    Finally “Save Dashboard” and once in the dashboard you should see something like this. Notice how I have clicked in the “Dashboards” button and the existing dashboards (which is only “second” in this case) are shown

    In the next article we will focus on how to create aggregate multiple metrics into a overall health number.

  • Grafana Stat panel with a String

    Grafana has become a very popular tool for monitoring systems and applications partly due to the amount of features and integrations it provides. The graphs it produces are beautiful. However sometimes you need to show a string or a value instead of a full graph. For that purpose Grafana provides the “Stat” panels. These are very often used to show a single value, such as the last reading or the moving average of a given metric.

    However sometimes you want to display a string. An example of this could be a “health” status, such as OK or CRITICAL

    or it could be some some other message you are collecting such as this

    We are going to show how to handle this scenario but for completeness let’s use some Python code. In this example we will be collecting some environment data and storing it into InfluxDB. On every interval we are going to:

    • read 2 numerical values (Temperature and Humidity)
    • infer a “Health” text value by comparing the 2 previous values against some threshold
    • write all 3 values to the database

    The code requires you to install the “influxdb” Python library. You can do so with “pip install influxdb”

    # Import the Influx client from the Python library
    from influxdb import InfluxDBClient
    
    # Create a connection to InfluxDB
    client = InfluxDBClient(host='localhost', port=8086)
    
    # Connect to the right database
    inf_db = "iot_database"
    client.switch_database(inf_db)
    
    while True:
        # Read the data from your sensors or a REST API ...
        # Let's say we got:
        temp = 25
        hum  = 60
    
        # Depending on some predefined thresholds we could derive the Health
        if temp < 35 and hum < 80:
            h = "OK"
        else:
            h = "CRITICAL"
    
        # Then we write the data
        data = 'warehouse Temperature={},Humidity={},Health={}'.format(temp,hum,h)
        client.write([data],{'db':inf_db},204,'line')
    
        # Pause for 5 seconds until the next iteration
        time.sleep(5)

    Now let’s see how we can use the “Health” text value in Grafana. Add a “Stat” panel to your dashboard and on the query section you need to “format as table” and select the “last” value.

    At this point the panel will still display “No data”. So you need to go to the “Options” section on the right pane and scroll down to “Fields”. By default, “Numeric Fields” will be selected, but you need to select “last”

    Additionally you might want to set “Graph mode” to “None”. By default Stat panels will want to show a graph as well as the value but in this case there is no graph anyway because the “Health” field is not numerical.

    Finally you might want to apply a color code to the text in your “Stat”. Unfortunately threshold-based coloring won’t work because Health contains not numerical values. Don’t despair, we can still color them to our liking by using “Value mappings”. You will find this option at the very bottom of the “Options” pane.

    You can add a “new value mapping” for each type of health status your code is producing, ex: OK, CRITICAL … Specify the value to match on the left and the color on the right as shown above

    In our simple code we had only 2 different health status hence our resulting value mappings is as follows

    So now when the sensor data is above the threshold we write Health = CRITICAL and it displays as follows:

    In a future post I will share some tips to build a hierarchy of dashboards including a front dashboard to be used in your control center with a visual snapshot of your entire operation

  • Creating ServiceNow Incidents via REST API

    Creating ServiceNow Incidents via REST API

    In this article we will explore how to create incidents in ServiceNow using the REST API. This article was a stepping stone for this video that shows how to integrate ServiceNow, Microsoft Teams and alerts from infrastructure. You might also be interested in the second post in this series: ServiceNow CMDB REST API tutorial

    The very first thing to do when working with a REST API is to get your hands on the reference guide and hopefully the getting started guide if there is one. The first thing to look for is how to authenticate with the API and whether there are any requirements for special headers or things like that. Afterwards things tend to flow faster and easier. Fortunately, ServiceNow supports basic authentication so there is no learning curve there, although it does support more secure authentication through OAuth if you need it

    In terms of documentation the online help is great. Here you can get an overview of the API. And this is the starting point for the online REST API reference. There are many branches or child API’s hanging of this root. In the screenshot below you can see how, for every call, the online help shows the URL (default or for a specific version) and parameters (path, query and request body). Further down it shows headers for the request and the response and even two coding examples for curl and python … as complete as it gets

    But the tool you will learn to love very quickly is the “REST API Explorer”. Please note that this is a tool that you can only access from within your instance. Once you log in go to “System Web Services” and then locate “REST API Explorer” as shown here

    You will then end up with a menu like this. Notice how you can select the specific child API in the top-left corner and the version of your environment.

    ServiceNow stores all data in tables. This is also true for Incidents which unsurprisingly are stored in the “incident” table. To manipulate tables we need to use the Table API. Different HTTP methods will enable us to do the various CRUD operations. For example if we want to create an incident we will have to use the POST method. Notice in the previous image how I have selected the “Table API” and within that API the “Create a record (POST)” call. Then on the right I have selected the “incident” table.

    When you scroll down you can see a dialog that allows you to build the request body for the incident creation. With the drop-down menu you can select from all the available fields. As you select new fields and assign values, the REST API Explorer builds the body for you in the text box immediately below. At the very bottom you can generate code in multiple languages

    We have grouped the API calls we are using in this article into a Postman collection. You can download the collection from the following the following GitHub repo :

    https://github.com/cermegno/postman-servicenow

    The collection uses 2 variables that you must add to an environment. If you are new to Postman environments check out this older article:

    • {{pwd}}. This is the password for the “admin” user of your ServiceNow instance. If you need to use a different user, you can change it in the collection settings
    • {{instance}}. This is your ServiceNow instance name, i.e. excluding the “.service-now.com” suffix. If you don’t have one or you cannot test this with your production instance, you can open your very own developer instance with ServiceNow

    The collection provides 4 calls and a saved example for each call:

    • Get details for all incidents. This will produce a 98 line JSON structure for each incident
    • Get details for a single incident. This requires you to pass the “sys_id” of the incident as part of the URL as shown below. You can get the “sys_id” for a specific incident from the body of the response during the creation (POST) operation
    • Create incident (POST). The JSON Body parameter can take “a lot” of fields but you can start small. For example the following Body produces the incident below:
    • Modify incident (PUT). This will allow you to make changes to incidents. In particular you can use it to resolve/close incidents by setting the state to “7” as shown in the screenshot below. This call also requires you to pass the “sys_id” of the incident you are modifying as part of the URL

    In this article we have used the REST API to interact with ServiceNow because this is the way I will do it in the upcoming video demo. But depending on what you are trying to do you might want to use Ansible. In that case you can use the official Ansible modules provided by ServiceNow themselves. The collection is available in Ansible Galaxy and provides just two modules designed to interact with ServiceNow tables. Follow the instructions in the Ansible Galaxy page to install the dependencies including the “pysnow” Python library

    As a next step you can visit the second post in this series: ServiceNow CMDB REST API tutorial

    We hope you find this article helpful. Let us know your thoughts in the comment section.