Tag: docker

Keep a container alive to troubleshoot it
Containers are a great way to package code and dependencies in a portable package. This is fantastic in a world where dependencies keep getting bigger and more complex. But we are not here to discuss their benefits. If Google brought you here, more than likely you know already what a container is and you are trying to troubleshoot one. If that’s the case, I hope you find useful the tip I am about to tell you.

To understand the solution we must first remember that containers are designed to run a task as soon as they start. You define such task with either CMD or ENTRYPOINT. The problem is that if the task dies the container dies too and you can’t log in into it to find out what happened. Sometimes “docker logs” will give you a hint of what’s happening but other times you wish you could simply log in to the container and check things out, but of course you can’t because the container is dead

A solution I have become fond of is to override the ENTRYPOINT with a process that keeps the container alive and then open a terminal session into it. Let me give you an example.

IMPORTANT: I am using Podman but all of this will work perfectly fine with Docker as well. This is the “Dockerfile“. Notice how the main task for this container is “python app.py“
```
FROM python:3.10.5
WORKDIR /app
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 7860
CMD ["python", "app.py"]
```
This is the code contained in “app.py“. It is a simple Gradio chatbot. It gets the users prompt and sends an API call to to get a response.
```
import gradio as gr
import os, requests, import urllib3
urllib3.disable_warnings()

appurl = os.environ["APP_URL"]

def give_response(query, history):
    payload = {"query": query}
    response = requests.post(appurl, json=payload)
    return response.json()["response"]

demo = gr.ChatInterface(
              give_response,
              type = "messages",
              title="My first Chatbot",
              description="Ask me a question, don't be shy")

demo.launch(server_name="0.0.0.0")
```
As you can see the code requires us to define an environment variable “APP_URL” with the URL to send the request to. Let’s say that we don’t define the variable and attempt to run the container.
```
pi@piper1:~$ podman run -d -p 7860:7860 localhost/blog:v1
19dc02db0b2e06c51756c7f55a3eb861c11f831a21574efdcd2e00081c9191a1
pi@piper1:~$ podman ps -a
CONTAINER ID  IMAGE              COMMAND        CREATED        STATUS      NAMES
19dc02db0b2e  localhost/blog:v1  python app.py  3 seconds ago  Exited (1)  dry_rice
```
As expected the container fails but here is the trick … we run it again but we use the “–entrypoint” argument to run “tail -f /dev/null“. The container is not changing we are simply overriding “python app.py” with this “tail” command which stays running and in doing so keeps the container alive. Notice the “single quote” around the square brackets.
```
pi@piper1:~$ podman run -d --entrypoint='["tail", "-f", "/dev/null"]' -p 7860:7860 localhost/blog:v1
e1031d4c2ecbf3d4ec244aef262e71fb2f4e227154b6e22eabf634e4e053f4a0
pi@piper1:~$ podman ps
CONTAINER ID  IMAGE              STATUS        PORTS                   NAMES
e1031d4c2ecb  localhost/blog:v1  Up 6 seconds  0.0.0.0:7860->7860/tcp  sour_soup
```
Now we can login into the container and do whatever checks we need to do. We can even run “python app.py” and see what’s going on live.
```
pi@piper1:~$ podman exec -it e1031d4c2ecb /bin/bash
root@e1031d4c2ecb:/app# ls -l
total 12
-rw-r--r-- 1 root root 518 Mar 24 04:35 Dockerfile
-rw-r--r-- 1 root root 566 Mar 24 04:35 app.py
-rw-r--r-- 1 root root 897 Mar 24 04:35 requirements.txt
root@e1031d4c2ecb:/app# python3 app.py
Traceback (most recent call last):
  File "/app/app.py", line 7, in <module>
    appurl = os.environ["APP_URL"]
  File "/usr/local/lib/python3.10/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'APP_URL'
```
This was a simplistic example but you get point. There could be a file or a path missing, a typo, a permissions issue … by using this trick you can troubleshoot interactively inside the container. Once you know what the issue is you can fix your files and rebuild the container image.

The next question is, can you do this in Kubernetes as well? Yes, you can add the “command” to the “Deployment”, not to the “Pod”. This is a list of strings as you see in the last line in the simplified manifest below.
```
apiVersion: v1
kind: Deployment
metadata:
  name: blog
spec:
  containers:
  - name: blog-container
    image: debian
    command: ["tail", "-f", "/dev/null"]
```
As soon as I save the changes to the deployment manifest, Openshift creates a new pod and kills the old one. Then I can terminal into the new pod and browse around. Notice I can even launch the application from the terminal session.
```
$ pwd         
/app
$ ls -l
total 12
-rw-r--r--. 1 root root 518 Mar 24 04:35 Dockerfile
-rw-r--r--. 1 root root 566 Mar 24 04:35 app.py
-rw-r--r--. 1 root root 897 Mar 24 04:35 requirements.txt
$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
1000770+       1       0  0 00:00 ?        00:00:00 tail -f /dev/null
1000770+       7       0  0 00:00 pts/0    00:00:00 sh -i -c TERM=xterm sh
1000770+      13       7  0 00:00 pts/0    00:00:00 sh
1000770+      90      13  0 00:02 pts/0    00:00:00 ps -ef

$ python3 app.py
* Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
```
Of course, this is only intended for troubleshooting. Once you find out what’s wrong you can fix the image and manifest and deploy them again.
Spread The Word:
Tweet
Email
WhatsApp
More
Print
Reddit
Like this:
Like Loading…
May 8, 2025

Podman build fails with /etc/passwd: permission denied

A short post today to share an issue that troubled me for a while. Hopefully it can help you too.

Basically I was trying to build a new container image using Podman in Ubuntu. These are my versions:

Ubuntu 24.04.2 LTS
podman version 4.9.3

My Dockerfile is straight forward

FROM python:3.10.5
WORKDIR /app
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt
CMD ["python", "app.py"]

When I tried to create the container image I get the following output

(env) user1@mytpl:~/Desktop/myapp$ podman build -t myapp:v1 .
STEP 1/6: FROM python:3.10.5
Resolved "python" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/python:3.10.5...
Getting image source signatures
Copying blob 588423e31bcf done   |
Copying blob 001c52e26ad5 done   |
Copying blob 2068746827ec done   |
Copying blob d9d4b9b6e964 done   |
Copying blob 8a335986117b done   |
Copying blob 9daef329d350 done   |
Copying blob ecb6a3f01c0d done   |
Copying blob 00d40f20f0cf done   |
Copying blob 5d588b4f3b55 done   |
Error: creating build container: copying system image from manifest list: writing blob: adding layer with blob "sha256:001c52e26ad57e3b25b439ee0052f66           92e5c0f2d5d982a00a8819ace5e521452": processing tar file(open /etc/passwd: permission denied): exit status 1

In the end I tracked it down to a service called “mfetpd.service” which appears to be some Trellix endpoint security software.

(env) user1@mytpl:~/Desktop/myapp$ sudo systemctl status mfetpd.service
● mfetpd.service - Trellix Endpoint Security for Linux Threat Prevention
     Loaded: loaded (/usr/lib/systemd/system/mfetpd.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-03-24 14:35:44 +08; 5min ago
       Docs: man:mfetpd(8)
    Process: 30546 ExecStartPre=/opt/McAfee/ens/tp/scripts/aac-control-wrapper.sh systemd (code=exited, status=0/SUCCESS

The solution was to temporarily stop this service to build the container image

sudo systemctl stop mfetpd.service
podman build ...
sudo systemctl start mfetpd.service

Once the image “python:3.10.5” layer is downloaded I am able to build more containers using the same version of Python without having to pause the “mfetpd” service which makes my security team happy.

I hope this helps you same some of your precious time!

May 7, 2025

Ansible with DellEMC Storage: Part 7 – Install PowerStore Collection on AWX/Tower
This blog is the continuation of Ansible with DellEMC storage multi-part blog.

In the last (6th Part) of this blog series, we discussed how to prepare Ansible Tower/AWX with Dell EMC storage credentials.

In this blog post, we will install Dell EMC PowerStore collection on Ansible AWX and go through the next steps.

As we all know that Ansible has moved to Collections – a new ways of managing integrations and content management. Dell EMC has already started working towards this and have released several collections for multiple Dell EMC portfolio products, few of which are listed below.
- Dell EMC Ansible Modules collections for
Apart from this list you can find other Dell portfolio collections (like OpenManage) on this link

For the scope of this blog post we will focus on installing Dell EMC PowerStore Ansible collection on Ansible AWX. Technically, all the collections can be installed using similar steps.

As a Pre-Requisite, this blog post assumes that you have –
- Ansible AWX installed and running
- Access to operating system / machine having Ansible AWX installed
- Access to PowerStore storage system (with credentials)
Additionally, if you’re getting started with Ansible AWX and/or integration with Dell EMC’s storage products then you can follow this blog series to get started from scratch.

As part of the installation collection installation steps we need to Ansible AWX machine and then connect to the awx_task docker container.

Login to the AWX machine. You can list the running AWX containers using below command
```
[root@awx ~]# docker container list
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                  NAMES
6ced2eccbd7b        ansible/awx_task:11.2.0   "tini -- /bin/sh -c …"   13 months ago       Up 6 days           8052/tcp               awx_task
42b14fbd15ad        ansible/awx_web:11.2.0    "tini -- /bin/sh -c …"   13 months ago       Up 6 days           0.0.0.0:80->8052/tcp   awx_web
4c08c0e39128        memcached:alpine          "docker-entrypoint.s…"   13 months ago       Up 6 days           11211/tcp              awx_memcached
42224676c21a        redis                     "docker-entrypoint.s…"   13 months ago       Up 6 days           6379/tcp               awx_redis
37d0ca0c67bc        postgres:10               "docker-entrypoint.s…"   13 months ago       Up 6 days           5432/tcp               awx_postgres
```
Then connect to the awx_task container using below command
```
# docker exec -it awx_task bash
```
Next, install the PowerStore Ansible Modules collection in awx_task container
```
# ansible-galaxy collection install dellemc.powerstore
Process install dependency map
Starting collection install process
Installing 'dellemc.powerstore:1.2.1' to '/home/awx/.ansible/collections/ansible_collections/dellemc/powerstore'
```
Then logout from the container.
```
bash-4.4# exit
```
Now you have successfully installed the Dell EMC PowerStore Ansible Modules collection. Next step post installing collection are
- Create the PowerStore credentials on the Ansible AWX
  - For credentials creation you can follow our earlier blog post
- Create PowerStore Project – assuming you’ve PowerStore playbooks on content repo (like Git)
- Configure Ansible AWX Job template / Workflow template for storage task automation
All Dell EMC’s published collections comes with sample playbooks to test the functionality and also to get you started with integrations. When it comes to PowerStore you can see them under /home/awx/ansible-powerstore/dellemc_ansible/powerstore/samples directory
```
# cd /home/awx/ansible-powerstore/dellemc_ansible/powerstore/samples
# ls -l
-rw-r--r-- 1 root root 1892 Jun  4  2020 capacity_volumes.yml
-rw-r--r-- 1 root root 1042 Jun  4  2020 create_multiple_volumes_async.yml
-rw-r--r-- 1 root root  799 Jun  4  2020 create_multiple_volumes.yml
-rw-r--r-- 1 root root  790 Jun  4  2020 delete_multiple_volumes.yml
-rw-r--r-- 1 root root 1710 Jun  4  2020 find_empty_volume_groups.yml
-rw-r--r-- 1 root root 1141 Jun  4  2020 search_volumes.yml
```
You can re-use these sample playbooks to quickly get started with storage automation tasks. Sample playbooks in the collection has multiple variables like –
- array_ip
- user
- password
- verifycert
You can capture the storage credentials by creating Dell EMC storage credential type (screenshot below)

Ansible AWX – Dell EMC Storage Credential Type

Once Dell Storage credential type is created then you can add PowerStore array details and credentials using AWX credential manager.

Ansible AWX – Dell EMC Storage Credential

After adding PowerStore credential you can use the same in the AWX automation job template creation. Additional variables (like volume names, size, host etc.) can be captured using extra_vars

Ansible AWX – Job Template Creation

Additionally you can create survey to capture the required variables and also workflow visualizer to create multi-step breakdown of the automation tasks including but not limited storage automation. Below is the example of breaking down the storage provisioning workflow in the logical steps (like approval, quota management, provisioning, etc.)

Ansible AWX – Workflow Visualizer

Hope this helps everyone.

Update: Please note that the latest version of AWX has moved to Kubernetes (instead of Docker). Please use the below steps to install the PowerStore modules.
```
[root@awx ~]# kubectl -n awx exec -it awx-844c574f84-bc4ww -c awx-ee -- /bin/bash
bash-4.4$ ansible-galaxy collection install dellemc.powerstore -c
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/download/dellemc-powerstore-1.6.0.tar.gz to /home/runner/.ansible/tmp/ansible-local-3648wbuk4c_/tmprzm8bse6/dellemc-powerstore-1.6.0-ltamh_71
Installing 'dellemc.powerstore:1.6.0' to '/home/runner/.ansible/collections/ansible_collections/dellemc/powerstore'
dellemc.powerstore:1.6.0 was installed successfully
bash-4.4$
```
Spread The Word:
Tweet
Email
WhatsApp
More
Print
Reddit
Like this:
Like Loading…
July 4, 2021

Installation of AWX using Ansible fails with error – “Unable to load docker-compose. Try `pip install docker-compose`, ImportError: No module named zipp, ImportError: No module named configparser

Recently while I was trying to install AWX using Ansible I came across below error on Centos.

Docker-compose error while installing AWX

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to load docker-compose. Try `pip install docker-compose`. Error: Traceback (most recent call last):\n  File \"/tmp/ansible_docker_compose_payload_jNhEZ8/ansible_docker_compose_payload.zip/ansible/modules/cloud/docker/docker_compose.py\", line 483, in <module>\n  File \"/usr/lib/python2.7/site-packages/compose/cli/command.py\", line 12, in <module>\n    from .. import config\n  File \"/usr/lib/python2.7/site-packages/compose/config/__init__.py\", line 6, in <module>\n    from .config import ConfigurationError\n  File \"/usr/lib/python2.7/site-packages/compose/config/config.py\", line 51, in <module>\n    from .validation import match_named_volumes\n  File \"/usr/lib/python2.7/site-packages/compose/config/validation.py\", line 12, in <module>\n    from jsonschema import Draft4Validator\n  File \"/usr/lib/python2.7/site-packages/jsonschema/__init__.py\", line 33, in <module>\n    import importlib_metadata as metadata\n  File \"/usr/lib/python2.7/site-packages/importlib_metadata/__init__.py\", line 9, in <module>\n    import zipp\nImportError: No module named zipp\n"}

While this error was referring to missing pip package “docker-compose”, but when I tried to install the same it gave below error stating that it’s already installed.

[root@dw-test-1 installer]# pip install docker-compose
Requirement already satisfied (use --upgrade to upgrade): docker-compose in /usr/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): texttable<2,>=0.9.0 in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): requests<3,>=2.20.0 in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): dockerpty<1,>=0.4.1 in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): six<2,>=1.3.0 in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): docopt<1,>=0.6.1 in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): ipaddress<2,>=1.0.16; python_version < "3.3" in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): subprocess32<4,>=3.5.4; python_version < "3.2" in /usr/lib64/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): enum34<2,>=1.0.4; python_version < "3.4" in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): websocket-client<1,>=0.32.0 in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): jsonschema<4,>=2.5.1 in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): backports.shutil-get-terminal-size==1.0.0; python_version < "3.3" in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): backports.ssl-match-hostname<4,>=3.5; python_version < "3.5" in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): cached-property<2,>=1.2.0 in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): PyYAML<6,>=3.10 in /usr/lib64/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): docker[ssh]<5,>=3.7.0 in /usr/lib/python2.7/site-packages (from docker-compose)
Requirement already satisfied (use --upgrade to upgrade): urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): idna<3,>=2.5 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): chardet<4,>=3.0.2 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): certifi>=2017.4.17 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): pyrsistent>=0.14.0 in /usr/lib64/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): attrs>=17.4.0 in /usr/lib/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): importlib-metadata; python_version < "3.8" in /usr/lib/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): functools32; python_version < "3" in /usr/lib/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): paramiko>=2.4.2; extra == "ssh" in /usr/lib/python2.7/site-packages (from docker[ssh]<5,>=3.7.0->docker-compose)
Collecting configparser>=3.5; python_version < "3" (from importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose)
  Using cached https://files.pythonhosted.org/packages/e5/7c/d4ccbcde76b4eea8cbd73b67b88c72578e8b4944d1270021596e80b13deb/configparser-5.0.0.tar.gz
  Running setup.py (path:/tmp/pip-build-c4RmaK/configparser/setup.py) egg_info for package configparser produced metadata for project name unknown. Fix your #egg=configparser fragments.
  Requirement already satisfied (use --upgrade to upgrade): unknown from https://files.pythonhosted.org/packages/e5/7c/d4ccbcde76b4eea8cbd73b67b88c72578e8b4944d1270021596e80b13deb/configparser-5.0.0.tar.gz#sha256=2ca44140ee259b5e3d8aaf47c79c36a7ab0d5e94d70bd4105c03ede7a20ea5a1 in /usr/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose)
Collecting zipp>=0.5 (from importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose)
  Using cached https://files.pythonhosted.org/packages/ce/8c/2c5f7dc1b418f659d36c04dec9446612fc7b45c8095cc7369dd772513055/zipp-3.1.0.tar.gz
  Running setup.py (path:/tmp/pip-build-c4RmaK/zipp/setup.py) egg_info for package zipp produced metadata for project name unknown. Fix your #egg=zipp fragments.
  Requirement already satisfied (use --upgrade to upgrade): unknown from https://files.pythonhosted.org/packages/ce/8c/2c5f7dc1b418f659d36c04dec9446612fc7b45c8095cc7369dd772513055/zipp-3.1.0.tar.gz#sha256=c599e4d75c98f6798c509911d08a22e6c021d074469042177c8c86fb92eefd96 in /usr/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): contextlib2; python_version < "3" in /usr/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): pathlib2; python_version < "3" in /usr/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): pynacl>=1.0.1 in /usr/lib64/python2.7/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): cryptography>=2.5 in /usr/lib64/python2.7/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): bcrypt>=3.1.3 in /usr/lib64/python2.7/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): scandir; python_version < "3.5" in /usr/lib64/python2.7/site-packages (from pathlib2; python_version < "3"->importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in /usr/lib64/python2.7/site-packages (from pynacl>=1.0.1->paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.4.1->pynacl>=1.0.1->paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose)
You are using pip version 8.1.2, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

After trying multiple solutions on several forums and reinstalling everything from scratch finally I thought of upgrading pip.

[root@dw-test-1 installer]# pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 787kB/s
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
      Successfully uninstalled pip-8.1.2
Successfully installed pip-20.0.2

Post upgrading pip to latest version I reinstalled docker-compose.

[root@dw-test-1 installer]# pip install docker-compose
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: docker-compose in /usr/lib/python2.7/site-packages (1.25.5)
Requirement already satisfied: backports.shutil-get-terminal-size==1.0.0; python_version < "3.3" in /usr/lib/python2.7/site-packages (from docker-compose) (1.0.0)
Requirement already satisfied: six<2,>=1.3.0 in /usr/lib/python2.7/site-packages (from docker-compose) (1.9.0)
Requirement already satisfied: PyYAML<6,>=3.10 in /usr/lib64/python2.7/site-packages (from docker-compose) (3.10)
Requirement already satisfied: docker[ssh]<5,>=3.7.0 in /usr/lib/python2.7/site-packages (from docker-compose) (4.2.0)
Requirement already satisfied: dockerpty<1,>=0.4.1 in /usr/lib/python2.7/site-packages (from docker-compose) (0.4.1)
Requirement already satisfied: jsonschema<4,>=2.5.1 in /usr/lib/python2.7/site-packages (from docker-compose) (3.2.0)
Requirement already satisfied: requests<3,>=2.20.0 in /usr/lib/python2.7/site-packages (from docker-compose) (2.23.0)
Requirement already satisfied: enum34<2,>=1.0.4; python_version < "3.4" in /usr/lib/python2.7/site-packages (from docker-compose) (1.0.4)
Requirement already satisfied: websocket-client<1,>=0.32.0 in /usr/lib/python2.7/site-packages (from docker-compose) (0.57.0)
Requirement already satisfied: cached-property<2,>=1.2.0 in /usr/lib/python2.7/site-packages (from docker-compose) (1.5.1)
Requirement already satisfied: ipaddress<2,>=1.0.16; python_version < "3.3" in /usr/lib/python2.7/site-packages (from docker-compose) (1.0.16)
Requirement already satisfied: docopt<1,>=0.6.1 in /usr/lib/python2.7/site-packages (from docker-compose) (0.6.2)
Requirement already satisfied: subprocess32<4,>=3.5.4; python_version < "3.2" in /usr/lib64/python2.7/site-packages (from docker-compose) (3.5.4)
Requirement already satisfied: texttable<2,>=0.9.0 in /usr/lib/python2.7/site-packages (from docker-compose) (1.6.2)
Requirement already satisfied: backports.ssl-match-hostname<4,>=3.5; python_version < "3.5" in /usr/lib/python2.7/site-packages (from docker-compose) (3.5.0.1)
Requirement already satisfied: paramiko>=2.4.2; extra == "ssh" in /usr/lib/python2.7/site-packages (from docker[ssh]<5,>=3.7.0->docker-compose) (2.7.1)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose) (0.9.8)
Requirement already satisfied: pyrsistent>=0.14.0 in /usr/lib64/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose) (0.16.0)
Requirement already satisfied: attrs>=17.4.0 in /usr/lib/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose) (19.3.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/lib/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose) (1.6.0)
Requirement already satisfied: functools32; python_version < "3" in /usr/lib/python2.7/site-packages (from jsonschema<4,>=2.5.1->docker-compose) (3.2.3.post2)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->docker-compose) (2.9)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->docker-compose) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->docker-compose) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->docker-compose) (2020.4.5.1)
Requirement already satisfied: bcrypt>=3.1.3 in /usr/lib64/python2.7/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose) (3.1.7)
Requirement already satisfied: pynacl>=1.0.1 in /usr/lib64/python2.7/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose) (1.3.0)
Requirement already satisfied: cryptography>=2.5 in /usr/lib64/python2.7/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose) (2.9)
Requirement already satisfied: pathlib2; python_version < "3" in /usr/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose) (2.3.5)
Requirement already satisfied: contextlib2; python_version < "3" in /usr/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose) (0.6.0.post1)
Collecting zipp>=0.5
  Downloading zipp-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting configparser>=3.5; python_version < "3"
  Downloading configparser-4.0.2-py2.py3-none-any.whl (22 kB)
Requirement already satisfied: cffi>=1.1 in /usr/lib64/python2.7/site-packages (from bcrypt>=3.1.3->paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose) (1.14.0)
Requirement already satisfied: scandir; python_version < "3.5" in /usr/lib64/python2.7/site-packages (from pathlib2; python_version < "3"->importlib-metadata; python_version < "3.8"->jsonschema<4,>=2.5.1->docker-compose) (1.10.0)
Requirement already satisfied: pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko>=2.4.2; extra == "ssh"->docker[ssh]<5,>=3.7.0->docker-compose) (2.14)
Installing collected packages: zipp, configparser
Successfully installed configparser-4.0.2 zipp-1.2.0

This time zipp and configparser packages also got installed with docker-compose. This resolved my issue and I was able to go ahead with successful AWX installation.

In summary –

First upgrade pip to latest version
Then install docker-compose. Make sure zipp and configparser are getting installed with the same.

Cheers and happy automating!

April 12, 2020

Installation of docker fails on CentOS 8 with Error – package containerd.io-1.2.10-3.2.el7.x86_64 is excluded

I recently came across a strange issue while installing Docker on CentOS (version 8) machine.

Issue –

When I tried below options to install docker on my machine I got the same error every time.

Installation Commands Used

yum install -y -q docker-ce
or
yum install -y docker-ce.x86_64
or
yum install docker-ce

Error message received

Error:
 Problem: package docker-ce-3:19.03.8-3.el7.x86_64 requires containerd.io >= 1.2.2-3, but none of the providers can be installed
  - cannot install the best candidate for the job
  - package containerd.io-1.2.10-3.2.el7.x86_64 is excluded
  - package containerd.io-1.2.13-3.1.el7.x86_64 is excluded
  - package containerd.io-1.2.2-3.3.el7.x86_64 is excluded
  - package containerd.io-1.2.2-3.el7.x86_64 is excluded
  - package containerd.io-1.2.4-3.1.el7.x86_64 is excluded
  - package containerd.io-1.2.5-3.1.el7.x86_64 is excluded
  - package containerd.io-1.2.6-3.3.el7.x86_64 is excluded

Resolution –

To resolve this issue, first we need to manually install the containerd.io package.

[root@test_centos8 /]# yum install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
Last metadata expiration check: 0:02:53 ago on Thu 02 Apr 2020 09:29:41 AM UTC.
containerd.io-1.2.6-3.3.el7.x86_64.rpm                                                                                                                                 18 MB/s |  26 MB     00:01
Dependencies resolved.
======================================================================================================================================================================================================
 Package                                               Architecture                    Version                                                            Repository                             Size
======================================================================================================================================================================================================
Installing:
 containerd.io                                         x86_64                          1.2.6-3.3.el7                                                      @commandline                           26 M
Installing dependencies:
 container-selinux                                     noarch                          2:2.124.0-1.module_el8.1.0+272+3e64ee36                            AppStream                              47 k
 checkpolicy                                           x86_64                          2.9-1.el8                                                          BaseOS                                348 k
 libselinux-utils                                      x86_64                          2.9-2.1.el8                                                        BaseOS                                243 k
 policycoreutils                                       x86_64                          2.9-3.el8_1.1                                                      BaseOS                                377 k
 policycoreutils-python-utils                          noarch                          2.9-3.el8_1.1                                                      BaseOS                                250 k
 python3-audit                                         x86_64                          3.0-0.13.20190507gitf58ec40.el8                                    BaseOS                                 85 k
 python3-libselinux                                    x86_64                          2.9-2.1.el8                                                        BaseOS                                283 k
 python3-libsemanage                                   x86_64                          2.9-1.el8                                                          BaseOS                                127 k
 python3-policycoreutils                               noarch                          2.9-3.el8_1.1                                                      BaseOS                                2.2 M
 python3-setools                                       x86_64                          4.2.2-1.el8                                                        BaseOS                                600 k
 rpm-plugin-selinux                                    x86_64                          4.14.2-25.el8                                                      BaseOS                                 73 k
 selinux-policy                                        noarch                          3.14.3-20.el8                                                      BaseOS                                602 k
 selinux-policy-targeted                               noarch                          3.14.3-20.el8                                                      BaseOS                                 15 M
Enabling module streams:
 container-tools                                                                       rhel8

Transaction Summary
======================================================================================================================================================================================================
Install  14 Packages

Total size: 46 M
Total download size: 20 M
Installed size: 158 M
Downloading Packages:
(1/13): libselinux-utils-2.9-2.1.el8.x86_64.rpm                                                                                                                       2.4 MB/s | 243 kB     00:00
(2/13): container-selinux-2.124.0-1.module_el8.1.0+272+3e64ee36.noarch.rpm                                                                                            375 kB/s |  47 kB     00:00
(3/13): checkpolicy-2.9-1.el8.x86_64.rpm                                                                                                                              2.5 MB/s | 348 kB     00:00
(4/13): python3-audit-3.0-0.13.20190507gitf58ec40.el8.x86_64.rpm                                                                                                      2.5 MB/s |  85 kB     00:00
(5/13): policycoreutils-2.9-3.el8_1.1.x86_64.rpm                                                                                                                      3.0 MB/s | 377 kB     00:00
(6/13): python3-libsemanage-2.9-1.el8.x86_64.rpm                                                                                                                      2.2 MB/s | 127 kB     00:00
(7/13): policycoreutils-python-utils-2.9-3.el8_1.1.noarch.rpm                                                                                                         685 kB/s | 250 kB     00:00
(8/13): python3-libselinux-2.9-2.1.el8.x86_64.rpm                                                                                                                     810 kB/s | 283 kB     00:00
(9/13): rpm-plugin-selinux-4.14.2-25.el8.x86_64.rpm                                                                                                                   288 kB/s |  73 kB     00:00
(10/13): python3-setools-4.2.2-1.el8.x86_64.rpm                                                                                                                       1.1 MB/s | 600 kB     00:00
(11/13): selinux-policy-3.14.3-20.el8.noarch.rpm                                                                                                                      936 kB/s | 602 kB     00:00
(12/13): python3-policycoreutils-2.9-3.el8_1.1.noarch.rpm                                                                                                             1.3 MB/s | 2.2 MB     00:01
(13/13): selinux-policy-targeted-3.14.3-20.el8.noarch.rpm                                                                                                             3.0 MB/s |  15 MB     00:05
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                 2.8 MB/s |  20 MB     00:07
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                              1/1
  Installing       : python3-libselinux-2.9-2.1.el8.x86_64                                                                                                                                       1/14
  Installing       : libselinux-utils-2.9-2.1.el8.x86_64                                                                                                                                         2/14
  Installing       : policycoreutils-2.9-3.el8_1.1.x86_64                                                                                                                                        3/14
  Running scriptlet: policycoreutils-2.9-3.el8_1.1.x86_64                                                                                                                                        3/14
  Installing       : rpm-plugin-selinux-4.14.2-25.el8.x86_64                                                                                                                                     4/14
  Installing       : selinux-policy-3.14.3-20.el8.noarch                                                                                                                                         5/14
  Running scriptlet: selinux-policy-3.14.3-20.el8.noarch                                                                                                                                         5/14
  Running scriptlet: selinux-policy-targeted-3.14.3-20.el8.noarch                                                                                                                                6/14
  Installing       : selinux-policy-targeted-3.14.3-20.el8.noarch                                                                                                                                6/14
  Running scriptlet: selinux-policy-targeted-3.14.3-20.el8.noarch                                                                                                                                6/14
  Installing       : python3-libsemanage-2.9-1.el8.x86_64                                                                                                                                        7/14
  Installing       : python3-setools-4.2.2-1.el8.x86_64                                                                                                                                          8/14
  Installing       : python3-audit-3.0-0.13.20190507gitf58ec40.el8.x86_64                                                                                                                        9/14
  Installing       : checkpolicy-2.9-1.el8.x86_64                                                                                                                                               10/14
  Installing       : python3-policycoreutils-2.9-3.el8_1.1.noarch                                                                                                                               11/14
  Installing       : policycoreutils-python-utils-2.9-3.el8_1.1.noarch                                                                                                                          12/14
  Running scriptlet: container-selinux-2:2.124.0-1.module_el8.1.0+272+3e64ee36.noarch                                                                                                           13/14
  Installing       : container-selinux-2:2.124.0-1.module_el8.1.0+272+3e64ee36.noarch                                                                                                           13/14
  Running scriptlet: container-selinux-2:2.124.0-1.module_el8.1.0+272+3e64ee36.noarch                                                                                                           13/14
  Installing       : containerd.io-1.2.6-3.3.el7.x86_64                                                                                                                                         14/14
  Running scriptlet: containerd.io-1.2.6-3.3.el7.x86_64                                                                                                                                         14/14
  Running scriptlet: container-selinux-2:2.124.0-1.module_el8.1.0+272+3e64ee36.noarch                                                                                                           14/14
  Running scriptlet: containerd.io-1.2.6-3.3.el7.x86_64                                                                                                                                         14/14
  Verifying        : container-selinux-2:2.124.0-1.module_el8.1.0+272+3e64ee36.noarch                                                                                                            1/14
  Verifying        : checkpolicy-2.9-1.el8.x86_64                                                                                                                                                2/14
  Verifying        : libselinux-utils-2.9-2.1.el8.x86_64                                                                                                                                         3/14
  Verifying        : policycoreutils-2.9-3.el8_1.1.x86_64                                                                                                                                        4/14
  Verifying        : policycoreutils-python-utils-2.9-3.el8_1.1.noarch                                                                                                                           5/14
  Verifying        : python3-audit-3.0-0.13.20190507gitf58ec40.el8.x86_64                                                                                                                        6/14
  Verifying        : python3-libselinux-2.9-2.1.el8.x86_64                                                                                                                                       7/14
  Verifying        : python3-libsemanage-2.9-1.el8.x86_64                                                                                                                                        8/14
  Verifying        : python3-policycoreutils-2.9-3.el8_1.1.noarch                                                                                                                                9/14
  Verifying        : python3-setools-4.2.2-1.el8.x86_64                                                                                                                                         10/14
  Verifying        : rpm-plugin-selinux-4.14.2-25.el8.x86_64                                                                                                                                    11/14
  Verifying        : selinux-policy-3.14.3-20.el8.noarch                                                                                                                                        12/14
  Verifying        : selinux-policy-targeted-3.14.3-20.el8.noarch                                                                                                                               13/14
  Verifying        : containerd.io-1.2.6-3.3.el7.x86_64                                                                                                                                         14/14

Installed:
  containerd.io-1.2.6-3.3.el7.x86_64   container-selinux-2:2.124.0-1.module_el8.1.0+272+3e64ee36.noarch checkpolicy-2.9-1.el8.x86_64                         libselinux-utils-2.9-2.1.el8.x86_64
  policycoreutils-2.9-3.el8_1.1.x86_64 policycoreutils-python-utils-2.9-3.el8_1.1.noarch                python3-audit-3.0-0.13.20190507gitf58ec40.el8.x86_64 python3-libselinux-2.9-2.1.el8.x86_64
  python3-libsemanage-2.9-1.el8.x86_64 python3-policycoreutils-2.9-3.el8_1.1.noarch                     python3-setools-4.2.2-1.el8.x86_64                   rpm-plugin-selinux-4.14.2-25.el8.x86_64
  selinux-policy-3.14.3-20.el8.noarch  selinux-policy-targeted-3.14.3-20.el8.noarch

Complete!

Once containerd.io package is installed you can go ahead and install the docker without any error.

[root@test_centos8 /]# yum install docker-ce
Last metadata expiration check: 0:19:09 ago on Thu 02 Apr 2020 09:29:41 AM UTC.
Dependencies resolved.
======================================================================================================================================================================================================
 Package                                                Architecture                           Version                                         Repository                                        Size
======================================================================================================================================================================================================
Installing:
 docker-ce                                              x86_64                                 3:19.03.8-3.el7                                 docker-ce-stable                                  25 M
Installing dependencies:
 iptables                                               x86_64                                 1.8.2-16.el8                                    BaseOS                                           586 k
 libcgroup                                              x86_64                                 0.41-19.el8                                     BaseOS                                            70 k
 libnetfilter_conntrack                                 x86_64                                 1.0.6-5.el8                                     BaseOS                                            65 k
 libnfnetlink                                           x86_64                                 1.0.1-13.el8                                    BaseOS                                            33 k
 libnftnl                                               x86_64                                 1.1.1-4.el8                                     BaseOS                                            83 k
 docker-ce-cli                                          x86_64                                 1:19.03.8-3.el7                                 docker-ce-stable                                  40 M

Transaction Summary
======================================================================================================================================================================================================
Install  7 Packages

Total download size: 65 M
Installed size: 276 M
Is this ok [y/N]: y
Downloading Packages:
(1/7): libnetfilter_conntrack-1.0.6-5.el8.x86_64.rpm                                                                                                                  1.0 MB/s |  65 kB     00:00
(2/7): libcgroup-0.41-19.el8.x86_64.rpm                                                                                                                               1.0 MB/s |  70 kB     00:00
(3/7): libnfnetlink-1.0.1-13.el8.x86_64.rpm                                                                                                                           1.0 MB/s |  33 kB     00:00
(4/7): libnftnl-1.1.1-4.el8.x86_64.rpm                                                                                                                                2.3 MB/s |  83 kB     00:00
(5/7): iptables-1.8.2-16.el8.x86_64.rpm                                                                                                                               3.3 MB/s | 586 kB     00:00
(6/7): docker-ce-19.03.8-3.el7.x86_64.rpm                                                                                                                             8.2 MB/s |  25 MB     00:02
(7/7): docker-ce-cli-19.03.8-3.el7.x86_64.rpm                                                                                                                          10 MB/s |  40 MB     00:03
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                  14 MB/s |  65 MB     00:04
warning: /var/cache/dnf/docker-ce-stable-091d8a9c23201250/packages/docker-ce-19.03.8-3.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY
Docker CE Stable - x86_64                                                                                                                                              39 kB/s | 1.6 kB     00:00
Importing GPG key 0x621E9F35:
 Userid     : "Docker Release (CE rpm) <docker@docker.com>"
 Fingerprint: 060A 61C5 1B55 8A7F 742B 77AA C52F EB6B 621E 9F35
 From       : https://download.docker.com/linux/centos/gpg
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                              1/1
  Installing       : libnfnetlink-1.0.1-13.el8.x86_64                                                                                                                                             1/7
  Running scriptlet: libnfnetlink-1.0.1-13.el8.x86_64                                                                                                                                             1/7
  Installing       : libnetfilter_conntrack-1.0.6-5.el8.x86_64                                                                                                                                    2/7
  Running scriptlet: libnetfilter_conntrack-1.0.6-5.el8.x86_64                                                                                                                                    2/7
  Installing       : docker-ce-cli-1:19.03.8-3.el7.x86_64                                                                                                                                         3/7
  Running scriptlet: docker-ce-cli-1:19.03.8-3.el7.x86_64                                                                                                                                         3/7
  Installing       : libnftnl-1.1.1-4.el8.x86_64                                                                                                                                                  4/7
  Running scriptlet: libnftnl-1.1.1-4.el8.x86_64                                                                                                                                                  4/7
  Running scriptlet: iptables-1.8.2-16.el8.x86_64                                                                                                                                                 5/7
  Installing       : iptables-1.8.2-16.el8.x86_64                                                                                                                                                 5/7
  Running scriptlet: iptables-1.8.2-16.el8.x86_64                                                                                                                                                 5/7
  Running scriptlet: libcgroup-0.41-19.el8.x86_64                                                                                                                                                 6/7
  Installing       : libcgroup-0.41-19.el8.x86_64                                                                                                                                                 6/7
  Running scriptlet: libcgroup-0.41-19.el8.x86_64                                                                                                                                                 6/7
  Installing       : docker-ce-3:19.03.8-3.el7.x86_64                                                                                                                                             7/7
  Running scriptlet: docker-ce-3:19.03.8-3.el7.x86_64                                                                                                                                             7/7
  Verifying        : iptables-1.8.2-16.el8.x86_64                                                                                                                                                 1/7
  Verifying        : libcgroup-0.41-19.el8.x86_64                                                                                                                                                 2/7
  Verifying        : libnetfilter_conntrack-1.0.6-5.el8.x86_64                                                                                                                                    3/7
  Verifying        : libnfnetlink-1.0.1-13.el8.x86_64                                                                                                                                             4/7
  Verifying        : libnftnl-1.1.1-4.el8.x86_64                                                                                                                                                  5/7
  Verifying        : docker-ce-3:19.03.8-3.el7.x86_64                                                                                                                                             6/7
  Verifying        : docker-ce-cli-1:19.03.8-3.el7.x86_64                                                                                                                                         7/7

Installed:
  docker-ce-3:19.03.8-3.el7.x86_64     iptables-1.8.2-16.el8.x86_64             libcgroup-0.41-19.el8.x86_64     libnetfilter_conntrack-1.0.6-5.el8.x86_64     libnfnetlink-1.0.1-13.el8.x86_64
  libnftnl-1.1.1-4.el8.x86_64          docker-ce-cli-1:19.03.8-3.el7.x86_64

Complete!

Once docker is successfully installed you can run docker -v to check the installed version.

[root@test_centos8 /]# docker -v
Docker version 19.03.8, build afacb8b

Hope this helps everyone (and saves productive time).

April 2, 2020

Automating Kubernetes deployment on VMs using Ansible

In this post, we will discuss automating Kubernetes deployment using Ansible.

In my example, I have used CentOS VMs (on VMware) for deploying Kubernetes. But technically Kubernetes deployment steps don’t differ irrespective of the platform you use.

Before getting started to make sure you have

Ansible server up and running on the network. Also, make sure Ansible can reach the VMware environment.
Make sure you’ve added Ansible server SSH authentication keys into VMware virtual machine before converting the same into the template. Follow this blog post for steps.

Once you have the pre-requisites in place follow the below steps.

Step 1 – Clone my GitHub repository which consists of required playbooks and instructions.

[root@alb-ansible dw-pm-csi]# git clone https://github.com/waghmaredb/ansible-k8s
Cloning into 'ansible-k8s'…
remote: Enumerating objects: 41, done.
remote: Counting objects: 100% (41/41), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 41 (delta 12), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (41/41), done.
[root@alb-ansible dw-pm-csi]# cd ansible-k8s/
[root@alb-ansible ansible-k8s]# ls
k8s-deployment.yml README.md

Step 2 – Edit k8s-deployment.yml file and edit below lines from VARS

common environment details
#ntp_server: - Replace with your NTP server IP/hostname
domain: "" - Replace with your DOMAIN NAME
dns_server: - Replace with your DNS server IP/hostname
vmware environment details
vcenter_ip: - Replace with your vCenter server IP/hostname
vcenter_username: - Replace with vCenter admin account username
vcenter_password: - Replace with vCenter admin account password
vmware_datacenter: - Replace with VMware datacenter you want to use
vmware_cluster: - Replace with VMware cluster you want to use
vm_network: "" - Replace with VM network you want kubernetes VMs to connect
k8s_vm_folder: - Replace with VM folder in which you want to place kubernetes VMs
k8s_template_name: - Replace with VMware CentOS template name
K8S environment details
k8s_master_ip: 192.168.172.100 - Replace IP address with kubernetes master server IP address you want to use
k8s_network_netmask: 255.255.255.0 - Replace subnet mask with netmask of kubernetes network
k8s_network_gateway: 192.168.172.1 - Replace gateway with kubernetes network gateway
k8s_node1_ip: 192.168.172.101 - Repalce IP address with kubernetes node IP address
#k8s_node2_ip: 192.168.1.102
#k8s_node3_ip: 192.168.1.103
#k8s_node4_ip: 192.168.1.104
#k8s_node5_ip: 192.168.1.105
#k8s_node6_ip: 192.168.1.106
#k8s_node7_ip: 192.168.1.107
#k8s_node8_ip: 192.168.1.108

Step 3 – Edit the /etc/ansible hosts file and insert the Kubernetes environment details. Make sure IP address details are inline with your Kubernetes environment

[kube_cluster1]
k8s-master ansible_host=192.168.172.100 ansible_user=root
worker1 ansible_host=192.168.172.101 ansible_user=root
worker2 ansible_host=192.168.172.102 ansible_user=root
worker3 ansible_host=192.168.172.103 ansible_user=root
worker4 ansible_host=192.168.172.104 ansible_user=root

[master]
k8s-master ansible_host=192.168.172.100 ansible_user=root

[worker]
worker1 ansible_host=192.168.172.101 ansible_user=root
worker2 ansible_host=192.168.172.102 ansible_user=root
worker3 ansible_host=192.168.172.103 ansible_user=root
worker4 ansible_host=192.168.172.104 ansible_user=root

Step 4 – Run the k8s-deployment.yml playbook.

March 2, 2020

What is Container Storage Integration (CSI) and Why now?
Containers are everywhere and they are here to stay. They are great level-playing ground to break the infrastructure dependency and allow developers to release their code to any environment.

Containers also help customers to operate at greater scales with ability to quickly scaling up and down, patching with disruptions, withstand infrastructure component failures, moving easily from on-premises to public clouds, etc.

As per the survey from sysdig lifespan of containers and container images is also very short.

At this stage of the popularity of containers there are two thought processes in the container fan club – Persistent or Non-Persistent Containers

If you explore docker hub top downloads you’ll notice that 7/10 top downloads require data persistence (snippet below)

Let’s understand persistent containers in more details.

Prior to Container Storage Integration CSI, Kubernetes provided in-tree (ie as part of the core code) plugins to support volumes but that posed a problem in that storage vendors had to align to the Kubernetes release process to fix a bug or to release new features among other problems. This also means every storage vendor had their own process to present volumes to Kubernetes.

This heterogeneous non-standard integrations were one of the biggest reasons why CSI was created. CSI was developed as a standard for exposing block and file storage storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. With the adoption of the Container Storage Interface (CSI), the Kubernetes volume layer becomes truly extensible. Using CSI, third-party storage providers like DellEMC can write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code. This gives Kubernetes users more options for storage and makes the system more secure and reliable. Also this approach makes sure that every vendor has standard way of interacting with Kubernetes.

With CSI Kubernetes supports Persistent Volumes (PV). PVs life-cycle independent of any Kubernetes POD. Kubernetes supports 2 ways to provision PVs
- Static – Admin Pre-provisions / creates a number of PVs
Static PV provisioning
- Dynamic – Cluster “automatically” provisions a volume
Dynamic PV provisioning

No matter which is the method of PV provisioning it can support varying properties such as performance, QOS, backup policies, etc. These properties are defined by StorageClass

There are 3 access modes which are supported on PV. Storage volume cannot be mounted simultaneously in more than one access mode.
1. ReadWriteOnce (RWO) – Volume can be mounted as ready-write by a single node
2. ReadOnlyMany (ROX) – Volume can be mounted read-only by many nodes
3. ReadWriteMany (RWX) – Volume can be mounted as read-write by many nodes
Below is the summary of persistence

DellEMC CSI Support

DellEMC understands that that Enterprise applications require persistent storage. As of now (Nov 2019) DellEMC supports CSI plugins for below storage arrays
Below are few documents around DellEMC CSI integration
- DellEMC CSI Solution Brief
- DellEMC CSI Whitepaper
More blogs on CSI coming up 🙂
Spread The Word:
Tweet
Email
WhatsApp
More
Print
Reddit
Like this:
Like Loading…
November 27, 2019
Installing Kubernetes on CentOS
Part of my job is to talk about the latest geeky technologies and many times I also have to demonstrate the same – Kind of “Show me” discussions.

When I started working on getting my hands dirty on Kubernetes (aka K8S) I faced many issues to get started. Now I am at the level where deploying K8S isn’t a big deal at all. The reason I am writing this blog is that often more than not I always get into discussions where someone is just starting with the K8S journey and has the same queries and questions which I also had. Hopefully, this summary will help people to get started with K8S.

Before I get started it’s important to understand the K8S lingo 🙂 This will help understand the implementation steps. Also, note that the purpose of this page is not to re-iterate the K8S components and architecture. It’s purely intended to list the steps o have hassle-free K8S deployment

Make sure that the VMware template which will be used has Ansible master server SSH keys added before you convert the VM image to the template. I have already documented this process of enabling SSH-based authentication in this blog

Part 1 – Dependencies

Below are the list of dependencies which needs to be installed on all the K8S nodes (master and worker)
- Disable SELinux
```
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
```
- Turnoff swap (also remove swap entry from /etc/fstab)
```
swapoff -a
```
- Install Docker
```
yum -y install docker
systemctl enable docker
systemctl start docker
systemctl status docker
```
- Ensure net.bridge.bridge-nf-call-ip6tables is set to 1
- Ensure net.bridge.bridge-nf-call-iptables is set to 1
```
sudo bash -c 'cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF'
```
- Add Kubernetes YUM repository
```
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
```
- Install kubelet (I installed version 1.14)
- install kubeadm (I installed version 1.14)
- Start kubelet service
```
yum -y install kubelet kubeadm
```
Below listed dependencies are only for K8S master
- Install kubectl
```
yum -y install kubectl
kubectl version
```
Part 2 – Kubernetes Master

Below listed steps are for configuring K8S Master
- Initialize the cluster using below command
```
kubeadm init --pod-network-cidr=10.244.0.0/16 >> cluster_initialized.txt
```
- Create ~/.kube directory (chmod 0755)
- Copy admin.conf to user’s kube config
```
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
```
- Install Pod network. In my case I have used Flannel, but you can choose relevant cluster networking from certified options from this link.
```
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml >> pod_network_setup.txt
```
At this moment your K8S master is installed and configured. Next step is to configure worker nodes and add them into the K8S cluster

Part 3 – Kubernetes Worker

Final part is to have worker nodes configured and add them in K8S cluster. This steps involves running commands on Master and Worker nodes.
- Master node – For adding worker nodes into the K8S cluster we first need to get the join command from the master server. Run the below command on Master server
```
kubeadm token create --print-join-command
```
- Note/copy the join command output. We need to run this join command on all worker nodes.
Finally run below command on the K8S Master. If everything was successful then you should see list of all the nodes (Master and Worker) of your K8S cluster.
```
kubectl get nodes
```
This concludes K8S installation and configuration 🙂
Spread The Word:
Tweet
Email
WhatsApp
More
Print
Reddit
Like this:
Like Loading…
November 9, 2019