How-to use GlusterFS to back persistent volumes in Kubernetes

A mountain I keep walking around instead of climbing in my Kubernetes lab is storing persistent data, I kept avoiding it. Sure – in a lab, I can just throw it all out most of the time. But, what about when we really need it? I decided I would use GlusterFS to back my persistent volumes and I’ve got to say… My experience with GlusterFS was great, I really enjoyed using it, and it seems rather resilient – and best of all? It was pretty easy to get going and to operate. Today we’ll spin up a Kubernetes cluster using my kube-ansible playbooks, and use some newly included plays that also setup a GlusterFS cluster. With that in hand, our goal will be to setup the persistent volumes and claims to those volumes, and we’ll spin up a MariaDB pod that stores data in a persistent volume, important data that we want to keep – so we’ll make some data about Vermont beer as it’s very very important.

Update: Hey! Check it out – I have a new article about GlusterFS for kube. Worth a gander as well.

Requirements

First up, this article will use my spin-up Kubernetes on CentOS article as a basis. So if there’s any details you feel are missing from here – make sure to double check that article as it goes further in depth for the moving parts that make up the kube-ansible playbooks. Particularly there’s more detail that article on how to modify the inventories and what’s going on there, too (and where your ssh keys are, which you’ll need too).

Now, what you’ll need…

  • A CentOS 7.3 host capable of spinning up a few virtual machines
  • A machine where you can run Ansible (which can also be the same host if you like), and has git to clone our playbooks.

That’s what we’re going to base it on. If you’d rather not use virtual machines, that’s OK! But, if you choose to spin this up on bare metal, you’ll have to do all the OS install yourself (as you guessed, or maybe you’re all cool and using Bifrost or Spacewalk or something cool, and that’s great too). To make it interesting, I’d recommend at least 3 hosts (a master and two minions), and… There’s one more important part you’re going to have to do if you’re using baremetal – and that’s going to be to make sure there’s a couple empty partitions available. Read ahead first and see what it looks like here with the VMs. Those partitions you’ll have to format for GlusterFS. In fact – that’s THE only hard part of this whole process is that you’ve gotta have some empty partitions across a few hosts that you can modify.

Let’s get our Kubernetes cluster running.

Ok, step zero – you need a clone of my playbooks, so make a clone and move into it’s directory…

git clone --branch v0.0.6 https://github.com/redhat-nfvpe/kube-ansible.git

Since we’ve got that we’re going to do run the virt-host-setup.yml playbook which sets up our CentOS host so that it can create a few virtual machines. The defaults spin up 4 machines, and you can modify some of these preferences by going into the vars/all.yml if you please. Also, you’ll need to modify the inventory/virthost.inventory file to suit your environment.

ansible-playbook -i inventory/virthost.inventory virt-host-setup.yml

Once that is complete, on your virtual machine host you should see some machines running if you were to run

virsh list --all

The virt-host-setup will complete with a set of IP addresses, so go ahead and use those in the inventory/vms.inventory file, and then we can start our Kubernetes installation.

ansible-playbook -i inventory/vms.inventory kube-install.yml

You can check that Kubernetes is running successfully now, SSH into your Master (you’ll need to do other work there soon, too)

kubectl get nodes

And you should see the master and 3 minions, by default. Alright, Kubernetes is up.

Let’s get GlusterFS running!

So we’re going to use a few playbooks here to get this all setup for you. Before we do that, let me speak to what’s happening in the background, and we’ll take a little peek for ourselves with our setup up and running.

First of all, most of my work to automate this with Ansible was based on this article on installing a GlusterFS cluster on CentOS, which I think comes from Storage SIG (maybe). I also referenced this blog article from Gluster about GlusterFS with Kubernetes. Last but not least, there’s example implementations of GlusterFS From Kubernetes GitHub repo.

Next a little consideration for your own architectural needs is that we’re going to use the Kubernetes nodes as GlusterFS themselves. Additionally – then we’re running GlusterFS processes on the hosts themselves. So, this wouldn’t work for an all Atomic Host setup. Which is unfortunate, and I admit I’m not entirely happy with it, but it might be a premature optimization of sorts right now. However, this is more-or-less a proof-of-concept. If you’re so inclined, you might be able to modify what’s here to adapt it to a fully containerized deployment (it’d be a lot, A LOT, swankier). You might want to organize this otherwise, but, it was convenient to make it this way, and could be easily broken out with a different inventory scheme if you so wished.

Attach some disks

The first playbook we’re going to run is the vm-attach-disk playbook. This is based on this publicly available help from access.redhat.com. The gist is that we create some qcow images and attach them to our running guests on the virtual machine host.

Let’s first look at the devices available on our kube-master for instance, so list the block devices…

[centos@kube-master ~]$ lsblk | grep -v docker

You’ll note that there’s just a vda mounted on /.

Let’s run that playbook now and take a peek at it again after.

ansible-playbook -i inventory/virthost.inventory vm-attach-disk.yml

Now go ahead and look on the master again, and list those block devices.

[centos@kube-master ~]$ lsblk | grep -v docker

You should have a vdb that’s 4 gigs. Great, that’s what the playbook does, it does it across the 4 guests. You’ll note it’s not mounted, and it’s not formatted.

Configure those disks, and install & configure Gluster!

Now that our disks are attached, we can go ahead and configure Gluster.

ansible-playbook -i inventory/vms.inventory gluster-install.yml

Here’s what’s been done:

  • Physical volumes and volume groups created on those disks.
  • Disks formatted as XFS.
  • Partitions mounted on /bricks/brick1 and /bricks/brick2.
  • Nodes attached to GlusterFS cluster.
  • Gluster volumes created across the cluster.
  • Some yaml for k8s added @ /home/centos/glusterfs*yaml

Cool, now that it’s setup let’s look at a few things. We’ll head right to the master to check this out.

[root@kube-master centos]# gluster peer status

You should see three peers with a connected state. Additionally, this should be the case on all the minions, too.

[root@kube-minion-2 centos]# gluster peer status
Number of Peers: 3

Hostname: 192.168.122.14
Uuid: 3edd6f2f-0055-4d97-ac81-4861e15f6e49
State: Peer in Cluster (Connected)

Hostname: 192.168.122.189
Uuid: 48e2d30b-8144-4ae8-9e00-90de4462e2bc
State: Peer in Cluster (Connected)

Hostname: 192.168.122.160
Uuid: d20b39ba-8543-427d-8228-eacabd293b68
State: Peer in Cluster (Connected)

Lookin’ good! How about volumes available?

[root@kube-master centos]# gluster volume status

Which will give you a lot of info about the volumes that have been created.

Let’s try using GlusterFS (optional)

So this part is entirely optional. But, do you want to see the filesystem in action? Let’s temporarily mount a volume, and we’ll write some data to it, and see it appear on other hosts.

[root@kube-master centos]# mkdir /mnt/gluster
[root@kube-master centos]# ipaddr=$(ifconfig | grep 192 | awk '{print $2}')
[root@kube-master centos]# mount -t glusterfs $ipaddr:/glustervol1 /mnt/gluster/

Ok, so now we have a gluster volume mounted at /mnt/gluster – let’s go ahead and put a file in there.

[root@kube-master centos]# echo "foo" >> /mnt/gluster/bar.txt

Now we should have a file, bar.txt with the contents “foo” on all the nodes in the /bricks/brick1/brick1 directory. Let’s verify that on a couple nodes.

[root@kube-master centos]# cat /bricks/brick1/brick1/bar.txt 
foo

And on kube-minion-2…

[root@kube-minion-2 centos]# cat /bricks/brick1/brick1/bar.txt
foo

Cool! Nifty right? Now let’s clean up.

[root@kube-master centos]# rm /mnt/gluster/bar.txt
[root@kube-master centos]# umount /mnt/gluster/

Add the persistent volumes to Kubernetes!

Alright, so you’re all good now, it works! (Or, I hope it works for you at this point, from following 1,001 blog articles for how-to documents, I know sometimes it can get frustrating, but… I’m hopeful for you).

With that all set and verified a bit, we can go ahead and configure Kubernetes to get it all looking good for us.

On the master, as the centos user, look in the /home/centos/ dir for these files…

[centos@kube-master ~]$ ls glusterfs-* -lah
-rw-rw-r--. 1 centos centos  781 Apr 19 19:08 glusterfs-endpoints.json
-rw-rw-r--. 1 centos centos  154 Apr 19 19:08 glusterfs-service.json
-rw-rw-r--. 1 centos centos 1.6K Apr 19 19:11 glusterfs-volumes.yaml

Go ahead and inspect them if you’d like. Let’s go ahead and implement them for us.

[centos@kube-master ~]$ kubectl create -f glusterfs-endpoints.json 
endpoints "glusterfs-cluster" created
[centos@kube-master ~]$ kubectl create -f glusterfs-service.json 
service "glusterfs-cluster" created
[centos@kube-master ~]$ kubectl create -f glusterfs-volumes.yaml 
persistentvolume "gluster-volume-1" created
persistentvolume "gluster-volume-2" created
persistentvolume "gluster-volume-3" created
persistentvolume "gluster-volume-4" created
persistentvolume "gluster-volume-5" created

Now we can ask kubectl to show us the persistent volumes pv.

[centos@kube-master ~]$ kubectl get pv
NAME               CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM     STORAGECLASS   REASON    AGE
gluster-volume-1   600Mi      RWO           Delete          Available             storage                  18s
gluster-volume-2   300Mi      RWO           Delete          Available             storage                  18s
gluster-volume-3   300Mi      RWO           Delete          Available             storage                  18s
gluster-volume-4   100Mi      RWO           Delete          Available             storage                  18s
gluster-volume-5   100Mi      RWO           Delete          Available             storage                  18s

Alright! That’s good now, we can go ahead and put these to use.

Let’s create our claims

First, we’re going to need a persistent volume claim. So let’s craft one here, and we’ll get that going. The persistent volume claim is like “staking a claim” of land. We’re going to say “Hey Kubernetes, we need a volume, and it’s going to be this big”. And it’ll allocate it smartly. And it’ll let you know for sure if there isn’t anything that it can claim.

So create a file like so…

[centos@kube-master ~]$ cat pvc.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: null
  name: mariadb-data
spec:
  storageClassName: storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 250Mi
status: {}

And then use kubectl create to apply it.

[centos@kube-master ~]$ kubectl create -f pvc.yaml 
persistentvolumeclaim "mariadb-data" created

And now we can list the “persistent volume claims”, the pvc

[centos@kube-master ~]$ kubectl get pvc
NAME           STATUS    VOLUME             CAPACITY   ACCESSMODES   STORAGECLASS   AGE
mariadb-data   Bound     gluster-volume-2   300Mi      RWO           storage        20s

You’ll see that Kubernetes was smart about it, and of the volumes we created – it used juuuust the right one. We had a 600 meg claim, 300 meg claims, and a couple 100 meg claims. It picked the 300 meg claim properly. Awesome!

Now, let’s put those volumes to use in a Maria DB pod.

Great, now we have some storage we can use across the cluster. Let’s go ahead and use it. We’re going to use Maria DB cause it’s a great example of a real-world way that we’d want to persist data – in a database.

So let’s create a YAML spec for this pod. Make yours like so:

[centos@kube-master ~]$ cat mariadb.yaml 
---
apiVersion: v1
kind: Pod
metadata:
  name: mariadb
spec:
  containers:
  - env:
      - name: MYSQL_ROOT_PASSWORD
        value: secret
    image: mariadb:10
    name: mariadb
    resources: {}
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: mariadb-data
  restartPolicy: Always
  volumes:
  - name: mariadb-data
    persistentVolumeClaim:
      claimName: mariadb-data

Cool, now create it…

[centos@kube-master ~]$ kubectl create -f mariadb.yaml 

Then, watch it come up…

[centos@kube-master ~]$ watch -n1 kubectl describe pod mariadb 

Let’s make some persistent data! Err, put beer in the fridge.

Once it comes up, let’s go ahead and create some data in there we can pull back up. (If you didn’t see it in the pod spec, you’ll want to know that the password is “secret” without quotes).

This data needs to be important right? Otherwise, we’d just throw it out. So we’re going to create some data regarding beer.

You’ll note I’m creating a database called kitchen with a table called fridge and then I’m inserting some of the BEST beers in Vermont (and likely the world, I’m not biased! ;) ). Like Heady Topper from The Alchemist, and Lawson’s sip of sunshine, and the best beer ever created – Hill Farmstead’s Edward

[centos@kube-master ~]$ kubectl exec -it mariadb -- /bin/bash
root@mariadb:/# stty cols 150
root@mariadb:/# mysql -u root -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.1.22-MariaDB-1~jessie mariadb.org binary distribution

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> CREATE DATABASE kitchen;
Query OK, 1 row affected (0.02 sec)

MariaDB [(none)]> USE kitchen;
Database changed
MariaDB [kitchen]> 
MariaDB [kitchen]> 
MariaDB [kitchen]> CREATE TABLE fridge (id INT AUTO_INCREMENT, item VARCHAR(255), quantity INT, PRIMARY KEY (id));
Query OK, 0 rows affected (0.31 sec)

MariaDB [kitchen]> INSERT INTO fridge VALUES (NULL,'heady topper',6);
Query OK, 1 row affected (0.05 sec)

MariaDB [kitchen]> INSERT INTO fridge VALUES (NULL,'sip of sunshine',6);
Query OK, 1 row affected (0.04 sec)

MariaDB [kitchen]> INSERT INTO fridge VALUES (NULL,'hill farmstead edward',6); 
Query OK, 1 row affected (0.03 sec)

MariaDB [kitchen]> SELECT * FROM fridge;
+----+-----------------------+----------+
| id | item                  | quantity |
+----+-----------------------+----------+
|  1 | heady topper          |        6 |
|  2 | sip of sunshine       |        6 |
|  3 | hill farmstead edward |        6 |
+----+-----------------------+----------+
3 rows in set (0.00 sec)

Destroy the pod!

Cool – well that’s all well and good, we know there’s some beer in our kitchen.fridge table in MariaDB.

But, let’s destroy the pod, first – where is the pod running, which minion? Let’s check that out. We’re going to restart it until it appears on a different node. (We could create an anti-affinity and all that good stuff, but, we’ll just kinda jimmy it here for a quick demo.)

[centos@kube-master ~]$ kubectl describe pod mariadb | grep -P "^Node:"
Node:       kube-minion-2/192.168.122.43

Alright, you’ll see mine is running on kube-minion-2, let’s remove that pod and create it again.

[centos@kube-master ~]$ kubectl delete pod mariadb
pod "mariadb" deleted
[centos@kube-master ~]$ kubectl create -f mariadb.yaml 
[centos@kube-master ~]$ watch -n1 kubectl describe pod mariadb 

Watch it come up again, and if it comes up on the same node – delete it and create it again. I believe it happens round-robin-ish, so… It’ll probably come up somewhere else.

Now, once it’s up – let’s go and check out the data in it.

[centos@kube-master ~]$ kubectl exec -it mariadb -- mysql -u root -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 10.1.22-MariaDB-1~jessie mariadb.org binary distribution

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> USE kitchen;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [kitchen]> SELECT * FROM fridge;
+----+-----------------------+----------+
| id | item                  | quantity |
+----+-----------------------+----------+
|  1 | heady topper          |        6 |
|  2 | sip of sunshine       |        6 |
|  3 | hill farmstead edward |        6 |
+----+-----------------------+----------+
3 rows in set (0.00 sec)

Hurray! There’s all the beer still in the fridge. Phew!!! Precious, precious beer.

koko - Connect Containers together with virtual ethernet connections

Let’s dig into koko created by Tomofumi Hayashi. koko (the project’s namesake comes from “COntainer COnnector”) is a utility written in Go that gives us a way to connect containers together with “veth” (virtual ethernet) devices – a feature available in the Linux kernel. This allows us to specify interfaces that the containers use and link them together – all without using Linux bridges. koko has become a cornerstone of the zebra-pen project, an effort I’m involved in to analyze gaps in containerized NFV workloads, specifically it routes traffic using Quagga, and we setup all the interfaces using koko. The project really took a turn for the better when Tomo came up with koko and we implemented it in zebra-pen. Ready to see koko in action? Let’s jump in the pool!

Quick update note: This article was written before “koko” was named…. “koko”! It was previously named “vethcon” as it dealt primarily with “veth connections for containers” and that’s what we focus on in this article. Now, “vethcon” does more than just use “veth” interfaces, and henceforth it was renamed. Now, it can also do some cool work with vxlan interfaces to do what we’ll do here – but also across hosts! This article still focuses on using veth interfaces. I did a wholesale find-and-replace of “vethcon” with “koko” and everything should “just work”, but, just so you can be forewarned.

We’ll talk about the back-story for what veth interfaces are, and talk a little bit about Linux network namespaces. Then we’ll dig into the koko source itself and briefly step through what it’s doing.

Last but not least – what fun would it be if we didn’t fire up koko and get it working? If you’re less interested in the back story, just scroll down to the “Ready, set, compile!” section. From there you can get your hands on the keyboard and dive into the fun stuff. Our goal will be to compile koko, connect two containers with one another, look at those interfaces and get a ping to come across them.

We’ll just connect a couple containers together, but, using koko you can also connect network namespaces to containers, and network namespaces to network namespaces, too.

Another note before we kick this off – koko’s life has really just begun, it’s useful and functional as it is. But, Tomo has bigger and better ideas for it – there’s some potential in the future for creating vxlan interfaces (and given that the rename happened, those are in there at least as a prototype), and getting it working with CNI – but, there’s still experimentation to be done there, and I don’t want to spoil it by saying too much. So, as I’ve heard said before “That’s another story for another bourbon.”

Requirements

If you want to sing along – the way that I’m going through this is using a fresh install of CentOS 7. In my case I’m using the generic cloud image. Chances are this will be very similar with a RHEL or Fedora install. But if you want to play along the same exact way, spin yourself up a fresh CentOS 7 VM.

You’re also going to need a spankin’ fresh version of Docker. So we’ll install from the official Docker RPM repos and install a really fresh one.

The back-story

koko leverages “veth” – as evidenced by its name. veth interfaces aren’t exactly new, veth devices were proposed way back in ‘07. The original authors describe veth as:

Veth stands for Virtual ETHernet. It is a simple tunnel driver that works at the link layer and looks like a pair of ethernet devices interconnected with each other.

veth interfaces come in pairs, and that’s what we’ll do in a bit, we’ll pair them up together with two containers. If you’d like to see some diagrams of veth pairs in action – I’ll point you to this article from opencloudblog which has does a nice job illustrating it.

Another concept that’s important to the functioning of koko is “network namespaces”. Linux namespaces is the general concept here that allows network namespaces – in short they give us a view of resources that are limited to a “namespace”. Linux namespaces are a fundamental part of how containers function under Linux, it provides the over-arching functionality that’s necessary to segregate processes and users, etc. This isn’t new either – apparently it begun in 2002 with mount-type namespaces.

Without network namespaces, in Linux all of your interfaces and routing tables are all mashed together and available to one another. With network namespaces, you can isolate these from one another, so they can work independently from one-another. This will give processes a specific view of these interfaces.

Let’s look at the koko go code.

So, what’s under the hood? In essence, koko uses a few modules and then provides some handling for us to pick out the container namespace and assign veth links to the containers. Its simplicity is its elegance, and quite a good idea.

It’s worth noting that koko may change after I write this article, so if you’re following along with a clone of koko – you might want to know what point in the git history it exists, so I’ll point you towards browsing the code at commitish 35c4c58 if you’d like.

Let’s first look at the modules, then, I’ll point you through the code just a touch, in case you wanted to get in there and look a little deeper.

The libraries

And other things that are more utilitarian, such as package context, c-style getopts, and internal built-ins like os,fmt,net, etc.

Application flow

Note: some of this naming may have changed a bit with the koko upgrade

At it’s core, koko defines a data object called vEth, which gives us a structure to store some information about the connections that we’ll make.

It’s a struct and is defined as so:

// ---------------------------------------------------- -
// ------------------------------ vEth data object.  - -
// -------------------------------------------------- -
// -- defines a data object to describe interfaces
// -------------------------------------------------- -

type vEth struct {
    // What's the network namespace?
    nsName string
    // And what will we call the link.
    linkName string
    // Is there an ip address?
    withIPAddr bool
    // What is that ip address.
    ipAddr net.IPNet
}

In some fairly terse diagramming using asciiflow, the general application flow goes as follows… (It’s high level, I’m missing a step or two, but, it’d help you dig through the code a bit if you were to step through it)

main()
  +
  |
  +------> parseDOption()  (parse -d options from cli)
  |
  +------> parseNOption()  (parse -n options from cli)
  |
  +------> makeVeth(veth1, veth2) with vEth data objects
               +
               |
               +------>  getVethPair(link names)
               |             +
               |             |
               |             +------>  makeVethPair(link)
               |                          +
               |                          |
               |                          +----> netlink.Veth()
               |
               +------>  setVethLink(link) for link 1 & 2

Ready, set, compile!

Ok, first let’s get ready and install the dependencies that we need. Go makes it really easy on us – it handles its own deps and we basically will just need golang, git and Docker.


# Enable the docker ce repo
[centos@koko ~]$ sudo yum-config-manager     --add-repo     https://download.docker.com/linux/centos/docker-ce.repo

# Install the deps.
[centos@koko ~]$ sudo yum install -y golang git docker-ce

# Start and enable docker
[centos@koko ~]$ sudo systemctl start docker && sudo systemctl enable docker

# Check that docker is working
[centos@koko ~]$ sudo docker ps

Now let’s set the gopath and git clone the code.

# Set the go path
[centos@koko ~]$ rm -Rf gocode/
[centos@koko ~]$ mkdir -p gocode/src
[centos@koko ~]$ export GOPATH=/home/centos/gocode/

# Clone koko
[centos@koko ~]$ git clone https://github.com/redhat-nfvpe/koko.git /home/centos/gocode/src/koko

Finally, we’ll grab the dependencies and compile koko.

# Fetch the dependencies for koko
[centos@koko ~]$ cd gocode/
[centos@koko gocode]$ go get koko

# Now, let's compile it
[centos@koko ~]$ go build koko

Now you can go ahead and run the help if you’d like.

[centos@koko gocode]$ ./koko -h

Usage:
./koko -d centos1:link1:192.168.1.1/24 -d centos2:link2:192.168.1.2/24 #with IP addr
./koko -d centos1:link1 -d centos2:link2  #without IP addr
./koko -n /var/run/netns/test1:link1:192.168.1.1/24 <other>  

Make a handy-dandy little Docker image

Let’s make ourselves a handy Docker image that we can use – we’ll base it on CentOS and just add a couple utilities for inspecting what’s going on.

Make a Dockerfile like so:

FROM centos:centos7
RUN yum install -y iproute tcpdump

I just hucked my Dockerfile into tmp and built from there.

[centos@koko gocode]$ cd /tmp/
[centos@koko tmp]$ vi Dockerfile
[centos@koko tmp]$ sudo docker build -t dougbtv/inspect-centos .

Run your containers

Now you can spin up a couple containers based on those images…

Note that we’re going to run these with --network none as a demonstration.

Let’s do that now…

[centos@koko gocode]$ sudo docker run --network none -dt --name centos1 dougbtv/inspect-centos /bin/bash
[centos@koko gocode]$ sudo docker run --network none -dt --name centos2 dougbtv/inspect-centos /bin/bash

If you exec ip link on either of the containers you’ll see they only have a local loopback interfaces.

[centos@koko gocode]$ sudo docker exec -it centos1 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

That’s perfect for us for now.

Let’s give koko a run!

Ok, cool, so at this point you should have a couple containers named centos1 & centos2 running, and both of those with --network none so they only have the local loopback as mentioned.

[centos@koko gocode]$ sudo docker ps --format 'table     '
NAMES               IMAGE
centos2    dougbtv/inspect-centos
centos1    dougbtv/inspect-centos

Cool – now, let’s get some network between the two of these containers using vetcon… What we’re going to do is put the containers on a network, the /24 we’re going to choose is 10.200.0.0/24 and we’ll make network interfaces named net1 and net2.

You pass these into koko with colon delimited fields which is like -d {container-name}:{interface-name}:{ip-address/netmask}. As we mentioned earlier, since veths are pairs – you pass in the -d {stuff} twice for the pain, one for each container.

Note that the container name can either be the name (as we gave it a --name in our docker run or it can be the container id [the big fat hash]). The interface name must be unique – it can’t match another one on your system, and it must be different

So that means we’re going to execute koko like this. (Psst, make sure you’re in the ~/gocode/ directory we created earlier, unless you moved the koko binary somewhere else that’s handy.)

Drum roll please…

[centos@koko gocode]$ sudo ./koko -d centos1:net1:10.200.0.1/24 -d centos2:net2:10.200.0.2/24
Create veth...done

Alright! Now we should have some interfaces called net1 and net2 in the centos1 & centos2 containers respectively, let’s take a look by running ip addr on each container. (I took the liberty of grepping for some specifics)

[centos@koko gocode]$ sudo docker exec -it centos1 ip addr | grep -P "^\d|inet "
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    inet 127.0.0.1/8 scope host lo
28: net1@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    inet 10.200.0.1/24 scope global net1

[centos@koko gocode]$ sudo docker exec -it centos2 ip addr | grep -P "^\d|inet "
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    inet 127.0.0.1/8 scope host lo
27: net2@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    inet 10.200.0.2/24 scope global net2

As you can see there’s an interface called net1 for the centos1 container, and it’s assigned the address 10.200.0.1. It’s companion, centos2 has the net2 address, assigned 10.200.0.2.

That being said, let’s exec a ping from centos1 to centos2 to prove that it’s in good shape.

Here we go!

[centos@koko gocode]$ sudo docker exec -it centos1 ping -c5 10.200.0.2
PING 10.200.0.2 (10.200.0.2) 56(84) bytes of data.
64 bytes from 10.200.0.2: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 10.200.0.2: icmp_seq=2 ttl=64 time=0.068 ms
64 bytes from 10.200.0.2: icmp_seq=3 ttl=64 time=0.055 ms
64 bytes from 10.200.0.2: icmp_seq=4 ttl=64 time=0.054 ms
64 bytes from 10.200.0.2: icmp_seq=5 ttl=64 time=0.052 ms

--- 10.200.0.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.052/0.058/0.068/0.009 ms

Alright, looking good with a ping, just to couple check, let’s also see that we can see it with a tcpdump on centos2. So, bring up 2 ssh sessions to this host (or, if it’s local to you, two terminals will do well, or however you’d like to do this).

And we’ll start a TCP dump on centos2 and we’ll exec the same ping command as above on centos1

And running that, we can see the pings going to-and-fro!

[centos@koko ~]$ sudo docker exec -it centos2 tcpdump -nn -i net2 'icmp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on net2, link-type EN10MB (Ethernet), capture size 65535 bytes
12:21:39.426020 IP 10.200.0.1 > 10.200.0.2: ICMP echo request, id 43, seq 1, length 64
12:21:39.426050 IP 10.200.0.2 > 10.200.0.1: ICMP echo reply, id 43, seq 1, length 64
12:21:40.425953 IP 10.200.0.1 > 10.200.0.2: ICMP echo request, id 43, seq 2, length 64
12:21:40.425983 IP 10.200.0.2 > 10.200.0.1: ICMP echo reply, id 43, seq 2, length 64
12:21:41.425898 IP 10.200.0.1 > 10.200.0.2: ICMP echo request, id 43, seq 3, length 64
12:21:41.425925 IP 10.200.0.2 > 10.200.0.1: ICMP echo reply, id 43, seq 3, length 64
12:21:42.425922 IP 10.200.0.1 > 10.200.0.2: ICMP echo request, id 43, seq 4, length 64
12:21:42.425949 IP 10.200.0.2 > 10.200.0.1: ICMP echo reply, id 43, seq 4, length 64
12:21:43.425870 IP 10.200.0.1 > 10.200.0.2: ICMP echo request, id 43, seq 5, length 64
12:21:43.425891 IP 10.200.0.2 > 10.200.0.1: ICMP echo reply, id 43, seq 5, length 64

(BTW, hit ctrl+c when you’re done with that tcpdump.)

Cool!!! …Man, sometimes when you’re working on networking goodies, the satisfaction of a successful ping is like no other. Ahhhh, feels so good.

Thank you, Tomo!

A big thanks goes out to Tomo for coming up with this idea, and then implementing it quite nicely in Go. It’s a well made utility built from an impressive idea. Really cool, I’ve enjoyed getting to utilitize it, and I hope it comes in handy to others in the future too.

You had ONE JOB -- A Kubernetes job.

Let’s take a look at how Kubernetes jobs are crafted. I had been jamming some kind of work-around shell scripts in the entrypoint* for some containers in the vnf-asterisk project that Leif and I have been working on. And that’s not perfect when we can use Kubernetes jobs, or in their new parlance, “run to completion finite workloads” (I’ll stick to calling them “jobs”). They’re one-shot containers that do one thing, and then end (sort of like a “oneshot” of systemd units, at least how we’ll use them today). I like the idea of using them to complete some service discovery for me when other pods are coming up. Today we’ll fire up a pod, and spin up a job to discover that pod (by querying the API for info about it), and put info into etcd. Let’s get the job done.

This post also exists as a gist on github where you can grab some files from, which I’ll probably reference a couple times.

* Not everyone likes having a custom entrypoint shell script, some people consider it a bit… “Jacked up”. But, personally I don’t depending on circumstance. Sometimes I think it’s a pragmatic solution. So it’s not always bad – it depends on the case. But, where we can break things up and into their particular places, it’s a GoodThing(TM).

Let’s try firing up a kubernetes job and see how it goes. We’ll use the k8s jobs documentation as a basis. But, as you’ll see we’ll need a bit more help as

Some requirements.

You’ll need a Kubernetes cluster up and running here. If you don’t, you can spin up k8s on centos with this blog article.

An bit of an editorial is that…. Y’know… OpenShift Origin kind of makes some of these things a little easier compared to vanilla K8s, especially with manging permissions and all that good stuff for the different accounts. It’s a little more cinched down in some ways (which you want in production), but, there’s some great considerations with oc to handle some of what we have to look at in more fine-grained detail herein.

Running etcd.

You can pick up the YAML for this from the gist, and it should be easy to fire up etcd with:

kubectl create -f etcd.yaml

Assuming you’ve setup DNS correctly to resolve from the master (get the DNS pod IP, and put it in your resolve.conf and search cluster.local – also see scratch.sh in the gist), you can check that it works…

# Set the value of the key "message" to be "sup sup"
[centos@kube-master ~]$ curl -L -X PUT http://etcd-client.default.svc.cluster.local:2379/v2/keys/message -d value="sup sup"
{"action":"set","node":{"key":"/message","value":"sup sup","modifiedIndex":9,"createdIndex":9}}

# Now retrieve that value.
[centos@kube-master ~]$ curl -L http://etcd-client.default.svc.cluster.local:2379/v2/keys/message 
{"action":"get","node":{"key":"/message","value":"sup sup","modifiedIndex":9,"createdIndex":9}}

Authenticating against the API.

At least to me – etcd is easy. Jobs are easy. The hardest part was authenticating against the API. So, let’s step through how that works quickly. It’s not difficult, I just didn’t have all the pieces together at first.

The idea here is that Kubernetes puts the default service account’s API token into a file in the container in the pod @ /var/run/secrets/kubernetes.io/serviceaccount/token for the default service account in the default namespace. We then present that to the API in our curl command.

But, It did get me to read about the kubectl proxy command, service accounts, and accessing the cluster. When really what I needed was just a bit of a tip from this stackoverflow answer.

First off, you can see that you have the api running with

kubectl get svc --all-namespaces | grep -i kubernetes

Great, if you see the line there that means you have the API running, and you’ll also be able to access it with DNS, which makes things a little cleaner.

Now that you can see that, we can go and access it… let’s run a pod.

kubectl run demo --image=centos:centos7 --command -- /bin/bash -c 'while :; do sleep 10; done'

Alright, now that you’ve got this running (look for it with kubectl get pods), we can enter that container and query the API.

Let’s do that just prove it.

[centos@kube-master ~]$ kubectl exec -it demo-1260169299-96zts -- /bin/bash

# Not enough columns for me...
[root@demo-1260169299-96zts /]# stty rows 50 cols 132

# Pull up the service account token.
[root@demo-1260169299-96zts /]# KUBE_TOKEN=$(</var/run/secrets/kubernetes.io/serviceaccount/token)

# Show it if you want.
[root@demo-1260169299-96zts /]#  echo $KUBE_TOKEN

# Now you can query the API
[root@demo-1260169299-96zts /]# curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://kubernetes.default.svc.cluster.local

Now, we have one job to do…

And that’s to create a job. So let’s create a job and we’ll put the IP address of this “demo pod” into etcd. In theory we’d use this with something else to discover where it’s based.

We’ll figure out the IP address of the pod by querying the API. If you’d like to dig in a little bit and get your feet with the Kube API, may I suggest this article from TheNewStack on taking the API for a spin.

Why not just query always the API? Well. You could do that too. But, in my case we’re going to generally standardize around using etcd. In part because in the full use-case we’re going to also store other metadata there that’s not “just the IP address”.

So, we can query the API directly to find out the fact we’re looking for, so let’s do that just to test out that our results are OK in the end.

I’m going to cheat here and run this little test from the master (instead of inside a container), it should work if you’re deploying using my playbooks.

# Figure out the pod name
[centos@kube-master ~]$ podname=$(curl -s http://localhost:8080/api/v1/namespaces/default/pods | jq ".items[] .metadata.name" | grep -i demo | sed -e 's/"//g')
[centos@kube-master ~]$ echo $podname
demo-1260169299-96zts

# Now using the podname, we can figure out the IP
[centos@kube-master ~]$ podip=$(curl -s http://localhost:8080/api/v1/namespaces/default/pods/$podname | jq '.status.podIP' | sed -s 's/"//g')
[centos@kube-master ~]$ echo $podip
10.244.2.11

Alright, that having been proven out we can create a job to do this for us, now too.

Let’s go and define a job YAML definition. You’ll find this one borrows generally heavily from the documentation, but, mixes up some things – especially it uses a customized centos:centos7 image of mine that has jq installed in it, it’s called dougbtv/jq and it’s available on dockerhub.

Also available in the gist, here’s the job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: hoover
spec:
  template:
    metadata:
      name: hoover
    spec:
      containers:
      - name: hoover
        image: dougbtv/jq
        command: ["/bin/bash"]
        args:
          - "-c"
          - >
            KUBE_TOKEN=$(</var/run/secrets/kubernetes.io/serviceaccount/token) &&
            podname=$(curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods | jq '.items[] .metadata.name' | grep -i demo | sed -e 's/\"//g') && 
            podip=$(curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/$podname | jq '.status.podIP' | sed -s 's/\"//g') &&
            echo "the pod is @ $podip" &&
            curl -L -X PUT http://etcd-client.default.svc.cluster.local:2379/v2/keys/podip -d value="$podip"
      restartPolicy: Never

Let’s create it.

[centos@kube-master ~]$ kubectl create -f job.yaml 

It’s named “hoover” as it’s kinda sucking up some info from the API to do something with it.

So look for it in the list of ALL pods.

[centos@kube-master ~]$ kubectl get pods --show-all
NAME                    READY     STATUS      RESTARTS   AGE
demo-1260169299-96zts   1/1       Running     0          1h
etcd0                   1/1       Running     0          12d
etcd1                   1/1       Running     0          12d
etcd2                   1/1       Running     0          12d
hoover-fkbjj            0/1       Completed   0          20s

Now we can see what the logs are from it. It’ll tell us what the IP is.

[centos@kube-master ~]$ kubectl logs hoover-fkbjj

That all said and done… we can complete this by seeing that the value made it to etcd.

[centos@kube-master ~]$ curl -L http://etcd-client.default.svc.cluster.local:2379/v2/keys/podip
{"action":"get","node":{"key":"/podip","value":"10.244.2.11","modifiedIndex":10,"createdIndex":10}}

Voila!

A (happy happy joy joy) ansible-container hello world!

Today we’re going to explore ansible-container, a project that gives you Ansible workflow for Docker. It provides a method of managing container images using ansible commands (so you can avoid a bunch of dirty bash-y Dockerfiles), and then provides a specification of “services” which is eerily similar (on purpose) to docker-compose. It also has paths forward for managing the instances of these containers on Kubernetes & OpenShift – that’s pretty tight. We’ll build two images “ren” and “stimpy”, which contain nginx and output some Ren & Stimpy quotes so we can get a grip on how it’s all put together. It’s better than bad – it’s good!

These steps were generally learned from dually the ansible-container demo github page and from the getting started guide. It also leverages this github project with demo ansible-container files I created which has all the files you need so you don’t have to baby them all in an editor.

My editorial is that… This is really a great project. However, I don’t consider it the be-all-end-all. I think it has an awesome purpose in the context of a larger ansible project. It’s squeaky clean when you use it that way. Except for the directory structure which I find a little awkward. Maybe I’m doing that part slightly wrong, it’s not terrible. I also think that Dockerfiles have their place. I like them, and in terms of some simpler apps (think, a Go binary) ansible-container is overkill, and your run of the mill pod spec when using k8s, raw and unabstracted isn’t so bad to deal with – in fact, it may be confusing in some places to abstract that. So, choose the right tool for the job is my advice. A-And I’d like a bike, and a Betsy Wetsherself doll, and a Cheesy-Bake Oven, and a Pulpy The Pup doll, and a gajillion green army men.

Ok, enough editorializing – let’s get some space madness and move onto getting this show on the road!

Requirements

Fairly simple – as per usual, we’re using a CentOS 7.3 based virtual machine to run all these on. Feel free to use your workstation, but, I put this all in a VM so I could isolate it, and see what you needed given a stock CentOS 7.3 install. Just as a note, my install is from a CentOS 7.3 generic cloud, and the basics are based on that.

Also – you need a half dozen gigs free of disk, and a minimum of 4 gigs of memory. I had a 2 gig memory VM and it was toast (ahem, powdered toast) when I went to do the image builds, so, keep that in mind.

Since I have a throw-away VM, I did all this as root, you can be a good guy and use sudo if you please.

Install Docker!

Yep, you’re going to need a docker install. We’ll just use the latest docker from CentOS repos, that’s all we need for now.

yum install -y docker
systemctl enable docker
systemctl start docker
docker images
docker -v

Install ansible-container

We’re going to install ansible-container from source so that we can have the 0.3.0 version (because we want the docker-compose v2 specification)

Now, go and install the every day stuff you need (like, your editor). I also installed tree so I could use it for the output here. Oh yeah, and you need git!

So we’re going to need to update some python-ish things, especially epel to get python-pip, then update python-pip, then upgrade setuptools.

yum install -y epel-release
yum install -y python-pip git
pip install --upgrade pip
pip install -U setuptools

Now, let’s clone the project and install it. These steps were generally learned from these official installation directions.

git clone https://github.com/ansible/ansible-container.git
cd ansible-container/
python ./setup.py install

And now you should have a version 0.3.0-ish ansible-container.

[user@host ansible-container]$ ansible-container version
Ansible Container, version 0.3.0-pre

Let’s look at ansible-container init

In a minute here we’re going to get into using a full-out project, but, typically when you start a project, there’s a few things you’re going to do.

  1. You’re going to use ansible-container init to scaffold the pieces you need.
  2. You’ll use ansible-container install some.project to install ansible galaxy modules into your project.

So let’s give that a test drive before we go onto our custom project.

Firstly, make a directory to put this in as we’re going to throw it out in a minute.

[user@host ~]$ cd ~
[user@host ~]$ mkdir foo
[user@host ~]$ cd foo/
[user@host foo]$ ansible-container init
Ansible Container initialized.

Alright, now you can see it created a ./ansible/ directory there, and it has a number of files therein.

Installing ansible galaxy modules for ansible-container

Now let’s say we’re going to install an nginx module from ansible galaxy. We’d do it like so…

[user@host foo]$ ansible-container install j00bar.nginx-container

Note that this will take a while the first time because it’s pulling some ansible-specific images.

Once it’s done with the pull, let’s inspect what’s there.

Inspecting what’s there.

Let’s take a look at what it looks like with ansible-container init and then an installed role.

[user@host foo]$ tree
.
└── ansible
    ├── ansible.cfg
    ├── container.yml
    ├── main.yml
    ├── meta.yml
    ├── requirements.txt
    └── requirements.yml

1 directory, 6 files

Here’s what each file does.

  • container.yml this is a combo of both inventory and “docker-compose”
  • main.yml this is your main playbook which runs plays against the defined containers in the container.yml
  • requirements.{txt,yml} is your python & role deps respectively
  • meta.yml is for ansible galaxy (should you publish there).
  • ansible.cfg is your… ansible config.

Let’s make our own custom playbooks & roles!

Alright, so go ahead and move back to home and clone my demo-ansible-container repo.

The job of this role is to create two nginx instances (in containers, naturally) that each serve their own custom HTML file (it’s more like a text file).

So let’s clone it and inspect a few things.

$ cd ~
$ git clone https://github.com/dougbtv/demo-ansible-container.git
$ cd demo-ansible-container/

Inspecting the project

Now that we’re in there, let’s show the whole directory structure, it’s basically the same as earlier when we did ansible-container init (as I started that way) plus it adds a ./ansible/roles/ directory which contains roles just as you’d have in your run-of-the-mill ansible project.

.
├── ansible
│   ├── ansible.cfg
│   ├── container.yml
│   ├── main.yml
│   ├── meta.yml
│   ├── requirements.txt
│   ├── requirements.yml
│   └── roles
│       ├── nginx-install
│       │   └── tasks
│       │       └── main.yml
│       ├── ren-html
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       └── index.html
│       └── stimpy-html
│           ├── tasks
│           │   └── main.yml
│           └── templates
│               └── index.html
└── README.md

You’ll note there’s everything we had before, plus three roles.

  • nginx-install: which install (and generally configures) nginx
  • ren-html & stimpy-html: which places specific HTML files in each container

Now, let’s look specifically at the most important pieces.

First, our container.yml

[user@host demo-ansible-container]$ cat ansible/container.yml 
version: '2'
services:
  ren:
    image: centos:centos7
    ports:
      - "8080:80"
    # user: nginx
    command: "nginx" # [nginx, -c, /etc/nginx/nginx.conf]
    dev_overrides:
      ports: []
      command: bin/false
    options:
      kube:
        runAsUser: 997
        replicas: 2
      openshift:
        replicas: 3
  stimpy:
    image: centos:centos7
    ports:
      - "8081:80"
    # user: nginx
    command: [nginx, -c, /etc/nginx/nginx.conf]
    dev_overrides:
      ports: []
      command: bin/false
    options:
      kube:
        runAsUser: 997
        replicas: 2
      openshift:
        replicas: 3
registries: {}

Whoa whoa, whoa Doug! There’s too much there. Yeah, there kind of is. I also put in some goodies to tempt you to look further ;) So, you’ll notice this looks very very much like a docker-compose yaml file.

Mostly though for now, looking at the services section, there’s two services listed ren & stimpy.

These comprise the inventory we’ll be using. And they specify things like… What ports we’re going to run the containers on, especially we’ll be using ports 8080 and 8081 which both map to port 80 inside the container.

Those are the most important for now.

So let’s move onto looking at the main.yml. This is sort of your site playbook for all your containers.

[user@host demo-ansible-container]$ cat ansible/main.yml 
- hosts: all
  gather_facts: false
- hosts: 
    - ren
    - stimpy
  roles:
    - role: nginx-install
- hosts: ren
  roles:
    - role: ren-html
- hosts: stimpy
  roles:
    - role: stimpy-html

So, looks like any other ansible playbook, awesome! The gist is that we use the “host” names ren & stimpy and we run roles against them.

You’ll see that both ren & stimpy have nginx installed into them, but, then use a specific role to install some HTML into each container image.

Feel free to deeply inspect the roles if you so please, they’re simple.

Onto the build!

Now that we’ve got that all setup, we can go ahead and build these container images.

Let’s do that now. Make sure you’re in the ~/demo-ansible-container working dir and not the ~/demo-ansible-container/ansible dir or this won’t work (one of my pet peeves with ansible-container, tbh)

[user@host demo-ansible-container]$ ansible-container build

You’ll see that it spins up some containers and then runs those plays, and you can see it having some specificity to each “host” (each container, really).

When it’s finished it will go and commit the images, to save the results of what it did to the images.

Let’s look at the results of what it did.

[user@host demo-ansible-container]$ docker images
REPOSITORY                                    TAG                 IMAGE ID            CREATED             SIZE
demo-ansible-container-ren                    20170316142619      ba5b90f9476e        5 seconds ago       353.9 MB
demo-ansible-container-ren                    latest              ba5b90f9476e        5 seconds ago       353.9 MB
demo-ansible-container-stimpy                 20170316142619      2b86e0872fa7        12 seconds ago      353.9 MB
demo-ansible-container-stimpy                 latest              2b86e0872fa7        12 seconds ago      353.9 MB
docker.io/centos                              centos7             98d35105a391        16 hours ago        192.5 MB
docker.io/ansible/ansible-container-builder   0.3                 b606876a2eaf        12 weeks ago        808.7 MB

As you can see it’s got it’s special ansible-container-builder image which it uses to bootstrap our images.

Then we’ve got our demo-ansible-container-ren and demo-ansible-container-stimpy each with two tags. One for latest and then anotehr tag with the date and time.

And we run it.

Ok, everything’s built, let’s run it.

ansible-container run --detached --production

You can run without –production and it will just run /bin/false in the container, which may be confusing, but, it’s basically a no-operation and you could use it to inspect the containers in development if you wanted.

When that completes, you should see two containers running.

[user@host demo-ansible-container]$ docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS              PORTS                  NAMES
7b160322dc26        demo-ansible-container-ren:latest      "nginx"                  27 seconds ago      Up 24 seconds       0.0.0.0:8080->80/tcp   ansible_ren_1
a2f8dabe8a6f        demo-ansible-container-stimpy:latest   "nginx -c /etc/nginx/"   27 seconds ago      Up 24 seconds       0.0.0.0:8081->80/tcp   ansible_stimpy_1

Great! Two containers up running on ports 8080 & 8081, just as we wanted.

Finally, verify the results.

You can now see you’ve got Ren & Stimpy running, let’s see what they have to say.

[user@host demo-ansible-container]$ curl localhost:8080
You fat, bloated eeeeediot!

[user@host demo-ansible-container]$ curl localhost:8081
Happy Happy Happy! Joy Joy Joy!

And there we go, two images built, two containers running, all with ansible instructions on how they’re built!

Makes for a very nice paradigm to create images & spin up containers in the context of an Ansible project.

Kuryr-Kubernetes will knock your socks off!

Seeing kuryr-kubernetes in action in my “Dr. Octagon NFV laboratory” has got me feeling that barefoot feeling – and henceforth has completely knocked my socks off. Kuryr-Kubernetes provides Kubernetes integration with OpenStack networking, and today we’ll walk through the steps so you can get your own instance up of it up and running so you can check it out for yourself. We’ll spin up kuryr-kubernetes with devstack, create some pods and a VM, inspect Neutron and verify the networking is working a charm.

As usual with these blog posts, I’m kind of standing on the shoulders of giants. I was able to get some great exposure to kuryr-kubernetes through Luis Tomas’s blog post. And then a lot of the steps here you’ll find familiar from this OpenStack superuser blog post. Additionally, I always wind up finding a good show-stopper or two, and Antoni Segura Puimedon (celebdor) was a huge help in diagnosing my setup, which I greatly appreciated.

Requirements

You might be able to do this with a VM, but, you’ll need some kind of nested virtualization – because we’re going to spin up a VM, too. In my case, I used baremetal and the machine is likely overpowered (48 gig RAM, 16 cores, 1TB spinning disk). I’d recommend no less than 4-8 gigs of RAM and at least a few cores, and maybe 20-40 gigs free (which is still overkill)

One requirement that’s basically for sure is a CentOS 7.3 (or later) install somewhere. I assume you’ve got this setup. Also, make sure it’s pretty fresh because I’ve run into problems with devstack where I tried to put it on an existing machine and it fought with say an existing Docker install.

That box needs git, and maybe your favorite text editor (and I use screen).

Get your devstack up and kickin’

The gist here is that we’ll clone devstack, setup the stack user, create a local.conf file, and then kick off the stack.sh

So here’s where we clone devstack, use it to create a stack user, and move the devstack clone into the stack user’s home and then assume that user.

[root@droctagon3 ~]# git clone https://git.openstack.org/openstack-dev/devstack
[root@droctagon3 ~]# cd devstack/
[root@droctagon3 devstack]# ./tools/create-stack-user.sh 
[root@droctagon3 devstack]# cd ../
[root@droctagon3 ~]# mv devstack/ /opt/stack/
[root@droctagon3 ~]# chown -R stack:stack /opt/stack/
[root@droctagon3 ~]# su - stack
[stack@droctagon3 ~]$ pwd
/opt/stack

Ok, now that we’re there, let’s create a local.conf to parameterize our devstack deploy. You’ll note that my config is a portmanteau of Luis’ and from the superuser blog post. I’ve left in my comments even so you can check it out and compare against the references. Go ahead and put this in with an echo heredoc or your favorite editor, here’s mine:

[stack@droctagon3 ~]$ cd devstack/
[stack@droctagon3 devstack]$ pwd
/opt/stack/devstack
[stack@droctagon3 devstack]$ cat local.conf 
[[local|localrc]]

LOGFILE=devstack.log
LOG_COLOR=False

# HOST_IP=CHANGEME
# Credentials
ADMIN_PASSWORD=pass
MYSQL_PASSWORD=pass
RABBIT_PASSWORD=pass
SERVICE_PASSWORD=pass
SERVICE_TOKEN=pass
# Enable Keystone v3
IDENTITY_API_VERSION=3

# Q_PLUGIN=ml2
# Q_ML2_TENANT_NETWORK_TYPE=vxlan

# LBaaSv2 service and Haproxy agent
enable_plugin neutron-lbaas \
 git://git.openstack.org/openstack/neutron-lbaas
enable_service q-lbaasv2
NEUTRON_LBAAS_SERVICE_PROVIDERV2="LOADBALANCERV2:Haproxy:neutron_lbaas.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default"

enable_plugin kuryr-kubernetes \
 https://git.openstack.org/openstack/kuryr-kubernetes refs/changes/45/376045/12

enable_service docker
enable_service etcd
enable_service kubernetes-api
enable_service kubernetes-controller-manager
enable_service kubernetes-scheduler
enable_service kubelet
enable_service kuryr-kubernetes

# [[post-config|/$Q_PLUGIN_CONF_FILE]]
# [securitygroup]
# firewall_driver = openvswitch

Now that we’ve got that set. Let’s just at least take a look at one parameters. The one in question is:

enable_plugin kuryr-kubernetes \
 https://git.openstack.org/openstack/kuryr-kubernetes refs/changes/45/376045/12

You’ll note that this is version pinned. I ran into a bit of a hitch that Toni helped get me out of. And we’ll use that work-around in a bit. There’s a patch that’s coming along that should fix this up. I didn’t have luck with it yet, but, just submitted the evening before this blog post.

Now, let’s run that devstack deploy, I run mine in a screen, that’s optional for you, but, I don’t wanna have connectivity lost during it and wonder “what happened?”.

[stack@droctagon3 devstack]$ screen -S devstack
[stack@droctagon3 devstack]$ ./stack.sh 

Now, relax… This takes ~50 minutes on my box.

Verify the install and make sure the kubelet is running

Alright, that should finish up and show you some timing stats and some URLs for your devstack instances.

Let’s just mildly verify that things work.

[stack@droctagon3 devstack]$ source openrc 
[stack@droctagon3 devstack]$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

Great, so we have some stuff running at least. But, what about Kubernetes?

It’s likely almost there.

[stack@droctagon3 devstack]$ kubectl get nodes

That’s going to be empty for now. It’s because the kubelet isn’t running. So, open the devstack “screens” with:

screen -r

Now, tab through those screens, hit Ctrl+a then n, and it will go to the next screen. Keep going until you get to the kubelet screen. It will be at the lower left hand size and/or have an * next to it.

It will likely be a screen with “just a prompt” and no logging. This is because the kubelet fails to run in this iteration, but, we can work around it.

First off, get your IP address, mine is on my interface enp1s0f1 so I used ip a and got it from there. Now, put that into the below command where I have YOUR_IP_HERE

Issue this command to run the kubelet:

sudo /usr/local/bin/hyperkube kubelet\
        --allow-privileged=true \
        --api-servers=http://YOUR_IP_HERE:8080 \
        --v=2 \
        --address='0.0.0.0' \
        --enable-server \
        --network-plugin=cni \
        --cni-bin-dir=/opt/stack/cni/bin \
        --cni-conf-dir=/opt/stack/cni/conf \
        --cert-dir=/var/lib/hyperkube/kubelet.cert \
        --root-dir=/var/lib/hyperkube/kubelet

Now you can detach from the screen by hitting Ctrl+a then d. You’ll be back to your regular old prompt.

Let’s list the nodes…

[stack@droctagon3 demo]$ kubectl get nodes
NAME         STATUS    AGE
droctagon3   Ready     4s

And you can see it’s ready to rumble.

Build a demo container

So let’s build something to run here. We’ll use the same container in a pod as shown in the superuser article.

Let’s create a python script that runs an http server and will report the hostname of the node it runs on (in this case when we’re finished, it will report the name of the pod in which it resides)

So let’s create those two files, we’ll put them in a “demo” dir.

[stack@droctagon3 demo]$ pwd
/opt/stack/devstack/demo

Now make the Dockerfile:

[stack@droctagon3 demo]$ cat Dockerfile 
FROM alpine
RUN apk add --no-cache python bash openssh-client curl
COPY server.py /server.py
ENTRYPOINT ["python", "server.py"]

And the server.py

[stack@droctagon3 demo]$ cat server.py 
import BaseHTTPServer as http
import platform

class Handler(http.BaseHTTPRequestHandler):
  def do_GET(self):
    self.send_response(200)
    self.send_header('Content-Type', 'text/plain')
    self.end_headers()
    self.wfile.write("%s\n" % platform.node())

if __name__ == '__main__':
  httpd = http.HTTPServer(('', 8080), Handler)
  httpd.serve_forever()

And kick off a Docker build.

[stack@droctagon3 demo]$ docker build -t demo:demo .

Kick up a Pod

Now we can launch a pod given that, we’ll even skip the step of making a yaml pod spec since this is so simple.

[stack@droctagon3 demo]$ kubectl run demo --image=demo:demo

And in a few seconds you should see it running…

[stack@droctagon3 demo]$ kubectl get pods
NAME                    READY     STATUS    RESTARTS   AGE
demo-2945424114-pi2b0   1/1       Running   0          45s

Kick up a VM

Cool, that’s kind of awesome. Now, let’s create a VM.

So first, download a cirros image.

[stack@droctagon3 ~]$ curl -o /tmp/cirros.qcow2 http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img

Now, you can upload it to glance.

glance image-create --name cirros --disk-format qcow2  --container-format bare  --file /tmp/cirros.qcow2 --progress

And we can kick off a pretty basic nova instance, and we’ll look at it a bit.

[stack@droctagon3 ~]$ nova boot --flavor m1.tiny --image cirros testvm
[stack@droctagon3 ~]$ openstack server list -c Name -c Networks -c 'Image Name'
+--------+---------------------------------------------------------+------------+
| Name   | Networks                                                | Image Name |
+--------+---------------------------------------------------------+------------+
| testvm | private=fdae:9098:19bf:0:f816:3eff:fed5:d769, 10.0.0.13 | cirros     |
+--------+---------------------------------------------------------+------------+

Kuryr magic has happened! Let’s see what it did.

So, now Kuryr has performed some cool stuff, we can see that it created a Neutron port for us.

[stack@droctagon3 ~]$ openstack port list --device-owner kuryr:container -c Name
+-----------------------+
| Name                  |
+-----------------------+
| demo-2945424114-pi2b0 |
+-----------------------+
[stack@droctagon3 ~]$ kubectl get pods
NAME                    READY     STATUS    RESTARTS   AGE
demo-2945424114-pi2b0   1/1       Running   0          5m

You can see that the port name is the same as the pod name – cool!

And that pod has an IP address on the same subnet as the nova instance. So let’s inspect that.

[stack@droctagon3 ~]$ pod=$(kubectl get pods -l run=demo -o jsonpath='{.items[].metadata.name}')
[stack@droctagon3 ~]$ pod_ip=$(kubectl get pod $pod -o jsonpath='{.status.podIP}')
[stack@droctagon3 ~]$ echo Pod $pod IP is $pod_ip
Pod demo-2945424114-pi2b0 IP is 10.0.0.4

Expose a service for the pod we launched

Ok, let’s go ahead and expose a service for this pod. We’ll expose it and see what the results are.

[stack@droctagon3 ~]$ kubectl expose deployment demo --port=80 --target-port=8080
service "demo" exposed
[stack@droctagon3 ~]$ kubectl get svc demo
NAME      CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
demo      10.0.0.84    <none>        80/TCP    13s
[stack@droctagon3 ~]$ kubectl get endpoints demo
NAME      ENDPOINTS       AGE
demo      10.0.0.4:8080   1m

And we have an LBaaS (load balancer as a service) which we can inspect with neutron…

[stack@droctagon3 ~]$ neutron lbaas-loadbalancer-list -c name -c vip_address -c provider
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+------------------------+-------------+----------+
| name                   | vip_address | provider |
+------------------------+-------------+----------+
| Endpoints:default/demo | 10.0.0.84   | haproxy  |
+------------------------+-------------+----------+
[stack@droctagon3 ~]$ neutron lbaas-listener-list -c name -c protocol -c protocol_port
[stack@droctagon3 ~]$ neutron lbaas-pool-list -c name -c protocol
[stack@droctagon3 ~]$ neutron lbaas-member-list Endpoints:default/demo:TCP:80 -c name -c address -c protocol_port
[stack@droctagon3 ~]$ neutron lbaas-member-list Endpoints:default/demo:TCP:80 -c name -c address -c protocol_port

Scale up the replicas

You can now scale up the number of replicas of this pod, and Kuryr will follow along in suit. Let’s do that now.

[stack@droctagon3 ~]$ kubectl scale deployment demo --replicas=2
deployment "demo" scaled
[stack@droctagon3 ~]$ kubectl get pods
NAME                    READY     STATUS              RESTARTS   AGE
demo-2945424114-pi2b0   1/1       Running             0          14m
demo-2945424114-rikrg   0/1       ContainerCreating   0          3s

We can see that more ports were created…

[stack@droctagon3 ~]$ openstack port list --device-owner kuryr:container -c Name -c 'Fixed IP Addresses'
[stack@droctagon3 ~]$ neutron lbaas-member-list Endpoints:default/demo:TCP:80 -c name -c address -c protocol_port

Verify connectivity

Now – as if the earlier goodies weren’t fun, this is the REAL fun part. We’re going to enter a pod, e.g. via kubectl exec and we’ll go ahead and check out that we can reach the pod from the pod, and the VM from the pod, and the exposed service (and henceforth both pods) from the VM.

Let’s do it! So go and exec the pod, and we’ll give it a cute prompt so we know where we are since we’re about to enter the rabbit hole.

[stack@droctagon3 ~]$ kubectl get pods
NAME                    READY     STATUS    RESTARTS   AGE
demo-2945424114-pi2b0   1/1       Running   0          21m
demo-2945424114-rikrg   1/1       Running   0          6m
[stack@droctagon3 ~]$ kubectl exec -it demo-2945424114-pi2b0 /bin/bash
bash-4.3# export PS1='[user@pod_a]$ '
[user@pod_a]$ 

Before you continue on – you might want to note some of the IP addresses we showed earlier in this process. Collect those or chuck ‘em in a note pad and we can use them here.

Now that we have that, we can verify our service locally.

[user@pod_a]$ curl 127.0.0.1:8080
demo-2945424114-pi2b0

And verify it with the pod IP

[user@pod_a]$ curl 10.0.0.4:8080
demo-2945424114-pi2b0

And verify we can reach the other pod

[user@pod_a]$ curl 10.0.0.11:8080
demo-2945424114-rikrg

Now we can verify the service, note how you get different results from each call, as it’s load balanced between pods.

[user@pod_a]$ curl 10.0.0.84
demo-2945424114-pi2b0
[user@pod_a]$ curl 10.0.0.84
demo-2945424114-rikrg

Cool, how about the VM? We should be able to ssh to it since it uses the default security group which is pretty wide open. Let’s ssh to that (reminder, password is cubswin:)) and also set the prompt to look cute.

[user@pod_a]$ ssh cirros@10.0.0.13
The authenticity of host '10.0.0.13 (10.0.0.13)' can't be established.
RSA key fingerprint is SHA256:Mhz/s1XnA+bUiCZxVc5vmD1C6NoeCmOmFOlaJh8g9P8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.13' (RSA) to the list of known hosts.
cirros@10.0.0.13's password: 
$ export PS1='[cirros@vm]$ '
[cirros@vm]$ 

Great, so that definitely means we can get to the VM from the pod. But, let’s go and curl that service!

[cirros@vm]$ curl 10.0.0.84
demo-2945424114-pi2b0
[cirros@vm]$ curl 10.0.0.84
demo-2945424114-rikrg

Voila! And that concludes our exploration of kuryr-kubernetes for today. Remember that you can find the Kuryr crew on the openstack mailing lists, and also in Freenode @ #openstack-kuryr.