Let's try Kuryr!

We’re going to spin up a simple implementation of Kuryr. Kuryr is a Docker network driver for Neutron that I’m pretty excited about. And especially pumped to get it to run on OpenShift (and doubly so there, OpenShift on OpenStack). Of all of the things that’s gotten me exicted is this video from OpenStack Summit Austin. We’ll use an existing openstack to spin up an instance, and in that instance we’ll run devstack (I guess that would be Quintiple-O in my case as this is a devstack ontop of Triple-O). On that box, we’ll run kuryr. I recently visited the crew in #openstack-kuryr on Freenode and I found that they’re currently working on kuryr for openshift-on-openstack (awesome!). I can’t hardly wait.

Let’s get devstack up.

To get started I used a portmanteau of devstack quick start and the all in one single vm with cloud-init instructions.

If you don’t want to use this all-in-one-vm method ontop of an existing openstack, just go ahead and follow the quick start on your own and skip down to the “Onto Kuryr!” section.

So I went ahead and made a file devstack-init.txt based on the exmaple I found in the all in one single vm with cloud-init instructions and I also referenced this blog article Because I like a few extra references. In short, this adds a user named stack, tosses a shell script in their home dir, and then runs that script to install git, clone devstack and then set a few passwords in local.conf.

[stack@undercloud ~]$ cat devstack-init.txt 

  - default
  - name: stack
    lock_passwd: False
    sudo: ["ALL=(ALL) NOPASSWD:ALL\nDefaults:stack !requiretty"]
    shell: /bin/bash

  - content: |
        sudo yum install -y nano git
        sudo chown stack:stack /home/stack
        cd /home/stack
        git clone https://git.openstack.org/openstack-dev/devstack
        cd devstack
        echo '[[local|localrc]]' > local.conf
        echo ADMIN_PASSWORD=password >> local.conf
        echo DATABASE_PASSWORD=password >> local.conf
        echo RABBIT_PASSWORD=password >> local.conf
        echo SERVICE_PASSWORD=password >> local.conf
    path: /home/stack/start.sh
    permissions: 0755

  - su -l stack ./start.sh

And then I booted specifying the –user-data like:

[stack@undercloud ~]$ internal_net=$(neutron net-list | awk ' /int/ {print $2;}')
[stack@undercloud ~]$ nova boot --flavor m1.ramheavy --key-name atomic_key --nic net-id=$internal_net --user-data devstack-init.txt --image centos-generic devstack

(A few notes: I made my own flavor for this, you can sub in any flavor you want, recommending 4+ gigs, I used 16 gigs cause I’ve got the overhead. Also, the image is a CentOS Generic Cloud image. I also assigned a floating IP so I could access the box from an external network, and I used a key that I keep handy around for it. You can see some of my previous blog entries if you need help with uploading a glance image, creating nova keys, or creating internal/external networks.)

Let that bad boy boot right up, and give ‘er some time to install devstack. While it’s working you can go ahead and tail the cloud-init logs a la:

[user@your_laptop ~]$ ssh -i atomic_key centos@
[centos@devstack ~]$ sudo tail -f /var/log/cloud-init.log 

When it’s finished it’ll give you some information about where Horizon & Keystone are and all that good stuff, such as:

Dec  1 17:39:17 localhost cloud-init: This is your host IP address:
Dec  1 17:39:17 localhost cloud-init: This is your host IPv6 address: ::1
Dec  1 17:39:17 localhost cloud-init: Horizon is now available at
Dec  1 17:39:17 localhost cloud-init: Keystone is serving at
Dec  1 17:39:17 localhost cloud-init: The default users are: admin and demo
Dec  1 17:39:17 localhost cloud-init: The password: password

Being used to TripleO I was expecting to find a stackrc file in the stack user’s home. Not so, or, at least not immediately apprent just obviously looking for that file.

That having been said and I might be missing something I made my own cloudrc file and you can too (doesn’t take long), I sourced it, and then I ran neutron net-list just to see if things are working

-bash-4.2$ whoami
-bash-4.2$ pwd
-bash-4.2$ cat cloudrc 
export OS_NO_CACHE=True
export OS_AUTH_URL=
export NOVA_VERSION=1.1
export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_TENANT_NAME=admin

-bash-4.2$ source cloudrc 
-bash-4.2$ neutron net-list
| id                                   | name    | subnets                                                  |
| 3d2435a1-1470-4032-afb6-f9235c2024f6 | private | 969eb0eb-f799-4978-9f8a-97846017e7d9 fdca:e1d0:c7a0::/64 |
|                                      |         | b21aa09e-e042-4da5-955f-432130dd697c         |
| 9ae5a213-9be9-4016-8e79-4fcf6d6a0fa6 | public  | 18c4c828-ae79-4260-a9ec-53152a6a7d3a       |
|                                      |         | 596ebe49-afc5-4532-8459-543062f2c52d 2001:db8::/64       |

Onto Kuryr!

Alright, so that’s looking great, let’s try kuryr. We’re going to be using Kuryr-libnetwork documentation

We’re going to need a Docker install, so let’s install from Docker’s RPM repo.

sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
name=Docker Repository

And then install docker and enable and start the service

-bash-4.2$ sudo yum install -y docker-engine
-bash-4.2$ sudo systemctl enable docker
-bash-4.2$ sudo systemctl start docker

Then we’re gonna pull the Kuryr image instead of building it for now, we’ll learn more about it, and how it’s built (probably a lot more) later on.

docker pull kuryr/libnetwork:latest

Then prepare docker to find the driver…

sudo mkdir -p /usr/lib/docker/plugins/kuryr
[centos@devstack ~]$ sudo mkdir -p /usr/lib/docker/plugins/kuryr
[centos@devstack ~]$ sudo curl -o /usr/lib/docker/plugins/kuryr/kuryr.spec \
[centos@devstack ~]$ sudo systemctl restart docker

Prepare the docker run command…

docker run --name kuryr-libnetwork \
  --net=host \
  --cap-add=NET_ADMIN \
  -e SERVICE_USER=admin \
  -e SERVICE_PASSWORD=password \
  -e USER_DOMAIN_NAME=Default \
  -v /var/log/kuryr:/var/log/kuryr \
  -v /var/run/openvswitch:/var/run/openvswitch \

Not that I changed a few things here – first, my password, and I specified the IDENTITY_URL to be for Keystone v3. I also curled it to double check it was the right endpoint. I left SERVICE_DOMAIN_NAME & USER_DOMAIN_NAME as default.

That’ll be running in the foreground, looks like it’s running for me, with something like:

[root@devstack centos]# docker run --name kuryr-libnetwork \
>   --net=host \
>   --cap-add=NET_ADMIN \
>   -e SERVICE_USER=admin \
>   -e SERVICE_PASSWORD=password \
>   -e SERVICE_DOMAIN_NAME=Default \
>   -e USER_DOMAIN_NAME=Default \
>   -e IDENTITY_URL= \
>   -v /var/log/kuryr:/var/log/kuryr \
>   -v /var/run/openvswitch:/var/run/openvswitch \
>   kuryr/libnetwork
*** Starting uWSGI 2.0.13 (64bit) on [Thu Dec  1 19:09:39 2016] ***
compiled with version: 5.3.0 on 16 May 2016 19:00:18
os: Linux-3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016
nodename: devstack
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /
detected binary path: /usr/sbin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
your memory page size is 4096 bytes
detected max file descriptor number: 65536
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :23750 fd 3
Python version: 2.7.12 (default, Jun 29 2016, 08:57:23)  [GCC 5.3.0]
Python main interpreter initialized at 0x7fecd267a580
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 249168 bytes (243 KB) for 4 cores
*** Operational MODE: preforking+threaded ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x7fecd267a580 pid: 8 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 8)
spawned uWSGI worker 1 (pid: 12, cores: 2)
spawned uWSGI worker 2 (pid: 13, cores: 2)

Alright, that’s all well and good, but, now that it’s running…. how do we make it “do stuff”? Let’s look at that.

Let’s use the documentation from the README on the github page make a network with it:

[root@devstack centos]# docker network create --driver kuryr test_net
[root@devstack centos]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
bfb511a6c689        bridge              bridge              local               
dfd800889f92        host                host                local               
37d8a8e9fc82        none                null                local               
52abdbb7f76f        test_net            kuryr               local               
[root@devstack centos]# 

Wooooord! That makes me excited, looks like there’s a lot of things working in harmony if we can get to this point.

You can also inspect it if you’d like

[root@devstack centos]# docker network inspect test_net

Looks like for right now we’re going to have an isolated subnet using this driver, but, that’s OK. Let’s spin up a couple instances and ping one from the other.

First instance will be a dummy and then we’ll spin up an extra and ping the first.

[root@devstack centos]# docker pull centos:centos7
[root@devstack centos]# docker run --network=test_net -d centos:centos7 /bin/bash -c 'sleep 500'
[root@devstack centos]# docker run --network=test_net -it --rm centos:centos7 /bin/bash

Ok, now that you have two containers up. The first is a dummy that will be up for 8 minutes. The second one will dump you into an interactive bash shell.

From another shell, let’s inspect the test_net we created with a Kuryr driver earlier.

[root@devstack centos]# docker network inspect test_net | grep -i IPv4
                "IPv4Address": "",
                "IPv4Address": "",

And we’ll go into the interactive shell and we’ll ping both for phun and prophit.

[root@dc83ef2da5fc /]# ping -c2 && ping -c2
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.321 ms
64 bytes from icmp_seq=2 ttl=64 time=0.087 ms

--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.087/0.204/0.321/0.117 ms
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from icmp_seq=2 ttl=64 time=0.041 ms

--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.024/0.032/0.041/0.010 ms

Woo hoo! We can ping containers over it! (Anti-climatic, yet, somehow satisfying.)

That’ll be all for now, in coming articles we’ll look at how to actually do something useful with it. And even better, we’ll start to look at how cool kuryr with k8s intergration could be!!! That’s the really exciting stuff.

Specifying the root device with Ironic

So I was wondering why I got the fabled No valid host was found. There are not enough hosts available when adding a compute node to my cluster. And I spun my wheels for a while, and I saw with a ironic node-show ${uuid} that well… There was only 13 gig of local disk on the node. Wat!?!1! Yeah. That’s not right, there’s a 120 gig SSD in there. Is the disk bad? Let me boot off the USB stick on the box and try to use the HDD. Wait, I can’t boot off that USB thumb drive? Yikes. Ohhhh, during deploy the USB stick got wiped! Henceforth, why you should specify a root device if you’ve got multiple storage devices.

So, I used the setting the root device for deployment docs to go ahead and specify what the root device is. I also found that the TripleO docs on baremetal overclouds has a reference to this in the “Enrolling nodes” section.

Well first question I asked, what disk is it. There’s a recipe for that, and you can see my /dev/sda is, well… Dangnabbit, the USB stick.

[stack@undercloud ~]$ openstack baremetal introspection data save 59a6fe44-899c-4907-ad8d-b104754a9a6a | jq '.inventory.disks'
    "rotational": true,
    "vendor": "Kingston",
    "name": "/dev/sda",
    "wwn_vendor_extension": null,
    "wwn_with_extension": null,
    "model": "DT 101 G2",
    "wwn": null,
    "serial": "001372995FC8EC3025BF0031",
    "size": 15606349824
    "rotational": false,
    "vendor": "ATA",
    "name": "/dev/sdb",
    "wwn_vendor_extension": null,
    "wwn_with_extension": "0x5f8db4c391600cf2",
    "model": "PNY CS1311 120GB",
    "wwn": "0x5f8db4c391600cf2",
    "serial": "PNY39162193120100CF2",
    "size": 120034123776

I thought we may need to go and specify some property in our instackenv. So let’s do that now, and we’ll try to import a single node to ironic. (Which is totally an experiment for me, I’ve only done my entire inventory at once, but, I don’t wanna go too many steps backwards in my deployment right now). And just for the record – the experiment failed. I couldn’t get that to import properly with openstack baremetal import single_node.json. So I’ve selected a different route. Which we’ll see below. However, I’m putting this into my playbook for dr. octagon for now as an experiment, we’ll see next time I import the entire instackenv.

I’m going to choose the serial number as the parameter I will use. And I had a simple single_node.json file that looked like:

  "nodes": [{
    "pm_type": "pxe_ipmitool",
    "mac": [
    "cpu": "2",
    "memory": "48130",
    "disk": "120",
    "root_device": {
      "serial": "PNY39162193120100CF2"
    "arch": "x86_64",
    "pm_user": "stack",
    "pm_password": "redhatz",
    "capabilities": "profile:compute,boot_option:local",
    "pm_addr": ""

I’m going to update the node manually for now.

[stack@undercloud ~]$ ironic node-update 59a6fe44-899c-4907-ad8d-b104754a9a6a add properties/root_device='{"serial": "PNY39162193120100CF2"}'

You can see that if you do a ironic node-show ${uuid} now, too.

I also re-ran introspection, with a process I’d’ summarize as:

[stack@undercloud ~]$ ironic node-set-provision-state 59a6fe44-899c-4907-ad8d-b104754a9a6a manage
[stack@undercloud ~]$ openstack baremetal introspection start 59a6fe44-899c-4907-ad8d-b104754a9a6a
[stack@undercloud ~]$ ironic node-set-provision-state 59a6fe44-899c-4907-ad8d-b104754a9a6a provide

In the future what I’m going to do, to make for a sure thing configure boot with:

openstack help baremetal configure boot --root-device-minimum-size 40

That should do it. Now…. To flash another live O/S on that thumb drive ;)

TripleO Overcloud deploy failed? Much sadness. Debugging your isolated network configuration

In setting up my home lab, Dr. Octagon I’ve been really stubbing my toe here and there getting the heat templates for network isolation just right. I still haven’t come exactly to the conclusions that I need to come to, but in the process I’ve gained a better understanding of the process for debugging what’s wrong with the templates in the process.

Some things have just been kicking me square in the keister and causing me much sadness.

I’ve also collected a fair number of links & resources for helping you out in the meanwhile, too. Which I thought I’d share.

Network Isolation Resources


Some other references

Through the process I also referenced cybertron’s blog and his tripleo-network-templates. He admits that they’re out of date, but, was worthwhile to see another example (even if it didn’t work off the shelf for me).

Factory reset & configure NetGear GSM7224 (without ezconfig)

So I am in the process of reconfiguring my home lab setup, which is mostly @ Laboratory B (Burlington Vermont’s hackerspace). And I needed a new managed switch, so, I bought an old managed switch (fancy that.) And it’s old, and the documentation on it is… quite garbage. It’s all there, but, good luck finding what you need in it. So I figured I’d save my steps in case I need to do this again.

First thing for me was that… Well I followed the manual, but, I couldn’t get it to listen on the default IP, so I had to console in via serial.

[root@whistler doug]# screen /dev/ttyS0 9600

And then I logged in (username: admin / password: {blank})

But I tried the ezconfig command, what just came back with what looked like a syntax error. Oh well, like it was an unrecognize command.

So I tried to factory reset it by just hitting the “reset” and doing a factory reset at the console, which didn’t help me with much. (You get a short chance for it to allow you “start [the] boot menu”)

Verifying Operational Code CRC in Flash...100%
Select an option. If no selection in 10 seconds then
operational code will start.

1 - Start operational code.
2 - Start Boot Menu.
Select (1, 2):2

Boot Menu
Options available
 1 - Start operational code
 2 - Change baud rate
 3 - Retrieve event log using XMODEM (64KB)
 4 - Load new operational code using XMODEM
 5 - Run Flash Diagnostics
 6 - Update Boot Code
 7 - Reset the system
 8 - Restore Configuration to factory defaults (delete config files) and reset
 9 - Display operational code vital product data
[Boot Menu] 1

Figured out I could show what was there.

(GSM7224) #show network

IP Address.....................................
Subnet Mask....................................
Default Gateway................................
Burned In MAC Address.......................... 00:0F:B5:FF:17:83
Network Configuration Protocol Current......... DHCP
Management VLAN ID............................. 1
Web Mode....................................... Enable
Java Mode...................................... Enable

And then pull up the help for the command to configure, which didn’t tell me every one of the parameters, helpful.

(GSM7224) #network parms ?

<ipaddr>                 Enter the IP Address.

So you actually issue it like so:

(GSM7224) #network parms
Network protocol must be none to set ip address.

But yeah you gotta unset the network protcol, turns out.

(GSM7224) #network protocol none

Changing protocol mode will reset ip configuration.
Are you sure you want to continue? (y/n)y

And then issue the command.

(GSM7224) #network parms

(GSM7224) #show network

IP Address.....................................
Subnet Mask....................................
Default Gateway................................
Burned In MAC Address.......................... 00:0F:B5:FF:17:83
Network Configuration Protocol Current......... None
Management VLAN ID............................. 1
Web Mode....................................... Enable
Java Mode...................................... Enable


virt-manager with X11 forwarding

Sometimes you just wanna see your machines in virt-manager, and the console is handy, too, right? virsh is awesome and all, but, sometimes I just like having the GUI. For me with a default CentOS 7 minimal install and using oooq (Triple-O Quick start) there’s a few extra steps that I needed.

First things first, if you ssh to your box with x11 forwarding like so:

ssh -Y stack@eendracht

But, I’d get a

X11 forwarding request failed on channel 0

So I found out that I needed to change /etc/ssh/sshd_config specifically I needed to have this line:

X11UseLocalhost no

And I also needed to install xauth

sudo yum install -y xauth

And then systemctl restart sshd, and… again

ssh -Y stack@eendracht

And away you go! But, tip and word to the wise… With virt-manager, it’s not going to look for your “user session” by default. So go ahead and in virt-manager GUI go and browse to “File -> Add Connection…” and from there in “Hypervisor” dropdown choose the “QEMU/KVM user session”. Now you should see the hosts that oooq created under the stack user.

virt-manager user session

I used to like to just use the virt-manager local to my machine and use an SSH session, however, I haven’t yet found a way to directly do that AND also use the user session.