06 May 2023
In today’s tutorial, we’re going to install Stable Diffusion on Fedora 38.
I’m putting together a lab machine for GPU workloads. And the first thing I wanted to do was get Stable Diffusion running, and I’m also hopeful to start using it for training LoRA’s, embeddings, maybe even a fine tuning checkpoint (we’ll see).
Fedora is my default home server setup, and I didn’t find a direct guide on how to do it, although it’s not terribly different from other distros
…Oddly enough I actually fired this up with Fedora Workstation.
Requirements
- An install of Fedora 38
- A nVidia GPU (if someone has insight on AMD GPUs, and wants to provide instructions, hit me up and I’ll update the article)
Installing Automatic Stable Diffusion WebUI on Fedora 38
I’m going to be using Vladmanic’s fork of Automatic1111 sd webui: https://github.com/vladmandic/automatic
Clone it.
Fedora 38 ships with Python 3.11, but some dependency for stable diffusion requires python 3.11, which will require a few extra steps.
Install python 3.10
Also, before you install CUDA, do a dnf update (otherwise I wound up with mismatched deps for NetworkManager and couldn’t boot off a new kernel, and I had to wheel up a crash cart, just kidding I don’t have a crash cart or a KVM for my Linux lab so it’s much more annoying where I move my server to my workstation area, luckily I just have a desktop server lab)
Install CUDA Toolkit (link is for F37 RPM, but it worked fine on F38)
And – follow the instructions there. You might need to reboot now.
Make a handler script to export the correct python version… I named mine user-webui.sh
#!/bin/bash
export python_cmd=python3.10
screen -S ./webui.sh --listen
NOTE: I fire it up in screen. If you don’t have Stockholm Syndrome for screen you can decide to not be a luddite and modify it to use tmux. And if you need a cheat sheet for screen, there you go. I also use the --listen flag because I’m going to connect to this from other machines on my network.
Then run the ./user-webui.sh once to get the venv, it will likely fail at this point. Or if you’re a smarter python user, create the venv yourself.
Then enter the venv.
Then ensurepip…
And now you can fire up the script!
14 Apr 2023
Have you played with ChatGPT yet? Ummm, yeah, who hasn’t!? I have pirate-styled rap battles to make! So let’s get right to the point so we can get back to generating rap-battles as soon as possible!
Today we’re going to run a LLM (Large Language Model) locally on one of our own machines, and we’re going to set it up so that we can interface with it via API, and we’ll even write a small program to test it out.
I have some ideas where I want to take some software I’m building and hook it up to one of these, later I’d like to train it on custom data and then query it. Maybe even have some like real-ish-time data fed to it and then be able to query it in near-real-time too. I also have some ideas for populating it with logs and then being like “yo, tell me what’s up with this machine?”
But yeah, instead of relying on a GPT service, I want to run the LLM myself, using open source tools.
Pre-requisites
Ok, so I’m not going to go deep into the details of the installation, so I’m just going to give some pointers. It’s not necessarily rocket science
First up, we’re going to install a webUI, OobaBooga: https://github.com/oobabooga/text-generation-webui
This is one of the few times I’m going to say the word “windows” on this blog, but, I actually installed mine on windows, because it’s a windows box that’s an art and music workstation where I’ve got my decent GPU (for gaming and also for stable diffusion and then associated windoze-y art tools). I follow this youtube video by @TroubleChute. I even used his opinionated script to automatically install Vicuna!
But you can also just install it with the instructions on the README, which appears to be really straight forward.
The Model we’re going to use, Vicuna, you can find it @ https://vicuna.lmsys.org/ – the thing that’s interesting about Vicuna is that it’s trained on crowd-sourced GPT output, and claims be 90% as good as GPT, which seems like a lofty statement. But so far it does seem like it’s pretty decent, even if it does parrot a lot of the kind of “walled garden” stuff that ChatGPT says (“As an AI language model, I can’t tell you what you asked me for” kind of stuff.)
Quick tip: After you install it and you’re playing with it on the WebUI, assuming you’ve installed Vicuna… Go to the bottom of the chat and find the radio button for “mode” and switch it to “instruct” and then in the “instruction template” drop down, select “Vicuna” – this will parse the output from the LLM so it makes more sense in context with your queries/conversation/prompts to it.
Well, first, in your text-generation-webui git clone – go and check out the ./extensions/api folder and make sure there’s a script there, there should be by default.
Next, we’re going to enable that by adding a flag to our startup script.
In your startup script, add a --extensions api field, and also a --listen flag (if you want to listen on all addresses and not just your loopback so you can access the api from your LAN/WAN).
So now, for example, mine looks like:
python server.py --extensions api --listen --auto-devices --chat --model anon8231489123_vicuna-13b-GPTQ-4bit-128g --wbits 4 --groupsize 128
Now your api should be ready to query.
Calling the API
Then I whipped together a node.js script (with ChatGPT’s help!) to query the API.
I’ve got the script here in this github gist.
Lucky you, the script asks for suggestions for cheese. If you ask me, I’d tell you have to have a Bayley Hazen blue, or maybe the Cabot Clothbound both from Jasper Hill Farms (they’ve go the raddest cheese cave, super good cheese)
And you can just run that with
node oobabooga_api_query.js
And away you go. Or if you don’t like Node, just have ChatGPT re-write it for you in assembly or something!
(The script still needs a little help parsing, but it’s close! I think it’s kind of missing the kind of “mode” + “template” from the UI, so that’s next to figure out)
B-b-b-bonus hip hop battle
Just for you, I asked it to:
write a rap battle about api clients written in node.js vs. assembly
And it gave me:
Verse 1 (Node.js):
I'm Node, I'm fast and sleek,
When it comes to APIs, I can't be beat!
With JavaScript at my core,
I make building apps a breeze, no need to feel bore-ot.
Chorus:
API clients, we rule the land,
Node.js or Assembly, which one takes the stand?
We may not be as old school,
But when it comes to speed and coolness, we'll never lack cool.
Verse 2 (Assembly):
I may be older than you think,
But don't underestimate me, I still have some tricks.
With raw power and precision,
My code runs smoothly without any friction.
Chorus:
API clients, we rule the land,
Node.js or Assembly, which one takes the stand?
We may not be as old school,
But when it comes to speed and coolness, we'll never lack cool.
Outro:
In the end, it doesn't matter who wins this battle,
Because both Node.js and Assembly are great for different purposes, so let's celebrate!
Thanks for your “everyone gets an award!” style indifference LLM!
…But not too shabby for something running on my local!
14 Jul 2022
The gist of Chainsaw CNI (brum-brum-brum-brum-brrrrrrrrr) is it’s a CNI plugin that runs in a CNI chain (more on that soon), and it allows you to run arbitrary ip commands against your Kubernetes pods to either manipulate or inspect networking. You can do this at run-time by annotating a pod with the commands you want to run.
For example, you can annotate a pod with:
k8s.v1.cni.cncf.io/chainsaw: >
["ip route","ip addr"]
And then get the output of ip route and ip addr for your pod.
I named it Chainsaw because:
- It works using CNI Chains.
- It’s powerful, but kind of dangerous.
Today, we’re going to:
- Talk about why I made it.
- Look at what CNI chains are.
- See what the architecture is comprised of.
- And of course, engage the choke, pull the rope start and fire up this chainsaw.
We’ll be using it with network attachment definitions – that is, the custom resource type that’s used by Multus CNI
Why do you say it’s dangerous? Well, like a chainsaw, you do permanent harm to something. You could totally turn off networking for a pod. Or, potentially you open up a way for some user of your system to do something more privileged than you thought. I’m still thinking about how to better address this part, but for now… I’d advise that you use it carefully, and in lab situations rather than production before these aspects are more fully considered.
Also, as an aside… I am a physical chainsaw user. I have one and, I use it. But I’m appropriately afraid of it. I take a long long time to think about it before I use it. I’ve watched a bunch of videos about it, but I really want to take a Game Of Logging course so I can really operate it safely. Typically, I’m just using my Silky Katanaboy (awesome Japanese pull saw!) for trail work and what not.
Last but not least, a quick disclaimer: This is… a really new project. So it’s missing all kinds of stuff you might take for granted: unit tests, automatic builds, all that. Just a proof of concept, really.
Why, though?
I was originally inspired by this hearing this particular discussion:
Person: “Hey I want to manipulate a route on a particular pod”
Me: “Cool, that’s totally possible, use the route override CNI” (it’s another chained plugin!)
Person: “But I don’t want to manipulate the net-attach-def, there’s tons of pods using them, and I only want to manipulate for a specific site, so I want to do it at runtime, adding more net-attach-defs makes life harder”.
Well, this kinda bothered me! I talked to a co-worker who said “Sure, next they’re going to want to change EVERYTHING at runtime!”
I was thinking: “hey, what if you COULD change whatever you wanted at runtime?”
And I figured, it could be a really handy tool, even if just for CNI developers, or network tinkerers as it may be.
CNI Chains
┌──────────────────┐ ┌────────────────┐
│ │ │ │
│ │ ┌───────────┐ │ │
│ CNI Plugin A │ │ │ │ CNI Plugin B │
│ ├───► cni result├───► │
│ │ │ │ │ │
│ │ └───────────┘ │ │
└──────────────────┘ └────────────────┘
CNI chains are… sometimes confusing to people. But, they don’t need to be, it’s basically as simple as saying, “You can chain as many CNI plugins together as you want, and each CNI plugin gets all the CNI results of the plugin before it”
This functionality was introduced in CNI 0.3.0 and is available in all later versions of CNI, naturally.
You can tell if you have a CNI plugin chain by looking at your CNI configuration, if the top level JSON has the "type" field – then it’s not a chain.
If it has the "plugins": [] array – then it’s a chain of plugins, and will run in the order within the array. As of CNI 1.0, you’ll always be using the plugins field, and always have chains, even if a “chain of one”.
Why do you use chained plugins? The best example I can usually think of is the Tuning Plugin. Which allows you to set network sysctls, or manipulate other parameters of networks – such as setting an interface into promiscuous mode. This is done typically after the work of your main plugin, which is going to do the plumbing to setup the networking for you (e.g. say, a vxlan tunnel, or a macvlan interface, etc etc).
The architecture
Not a whole lot to say, but it’s a “sort of thick plugin” – thick CNI plugins are those that have a resident daemon, as opposed to “thin CNI plugins” – which run as a one-shot (all of the reference CNI plugins are one shots). But in this case, we just use the daemonset that’s resident for looking at the log output, for inspecting our results.
Other than that, it’s similar to Multus CNI in that it knows how to talk to the k8s API and get the annotations, and it uses a generated kubeconfig to authorize itself against the k8s API
Let’s get to using it!
Requirements:
- A k8s cluster, the newer the beter.
- Multus CNI must be installed
That’s about it. Don’t use a production cluster ;)
So go ahead and clone dougbtv/chainsaw-cni.
Then create the daemonset with:
kubectl create -f deployments/daemonset.yaml
NOTE: Are you an openshift user? Use the deployments/daemonset_openshift.yaml deployment instead :thumbsup:
Now, let’s create a net-attach-def which implements chainsaw in a chain – note the plugins array!
Also note the use of the special token CURRENT_INTERFACE which will use the current interface name as opposed to you having to know it in advance.
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: test-chainsaw
spec:
config: '{
"cniVersion": "0.4.0",
"name": "test-chainsaw-chain",
"plugins": [{
"type": "bridge",
"name": "mybridge",
"bridge": "chainsawbr0",
"ipam": {
"type": "host-local",
"subnet": "192.0.2.0/24"
}
}, {
"type": "chainsaw",
"foo": "bar"
}]
}'
---
apiVersion: v1
kind: Pod
metadata:
name: chainsawtestpod
annotations:
k8s.v1.cni.cncf.io/networks: test-chainsaw
k8s.v1.cni.cncf.io/chainsaw: >
["ip route add 192.0.3.0/24 dev CURRENT_INTERFACE", "ip route"]
spec:
containers:
- name: chainsawtestpod
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: alpine
Next, check what node the pod is running with:
You can then find the output from the results of the ip commands from the chainsaw daemonset that is running on that node, e.g.
kubectl get pods -n kube-system -o wide | grep -iP "status|chainsaw"
And looking at the logs for the daemonset pod that correlates to the node on which the pod resides, for example:
kubectl logs kube-chainsaw-cni-ds-kgx69 -n kube-system
You’ll see that we have added a route to 192.0.3.0/24 and then show the IP route output!
So my results look like:
Detected commands: [route add 192.0.3.0/24 dev CURRENT_INTERFACE route]
Running ip netns exec 901afa16-48e7-4f22-b2b1-7678fa3e9f5e ip route add 192.0.3.0/24 dev net1 ===============
Running ip netns exec 901afa16-48e7-4f22-b2b1-7678fa3e9f5e ip route ===============
default via 10.129.2.1 dev eth0
10.128.0.0/14 dev eth0
10.129.2.0/23 dev eth0 proto kernel scope link src 10.129.2.64
172.30.0.0/16 via 10.129.2.1 dev eth0
192.0.2.0/24 dev net1 proto kernel scope link src 192.0.2.51
192.0.3.0/24 dev net1 scope link
224.0.0.0/4 dev eth0
14 May 2021
If you’re looking at developing (or debugging!) CNI plugins, you’re going to need a workflow for developing CNI plugins – something that really lets you get in there, and see exactly what a CNI plugin is doing. You’re going to need a bit of a swiss army knife, or something that slices, dices, and makes juilienne fries. cnitool is just the thing to do the job. Today we’ll walk through setting up cnitool, and then we’ll make a “dummy” CNI plugin to use it with, and we’ll run a reference CNI plugin.
We’ll also cover some of the basics of the information that’s passed to and from the CNI plugins and CNI itself, and how you might interact with that information, and how you might inspect a container that’s been plumbed with interfaces as created by a CNI plugin.
In this article, we’ll do this entirely without interacting with Kubernetes (and save it for another time!). And we actually do it without a container runtime at all – no docker, no crio. We just create the network namespace by hand. But the same kind of principles apply with both a container runtime (docker, crio) or a container orchestration enginer (e.g. k8s)
You might remember my blog article about a workflow for developing CNI plugins. That article uses the docker-run.sh, which is still totally valid. You might look at it for a reference, but CNI tool gives a bit more granularity.
Prerequisites
- Golang installed and configured on your system.
- I used a Fedora environment, these steps probably work elsewhere.
Basically, all the steps necessary to install cnitool are available in the cnitool README. I’ll summarize them here, but, it may be worth a reference.
Install cnitool…
go get github.com/containernetworking/cni
go install github.com/containernetworking/cni/cnitool
You can test if it’s in your path and operational with:
Next, we’ll compile the “reference CNI plugins” – these are a series of plugins that are offered by the CNI maintainers that create network interfaces for pods (as well as provide a number of “meta” type plugins that alter the properties, attributes, and what not of a particular container’s network). We also set our CNI_PATH variable (which is used by cnitool to know where these plugin executables are)
git clone https://github.com/containernetworking/plugins.git
cd plugins
./build_linux.sh
export CNI_PATH=$(pwd)/bin
echo $CNI_PATH
Alright, you’re basically all setup at this point.
We’ll need to create a CNI configuration. For testing purposes, we’re going to create a configuration for the bridge CNI.
Create a directory and file at /tmp/cniconfig/10-myptp.conf with these contents:
{
"cniVersion": "0.4.0",
"name": "myptp",
"type": "ptp",
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "172.16.29.0/24",
"routes": [{
"dst": "0.0.0.0/0"
}]
}
}
And then set your CNI configuration directory by exporting this variable as:
export NETCONFPATH=/tmp/cniconfig/
First we create a netns – a network namespace. This is kind of a privately sorta-jailed space in which network components live, and is the basis of networking in containers, “here’s your private namespace in which to do your network-y things”. This, from a CNI point of view, is equivalent to the “sandbox” which is the basis container of pods that run in kubernetes. In k8s we’d have one or more containers running inside this sandbox, and they’d share the networks as in this network namespace.
sudo ip netns add myplayground
You can go and list them to see that it’s there…
sudo ip netns list | grep myplayground
Now we’re going to run cnitool with sudo so it has the appropriate permissions, and we’re going to need to pass it along our environment variables and our path to cnitool (if your root user doesn’t have a go environment, or isn’t configured that way), for me it looks like:
sudo NETCONFPATH=$(echo $NETCONFPATH) CNI_PATH=$(echo $CNI_PATH) $(which cnitool) add myptp /var/run/netns/myplayground
Let’s breakdown what this is doing more or less…
NETCONFPATH=$(echo $NETCONFPATH) CNI_PATH=$(echo $CNI_PATH) sets our environment variables to tell tool$(which cnitool) figures out the path of cnitool so that inside your sudo environment, you don’t need your GOPATH (you’re rad if you have that setup, though)add myptp /var/run/netns/myplayground says that add is the CNI method which is being invoked, myptp is our configuration, and the /var/run/... is the path to the netns that we created.
You should get some output that looks like:
{
"cniVersion": "0.4.0",
"interfaces": [
{
"name": "veth20b2acac",
"mac": "62:22:15:72:b2:29"
},
{
"name": "eth0",
"mac": "42:48:16:0b:e9:98",
"sandbox": "/var/run/netns/myplayground"
}
],
"ips": [
{
"version": "4",
"interface": 1,
"address": "172.16.29.3/24",
"gateway": "172.16.29.1"
}
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"dns": {}
}
You can then actually do a ping out that interface, with:
sudo ip -n myplayground addr
sudo ip netns exec myplayground ping -c 1 4.2.2.2
And you can use nsenter to more interactively play with it, too…
sudo nsenter --net=/var/run/netns/myplayground /bin/bash
[root@host dir]# ip a
[root@host dir]# ip route
[root@host dir]# ping -c 5 4.2.2.2
What we’re going to do is create a shell script that is a CNI plugin. You see, CNI plugins can be executables of any variety – they just need to be able to read from stdin, and write to stdout and stderr.
This is kind of a blank slate for a CNI plugin that’s made with bash. You could use this approach, but, in reality – you’ll probably write these applications with go. Why? Well, especially because there’s the CNI libraries (especially libcni) which you would use to be able to express some of these ideas about CNI in a more elegant fashion. Take a look at how Multus uses CNI’s skel (skeletal components, for the framework of your CNI plugin) in its main routine to call the methods as CNI has called them. Just read through Multus’ main.go and look how it imports skel and then using skel calls our method to add when CNI ADD is used.
First, let’s make a cni configuration for our dummy plugin. I made mine at /tmp/cniconfig/05-dummy.conf.
{
"cniVersion": "0.4.0",
"name": "mydummy",
"type": "dummy"
}
There’s not a lot to pay attention to here, the most important things are:
- the
type field which must have the same name as our executable on disk – which are both going to be dummy
- the
name field is the name we’ll reference in our cnitool command, which will be mydummy.
Now, in the path where we have our reference CNI plugins, lets add another file, name it dummy, and then make sure its executable. In my case I did a:
vi ./bin/dummy
chmod 0755 ./bin/dummy
I made mine with the contents from this gist.
The first thing to note is that the majority of this file is to actually just setup some logging for looking at the CNI parameters, and all the magic happens in the last 3-4 lines.
Mainly, we want to output 3 environment using these three lines. These are some environment variables that are sent to us from CNI and that a CNI plugin can use to figure out the netns, the container id, and the CNI command.
Importantly – since we have this DEBUG variable turned on, we’re outputting via stderr… if there’s any stderr output during a CNI plugin run, this is considered a failure, as that’s what you’re supposed to do when you error out, is output to stderr.
And last but not least, we output a CNI result at the bottom line, which calls this function which outputs a (sorta kinda realistic) CNI result.
You can turn that off, but we have it on for demonstrative purposes so you can easily see the what those variables are.
So, let’s run it!
sudo NETCONFPATH=$(echo $NETCONFPATH) CNI_PATH=$(echo $CNI_PATH) $(which cnitool) add mydummy /var/run/netns/dummyplayground
And you can see output that looks like:
CNI method: ADD
CNI container id: cnitool-06764c511c35893f831e
CNI netns: /var/run/netns/dummyplayground
{
"cniVersion": "0.4.0",
"interfaces": [
{
"name": "dummy"
}
],
"dns": {}
}
Here we’ll see that there’s a lot of information that we as humans already know, since we’re executing CNI tool, but it demonstrates how a CNI plugin interacts with this information, it’s telling us that it:
- Knows that we’re doing a CNI
ADD operation.
- We’re using a netns that’s called
dummyplayground
- It’s outputting a CNI result.
These are the general basics of what a CNI plugin needs in order to operate. And then… from there, the sky’s the limit. A more realistic plugin might
And to learn a bit more, you might think about looking at some of the reference CNI plugins, and see what they do to create interfaces inside these network namespaces.
But what if my CNI plugins interacts with Kubernetes!?
…And that’s for next time! You’ll need a Kubernetes environment of some sort.
27 Nov 2019
Something that’s a real challenge when you’re trying to attach multiple networks to pods in Kubernetes is trying to get the right IP addresses assigned to those interfaces. Sure, you’d think, “Oh, give it an IP address, no big deal” – but, turns out… It’s less than trivial. That’s why I came up with the IP Address Management (IPAM) plugin that I call “Whereabouts” – you can think of it like a DHCP replacement, it assigns IP addresses dynamically to interfaces created by CNI plugins in Kubernetes. Today, we’ll walk through how to use Whereabouts, and highlight some of the issues that it overcomes. First – a little background.
The “multi-networking problem” in Kubernetes is something that’s been near and dear to me. Basically what it boils down to is the question “How do you access multiple networks from networking-based workloads in Kube?” As a member of the Network Plumbing Working Group, I’ve helped to write a specification for how to express your intent to attach to multiple networks, and I’ve contributed to Multus CNI in the process. Multus CNI is a reference implementation of that spec and it gives you the ability to create additional interfaces in pods, each one of those interfaces created by CNI plugins. This kind of functionality is critical for creating network topologies that provide control and data plane isolation (for example). If you’re a follower of my blog – you’ll know that I’m apt to use telephony examples (especially with Asterisk!) usually to show how you might isolate signal, media and control.
I’ll admit to being somewhat biased (being a Multus maintainer), but typically I see community members pick up Multus and have some nice success with it rather quickly. However, sometimes they get tripped up when it comes to getting IP addresses assigned on their additional interfaces. Usually they start by using the quick-start guide). The examples for Multus CNI are focused on a quick start in a lab, and for IP address assignment, we use the host-local reference plugin from the CNI maintainers. It works flawlessly for a single node.
But… Once they get through the quickstart guide in a lab, they’re like “Great! Ok, now let’s exapand the scale a little bit…” and once that happens, they’re using more than one node, and… It all comes crumbling down.
See – the reason why host local doesn’t work across multiple nodes is actually right in the name “host-local” – the storage for the IP allocations is local to each node. That is, it stores which IPs have been allocated in a flat file on the node, and it doesn’t know if IPs in the same range have been allocated on a different node. This is… Frustrating, and really the core reasoning behind why I originally created Whereabouts. That’s not to say there’s anything inherently wrong with host-local, it works great for the purpose for which its designed, and its purview (from my view) is for local configurations for each node (which isn’t necessarily the paradigm that’s used with a technology like Multus CNI where CNI configurations aren’t local to each node).
Of course, the next thing you might ask is “Why not just DHCP?” and actually that’s what people typically try next. They’ll try to use the DHCP CNI plugin. And you know, the DHCP CNI plugin is actually pretty great (and aside from the README, these rkt docs kind of explain it pretty well in the IP Address management section). But, some of it is less than intuitive. Firstly, it requires two parts – one of which is to run the DHCP CNI plugin in “daemon mode”. You’ve gotta have this running on each node, so you’ll need a recipe to do just that. But… It’s “DHCP CNI Plugin in Daemon Mode” it’s not a “DHCP Server”. Soooo – if you don’t already have a DHCP server you can use, you’ll also need to setup a DHCP server itself. The “DHCP CNI Plugin in Daemon Mode” just gives you a way to listen to for DHCP messages.
And personally – I think managing a DHCP server is a pain in the gluteous maximus. And it’s the beginning of ski season, and I’m a telemark skier, so I have enough of those pains.
I’d also like to give some BIG THANKS! I’d like to point out that Christopher Randles has made some monstrous contributions to Whereabouts – especially but not limited to the engine which provides the Kubernetes-backed data store (Thanks Christopher!). Additionally, I’d also like to thank Tomofumi Hayashi who is the author of the static IPAM CNI plugin. I originally based Whereabouts on the structure of the static IPAM CNI plugin as it had all the basics, and also I could leverage what was built there to allow Whereabouts users to also use the static features alongside Whereabouts.
How Whereabouts works
From a user perspective, it’s pretty easy – basically, you add a section to your CNI configuration(s). The CNI specification has a construct for “ipam” – IP Address management.
Here’s an example of what a Whereabouts configuration looks like:
"ipam": {
"type": "whereabouts",
"datastore": "kubernetes",
"kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
"range": "192.168.2.0/24"
}
Here, we’re essentially saying:
- We choose
whereabouts as a value for type which defines which IPAM plugin we’re calling.
- We’d like to use
kubernetes for our datastore (where we’ll store the IP addresses we’ve allocated) (and we’ll provide a kubeconfig for it, so Whereabouts can access the kube API)
- And we’d like an IP address
range that’s a /24 – we’re asking Whereabouts to assign us IP addresses in the range of 192.168.2.1 to 192.168.2.255.
Behind the scenes, honestly… It’s not much more complex than what you might assume from the exposed knobs from the user perspective. Essentially – it’s storing the IP address allocations in a data store. It can use the Kubernetes API natively to do so, or, it can use an etcd instance. This provides a method to access what’s been allocated across the cluster – so you can assign IP addresses across nodes in the cluster (unlike being limited to a single host, with host-local). Otherwise, regarding internals – I have to admit it was kind of satisfying to program the logic to scan through IP address ranges with bitwise operations, ok I’m downplaying it… Let’s be honest, it was super satisifying.
Requirements
- A Kubernetes Cluster v1.16 or later
- You need a default network CNI plugin installed (like Flannel [or Weave, or Calico, etc, etc])
- Multus CNI
- I’ll cover a basic installation here, so you don’t need to have it right now. But, if you already have it installed, you’ll save a step.
- If you’re using OpenShift – you already have all of the above out of the box, so you’re all set.
Essentially, all of the commands will be run from wherever you have access to kubectl.
Let’s install Multus CNI
You can always refer to the quick start guide if you’d like more information about it, but, I’ll provide the cheat sheet here.
Basically we just clone the Multus repo and then apply the daemonset for it…
git clone https://github.com/intel/multus-cni.git && cd multus-cni
cat ./images/multus-daemonset.yml | kubectl apply -f -
You can check to see that it’s been installed by watching the pods for it come up, with watch -n1 kubectl get pods --all-namespaces. When you see the kube-multus-ds-* pods in a Running state you’re good. If you’re a curious type you can check out the contents (on any or all nodes) of /etc/cni/net.d/00-multus.conf to see how Multus was configured.
Let’s fire up Whereabouts!
The installation for it is easy, it’s basically the same as Multus, we clone it and apply the daemonset. This is copied directly from the Whereabouts README.
git clone https://github.com/dougbtv/whereabouts && cd whereabouts
kubectl apply -f ./doc/daemonset-install.yaml -f ./doc/whereabouts.cni.k8s.io_ippools.yaml
Same drill as above, just wait for the pods to come up with watch -n1 kubectl get pods --all-namespaces, they’re named whereabouts-* (usually in the kube-system namespace).
Time for a test drive
The goal here is to create a configuration to add an extra interface on a pod, add a Whereabouts configurations to that, spin up two pods, have those pods on different nodes, and show that they’ve been assigned IP addresses as we’ve specified.
Alright, what I’m going to do next is to give my nodes some labels so I can be assured that pods wind up on different nodes – this is mostly just used to illustrate that Whereabouts works with multiple nodes (as opposed to how host-local works).
$ kubectl get nodes
$ kubectl label node kube-whereabouts-demo-node-1 side=left
$ kubectl label node kube-whereabouts-demo-node-2 side=right
$ kubectl get nodes --show-labels
Now what we’re going to do is create a NetworkAttachmentDefinition – this a custom resource that we’ll create to express that we’d like to attach an additional interface to a pod. Basically what we do is pack a CNI configuration inside our NetworkAttachmentDefinition. In this CNI configuration we’ll also include our whereabouts config.
Here’s how I created mine:
cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.0",
"name": "whereaboutsexample",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "whereabouts",
"datastore": "kubernetes",
"kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
"range": "192.168.2.225/28",
"log_file" : "/tmp/whereabouts.log",
"log_level" : "debug"
}
}'
EOF
What we’re doing here is creating a NetworkAttachmentDefinition for a macvlan-type interface (using the macvlan CNI plugin).
NOTE: If you’re copying and pasting the above configuration (and I hope you are!) make sure you set the master parameter to match the name of a real interface name as available on your nodes.
Then we specify an ipam section, and we say that we want to use whereabouts as our type of IPAM plugin. We specify where the kubeconfig lives (this gives whereabouts access to the Kube API).
And maybe most important to us as users – we specify the range we’d like to have IP addresses assigned in. You can use CIDR notation here, and… If you need to use other options to exclude ranges, or other range formats – check out the README’s guide to the core parameters.
After we’ve created this configuration, we can list it too – in case we need to remove or change it later, such as:
$ kubectl get network-attachment-definitions.k8s.cni.cncf.io
Alright, we have all our basic setup together, now let’s finally spin up some pods…
Note that we have annotations here that include k8s.v1.cni.cncf.io/networks: macvlan-conf – that value of macvlan-conf matches the name of the NetworkAttachmentDefinition that we created above.
Let’s create the first pod for our “left side” label:
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod-left
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod-left
command: ["/bin/bash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: dougbtv/centos-network
nodeSelector:
side: left
EOF
And again for the right side:
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod-right
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod-right
command: ["/bin/bash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: dougbtv/centos-network
nodeSelector:
side: right
EOF
I then wait for the pods to come up with watch -n1 kubectl get pods --all-namespaces or I look at the details of one pod with watch -n1 'kubectl describe pod samplepod-left | tail -n 50'
Also – you’ll note if you kubectl get pods -o wide the pods are indeed running on different nodes.
Once the pods are up and in a Running state, we can interact with them.
The first thing I do is check out that the IPs have been assigned:
$ kubectl exec -it samplepod-left -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
link/ether 3e:f7:4b:a1:16:4b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.2.4/24 scope global eth0
valid_lft forever preferred_lft forever
4: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether b6:42:18:70:12:6e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.2.225/28 scope global net1
valid_lft forever preferred_lft forever
You’ll note there’s three interfaces, a local loopback, an eth0 that’s for our “default network” (where we have pod-to-pod connectivity by default), and an additional interface – net1. This is our macvlan connection AND it’s got an IP address assigned dynamically by Whereabouts. In this case 192.168.2.225
Let’s check out the right side, too:
$ kubectl exec -it samplepod-right -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
link/ether 96:28:58:b9:a4:4c brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.1.3/24 scope global eth0
valid_lft forever preferred_lft forever
4: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 7a:31:a7:57:82:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.2.226/28 scope global net1
valid_lft forever preferred_lft forever
Great, we’ve got another dynamically assigned address that does not collide with our already reserved IP address from the left side! Our address on the right side here is 192.168.2.226.
And while connectivity is kind of outside the scope of this article – in most cases it should generally work right out the box, and you should be able to ping from one pod to the next!
[centos@kube-whereabouts-demo-master whereabouts]$ kubectl exec -it samplepod-right -- ping -c5 192.168.2.225
PING 192.168.2.225 (192.168.2.225) 56(84) bytes of data.
64 bytes from 192.168.2.225: icmp_seq=1 ttl=64 time=0.438 ms
64 bytes from 192.168.2.225: icmp_seq=2 ttl=64 time=0.217 ms
64 bytes from 192.168.2.225: icmp_seq=3 ttl=64 time=0.316 ms
64 bytes from 192.168.2.225: icmp_seq=4 ttl=64 time=0.269 ms
64 bytes from 192.168.2.225: icmp_seq=5 ttl=64 time=0.226 ms
And that’s how you can determine your pods Whereabouts (by assigning it a dynamic address without the pain of runnning DHCP!).