kubernetes

You are currently browsing articles tagged kubernetes.

image In previous posts, we talked about running skyDNS and Heapster on your Kubernetes cluster.  In this post, I want to talk about the last of the cluster ‘addons’ available today in the Kubernetes repository.  This add on is a combination of Fluentd, Elasticsearch, and Kibana that makes a pretty powerful logging aggregation system on top of your Kubernetes cluster.   One of the major struggles with any large deployment is logging. Having a central place to aggregate logs makes troubleshooting and analysis considerably easier to do.   That being said, let’s jump right into the configuration.

Note: I have an open PR on this addon to make it a little more flexible from a configuration perspective.  Namely, I want to be able to specify the port and protocol used by the API server to access the backend service when using the API server as a service proxy.  That being said, some of my pod/controller definitions will be different from what you see on GitHub.  I’ll point out the differences below when we come across them.  The PR got merged!  We still need to slightly tweak the controller definition but other than that it should be identical to what you see out on the official repo.  More info on the tweaks below when we talk about the controller definition, see ‘Note v2:’.

The first step is to have the Kubernetes nodes collect the logs.  This is done with a local Fluentd container running on each node.  To get this to run on each host, we’ll use a local manifest on each node to tell it to run the container.  If you’ve been reading my other posts, you’ll recall that in the initial cAdvisor deployment we used a manifest to deploy that local container before it got integrated into the kubelet.  If you still have the kubelet service configured for manifests you’re all set and can skip to the next part where we define the manifest.  If you don’t and are starting from scratch, follow the below directions…

Note: If you’re going off of my saltstack repo for my lab build this is entire build is already integrated for you.

The first thing we need to do is to tell the kubelet process to look for manifests.  Manifests are almost identical to pod definitions and they define a container you want the kubelet process to run when it starts.  Sort of like a locally significant pod that the kubelet keeps an eye on and ensures that it’s always running.  So let’s checkout our kubelet systemd unit file…

Notice the line in the service config that defines a ‘config’ directory.  That tells the kubelet process to look in ‘/etc/kubernetes/manifests’ for any manifest definition files.  If you don’t have this line in your service definition, you’ll need to add it.  So let’s look at the manifest definition we’ll be using for Fluentd…

This should look very similar to you since it’s essentially a pod definition we would deploy into the Kubernetes cluster.  Once you have the manifest in place and the kubelet service updated, restart the kubelet so it picks up the new config.  Once it has, you should see the individual hosts download and starting the container defined in the manifest…

image
Good, so now what exactly does this container do?  If we look at the code out on GitHub, we can see that the Fluentd config is grabbing logs from a couple of places…

image The first two sections tell Fluentd to grab the container logs.  We do this by wildcarding all of the logs in ‘var/lib/docker/containers/*’.  The last two sections grab the kubelet log file off of each local host.  So we’re grabbing all of these logs and sending them to ‘elasticsearch-logging.default’.  So what is that host?  A Kubernetes service of course.  Let’s define that service…

So this service will get registered in DNS (requires the skyDNS addon) and that’s how Fluentd will reach ElasticSearch.  This of course means that we need to define the pod that will match the given service selector.  Let’s do that now…

Easy enough.  So all of the Fluent containers will send the their logs to this ElasticSearch pod.  The next step is to create a Kibana frontend so that we can view and search the logs.  Let’s define that..

Note: This is where the changes I made in my PR come into play.  I needed a way to tell the pod what port and protocol to use on the API server.  The definition from GitHub assumes you’re running the API server on port 443 and doesn’t allow you to configure this service to be accessed over another port or protocol.  I added in the ENV variable section and also rebuilt the Docker image to take the new variables into account.  

Note v2: The above note is still accurate, but please note that this is now the official controller definition from the Kubernetes project.  The only tweaks made were changing the SCHEME to ‘http’ and inserting the ‘:8080’ since that’s what my API server is listening on.  The official image also got updated so you don’t need to use my any more.

This definition assumes you’ll be accessing the Kibana frontend through the API server proxy on HTTP.  So now that we have the frontend pod defined, we need to define a service to access it so we can use the API server proxy…

Once you define all 4 components into the cluster, you should be able to access Kibana through the following URL…

Note: You NEED the trailing slash in the URL!!!

image
Pretty slick huh?  Now let’s see if it actually works.  To do this, I’ll load a pod definition into Kubernetes that launches a test web pod I created. The pod definition looks like this…

Once the pods loads, let’s go to the local node it’s running on and see if it generated any logs that Docker caught…

image
Looks like it did, so let’s go back to the Kibana frontend and search for these logs…

image
Awesome!  So it’s working as expected.  So as you can see, this would be a pretty handy tool to have as your cluster continues to grow.  Some pretty awesome open source logging functionality here.

Tags: ,

Note: Using SaltStack to deploy Kubernetes is something that I’ve been working on considerably since this was first posted.  My most recent iteration has a considerable amount of template automation built into it and is out on github – https://github.com/jonlangemak/saltstackv2  

I Would love to hear feedback from anyone that tries it!  New blog post coming on this soon hopefully with an end to end walk through.  

The more I play around with Docker and Kubernetes the more I find myself needing to rebuild my lab.  Config file changes are done all over the place, permissions change, some binaries are added or updated, and things get out of sync.  I always like to figure out how things work and then rebuild ‘the right way’ to make sure I know what I’m talking about.  The process of rebuilding the lab takes quite a bit of time and was generally annoying.  So I was looking for a way to automate some of the rebuild.  Having some previous experience with Chef, I thought I might give that a try but I never got around to it.  Then one day I was browsing the Kubernetes github repo and noticed that there was already a fair amount of SaltStack files out in the repo.  I had heard about SaltStack, but had no idea what it was so I thought I’d give it a try and see if it could help me with my lab rebuilds.

Make a long story short, it helps, A LOT.  While I know I’ve only scratched the surface the basic SaltStack implementation I came up with saved me some serious time.  Not only was the rebuild faster, ongoing config changes are much easier as well.  If I need to deploy a new binary or file to a set of minions I just add it into the SaltStack config and let it take care of pushing it out to the nodes.

So in this post, I’m going to outline how I used SaltStack to rebuild the lab I’ve been using in my other Kubernetes posts.  The SaltStack config I came up with works, but I’m sure it can be optimized and I hope to continue to refine it as I learn more about SaltStack.  However – I’m not going to spend much time explaining what SaltStack is actually doing to make this all happen.  I know, that seems counterintuitive, but in this post I want to focus on installing Salt and showing you that it can solve a problem in an awesome way.  In the next post, we’ll dig into what SaltStack is actually doing on the backend as well as review the Salt configuration files that I’ll be using in this post.

So just as a refresher, our lab looks like this…

Note: Same goes as before.  The servers are already configured with a name, IP address, DNS server definition, and gateway.

image
We’ll be using the server kubbuild to build the Kubernetes binaries and it will also serve as the Salt master node.  The remaining 5 server will all be Salt minions.  Since we’re starting from scratch, the first thing we’ll do is clone the Kubernetes repo and build the binaries we’ll need to run out cluster…

Note: These are the exact same steps from the first blog we did Kubernetes 101 – The build.  I’m copying them below just for reference.

Let’s start by downloading the tools we’ll need to build and distribute the code and configuring the services required…

So at this point, we have a fairly basic system running Apache and docker.  You might be wondering why we’re running docker on this system.   The kubernetes team came up with a pretty cool way to build the kubernetes binaries.  Rather than having you build a system that had all of the required tools (Namely GO) the build script downloads a golang container that the build process runs inside of.  Pretty cool huh? Let’s get started on the kubernetes build by copying the kubernetes repo from github…

This will copy the entire repository down from github.  If you want to put the repository somewhere specific make sure you CD over to that directory before running the git clone.  Since this machine is just being used for this explicit purpose I’m ok with the repository being right in the root folder. Once the clone completes we can build the code.  Like I mentioned, the kubernetes team offers several build scripts that you can use to generate the required binaries to run kubernetes.  We’re going to use the ‘build-cross’ script that builds all the binaries for all of the platforms.  There’s more info on github about the various build scripts here.

This will kick off the build process.  It will make sure you have docker installed and then prompt you to download the golang container if you don’t already have it.  This piece of the build can take some time.  The golang container is over 400 meg and the build process itself takes some time to complete.  Go get a cup of coffee and wait for this step to complete…

image
When the build is complete, you should get some output indicating that the build was successful.

Now this is where things are going to change from the initial build post we did.  We’ll start by installing the Salt master on kubbuild.  To do that, we’ll need the EPEL repository which has the YUM package for the Salt master install.  Once we have the EPEL repo we can use YUM to install the salt-master package on kubbuild…

Once it’s installed, we need to make a small configuration change to allow Salt to distribute files to the minions.  Let’s edit the file ‘/etc/salt/master’ and look for the section called ‘File Server Settings’.  Uncomment the section of the config I highlight below…

image
This tells Salt that the file root is the directory ‘/srv/salt’.  Save the file and exit.  The last thing we have to do is enable and start the salt-master service…

Now our master is up and running so the next step is to get the Salt minions online.  The install process for the minions is pretty similar.  Start by installing the required packages…

Once the client is installed, we need to tell the client who the Salt master is.  This is done by defining the master in ‘/etc/salt/minion’.  Uncomment the line that starts with ‘master:’ and set it to the name of the Salt master server.  My config looks like this…

image
Once that’s defined, save the file and then enable and start the salt-minion service…

Perform the same Salt minion config on each of the 5 servers and you’re done!  Salt is fully configured.  To see if we were successful, let’s head back to the kubbuild server and see if the master can talk to the minions…

image
Awesome!  This is a good sign.  All of the minions are listed but before we can talk to them, the master needs to accept their keys.  All Salt communication is encrypted and before the master can talk to the minions the master needs to approve of the key the minion is using.  To do this, we’ll use the ‘salt-key –A’ command to accept all of the keys…

image
We can see that it prompts us to ensure we want to accept all of the keys and after we agree all of the minions now show up under the ‘Accepted Keys’ category.  To make sure we have full connectivity we can issue a test command from the master…

image

Looks good!  Now let’s move onto the fun part.  Let’s clone my github repo onto kubbuild…

This should create the folder ‘salt’ under the srv folder.  Since we’re going to use Salt to distribute the binaries to the nodes we need to copy the binaries to a location that Salt can find them. In this case, that would be the salt file root that we defined in the master config file.  So let’s create a directory called ‘kube_binaries’ under the ‘/srv/salt’ directory and copy the binaries into that folder…

So now we have everything in place.  To build the Kubernetes lab cluster tell Salt to run by issuing the following command…

Don’t worry about what all this means at the moment.  We’ll cover all of that in the next post.  For now, sit back and watch the magic happen.  As Salt runs, you should start seeing messages scroll across the screen of the master indicating the current status of the run.   Something that looks like this…

image
When it’s all done, you can scroll back up through the output and make sure nothing failed.  If things look good, let’s jump over to the kubmasta server and see where things stand…

image
Nice!  So it looks like our lab setup is working.  Let’s deploy our sky-dns pod to it and see what happens…

image
Hard to see there but the pod loaded and is running as expected.  Salt just deployed a fully working Kubernetes cluster on 5 machines!  Pretty slick!

So I know at this point I’m portraying Salt as black magic that just works.  I hate to end it there, but the point of this post was just to show you what Salt can do.  The next post will cover the details of how Salt made this all happen.  And like I said, I’ve just scratched the surface of Salt’s capabilities.  I’m sure there will be more posts to come as I learn more about what Salt can do!

Tags: , ,

NOTE: Kubernetes has evolved! The cAdvisor module is now integrated directly into the Kubelet binary.  These directions are no longer entirely accurate because of this.  The cAdvisor manifest does not need to be deployed.  If you’re looking for current info on deploying Kubernetes, please reference the related config files in my GitHub salt repo.  I’ll be doing my best to keep those as up to date as possible.

If you’ve spent some time with Kubernetes, or docker in general, you probably start to wonder about performance.  Moreover, you’re probably wondering how to gauge performance of he overall host as well as the containers running on it.  This is where cAdvisor comes in.  CAdvisor is a open source tool for monitoring docker and it’s running containers.  The best part about cAdvisor is that it has native docker support and is super easy to integrate into an existing Kubernetes cluster.  Additionally, cAdvisor runs in a container (starting to see why docker is awesome?) so the configuration changes required on the host are super minimal.  AKA – You just need to tell the host to run the container.

In addition to installing cAdvisor on our bare metal Kubernetes cluster, we’re also going to install another awesome open source Google tool call Heapster.  Heapster gathers all of the data from each Kubernetes node via cAdvisor and puts it all together for you in one spot.

So let’s get started with installing cAdvisor…

The cAdvisor container needs to run on each host you want cAdvisor to monitor.  While we could do this through the Kubernetes master, it’s easier just to tell each host to run the container when it boots.  Luckily, the Kubernetes-Kubelet service can be configured to read and process local ‘manifests’.  Manifests are VERY similar to a pod configuration in Kubernetes.  In fact, they’re identical.  You essentially just remove the piece of the pod config telling Kubernetes that it’s a pod since by default, a manifest is a pod.  So the first thing we need to do is edit our Kubelet service configuration to tell it to look for local manifests.  Let’s edit our service definition located here…

Our new config will look like this…

Notice that all we added was the line indicating that the service should look for it’s configuration in the directory ‘/etc/kubernetes/manifests’.  This directory shouldn’t exist since we’ve never used it before so let’s save this file, and create the directories that we need…

The next step is to create the manifest.  So let’s move into our new folder and create a file called ‘cadvisor.manifest’.  In this file, let’s put the following configuration…

So this format should look pretty familiar to us from pod definitions we’ve seen in the past.  We can see that the container is mounting a number of volumes from which it will read the data it needs for cAdvisor to pull statistics.  Now that we have everything in place, let’s tell systemd to reload the service.  We’ll do so using these two commands…

Once we do that, the kubelet service will read the manifest file and start the process of downloading the starting the cAdvisor container.  It’s a small container image so within a matter of seconds we should see it running on kubminion1…

image
Once it’s running, all we have to do is browse to the host on port 4914 and we should see the cAdvisor page…

image

Scroll down further and we’ll see some interesting data about CPU and memory usage…

image
We can also dive into any running container and check to see how it’s running by scrolling up to the top and picking the ‘Containers’ link and then selecting the container we want to look at.  Pretty cool stuff!

Now let’s do this same configuration on our other 3 hosts.  When you’re done, verify that you can connect to the cAdvisor status page on all 4 hosts.  Now let’s configure Heapster…

Note: Heapster relies on the service DNS resolution.  If you don’t have that running yet, see this.

This is the awesome part.  I mean, the really awesome part.  Heapster uses two pods and three services deployed into the cluster for it to work.  The first pod is the heapster pod which does all of the data collection.  The second pod runs a database server (InfluxDB) along with a web front end to visualize the stats (Grafana).  That being said, now that cAdvisor is deployed, the hard work for us is over.  Now its just a matter of deploying some pods and services into the cluster and letting the magic happen.  Let’s look at the pod definitions we’ll be using…

Note(s): I had some issues in the latest release of Grafana.  There were two issues…

First – It turns out that the API port wasn’t configurable in the official release.  I suspect this is another difference between running this on bare metal and running this in GCE.  Regardless, I opened a pull request and had the code I used to fix this merged into the project (yay me!).  That being said, we can use the revised templates from GitHub, but the docker image isn’t updated yet with the fix I made.  I’ve got a docker image out there that does have the fix so I’ll be using that rather than the official released image.  I reached out to the guys that manage this to see when the official docker image will be updated. When that happens, we can go back to using the original containers from the project.

Second – We need to use a newer version of the Heapster container than what is specified in the pod template.  This is due to a bug that I’m only seeing on the bare metal side of things and isn’t reproducible on the GCE side.  They released a newer image that fixes this issue for me but the issue still needs to be debugged further.  I’m assuming that sooner or later the template will be updated to specify the new version but for now we need to update it from Heapster version .7 to version .8

All of the templates I show below have the two changes I listed above already made.  If you download the pod templates off of GitHub  directly you’ll need to make these changes yourself.  

The new deployment files (and associated containers) out on the Heapster GitHub page work as is!  The only change I had to make was to set my KUBERNETES_API_PORT port to 8443 in the InfluxDB/Grafana controller file.  The service and controller files located below are not updated so please see the new YAML files on the GitHub page located here https://github.com/GoogleCloudPlatform/heapster/tree/master/deploy  

 

The Grafana Kubernetes service (grafana-service.yaml)…

The Heapster Kubernetes service (heapster-service.yaml)…

The InfluxDB Kubernetes service (influxdb-service.yaml)…

The InfluxDB and Grafana Replication controller configuration (influxdb-grafana-controller.yaml)…

The Heapster Replication controller configuration (heapster-controller.yaml)…

Once we have the configuration files on our Kubernetes master we can deploy them into the cluster like this…

Once the pods deploy, we should be able to browse to the Grafana front end at this URL…

https://kubmasta:8443/api/v1beta1/proxy/services/monitoring-grafana/

Note: In my case, my Kubernetes master is called ‘kubmasta’ and the API service is listening on port 8443.  Since you’re using HTTPS we’re also going to get cert warnings on your initial connection to the API server.

Notice that in this case we aren’t connecting directly to the pod providing the service.  Rather, we’re connecting through the API server to the backend service.  This seems to work well but I’m still not 100% sure on the mechanics behind how this works but I’m working to get more info on it.  Either way, the Grafana frontend and default Kubernetes dashboard is pretty awesome…

image
As I mentioned above, you’re going to get cert errors unless you have the certificates sorted out.  An alternative would be to tell Grafana to use HTTP rather than HTTPS.  This cold be done by changing the Grafana replication controller config to something like this…

With this configuration, we could access the Grafana frontend on HTTP by accessing this URL…

http://kubmasta:8080/api/v1beta1/proxy/services/monitoring-grafana/

Either way, pretty cool stuff.  I hope you’re starting to see the power of this model.  Deploying an entire platform like Heapster was done with 5 command on the Kubernetes master.

Tags: ,

« Older entries § Newer entries »