CoreOS – Getting your second (and 3rd) host online

Quick note: For the sake of things formatting correctly on the blog I’m using a lot of screenshots of config files rather than the actual text.  All of the text for the cloud-configs can be found on my github account here – https://github.com/jonlangemak/coreos/

Now that we have our first CoreOS host online we can start the cool stuff.  We’re going to pick up where we left off in my last post with our first installed CoreOS host which we called ‘coreOS1’.  Recall we had started with a very basic cloud-config that looked something like this…

All this really did was get us up and running with a DHCP address and the base system services running.  Since we’re looking to do a bit more in this post, we need to add some sections to the cloud-config.  Namely, I’m going to configure a static IP address for the host, configure and start etcd, and configure and start fleet.  So here’s what my new cloud config for coreOS1 will look like…

So there’s a lot more in this cloud-config.  This config certainly deserves some explaining.  However, in this post, I want to just get a couple more hosts online and get them clustered.  So bear with me, I promise follow up posts are going to dig much further into CoreOS services and the cloud-config itself.  There are two items that I need to cover right now to get us going.  The first is the etcd discovery URL.  Note that the URL has a syntax of…

https://discovery.etcd.io/e87776d4ad4459a799d267488b61ed50

The bold piece of the URL is a unique key.  This NEEDS to be the same on each host that you want to be part of the cluster.  To generate the URL, you can use this URL…

https://discovery.etcd.io/new

For now, just know that each time you build a cluster you need to generate a NEW etcd discovery URL and that all of the host in the same cluster need to use the same discovery URL.  Again, much more to come on this when we dive into etcd.

Note: The discovery URL is used by the hosts for clustering.  Since this is an external URL, the hosts require internet access for this to work.

The second thing we need to know for our new cloud config is the servers interface name.  Note that the cloud-config specifics a interface name to match on.  In our case, this is ‘ens18’.  This can be found just by doing a quick ‘ifconfig’ on the host…

For now, that’s all we need to keep moving forward with building additional hosts.

So now that we have a more robust cloud-config, how do we get that on the first host that we built?  The cloud-config used to build the current iteration of the host can be found in this location…

/var/lib/coreos-install/user_data

The file is protected so we’ll have to use the following command to edit it…

sudo vi /var/lib/coreos-install/user_data

You should see a copy of your original cloud-config as shown below…

Here you could manually edit the file but I prefer to load a copy of the file that I know has been validated.  That being said, I would exit out of vi and run the following command to copy a new cloud-config from my local tools server…

sudo wget -O /var/lib/coreos-install/user_data http://tools/cloud-config-1.yml

Next we need to tell the CoreOS host to load the new cloud-configuration file.  This can be done either via command or by simply rebooting the host.  Let’s use the cloud-init command to load the file without a reboot…

sudo coreos-cloudinit --from-file /var/lib/coreos-install/user_data

So let’s give it a try…

Now let’s check and see if the services are now running as we expect…

Note: Again, indulge me here, we’re going to cover this stuff in much greater detail in future posts.

We’ve now verified that the two services we need for CoreOS clustering are running.  The next step is to configure our second and third hosts.  These will have identical cloud-configuration files with the exception of the hostname and IP address configuration.  The cloud-configs will look like this…

Just like we did above, ensure that you set the discovery URL the same in each cloud-config and verify the network interface name.  You can see for my host coreOS3 that this is different than the others since it’s a different type of machine.

For hosts coreOS2 and coreOS3 Im going to load the cloud-config during the install much like we did in the first post.  If you need some refreshing pop back over to the first post and follow those directions.

<Time Passes as you install the 2nd and 3rd host>

I’ll assume at this point you have all 3 hosts online.  You should be able to log into each host independently and verify that etcd and fleet are running by issues these two commands…

systemctl status etcd
systemctl status fleet

Fleet relies on etcd so if you’re having issues with etcd its very likely fleet wont be working either.  Please make sure that both services are running on each of your 3 hosts before continuing.  You should be able to SSH into each of the 3 hosts directly and verify the services are running as expected.

At this point, if all has gone well, you should be able to see if your devices are clustered by running this command…

fleetctl list-machines

When run you should see this…

If you do, then we’ve succeeded in creating out first CoreOS cluster!  Note that fleet and etcd are distributed services so you can run fleet and etcd command on any of the cluster hosts and you should see the same response.  Pretty slick huh?

In the next post (coming very soon!) we’ll deploy our first application using fleet to our cluster.  After we wrap up the fleet demo we’ll be diving into some of the specifics of how all of this works with deep dives on systemd, etcd, and fleet.