Kubernetes Networking 101 – Ingress resources

I called my last post ‘basic’ external access into the cluster because I didn’t get a chance to talk about the ingress object.  Ingress resources are interesting in that they allow you to use one object to load balance to different back-end objects.  This could be handy for several reasons and allows you a more fine-grained means to load balance traffic.  Let’s take a look at an example of using the Nginx ingress controller in our Kubernetes cluster.

To demonstrate this we’re going to continue using the same lab that we used in previous posts but for the sake of level setting we’re going to start by clearing the slate.  Let’s delete all of the objects in the cluster and then we’ll start by build them from scratch so you can see every step of the way how we setup and use the ingress.

Since this will kill our net-test pod, let’s start that again…

Recall that we used this pod as a testing endpoint so we could simulate traffic originating from a pod so it’s worth keeping around.

Alright – now that we have an empty cluster the first thing we need to do is build some things we want to load balance to. In this case, we’ll define two deployments. Save this definition as a file called back-end-deployment.yaml on your Kubernetes master…

Notice how we defined two deployments in the same file and separated the definitions with a ---. Next we want to define a service that can be used to reach each of these deployments. Save this service definition as back-end-service.yaml

Notice how the service selectors are looking the specific labels that match each pod. In this case, we’re looking for app and version with the version differing between each deployment. We can now deploy these definitions and ensure everything is created as expected…

I created a new folder called ingress to store all these definitions in.

These deployments and services represent the pods and services that we’ll be doing the actual load balancing to. They appear to have deployed successfully so let’s move onto the next step.

Next we’ll start building the actual ingress. To start with we need to define what’s referred to as a default back-end. Default back-ends serve as the default endpoint that the ingress will send traffic to in the event it doesn’t match any other rules. In our case the default back-end will consist of a deployment and a service that matches the deployed pods to make them easily reachable. First define the default back-end deployment. Save this as a file called default-back-end-deployment.yaml

Next lets define the service that will match the default back-end. save this file as default-back-end-service.yaml

Now let’s deploy both the definition for the default back-end deployment as well as the service…

Great! This looks just like we’d expect but let’s do some extra validation from our net-test pod to make sure the pods and services are working as expected before we get too far into the ingress configuration…

If you aren’t comfortable with services see my post on them here.

As expected pods can resolve the services by DNS name and we can successfully reach each service.   In the case of the default back-end we get a 404. Since all of the back end pods are reachable we can move on to defining the ingress itself. The Nginx ingress controller comes in the form of a deployment. The deployment definition looks like this…

Go ahead and save this file as nginx-ingress-controller-deployment.yaml on your server. However – before we can deploy this definition we need to deploy a config-map. Config-maps are a Kubernetes construct that are used to handle non-private configuration information. Since the Nginx ingress controller above expects a config-map, we need to deploy that before we can deploy the ingress controller…

In this case, we’re using a config-map to pass service level parameters to the pod. In this case, we’re passing the enable-vts-status: 'true' parameter which is required for us to see the VTS page of the Nginx load balancer. Save this as nginx-ingress-controller-config-map.yaml on your server and then deploy both the config-map and the Nginx ingress controller deployment…

Alright – so we’re still looking good here. The pod generated from the deployment is running. If you want to perform another quick smoke test at this point you can try connecting to the Nginx controller pod directly from the net-test pod. Doing so should result in landing at the default back-end since we have not yet told the ingress what it should do…

Excellent!  Next we need to define the ingress policy or ingress object. Doing so is just like defining any other object in Kubernetes, we use a YAML definition like the one below…

The rules defined in the ingress will be read by the Nginx ingress controller and turned into Nginx load balancing rules. Pretty slick right? In this case, we define 3 rules. Let’s walk through the rules one at a time from top to bottom.

  1. Is looking for http traffic headed to the host website8080.com.  If it receives traffic matching this host it will load balance it to the pods that match the selector for the service backend-svc-1
  2. Is looking for http traffic headed to the host website9090.com.  If it receives traffic matching this host it will load balance it to the pods that match the selector for the service backend-svc-2
  3. Is looking for traffic destined to the host website.com on multiple different paths…
    1. Is looking for http traffic matching a path of /eightyeighty.  If it receives traffic matching this path it will load balance it to the pods that match the selector for the service back-end-svc-1
    2. Is looking for http traffic matching a path of /ninetyninety.  If it receives traffic matching this path it will load balance it to the pods that match the selector for the service back-end-svc-2
    3. Is looking for http traffic matching a path of /nginx_status.  If it receives traffic matching this path it will load balance it to the pods that match the selector for the service nginx-ingress (not yet defined)

Those rules are pretty straight forward and things we’re used to dealing with on traditional load balancing platforms. Let’s go ahead and save this definition as nginx-ingress.yaml and deploy it to the cluster…

We can see that the ingress has been created successfully. If we would have been watching the logs of the Nginx ingress controller pod as we deployed the ingress we would have seen these log entries shortly after defining the ingress resource in the cluster…

The ingress controller is constantly watching the API server for ingress configuration. Directly after defining the ingress policy, the controller started building the configuration in the Nginx load balancer. Now that it’s defined, we should be able to do some preliminary testing within the cluster. Once again from our net-test pod we can run the following tests…

We can run tests from within the cluster by connecting directly to the pod IP address of the Nginx ingress controller which in this cases is 10.100.3.34. You’ll notice the first test fails and we end up at the default back-end. This is because we didnt pass a host header. In the second example we pass the website8080.com host header and get the correct response. In the third example we pass the website9090.com host header and also receive the response we’re expecting. In the fourth example we attempt to connect to website.com and receive once again the default back-end response of 404. If we then try the appropriate paths we’ll see we once again start getting the correct responses.

The last piece that’s missing is external access. In our case, we need to expose the ingress to the upstream network. Since we’re not running in a cloud environment, the best option for that would be with a nodePort type service like this…

I used a nodePort service here but you certainly could have also used the externalIP construct as well.  That would allow you to access the URLs on their normal port.  

Notice that this is looking for pods that match the selector nginx-ingress-lb and provides service functionality for two different ports. The first will be port 80 which we’re asking it provide on the host’s interface on port (nodePort) 30000. This will service the actual inbound requests to the websites. The second port is 18080 and we’re asking it to provide that on nodePort 32767. This will let us view the Nginx VTS monitoring page of the load balancer. Let’s save this definition as nginx-ingress-controller-service.yaml and deploy it to our cluster….

Now we should be able to reach all of our URLs from outside of the cluster either by passing the host header manually as we did above, or by creating DNS records to resolve the names to a Kubernetes node. If you want to access the Nginx VTS monitoring page through a web browser you’ll need to go the DNS route. I created local DNS zones for each domain to test and was then successful in reaching the website from my workstation’s browser…

If you added the DNS records you should also be able to reach the VTS monitoring page of the Nginx ingress controller as well…

When I first saw this, I was immediately surprised by something.  Does anything look strange to you?  I was surprised to see that the upstream pools listed the actual pod IP addresses.  Recall when we defined our ingress policy we listed the destinations as the Kubernetes services. My initial assumption was that the Nginx ingress controller would then simply be resolving the service name to an IP address and using the single service IP as it’s pool. That is – the ingress controller was just load balancing to a normal Kubernetes service.  Turns out that’s not the case. The ingress controller relies on the services to keep track of the pods but doesn’t actually use the service construct to get traffic to the pods. Since Kubernetes is keeping track of which pods are in a given service the ingress controller can just query the API server to get a list of pods that are currently alive and match the selector for the service. In this manner, traffic is load balanced directly to a pod rather than through a service construct. If we mapped out a request to one of our test websites through the Nginx ingress controller it would look something like this…

If you arent comfortable with how nodePort services work check out my last post.

In this case, I’ve pointed the DNS zones for website.com, website8080.com, and website9090.com to the host ubuntu-2. The diagram above shows a client session headed to website9090.com. Note that the client still believes that it’s TCP session (orange line) is with the host ubuntu-2 (10.20.30.72). The nodePort service is doing it’s job and sending the traffic over to the Nginx ingress controller. In doing so, the host hides the traffic behind it’s own source IP address to ensure the traffic returns successfully (blue line). This is entirely nodePort service functionality. What’s new is that when the Nginx pod talks to the back end pool, in this case 10.100.2.28, it does so directly pod to pod (green line).

As you can see – the ingress is allowing us to handle traffic to multiple different back ends now.  The ingress policy can be changed by editing the object using kubectl edit ingress nginx-ingress. So for instance, let’s say that we wanted to move the website8080.com to point to the pods that are selected by backend-svc-2 rather than backend-svc-1. Simply edit the ingress to look like this…

Then save the configuration and try to reconnect to website8080.com once again…

The Nginx ingress controller watches for changes in configuration on the API server and then implements those changes.  If we would have been watching the logs on the Nginx ingress controller container we would have seen something like this in the log entries after we made our change…

The ingress controller is also watching for changes to the service. For instance, if we now scaled our deploy-test-2 deployment we could see the Nginx pool size increase to account for the new pods. Here’s what VTS looks like before the change…

Then we can scale the deployment up with this command…

And after a couple of seconds VTS will show the newly deployed pods as part of the pool…

We can also modify properties of the controller itself by modifying the configMap that the controller is reading for it’s configuration. One of the more interesting options we can enable on the Nginx ingress controller is sticky sessions. Since the ingress is load balancing directly to the pods rather than to a service it’s possible for it to maintain sessions between the back-end pool members. We can enable this by editing the config map. kubectl edit configmap nginx-ingress-controller-conf and then add the highlighted line to the configMap…

Once again, the Nginx ingress controller will detect the change and reload it’s configuration. Now if we access website8080.com repeatedly from the same host we should see the load is sent to the same pod.  In this case the pod 10.100.0.25…

The point of this post was just to show you basics of how the ingress worked from a network perspective.  There are many other uses cases for them and many other ingress controllers to choose from besides Nginx. Take a look at the official documentation for them as well as these other posts that I found helpful during my research…

Jay Gorrell – Kubernetes Ingress

Daemonza – Kubernetes nginx-ingress-controller

  1. Smruti Ranjan Tripathy’s avatar

    Hi ,
    This post was extremely helpful. Thanks a lot for it. I am successfull in implemeting ingress .But I’am implememnting ingress in aws . The above configurations worked. I exposed the ingress controller using elb. I was able to access my various services. but I’m not able to access the nginx_status page publicly. I ‘m able to access that page from net-test pod. Any thoughts ?

    Reply

  2. Vikram’s avatar

    Great post!!!! Thanks a lot.

    After going through the setup, I am able to access the endpoints this way

    curl -H Host:jcia.federated.fds http://11.168.84.xx:30000 This is Web Server 1 running on 8080!

    where jcia.federated.fds is the dns resolvable name of the master
    This is the ingress spec part

    host: jcia.federated.fds
    http: paths:
    backend: serviceName:
    backend-svc-1 servicePort: 80

    I want to completely get the nodeip out of the way and want to access the webservice thro a single url like dns name of master/load balancer or VIP so that even if one node goes down, it doesnt affect the application.

    Could you please let me know how this can be achieved?

    Thanks once again

    Reply

  3. Vikram’s avatar

    Forgot to mention that 11.168.84.xx is my node ip and that is what I want to avoid in my access url.

    Reply

  4. Ben’s avatar

    I keep getting “ImagePullBackOff” when created image net_tools, even i deleted the pod and create again and again. Any ideas?

    $ kubectl get po
    NAME READY STATUS RESTARTS AGE
    deploy-test-1-3200654403-1fbbg 1/1 Running 0 24m
    deploy-test-1-3200654403-n3v4d 1/1 Running 0 20m
    deploy-test-2-3792706633-3038g 1/1 Running 0 24m
    deploy-test-2-3792706633-rws85 0/1 ImagePullBackOff 0 7m
    net-test-353103886-43gm4 0/1 ImagePullBackOff 0 2m

    Reply

    1. RM’s avatar

      ImagePullBackOff typically happens if the image reference is invalid

      Reply

Reply

Your email address will not be published. Required fields are marked *