Kubernetes Traffic Routing - Part 2 of 2

, KAZUHIRO TACHIBANA, CONTAINER

Overview

    This blog explains the details of Ingress, which is one of the ways to expose Pods running in K8S cluster to outside the world, what is Ingress and it’s fundamental concept and major features.
    By the end, you will learn what you benefit from Ingress.

What is Ingress?

    Ingress is a Kubernetes (a.k.a K8S) resource to define set of rules that allow and route inbound traffic to reach K8S cluster services. It can be configured to give Service externally-reachable urls, load balance traffic, terminate SSL, offer name based virtual hosting etc.

Concept

    Even though there are multiple ways to expose Service and Pods running in K8S cluster to outside the world in order to route inbound traffic, because they take the principle nothing but port based proxy mechanism, it would be tough to manage ports in every Service to proxy inbound traffic or much cost to manage loadbalancer which is most likely L4.
    Ingress comes up with the idea to to expose Service and Pods on a single IP and single port, either based on the subdomain or may be on the path in the URL. Consequently, Ingress supports L7 Loadbalancer as opposed to L4 Loadbalancer.

Rule

    Rule in here may imply the definition how to route inbound traffic and send it to backend. There are three type to define rule Path Proxy, Name based Virtual Hosting and TLS.

Default Backend

    Default Backend stands for a Service capable of handling all inbound traffic that doesn’t match paths or hosts in Ingress. Default Backend is required to get implemented as the minimum features like below.

  • Serves a / for 404 page
  • Serves a /healthz on port 10254, as both liveness and readiness probe

Ingress Controller

    In order for the Ingress to work, K8S cluster must have an Ingress Controller running, which is an application that watch Ingress in K8S cluster and configure a balancer to apply the rules defined in Ingress. Ingress Controller is to satisfy an Ingress, simply creating the resource will have no effect.
    Ingress Controller is supposed to get rollout as a Pods. It contains a loadbalancer like nginx or HAProxy and controller daemon. The controller daemon receives the desired Ingress configuration from kube-apiserver. It generates an nginx or HAProxy configuration file and restarts the loadbalancer process for changes to take effect. In other words, Ingress Controller is a loadbalancer managed by K8S.
    Typically you will use an existing Ingress controller that controls a third party loadbalancer like nginx, HAProxy, Vulcand or Traefik.

Demo

    The demo shows how Ingress Controller work to route inbound traffic based on rule definition in Ingress under below context.

Environment

    The demo runs in the K8S of version 1.6.4 built with Docker of version 1.12.1.

Default Backend

    Default Backend is required to achieve expected Ingress functionality overall.
    The demo uses Default Backend published in here where managed by K8S.

Ingress Controller

    Ingress Controller implementation with third party loadbalancer can be anything of nginx, HAProxy, Vulcand or Traefik.
    nginx Ingress Controller is comparatively much mature than another and popular one that provides sticky sessions, websockets and more complex path rewrites function.
    The demo uses nginx Ingress Controller published in here where managed by K8S.

Service

    As the proof of concept, two Java based applications (Tomcat sample application and Jenkins) are used for the services running in K8S as diagrammed in below.

Yaml

    The demo uses the yaml files for rolling out each of the K8S resources.

default-backend-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

default-backend-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: kube-system
  labels:
    k8s-app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    k8s-app: default-http-backend
  type: ClusterIP

nginx-ingress-controller-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  labels:
    k8s-app: nginx-ingress-controller
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-controller
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.10
        name: nginx-ingress-controller
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ports:
        - containerPort: 80
          hostPort: 10081
        - containerPort: 443
          hostPort: 10443
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=kube-system/default-http-backend
        - --default-ssl-certificate=kube-system/tls-certificate
        - --apiserver-host=${K8S_MASTER_IP_ADDRESS}
        - --ingress-class=nginx

nginx-ingress-controller-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-controller
  namespace: kube-system
spec:      
  # Because of the type of NodePort, arbitrary port in specific range will be opened.
  type: NodePort
  ports:
    - port: 80
      name: http
    - port: 443
      name: https
  selector:
    k8s-app: nginx-ingress-controller

tomcat8-jenkins-deployment.yaml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    k8s-app: tomcat8-jenkins
  name: tomcat8-jenkins
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: tomcat8-jenkins
      name: tomcat8-jenkins
    spec:
      containers:
        -
          args:
          - daemon
          # Fetch from Docker Hub
          image: "skcho4docker/tomcat8-jenkins-path:ver1.6.3"
          name: tomcat8-jenkins-path
          ports:
            -
              containerPort: 8080
              protocol: TCP

tomcat8-sample-deployment.yaml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    k8s-app: tomcat8-sample
  name: tomcat8-sample
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: tomcat8-sample
      name: tomcat8-sample
    spec:
      containers:      
        -
          args:
          - daemon
          # Fetch from Docker Hub
          image: "skcho4docker/tomcat8-sample-path:ver1.0.0"
          name: tomcat8-sample-path
          ports:
            -
              containerPort: 8080
              protocol: TCP

tomcat8-jenkins-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: tomcat8-jenkins
  namespace: default
  labels:
    k8s-app: tomcat8-jenkins
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    k8s-app: tomcat8-jenkins
  type: ClusterIP

tomcat8-sample-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: tomcat8-sample
  namespace: default
  labels:
    k8s-app: tomcat8-sample
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    k8s-app: tomcat8-sample
  type: ClusterIP

tomcat8-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tomcat8-host
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  # Because the host is arbitrary, it is needed to append it with your actual NIC IP to host file in your local machine. 
  - host: tomcat.com
    http:
      paths:
      - path: /jenkins
        backend:
          serviceName: tomcat8-jenkins
          servicePort: 80
      - path: /sample
        backend:
          serviceName: tomcat8-sample
          servicePort: 80

Rollout

    The K8S resources defined in the yaml files described in above are rollout through K8S dashboard.
    Once they are on K8S running, the workload looks like as reflected in below.

    It can be confirmed that Ingress works as shown in below.

http://tomcat.com:31011/jenkins/

http://tomcat.com:31011/sample/

Summary

    Even though all of advantages of Ingress were not leveraged this time, Ingress worked as it was expected. L7 load balancer feature is definitely an additional feature that is worth experimenting with.
    Despite of the fact that Ingress needs to mature, from required feature perspective, it is enough to fulfill service level requirements for production environment. But, it is strongly recommended to explore Ingress feature even further for considering future adoption.
    Rakuten will keep an eyes on Ingress feature as it evolves in coming month.

Also, we are hiring, If you are interested in working as a container engineer, please contact us here