Kubernetes – mysql Stateful set

Posted: March 20, 2020 in kubernetes

StatefulSets in Kubernetes are used for applications where data consistency and replication is required (relational databases).

In stateful set each pod is assigned a unique ordinal number in the range of [0, N),and they are shut down in reverse order to ensure a reliable and repeatable deployment and runtime. The StatefulSet will not even scale until all the required pods are running, so if one dies, it recreates the pod before attempting to add additional instances to meet the scaling criteria.This ID sticks to the pod even when its rescheduled to another worker node. so pod retains the connection to the volume that holds the state of the database. In this way each replica pod in a stateful set has its state and data.

In this example, service named mysql will be exposed to client applications and service will pass write/read request to one of the node.Each node will store data in persistent volume (/mnt/mysql[01-02]) folders in Kubernetes host. Each volume is replicated to Kubernetes node ensuring data availability

Persistent volume claim is created for each pod

storage.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:

  name: localstorage

provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate
reclaimPolicy: Delete
allowVolumeExpansion: True

---

kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-01
  labels:
    type: local
spec:
  storageClassName: localstorage
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql01"
    type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-02
  labels:
    type: local
spec:
  storageClassName: localstorage
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql02"
    type: DirectoryOrCreate

mysql credentials are stored in env.example

MYSQL_PASSWORD=password
MYSQL_DATABASE=db
MYSQL_ROOT_PASSWORD=password
MYSQL_USER=user

And secret is created from this file and referenced in worload.yaml

kubectl create secret generic prod-secrets --from-env-file=env.example

services.yaml:

apiVersion: v1
kind: Service
metadata:
  name: mysql

spec:
  # Open port 3306 only to pods in cluster
  selector:
    app: mysql-container

  ports:
    - name: mysql
      port: 3306
      protocol: TCP
      targetPort: 3306
  type: ClusterIP

volumeClaimTemplate will automatically create persitent volume claim for each pod, workload.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-container
spec:
  serviceName: mysql
  replicas: 2
  selector:
    matchLabels:
      app: mysql-container
  template:
    metadata:
      labels:
        app: mysql-container
    spec:
      containers:
      - name: mysql-container
        image: mysql:dev
        imagePullPolicy: "IfNotPresent"
        envFrom:
          - secretRef:
             name: prod-secrets
        ports:
        - containerPort: 3306
        # container (pod) path
        volumeMounts:
          - name: mysql-persistent-storage
            mountPath: /var/lib/mysql

        resources:
          requests:
            memory: 300Mi
            cpu: 400m
          limits:
            memory: 400Mi
            cpu: 500m
      restartPolicy: Always

  volumeClaimTemplates:
    - metadata:
        name: mysql-persistent-storage
      spec:
        storageClassName: localstorage
        accessModes: ["ReadWriteOnce"]
        resources:
         requests:
          storage: 5Gi
        selector:
         matchLabels:
          type: local

After all above yaml files are applied, check microservices status:

kubectl get pvc
NAME                                         STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-persistent-storage-mysql-container-0   Bound    mysql-02   5Gi        RWO            localstorage   3h40m
mysql-persistent-storage-mysql-container-1   Bound    mysql-01   5Gi        RWO            localstorage   3h39m


kubectl get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                                                       STORAGECLASS   REASON   AGE
                                                                                           
mysql-01       5Gi        RWO            Retain           Bound    default/mysql-persistent-storage-mysql-container-1                                                                          localstorage            3h42m
mysql-02       5Gi        RWO            Retain           Bound    default/mysql-persistent-storage-mysql-container-0  

kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
mysql-container-0                   1/1     Running   0          8m17s
mysql-container-1                   1/1     Running   0          8m27s



kubectl get services
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
adminer-container   NodePort    10.109.247.8     <none>        8080:30050/TCP   2d5h
kubernetes          ClusterIP   10.96.0.1        <none>        443/TCP          3d2h
mysql               ClusterIP   10.111.175.116   <none>        3306/TCP         33m


 kubectl describe statefulset.apps/mysql-container
Name:               mysql-container
Namespace:          default
CreationTimestamp:  Fri, 20 Mar 2020 11:49:31 -0400
Selector:           app=mysql-container
Labels:             <none>
Annotations:        kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"mysql-container","namespace":"default"},"spec":{"replica...
Replicas:           2 desired | 2 total
Update Strategy:    RollingUpdate
  Partition:        824640705452
Pods Status:        2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=mysql-container
  Containers:
   mysql-container:
    Image:      mysql:dev
    Port:       3306/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     500m
      memory:  400Mi
    Requests:
      cpu:     400m
      memory:  300Mi
    Environment Variables from:
      prod-secrets  Secret  Optional: false
    Environment:    <none>
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
  Volumes:  <none>
Volume Claims:
  Name:          mysql-persistent-storage
  StorageClass:  localstorage
  Labels:        <none>
  Annotations:   <none>
  Capacity:      5Gi
  Access Modes:  [ReadWriteOnce]
Events:
  Type    Reason            Age                  From                    Message
  ----    ------            ----                 ----                    -------
  Normal  SuccessfulCreate  38m                  statefulset-controller  create Claim mysql-persistent-storage-mysql-container-2 Pod mysql-container-2 in StatefulSet mysql-container success
  Normal  SuccessfulCreate  38m                  statefulset-controller  create Pod mysql-container-2 in StatefulSet mysql-container successful
  Normal  SuccessfulDelete  37m                  statefulset-controller  delete Pod mysql-container-2 in StatefulSet mysql-container successful
  Normal  SuccessfulDelete  21m                  statefulset-controller  delete Pod mysql-container-1 in StatefulSet mysql-container successful
  Normal  SuccessfulCreate  21m (x7 over 3h54m)  statefulset-controller  create Pod mysql-container-1 in StatefulSet mysql-container successful
  Normal  SuccessfulDelete  21m                  statefulset-controller  delete Pod mysql-container-0 in StatefulSet mysql-container successful
  Normal  SuccessfulCreate  21m (x7 over 3h55m)  statefulset-controller  create Pod mysql-container-0 in StatefulSet mysql-container successful

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s