High Availability

To set up a multi-node-HA Mosquitto broker and Management Center using Helm charts, you'll first need a Kubernetes environment. For deploying a full fledged kubernetes cluster on multiple hosts, Kubeadm is an excellent choice. Kubeadm is a command-line tool in the Kubernetes ecosystem designed to facilitate the process of setting up and bootstrapping a Kubernetes cluster.(Discussed in Introduction section). This setup would deploy a 3 Mosquitto broker as a statefulsets. Also, a Management-Center pod and HA-proxy pod as a deployment entity. All the deployment would be deployed on multiple hosts. This deployment by default uses a NFS server to mount volumes. You would need to setup the NFS server before using this deployment.

Recommended Setup

1. 1 Control-plane node, 3 worker nodes and a NFS Server
2. You can name them control-plane, node-1, node-2, node-3, nfs-server. If you want to have different names for your machines you would have to uncompress the helm charts and change the hostnames entries of values.yaml. tar -xzvf mosquitto-multi-node-multi-host-0.1.0.tgzcd mosquitto-multi-node-multi-host
   • Change the values of hostname from node-1/node-2/node-3 to the names of your machines.
   • Go back to the parent directory: cd ../
   • Package the helm charts helm package mosquitto-multi-node-multi-host
   • Install the package. (Further details refer to installation is given below.)

HA-PROXY Configurations HA-proxy need to be configured accordingly for the kubernetes setup. For server m1, m2 and m3 needs to be configured in this case. Instead of using docker IP we would use DNS names to address the pods. For eg mosquitto-0.mosquitto.multinode.svc.cluster.local. Here mosquitto-0,mosquitto-1,mosquitto-2 are the name of individual mosquitto pods running as statefulsets. Each new pod would increase its pod-ordinal by 1. Rest can be defined as follows <pod-name>.<name-of-the-statefulset>.<namespace>.svc.cluster.local

global
    daemon
    maxconn 4096

frontend mqtt_frontend
    bind *:1883
    mode tcp
    default_backend mqtt_backend
    timeout client 10m

backend mqtt_backend
    timeout connect 5000
    timeout server 10m
    mode tcp
    option redispatch
    server m1 mosquitto-0.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions
    server m2 mosquitto-1.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions
    server m3 mosquitto-2.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions

Note: Make sure to add more entries to this haproxy.cfg file if you wish to add nodes to this cluster. You would have to reconfigure the HACONFIG in that case. You can refer to Add Node i.e point 7 in Installation section for tips on reconfiguring haconfig configmap.

Installation

Prerequisites: Kubernetes Cluster should be up and running.If if you are yet to setup the cluster, refer Kubernetes Cluster Setup. Once kubernetes cluster is up and running and you have installed all the configmaps through setup.sh, you can now follow these steps to install the multi-node Mosquitto broker setup:

1. Setup the folder on your local machine:

   • If not done previously, copy or setup the mosquitto-2.7-mmc-2.7-cluster-kubernetes repository to the NFS server and Control-plane. For both instances the path of the repository has to be the same. Also make sure to create a directory inside the repository on Control-plane named license that contains the license.lic file we provided you.

2. Setup NFS Server

   • Install necessary dependencies sudo apt-get updatesudo apt-get install nfs-kernel-server
   • Configure exports directory.Open the /etc/exports file on NFS-server. Expose the directories so that pods running on other nodes can access these directories.
     • You can use the following as a reference. Here we expose three mosquitto nodes and management center.

       /home/<user>/mosquitto-2.7-mmc-2.7-cluster-kubernetes/server1/mosquitto/data *(rw,sync,no_root_squash,no_subtree_check)
       /home/<user>/mosquitto-2.7-mmc-2.7-cluster-kubernetes/server2/mosquitto/data  *(rw,sync,no_root_squash,no_subtree_check)
       /home/<user>/mosquitto-2.7-mmc-2.7-cluster-kubernetes/server3/mosquitto/data *(rw,sync,no_root_squash,no_subtree_check)
       /home/<user>/mosquitto-2.7-mmc-2.7-cluster-kubernetes/server1/management-center/config *(rw,sync,no_root_squash,no_subtree_check)
   • Make sure all the data directories have adequate privileges so that mosquitto kubernetes pods can create additional directories inside these data directories. If you face any issues like permission denied you can try giving the 1000 ownership to all the relevant data dir using following command:

   sudo chown -R 1000:1000 /home/<user>/mosquitto-2.7-mmc-2.7-cluster-kubernetes/server1/mosquitto/data
   sudo chown -R 1000:1000 /home/<user>/mosquitto-2.7-mmc-2.7-cluster-kubernetes/server2/mosquitto/data
   sudo chown -R 1000:1000 /home/<user>/mosquitto-2.7-mmc-2.7-cluster-kubernetes/server3/mosquitto/data
   • Note: We provide ownership of 1000 as mosquitto kubernetes pods uses user id of 1000 by default.
   • Expose the directories using: sudo exportfs -a
   • Restart the kernel-server sudo systemctl restart nfs-kernel-server
   • Install neccessary nfs-common on other nodes so that they act as nfs-clients: sudo apt-get install nfs-common

3. Change Directory:

   • Navigate to the project directory (i.e multi-node-multi-host). cd mosquitto-2.7-mmc-2.7-cluster-kubernetes/kubernetes/multi-node-multi-host/

4. Install Helm Chart:

   • Use the following helm install command to deploy the setup to your Kubernetes cluster. Replace <release-name> with the desired name for your Helm release and <namespace> with your chosen Kubernetes namespace:

     helm install <release-name>  mosquitto-multi-node-multi-host-0.1.0.tgz   --set repoPath=$HOME -n <namespace>
   • repoPath: Set the repoPath flag to the path where this repo was cloned. The above command expects it to be at $HOME.
   • namespace: Set it to the namespace of your deployment.
   • Note: Ensure that you have a running Kubernetes cluster set up to run this deployment. We recommend setting up the kubernetes cluster using Kubeadm. You can also use our installation script to set up Kubernetes cluster. To setup the kubernetes cluster, follow the instructions in Kubernetes Cluster Setup.
   • Note: If you want to deploy the setup in a different namespace, make sure to pass a seperate flag --set namespace=<your-custom-namespace> along with the helm installation command
   • Note: By default the NFS server IP is set to 10.156.0.10 .You can change the NFS server IP by passing --set nfs=<your-nfs-ip> by passing it along with helm installation command.
   • So for eg: If you NFS IP is 10.10.10.10,your name namespace is test-namespace and your arbitrary release name is sample-release-name then your helm installation command should be:

   helm install sample-release-name  mosquitto-multi-node-multi-host-0.1.0.tgz   --set repoPath=$HOME -n test-namespace --set namespace=test-namespace --set nfs=10.10.10.10

5. You can monitor the running pods using the following command: kubectl get pods -o wide -n <namespace>

6. Open Applications http://localhost:31021 (MMC)

   • IP: ip-of-host-running-ha-proxy & Port: 31028 (HA)

7. To uninstall the setup: helm uninstall <release-name> -n <namespace>

8. Add Nodes

   • Configure your HAproxy by adding entries to the haproxy.cfg.For eg to add fourth node, add below line to haproxy.cfg: server m4 mosquitto-3.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions
   • Delete the haproxy configmap of kubernetes kubectl delete configmap haconfig -n <namespace>
   • Reconfigure HAconfig: kubectl create configmap haconfig -n <namespace> --from-file=<path-to-haproxy.cfg>
   • Uninstall helm package helm uninstall <release-name> -n <namespace>
   • Reinstall helm package so that new configmaps can come into effect(Use the same command you used to install it the first time with appropriate flags.Below eg is just a reference) helm install <release-name> mosquitto-multi-node-multi-host-0.1.0.tgz --set repoPath=$HOME -n <namespace>
   • In order to add nodes to existing cluster You can scale the kubernetes replica count of statefulset to desired number. For eg if you want to increase the number of nodes from default three nodes to five nodes you can use the following command: kubectl scale statefulsets mosquitto --replicas=5 -n <namespace>

• You can now add the broker connections and the additional nodes in the cluster through Management-center.

9. Remove Nodes
   • You can start by removing the node configuration from the existing cluster using the Management Center.
   • You can now remove the broker connections using the Management Center.
   • Then you can set the replica count to desired number. For eg if you want to decrease the number of nodes from five nodes to three nodes you can use the following command. kubectl scale statefulsets mosquitto --replicas=3 -n <namespace>

Kubernetes Cluster Setup

Dependencies and Prerequisites

• Docker
• Kubernetes Cluster with Kubeadm
• Helm

If you need to set up a Kubernetes cluster, you can refer to our installation script. If you plan on using your own cluster, you can skip to step 4. Follow these steps on your master/control-plane node :

1. Setup the ha-cluster setups folder:

   • Copy or setup the mosquitto-2.7-mmc-2.7-cluster-kubernetes repository to the NFS server and Control-plane. For both instances the path of the repository has to be the same. Also make sure to create a directory inside the repository on Control-plane named license that contains the license.lic file we provided you.

2. Choose Architecture Folder:

   • Depending on your host architecture, navigate to the corresponding folder:
     • For Debian AMD64:

       cd mosquitto-2.7-mmc-2.7-cluster-kubernetes/kubernetes/multi-node-multi-host/debian_amd64

3. Install Common Dependencies:

   • Run the following command to install the necessary dependencies on all the nodes(including control-plane node):

     bash common-installation-debian.sh

4. Install Master Dependencies:

   • Run the following command to install the necessary dependencies only on the master node (i.e control-plane node):

     bash  master-installation-debian.sh

5. Setup Kubernetes: IMPORTANT

   • Execute the setup script to configure Kubernetes:

     bash setup.sh
     • Follow the instructions of this file. This would create configmap for mosquitto.conf, create configmaps for your license file.
     • Make sure you have placed your license file in the required folder before running this script i.e at /home/<user>/mosquitto-2.7-mmc-2.7-cluster-kubernetes/license/license.lic (if license folder does not exist ,create it and add your license.lic file)
     • The default namespace for installation is multinode. You can change the namespace through the setup script. Script will prompt you to choose the namespace of your choice.

6. Setup Kubernetes Secrets:

   • This step is required for kubernetes to pull required docker images from the registry. You can set the secrets using the following command::

     kubectl create secret docker-registry  mosquitto-pro-secret  --docker-server=registry.cedalo.com --docker-username=<username> --docker-password=<password>  --docker-email=<email> -n <namespace>
   • namespace: namespace should be the same as the one you selected or enetered while running the setup.sh.
   • docker-username: Your docker username
   • docker-password: Your docker password
   • docker-email: Email registered for accessing docker registry

Further Useful Commands:

• If you want to change mosquitto.conf. Make the required changes in mosquitto.conf then delete the configmap and create a new one. Make sure to uninstall the deployment before making the change.
• You can uninstall the setup using the following command: helm uninstall <release-name> -n <namespace>
• To delete the configmap: kubectl delete configmap mosquitto-config -n <namespace>
• To reconfigure the configmap: kubectl create configmap mosquitto-config -n $namespace --from-file=<path-to-mosquitto.conf>
• If you want to customize the deployments, you can unzip the package using: tar -xzvf mosquitto-multi-node-multi-host-0.1.0.tgz
• Make the changes and repackage the folder mosquitto-multi-node-single-host using: helm package mosquitto-multi-node-multi-host

Usage

Once the installation is complete, you can start using the multi-node Mosquitto broker. Be sure to check the Mosquitto documentation for further details on configuring and using the broker