HA-Single-host

To set up a multi-node Mosquitto broker and Management Center on a single host using Helm charts, you'll first need a Kubernetes environment. For deploying kubernetes on a single host, Minikube is an excellent choice. Minikube is a lightweight Kubernetes distribution designed for local development and testing (Discussed in Introduction section).

This setup would deploy a 3 Mosquitto broker as statefulsets. Also, a Management-Center pod and HA-proxy pod as a deployment entity. All the deployment would be deployed on the same host.

Recommended Setup

1 MMC and 3 Mosquitto Brokers (single host)

HA-PROXY Configurations HA-proxy need to be configured accordingly for the kubernetes setup. For server m1, m2 and m3 needs to be configured in this case. Instead of using docker IP we would use DNS names to address the pods. For eg mosquitto-0.mosquitto.multinode.svc.cluster.local. Here mosquitto-0,mosquitto-1,mosquitto-2 are the name of individual mosquitto pods running as statefulsets. Each new pod would increase its pod-ordinal by 1. Rest can be defined as follows <pod-name>.<name-of-the-statefulset>.<namespace>.svc.cluster.local

Your setup folder comes along with a default configuration of haproxy config which is given below. This assumes that your using namespace name as "multinode". You can also change the namespace name if you want and the procedure to do it would be discussed at a later stage.

global
    daemon
    maxconn 4096

frontend mqtt_frontend
    bind *:1883
    mode tcp
    default_backend mqtt_backend
    timeout client 10m

backend mqtt_backend
    timeout connect 5000
    timeout server 10m
    mode tcp
    option redispatch
    server m1 mosquitto-0.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions
    server m2 mosquitto-1.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions
    server m3 mosquitto-2.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions

Note: Make sure to add more entries to this haconfig.yaml file if you wish to add nodes to this cluster. You would have to reconfigure the haconfig.yaml inside helm charts in that case.

Kubernetes Cluster Setup

If you need to set up a Kubernetes cluster, you can follow these steps:

Dependencies and Prerequisites:

• Docker
• Minikube
• Helm

1. Setup the folder on your local machine:

   • Copy or setup the mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes repository to your local machine. Also make sure to create a directory inside the repository named license that contains the license.lic file we provided you. So the relative path would be mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/license/license.lic.
   • Make sure all the data directories have adequate privileges so that mosquitto kubernetes pods can create additional directories inside these data directories. We provide 1000 ownership to the data directory of mosquitto servers and root ownership to config of management-center. The same ownership are also the default ownership of mosquitto pods and MMC pods.

       `sudo chown -R 1000:1000 /home/<user>/mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/server1/mosquitto/data`
       `sudo chown -R 1000:1000 /home/<user>/mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/server2/mosquitto/data`
       `sudo chown -R 1000:1000 /home/<user>/mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/server3/mosquitto/data`
       `sudo chown -R root:root /home/<user>/mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/server1/management-center/config`
   • Note: We provide ownership of 1000 as mosquitto kubernetes pods uses user id of 1000 by default and root to MMC pods.

2. Choose Architecture Folder:

   • Depending on your host architecture, navigate to the corresponding folder:
     • For Debian AMD64:

       cd mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/kubernetes/multi-node-single-host/debian_amd64
     • For Ubuntu AMD64:

       cd mosquitto-2.9-mmc-2.9-singlenode-kubernetes/kubernetes/multi-node-single-host/ubuntu_amd64

3. Install Pre-requisites:

   • Run the following command to install the necessary dependencies on the host. This script installs the dependencies like docker, minikube and helm. You can also install the dependencies separately on your own as well. To run the script use the following command:

     bash  install-dependencies.sh

4. Start Minikube:

   • On your local node: Start the minikube setup using the minikube start command mentioned below.

   • repoPath: Replace the repoPath variable with the path where the folder mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes resides on your local. For eg if it exists on /root/mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes therefore the repoPath would be /root or if exists on /home/demo/mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes then the repoPath would be /home/demo.

     minikube start --mount-string="<repoPath>/mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/:/mnt" --mount

5. Create a namespace

   • On your local node: Create a namespace in which you would want to deploy the application. The deployment folder is pre-configured for the namespace named multinode. If you want to use the default configuration you can create a namespace named multinode using the below command:
   • kubectl create namespace multinode
   • If you want to use a different namespace, use the command: kubectl create namespace <your-custom-namespace>. Replace <your-custom-namespace> with the name of the namespace you want to configure.

6. Create configmap for your license

   • On your local node: Create a configmap for your license key. You can create the configmap using the following command:
   • kubectl create configmap mosquitto-license -n <namespace> --from-file=<path-to-your-license-file>
   • Make sure the name of the configmap remains the same as mosquitto-license as this is required by the deployment files and statefulsets.
   • A sample configmap creation command would look something like this if the choosen namespace is multinode and the license file is at the path /home/<user>/mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/license/license.lic :
     • kubectl create configmap mosquitto-license -n multinode --from-file=/home/<user>/mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/license/license.lic

Installation

Prerequisites:

1. Minikube should be up and running. If you are yet to setup the minikube for single-node deployment, refer Kubernetes Setup.
2. You have successfully created the namespace and configmap for your license (i.e mosquitto-license).

Helm Installation

1. Change Directory:

   • Navigate to the project directory (i.e multi-node-single-host). cd mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/kubernetes/multi-node-single-host

2. Install Helm Chart:

   • Use the following helm install command to deploy the setup to your Kubernetes cluster. Replace <release-name> with the desired name for your Helm release and <namespace> with your chosen Kubernetes namespace:

     helm install <release-name> mosquitto-multi-node-single-host-0.1.0.tgz  -n <namespace> --set imageCredentials.registry=registry.cedalo.com --set imageCredentials.username=<username> --set imageCredentials.password=<password> --set imageCredentials.email=<email>
   • namespace: Set it to the namespace of your deployment.
   • Note: If you want to deploy the setup in a different namespace other than multinode, make sure to pass a separate flag --set namespace=<your-custom-namespace> along with the helm installation command.
   • imageCredentials.username: Your docker username provided by Cedalo team.
   • imageCredentials.password: Your docker password provided by Cedalo team.
   • imageCredentials.email: Registered e-mail for accessing docker registry.
   • Sample example: If your name namespace is test-namespace and your arbitrary release name is sample-release-name, username, password and email be demo-username, demo-password and demo@gmail.com then your helm installation command should be:

        helm install sample-release-name  mosquitto-multi-node-single-host-0.1.0.tgz   --set -n test-namespace --set namespace=test-namespace --set imageCredentials.registry=registry.cedalo.com --set imageCredentials.username=demo-username --set imageCredentials.password=demo-password --set imageCredentials.email=demo@gmail.com

3. You can monitor the running pods using the kubectl get pods -o wide -n <namespace> command. To observe the opened ports use kubectl get svc -n <namespace>.

4. To uninstall the setup: helm uninstall <release-name> -n <namespace>

5. To delete the minikube setup and delete all config: minikube stop && minikube delete

Your Mosquitto setup is now running with a three single mosquitto nodes and the Management Center.

Open Applications: MMC

• kubectl get nodes -o wide
• Get the node ip from the above command. This would not same as your host IP as Minikube setups a hypervisor between your host and Kubernetes pods.Usually the IP of minikube is 192.168.49.2, however it is always better to confirm using kubectl get nodes -o wide command.
• http://<"node-ip">:31021 (Open on your browser )
• Note: If MMC is not loading then you can following command: minikube service mmc -n <namespace>
• To create a cluster through MMC follow: these steps.

Connect to the Mosquitto

• Make sure you have created the cluster through Management-Center.
• minikube service ha -n <namespace>
• To subscribe to SYS topics: mosquitto_sub -h <minikube-node-ip> -p 31028 -u <username> -P <password> -t '$SYS/#'

Further Useful Commands:

• If you want to change mosquitto.conf, you can do so by uncompressing the helm chart, making the required changes and packaging the helm charts again. The detailed procedure is mentioned below:
  • tar -xzvf mosquitto-multi-node-single-host-0.1.0.tgz
    • cd mosquitto-multi-node-single-host/files/
    • Make changes to mosquitto.conf and save it.
    • Go back to the parent directory: cd ../
    • Package the helm chart to its original form using: helm package mosquitto-multi-node-single-host
    • Uninstall helm package helm uninstall <release-name> -n <namespace>
  • Reinstall the helm package using the same command you used the first time from the mosquitto-2.9-mmc-2.9-cluster-single-host-kubernetes/kubernetes/multi-node-single-host/ directory.

Create Cluster in Management Center

After you have completed the installation process, the last step is to configure the Mosquitto HA cluster. Access the Management Center and use the default credentials cedalo and password mmcisawesome.

• Make sure all three mosquitto nodes are connected in the connection menu. The HA proxy will only connect after the cluster is successfully set up.
• Navigate to Cluster Management and click NEW CLUSTER.
• Configure Name, Description and choose between Full-sync and Dynamic Security Sync.
• Configure IP address: Instead of private IP address we will use DNS address.
  • For node1: mosquitto-0.mosquitto.multinode.svc.cluster.local and select broker2 from drop-down
  • For node2: mosquitto-1.mosquitto.multinode.svc.cluster.local and select broker2 from drop-down
  • For node3: mosquitto-2.mosquitto.multinode.svc.cluster.local and select broker3 from drop-down
    • Replace "multinode" with your own namespace. If you have used the default one, use the mentioned configurations.
    • mosquitto-0 has to be mapped to the mosquitto-1 node in the MMC UI and so on.
• Click Save

Usage

Once the installation is complete, you can start using the multi-node Mosquitto broker on your single host. Be sure to check the Mosquitto documentation for further details on configuring and using the broker.