HA-Single-host

To set up a multi-node Mosquitto broker and Management Center on a single host using Helm charts, you'll first need a Kubernetes environment. For deploying kubernetes on a single host, Minikube is an excellent choice. Minikube is a lightweight Kubernetes distribution designed for local development and testing (Discussed in Introduction section).

This setup would deploy a 3 Mosquitto broker as statefulsets. Also, a Management-Center pod and HA-proxy pod as a deployment entity. All the deployment would be deployed on the same host.

Recommended Setup

1. MMC and 3 Mosquitto Brokers (single host)

HA-PROXY Configurations HA-proxy need to be configured accordingly for the kubernetes setup. For server m1, m2 and m3 needs to be configured in this case. Instead of using docker IP we would use DNS names to address the pods. For eg mosquitto-0.mosquitto.multinode.svc.cluster.local. Here mosquitto-0,mosquitto-1,mosquitto-2 are the name of individual mosquitto pods running as statefulsets. Each new pod would increase its pod-ordinal by 1. Rest can be defined as follows <pod-name>.<name-of-the-statefulset>.<namespace>.svc.cluster.local

global
    daemon
    maxconn 4096

frontend mqtt_frontend
    bind *:1883
    mode tcp
    default_backend mqtt_backend
    timeout client 10m

backend mqtt_backend
    timeout connect 5000
    timeout server 10m
    mode tcp
    option redispatch
    server m1 mosquitto-0.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions
    server m2 mosquitto-1.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions
    server m3 mosquitto-2.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions

Note: Make sure to add more entries to this haproxy.cfg file if you wish to add nodes to this cluster. You would have to reconfigure the HACONFIG in that case. You can refer to Add Node i.e point 7 in Installation section for tips on reconfiguring haconfig configmap.

Installation

Before proceeding with the actual installation of Mosquitto brokers we would need certain dependencies as a prerequisites for Minikube Kubernetes setup. You can install those dependencies using the install-dependencies.sh file in the shared single-node directory. You can find the required dependencies installation file based on your host configuration. If you have a host machine running debian AMD64 configuration, you can get debian-amd64 directory to install the packages. Main packages included are docker and helm-charts.

1. Setup the folder on your local machine:

   • Copy or setup the mosquitto-2.7-mmc-2.7-cluster-single-host-kubernetes repository to your local machine. Also make sure to create a directory inside the repository named license that contains the license.lic file we provided you.
   • If you face any permission issues then make sure all the data directories have adequate privilges so that mosquitto kubernetes pods can create additional directories inside these data directories. If you face any issues like permission denied you can try giving the 1000 ownership to all the relevant data dir using following command:

     sudo chown -R 1000:1000 /home/<user>/mosquitto-2.7-mmc-2.7-cluster-single-host-kubernetes/server1/mosquitto/data
     sudo chown -R 1000:1000 /home/<user>/mosquitto-2.7-mmc-2.7-cluster-single-host-kubernetes/server2/mosquitto/data
     sudo chown -R 1000:1000 /home/<user>/mosquitto-2.7-mmc-2.7-cluster-single-host-kubernetes/server3/mosquitto/data
     • Note: We provide ownership of 1000 as mosquitto kubernetes pods uses user id of 1000 by default.

2. Setup Minikube Kubernetes Setup

   • See section: Kubernetes Cluster Setup

3. Change Directory:

   • Navigate to the project directory (i.e multi-node-single-host). cd mosquitto-2.7-mmc-2.7-cluster-single-host-kubernetes/kubernetes/multi-node-single-host

4. Install Helm Chart:

   • Use the following helm install command to deploy the setup to your Kubernetes cluster. Replace <release-name> with the desired name for your Helm release and <namespace> with your chosen Kubernetes namespace:

     helm install <release-name> mosquitto-multi-node-single-host-0.1.0.tgz --set repoPath=$HOME -n <namespace>
   • repoPath: Set the repoPath flag to the path where this repo was cloned. The above command expects it to be at $HOME.
   • namespace: Set it to the namespace of your deployment.
   • Note: Ensure that you have a running Kubernetes cluster set up to run this deployment. We recommend setting up the kubernetes cluster using Minikube. You can also use our installation script to set up Kubernetes cluster. To setup the kubernetes cluster, follow the instructions in Kubernetes Cluster Setup.
   • Note: If you want to deploy the setup in a different namespace, make sure to pass a seperate flag --set namespace=<your-custom-namespace> along with the helm installation command.

5. Open Applications:

• kubectl get nodes -o wide
• Get the node ip from the above command. This would not same as your host as Minikube setups a hypervisor between your host and Kubernetes pods.Usually the IP of minikube is 192.168.49.2.
• http://<"node-ip">:31021 (Open on your browser )
• Note: If MMC is not loading then you can following command: minikube service mmc -n <namespace>
• IP: <"node-ip"> & Port: 31028 (HA)

6. To uninstall the setup: helm uninstall <release-name> -n <namespace>
7. To delete the minikube setup and delete all config: minikube stop && minikube delete

Kubernetes Cluster Setup

If you need to set up a Kubernetes cluster, you can use our installation script (recommended). If you plan on using your own cluster, you can skip to step 3. Follow these steps:

Dependencies and Prerequisites:

• Docker
• Minikube
• Helm

1. Choose Architecture Folder:

   • Depending on your host architecture, navigate to the corresponding folder:
     • For Debian AMD64:

       cd mosquitto-2.7-mmc-2.7-cluster-single-host-kubernetes/kubernetes/multi-node-single-host/debian_amd64

2. Install Pre-requisites:

   • Run the following command to install the necessary dependencies on the host:

     bash  install-dependencies.sh

3. Setup Kubernetes: IMPORTANT

   • Execute the setup script to configure Kubernetes:

     bash setup.sh
     • Follow the instructions of this file. This would install Minikube, create configmap for mosquitto.conf, create configmaps for your license file. Make sure you have placed your license file in the required folder before running this script.
     • The default namespace for installation is single-node. You can change the namespace through the setup script..

4. Setup Kubernetes Secrets:

   • This step is required for kubernetes to pull required docker images from the registry. You can set the secrets using the following command:

     kubectl create secret docker-registry  mosquitto-pro-secret  --docker-server=registry.cedalo.com --docker-username=<username> --docker-password=<password>  --docker-email=<email> -n <namespace>
   • namespace: namespace should be the same as the one you selected or enetered while running the setup.sh.
   • docker-username: Your docker username
   • docker-password: Your docker password
   • docker-email: Email registered for accessing docker registry

5. You can monitor the running pods using the following command: kubectl get pods -o wide -n <namespace>

Further Useful Commands:

• If you want to change mosquitto.conf. Make the required changes in mosquitto.conf then delete the configmap and create a new one. Make sure to uninstall the deployment before making the change.
• You can uninstall the setup using the following command: helm uninstall <release-name> -n <namespace>
• To delete the configmap: kubectl delete configmap mosquitto-config -n <namespace>
• To reconfigure the configmap (after making changes to mosquitto.conf): kubectl create configmap mosquitto-config -n $namespace --from-file=<path-to-mosquitto.conf>
• If you want to customize the deployments, you can unzip the package using: tar -xzvf mosquitto-multi-node-single-host-0.1.0.tgz
• Make the changes and repackage the folder mosquitto-multi-node-single-host using: helm package mosquitto-multi-node-single-host

Usage

Once the installation is complete, you can start using the multi-node Mosquitto broker on your single host. Be sure to check the Mosquitto documentation for further details on configuring and using the broker.