Skip to main content
Version: Mosquitto 2.9

High Availability Openshift

To set up a multi-node-HA Mosquitto broker and Management Center using Helm charts, you'll first need a running Openshift OKD cluster. Openshift offers lot of different features on top of Kubernetes. For details on Openshift and OKD you can refer to the Introduction page. For deploying a full fledged OKD cluster, you can follow the official Openshift OKD installation documentation. OKD can be mainly installed in two different fashion:

  1. IPI: Installer Provisioned Infrastructure
  2. UPI: User Provisioned Infrastructure

Installer Provisioned Infrastructure: Installer Provisioned Infrastructure (IPI) in OKD/OpenShift refers to a deployment model where the installation program provisions and manages all the components of the infrastructure needed to run the OpenShift cluster. This includes the creation of virtual machines, networking rules, load balancers, and storage components, among others. The installer uses cloud-specific APIs to automatically set up the infrastructure, making the process faster, more standardized, and less prone to human error compared to manually setting up the environment.

User Provisioned Infrastructure: User Provisioned Infrastructure (UPI) in OKD/OpenShift is a deployment model where users manually create and manage all the infrastructure components required to run the OpenShift cluster. This includes setting up virtual machines or physical servers, configuring networking, load balancers, storage, and any other necessary infrastructure components. Unlike the Installer Provisioned Infrastructure (IPI) model, where the installation program automatically creates and configures the infrastructure, UPI offers users complete control over the deployment process.

You are free to choose your own method among the two. You can also choose the cloud provider you want to deploy your solution on. Openshift OKD supports number of different cloud providers and also gives you an option to do bare-metal installation. In this deployment we went forward with UPI and deployed our infrastructure on Google Cloud Platform (GCP) using the Private cluster method mentioned here. Therefore, this solution is developed and tested on GCP, however it is unlikely that basic infrastructure would differ across different cloud providers.

A private cluster in GCP ensures that the nodes are isolated in a private network, reducing exposure to the public internet but again you are free to choose your own version of infrastructure supported by Openshift OKD. We will briefly discuss how the infrastructure looks like in our case so that you can have a reference for your own infrastructure.

OKD Infrastructure (During provisioning)

Figure 1: OKD Infrastructure on GCP during provisioning

The diagram depicts the deployment process for our OCP cluster on GCP, starting with the establishment of a bastion host. Bastion host is where we'll execute commands to configure the bootstrap node, then the Master nodes, and finally, the worker nodes in a separate subnet. Before initiating the bootstrap procedure, we set up the essential infrastructure components, including networks, subnetworks, an IAM service account, an IAM project, a Private DNS zone, Load balancers, Cloud NATs, and a Cloud Router.

Upon completing the bootstrap phase, we dismantled the bootstrap components from the cluster. Subsequently, we focussed on creating the worker nodes. After the worker nodes are operational, we set up a reverse proxy on the bastion host to facilitate local access to the OCP Console UI through our browser. To conclude, we confirm that all cluster operators are marked as ‘Available’. Once we done with the provisioning the architecture would look something like Figure 2. More detailed steps can be found in the official documentation. These discussed steps are all part of the official documentation.

OKD Infrastructure (Post provisioning)

Figure 2: OKD Infrastructure on GCP post provisioning

Note: This deployment involves setting up a private cluster, which means access to the cluster is limited to through the bastion host. Consequently, we avoid using public DNS for this installation, relying solely on a private DNS zone. To facilitate access to the external UI, we will employ a reverse proxy for this purpose.

We will also provision a NFS server in the same subnet as worker node. Therefore, this setup would deploy a 3 Mosquitto broker as a statefulsets, a Management-Center pod and HA-proxy pod as a deployment entity. These statefulsets and deployment pods would mount volumes from the NFS server. You would need to setup the NFS server before using this deployment.

Recommended Setup

  1. 3 Control-plane node, 2 worker nodes and a NFS Server
  2. Management center (MMC) is configured to have a node affinity that means the pod for MMC will spawn on a specific worker node. The default configuration expects names of the worker nodes to be openshift-test-rcjp5-worker-0 and openshift-test-rcjp5-worker-1. Given the nodes are named in similar fashion, MMC would be spawned on openshift-test-rcjp5-worker-0.
  3. If you want to have different names for your nodes you can also do that. You will have to adjust the hostnames of nodes in helm chart so that the MMC node affinity remains intact. To adjust the helm chart you will have to uncompress the helm charts and change the hostnames entries of values.yaml. You can do so using the following command:
    • tar -xzvf mosquitto-multi-node-multi-host-0.1.0.tgz
    • cd mosquitto-multi-node-multi-host
    • Change the values of hostname from openshift-test-rcjp5-worker-0/openshift-test-rcjp5-worker-1 to the names of your machines. For eg, openshift-test-rcjp5-worker-0 and openshift-test-rcjp5-worker-1 can be renamed as worker-node-0 and worker-node-1. Doing this, MMC would now be spawned on the node named worker-node-0.
    • Go back to the parent directory: cd ../
    • Package the helm chart to its original form using: helm package mosquitto-multi-node-multi-host

HA-PROXY Configurations HA-proxy need to be configured accordingly for the kubernetes setup. For eg: server m1, m2 and m3 needs to be configured in this case. Instead of using docker IP we would use DNS names to address the pods. For eg mosquitto-0.mosquitto.multinode.svc.cluster.local. Here mosquitto-0,mosquitto-1,mosquitto-2 are the name of individual mosquitto pods running as statefulsets. Each new pod would increase its pod-ordinal by 1. Rest can be defined as follows <pod-name>.<name-of-the-statefulset>.<namespace>.svc.cluster.local

Your setup folder comes along with a default configuration of haproxy config which is given below. This assumes that your using namespace name as "multinode". You can also change the namespace name if you want and the procedure to do it would be discussed at a later stage.

global
    daemon
    maxconn 4096

frontend mqtt_frontend
    bind *:1883
    mode tcp
    default_backend mqtt_backend
    timeout client 10m

backend mqtt_backend
    timeout connect 5000
    timeout server 10m
    mode tcp
    option redispatch
    server m1 mosquitto-0.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions
    server m2 mosquitto-1.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions
    server m3 mosquitto-2.mosquitto.multinode.svc.cluster.local:1883 check on-marked-down shutdown-sessions

Note: Make sure to add more entries to this haproxy.cfg configmap file if you wish to add nodes to this cluster. Once you finish the updating the haconfig configmap you can uninstall and reinstall the helm chart for the changes to come in affect. You can refer to Add Node i.e point 7 in Installation section for tips on reconfiguring haconfig configmap.

Openshift Cluster Setup and Configuration​

Dependencies and Prerequisites​

As we chose to use a private cluster, therefore master and worker nodes would not have access to the internet. Therefore, we will install the dependencies on the bastion node and would also deploy the application from the bastion node.

Prerequisites

  • Running Openshift OKD cluster by following the official documentation guide of Openshift
  • Helm
  • Bastion node with internet access (running Ubuntu in the example case).

After checking the environment prerequisites are set, follow we will prepare the Mosquitto environment:

  1. Setup the ha-cluster setups folder:

    • Copy or setup the mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift folder to the Bastion node. Also make sure to create a directory inside the copied folder on Bastion node named license that contains the license.lic file we provided you. So the relative path would be mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/license/license.lic.

  2. Create a namespace

    • Create a namespace in which you would want to deploy the application. The deployment folder is pre-configured for the namespace named multinode. If you want to use the default configuration you can create a namespace named multinode using the below command:
    • oc create namespace multinode
    • If you want to use a different namespace, use the command: oc create namespace <your-custom-namespace>. Replace <your-custom-namespace> with the name of the namespace you want to configure.

  3. Create configmap for your license

    • Create a configmap for your license key (same license you created ). You can create the configmap using the following command:
    • oc create configmap mosquitto-license -n <namespace> --from-file=<path-to-your-license-file>
    • Make sure the name of the configmap remains the same as mosquitto-license as this is required by the deployment files and statefulsets.
    • A sample configmap creation command would look something like this if the choosen namespace is multinode and the license file is at the path /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/license/license.lic :
      • oc create configmap mosquitto-license -n multinode --from-file=/root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/license/license.lic

  4. Setup NFS Server

    • Copy or setup the mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift folder to the NFS-Server.

    • Install necessary dependencies sudo apt-get update sudo apt-get install nfs-kernel-server

    • Configure exports directory. Open the /etc/exports file on NFS-server. Expose the directories so that pods running on other worker nodes can access these directories and mount the volumes.

      • The default starting point of the cluster is with 3 Mosquitto broker nodes, however we will configure and expose a total of 6 Mosquitto data directories along with a MMC config directory in the NFS server. As the provisioning of data directories on the NFS servers are not dynamic at the moment for On-premise setup, configuring three extra Mosquitto data directories makes the job easier later when you want to add further nodes.
      • The helm charts therefore is also configured in a fashion that they create total of 7 persistent volumes and persistent volume claims (6 for Mosquitto data directories and 1 for MMC). However, only three Mosquitto broker would be spinned up by default.
      • You can use the following as a reference. Here we expose six Mosquitto nodes and management center.
        /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server1/mosquitto/data *(rw,sync,no_root_squash,no_subtree_check)
        /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server2/mosquitto/data  *(rw,sync,no_root_squash,no_subtree_check)
        /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server3/mosquitto/data *(rw,sync,no_root_squash,no_subtree_check)
        /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server4/mosquitto/data *(rw,sync,no_root_squash,no_subtree_check)
        /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server5/mosquitto/data *(rw,sync,no_root_squash,no_subtree_check)
        /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server6/mosquitto/data *(rw,sync,no_root_squash,no_subtree_check)
        /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server1/management-center/config *(rw,sync,no_root_squash,no_subtree_check)

    • On your bastion node: Check the allocated user id for your namespace after you already created your desired namespace (step 2). You can check the allocated user id for your namespace by running the command oc describe namespace <namespace> where <namespace> is the namespace you chose while creating the namespace in step 2. For default namespace i.e multinode, the command would be oc describe namespace multinode.

      • The above command would output a response. A sample output could be like:

        Name:         multinode
        Labels:       kubernetes.io/metadata.name=multinode
                      pod-security.kubernetes.io/audit=restricted
                      pod-security.kubernetes.io/audit-version=v1.24
                      pod-security.kubernetes.io/warn=restricted
                      pod-security.kubernetes.io/warn-version=v1.24
        Annotations:  openshift.io/sa.scc.mcs: s0:c27,c4
                      openshift.io/sa.scc.supplemental-groups: 1000710000/10000
                      openshift.io/sa.scc.uid-range: 1000710000/10000
        Status:       Active

        No resource quota.
        No LimitRange resource.

      • Note down the value for openshift.io/sa.scc.uid-range. In this is case it is 1000710000. You would need these to give permissions to the exposed directories in NFS server and also while installing the helm chart.

    • The noted user id will now be used to make sure all the data directories have adequate privileges. This confirms that Mosquitto Kubernetes pods can create additional directories inside these data directories. Therefore, we will now give adequate permission using the user id we noted in the previous step to all the relevant data directories of Mosquitto server and config directory of MMC using following command:

      sudo chown -R 1000710000:1000710000 /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server1/mosquitto/data
      sudo chown -R 1000710000:1000710000 /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server2/mosquitto/data
      sudo chown -R 1000710000:1000710000 /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server3/mosquitto/data
      sudo chown -R 1000710000:1000710000 /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server4/mosquitto/data
      sudo chown -R 1000710000:1000710000 /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server5/mosquitto/data
      sudo chown -R 1000710000:1000710000 /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server6/mosquitto/data
      sudo chown -R 1000710000:1000710000 /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/server1/management-center/config
      • Note: We provide ownership of 1000710000 based on uid range of our namespace. Please note down and use the uid range of your own namespace from the previous step.

    • Expose the directories using: sudo exportfs -a

    • Restart the kernel-server sudo systemctl restart nfs-kernel-server

Installation​

Prerequisites:

  1. Openshift OKD Cluster should be up and running.
  2. You have successfully created the namespace and configmap for your license (i.e mosquitto-license).
  3. You have configured your NFS Server by exposing the directories.
  4. We can now deploy our Mosquitto application on openshift using two different strategies:

Installation using Helm Charts:​

Helm charts offer a comprehensive solution for configuring various Kubernetes resources—including stateful sets, deployment templates, services, and service accounts—through a single command, streamlining the deployment process.

  1. Setup the folder on your Bastion Node:

    • Make sure you have the mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift folder on the Bastion-node.

  2. Change Directory:

    • Navigate to the project directory (i.e multi-node-multi-host). cd mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/kubernetes/multi-node-multi-host/

  3. Install Helm Chart:

    • Use the following helm install command to deploy the Mosquitto application on to your OKD cluster. Replace <release-name> with the desired name for your Helm release and <namespace> with your chosen Kubernetes namespace:
      helm install <release-name>  mosquitto-multi-node-multi-host-0.1.0.tgz   --set repoPath=<root-path-to-openshift-folder> --set runAsUser=<namespace-alloted-user-id> --set nfs=<your-nfs-ip> -n <namespace>  --set imageCredentials.registry=registry.cedalo.com --set imageCredentials.username=<username> --set imageCredentials.password=<password> --set imageCredentials.email=<email>
    • repoPath: Set the repoPath flag to the path where the folder mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift resides on NFS server. In our case it exists on /root/mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift therefore the repoPath would be /root.
    • namespace: Set it to the namespace of your deployment.
    • Note: If you want to deploy the setup in a different namespace other than multinode, make sure to pass a separate flag --set namespace=<your-custom-namespace> along with the helm installation command. This custom namespace must have have been created in step 2 of Openshift Cluster Setup and Configuration.
    • namespace-alloted-user-id: Set it to user id you noted while setting up the NFS server (step 4 of Openshift Cluster Setup and Configuration) .
    • Note: You need to configure the IP of your NFS server by passing --set nfs=<your-nfs-ip> along with the helm installation command. Make sure you use the internal NFS ip accessible from within the OKD cluster and not the external IP exposed to the internet (in case you have one).
    • imageCredentials.username: Your docker username provided by Cedalo team.
    • imageCredentials.password: Your docker password provided by Cedalo team.
    • imageCredentials.email: Registered e-mail for accessing docker registry.
    • Sample example: If your NFS IP is 10.10.10.10,your user id for your namespace is 1000710000, mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift resides at the location /root on your nfs, your name namespace is test-namespace and your arbitrary release name is sample-release-name, username, password and email be demo-username, demo-password and demo@gmail.com then your helm installation command should be:
         helm install sample-release-name  mosquitto-multi-node-multi-host-0.1.0.tgz   --set repoPath=/root  --set runAsUser=1000710000 -n test-namespace --set namespace=test-namespace --set nfs=10.10.10.10 --set imageCredentials.registry=registry.cedalo.com --set imageCredentials.username=demo-username --set imageCredentials.password=demo-password --set imageCredentials.email=demo@gmail.com

  4. You can monitor the running pods using the oc get pods -o wide -n <namespace> command. To observe the opened ports use oc get svc -n <namespace>.

  5. To uninstall the setup: helm uninstall <release-name> -n <namespace>

Your Mosquitto setup is now running with three single mosquitto nodes and the Management Center. To finish the cluster setup, the Management Center offers a UI to create the Mosquitto HA Cluster. The Management Center is reachable from the localhost via port 31021.

  • To set up the cluster follow these steps.
  • To be able to access this UI outside of your localhost go here

Further possible Configurations​

  • Add Nodes

    • Configure your HA-proxy by adding entries to the haproxy.cfg.
    • haconfig configmap is part of the compressed Helm chart. Therefore, to edit the configmap, uncompress the helm chart using the following command:
      • tar -xzvf mosquitto-multi-node-multi-host-0.1.0.tgz
      • cd mosquitto-multi-node-multi-host/templates/
      • Edit haconfig.yaml by adding below mentioned line at end the the file server m4 mosquitto-3.mosquitto.<namespace>.svc.cluster.local:1883 check on-marked-down shutdown-sessions. Replace <namespace> with your chosen namespace. This config line assumes you are adding the fourth node.
    • Go back to the parent directory: cd ../
    • Package the helm chart to its original form using: helm package mosquitto-multi-node-multi-host
    • Uninstall helm package helm uninstall <release-name> -n <namespace>
    • We do not need to configure extra data directories in the NFS server till we exceed the count of 6 Mosquitto brokers as we already addressed this while configuring our NFS server.
    • Reinstall helm package so that new configmaps can come into effect (Use the same command you used to install it the first time with appropriate flags. Below command eg is just a reference) helm install <release-name> mosquitto-multi-node-multi-host-0.1.0.tgz --set repoPath=$HOME -n <namespace>
    • In order to add nodes to existing cluster, you can scale the kubernetes replica count of statefulset to desired number. For eg if you want to increase the number of nodes from default three nodes to four nodes you can use the following command: oc scale statefulsets mosquitto --replicas=4 -n <namespace>
    • You can now add the broker connections and the additional nodes in the cluster through Management-center.
      • Open your MMC, navigate to broker connections, click on NEW CONNECTION .
      • Fill in the details like ID(make sure it does not conflict with the existing ones), NAME: name of the connection (you can choose your own name).
      • URL: mqtt://mosquitto-x.mosquitto.<namespace>.svc.cluster.local:1885 . Replace x in mosquitto-x with a valid number based on the which node you are adding. If you are adding a fourth node and the name of your namespace is multinode then the entry would be mqtt://mosquitto-3.mosquitto.multinode.svc.cluster.local:1885.
      • Add your username and password given by the cedalo team, select/deselect further options and click CONNECT & SAVE.
    • Adding new node to existing cluster:
      • Navigate to Cluster Management and select your existing cluster.
      • Click Edit and select Add Node.
      • Enter DNS address in IP address field. For eg mosquitto-x.mosquitto.<namespace>.svc.cluster.local. Replace x in mosquitto-x with a valid number based on the which node you are adding. If you are adding a fourth node and the name of your namespace is multinode then the entry would be mosquitto-3.mosquitto.multinode.svc.cluster.local.
        • Select Add Node and click Save.

  • Remove Nodes

    • You can start by removing the node configuration from the existing cluster using the Management Center.
    • You can now remove the broker connections using the Management Center.
    • Then you can set the replica count to desired number. For eg if you want to decrease the number of nodes from five nodes to three nodes you can use the following command. oc scale statefulsets mosquitto --replicas=3 -n <namespace>

  • Further Useful Commands

    • If you want to change mosquitto.conf, you can do so by uncompressing the helm chart, making the required changes and packaging the helm charts again. The detailed procedure is mentioned below:
      • tar -xzvf mosquitto-multi-node-multi-host-0.1.0.tgz
      • cd mosquitto-multi-node-multi-host/files/
      • Make changes to mosquitto.conf and save it.
      • Go back to the parent directory: cd ../
      • Package the helm chart to its original form using: helm package mosquitto-multi-node-multi-host
      • Uninstall helm package helm uninstall <release-name> -n <namespace>
      • Reinstall the helm package using the same command you used the first time from the mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/kubernetes/multi-node-multi-host/ directory.

Installation using Openshift UI​

Prerequisites:

  1. Openshift OKD Cluster should be up and running.
  2. You have configured your NFS Server by exposing the directories.

Installation​

  1. Navigate to the project directory . cd mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift/kubernetes/multi-node-multi-host/mosquitto-ha-openshift-ui/

  2. Log in to your Openshift UI Console.

  3. Create Namespace:

    • Navigate to Administration tab from side bar.
    • Click Namespaces
    • Click on Create Namespace .
    • Enter the name of the namespace you want to use. By default, the templates are configured with the namespace named multinode. If you want to use the default namespace , enter the namespace as multinode.
    • Set Default network policy as No restrictions.
    • Click Create.

  4. Deploy Configmaps (License):

    • Navigate to Workloads tab from side bar.
    • Click ConfigMaps.
    • Click on Create ConfigMap.
    • Select Form view
    • Name the configmap as mosquitto-license, choose the configmap as immutable.
    • Add your license file in Key-value pair section.
    • Click Create.
  • Similarly repeat the steps for haconfig.yaml and mosquitto-config1.yaml and create two different services.
  1. Deploy Secrets (Image Pull):
    • Navigate to Workloads tab from side bar.
    • Click Secrets.
    • Click on Create Secret and select Image pull secret.
    • Enter Secret name as mosquitto-pro-secret.
    • Choose Authentication type as Image registry credentials.
    • Enter Registry server address as registry.cedalo.com.
    • Enter your username, passwordand email given by the Cedalo team.
    • Click Create
  2. Deploy Service Accounts:
  • Navigate to User Management tab from side bar.
  • Click Service Accounts.
  • Click on Create ServiceAccount.
  • Copy the content from service_account.yaml file in mosquitto-ha-openshift-ui folder. Starting with service account named ha.
  • Paste the content and make sure to change your namespace from multinode to your chosen namespace. (you can leave it as it is if it is default one).
  • Click create.
  • Repeat the steps for remaining service accounts namely mosquitto and mmc.
  • At the end of this activity, you will have three different service accounts namely ha , mosquitto and mmc .
  1. Deploy Configmaps:
    • Navigate to Workloads tab from side bar.
    • Click ConfigMaps.
    • Click on Create ConfigMap.
    • Copy the content from config-json.yaml file in mosquitto-ha-openshift-ui folder.
    • Paste the content and make sure to change your namespace from multinode to your chosen namespace. (you can leave it as it is if it is default one).
    • Click Create.
  • Similarly repeat the steps for haconfig.yaml and mosquitto-config1.yaml and create two different services.
  1. Deploy Services:
  • Navigate to Networking tab from side bar.
  • Click Services.
  • Click on Create Service.
  • Copy the content from service_ha.yaml.
  • Paste it in the Yaml editor and make sure to change your namespace from multinode to your chosen namespace. (you can leave it as it is if it is default one).
  • Click Create.
  • Similarly repeat the steps for service_mmc.yaml and service-statefulset.yaml and create two different services.
  • At the end of this activity, you will have three different services namely ha , mosquitto and mmc.
  1. Deploy Persistent Volume (PV):
  • Navigate to Storage tab from side bar.
  • Click PersistentVolumes.
  • Click on Create PersistentVolumes.
  • Copy the contents from pv-openshift.yaml. Start with copying the PV named mosquitto-pv-0 .
  • Paste it in the YAML editor and make sure to change your namespace from multinode to your chosen namespace. (you can leave it as it is if it is default one).
  • Make sure that the path to mosquitto-2.9-mmc-2.9-cluster-kubernetes-openshift folder on your nfs folder is correct. Default configuration expects the folder to be at /root .
  • Change the NFS server IP to your own NFS IP in each entry of Persistent Volume configuration.
  • Click Create.
  • Repeat the steps for creating remaining PVs: mosquitto-pv-1,mosquitto-pv-2,mosquitto-pv-3,mosquitto-pv-4,mosquitto-pv-5 and mmc-pv
  1. Deploy Persistent Volume Claims (PVCs):
  • Navigate to Storage tab from side bar.
  • Click PersistentVolumeClaim.
  • Click on Create PersistentVolumeClaim.
  • Copy the contents from pvc-openshift.yaml . Start with copying the PVC named mosquitto-data-mosquitto-0.
  • Paste it in the YAML editor and make sure to change your namespace from multinode to your chosen namespace. (you can leave it as it is if it is default one)
  • Click Create.
  • Repeat the steps for creating remaining PVCs: mosquitto-data-mosquitto-1,mosquitto-data-mosquitto-2,mosquitto-data-mosquitto-3,mosquitto-data-mosquitto-4,mosquitto-data-mosquitto-5 and mmc-config
  1. Deploy Mosquitto (Statefulset):
  • Navigate to Workloads tab from side bar.
  • Click StatefulSets.
  • Click on Create StatefulSet.
  • Copy the contents from statefulset-openshift.yaml,
  • Paste it in the YAML editor and make sure to change your namespace from multinode to your chosen namespace. (you can leave it as it is if it is default one).
  • Set the field runAsUser to the your own user id. Default one is set to 1000710000. You can find the user id of your namespace through the following command: oc describe namespace multinode. Refer NFS-Server section for more details.
  • Click Create. After this activity, you would see statefulset pods scaling from 0 to 3.
  1. Deploy MMC (Deployment):
  • Navigate to Workloads tab from side bar,
  • Click Deployments
  • Click on Create Deployment.
  • Copy the contents from deployment-mmc-openshift.yaml,
  • Paste it in the YAML editor and make sure to change your namespace from multinode to your chosen namespace. (you can leave it as it is if it is default one).
  • As MMC pod has node affinity , set the value of the field kubernetes.io/hostname and value field under nodeAffinity to the worker node hostname of where you want MMC pod to spin up. Default value is set to openshift-test-rcjp5-worker-0.
  • Set the field runAsUser to the your own user id. Default one is set to 1000710000. You can find the user id of your namespace through the following command: oc describe namespace multinode.
  • Click Create.
  1. Deploy HAProxy (Deployment):
  • Navigate to Workloads tab from side bar.
  • Click Deployments.
  • Click on Create Deployment.
  • Copy the contents from deployment-ha-openshift.yaml.
  • Paste it in the YAML editor and make sure to change your namespace from multinode to your chosen namespace. (you can leave it as it is if it is default one).
  • Click Create.

Create Cluster in Management Center​

After you have completed the installation process, the last step is to configure the Mosquitto HA cluster. Access the Management Center and use the default credentials cedalo and password mmcisawesome.

  • Make sure all three mosquitto nodes are connected in the connection menu. The HA proxy will only connect after the cluster is successfully set up.
  • Navigate to Cluster Management and click NEW CLUSTER.
  • Configure Name, Description and choose between Full-sync and Dynamic Security Sync.
  • Configure IP address: Instead of private IP address we will use DNS address.
    • For node1: mosquitto-0.mosquitto.multinode.svc.cluster.local and select broker2 from drop-down
    • For node2: mosquitto-1.mosquitto.multinode.svc.cluster.local and select broker2 from drop-down
    • For node3: mosquitto-2.mosquitto.multinode.svc.cluster.local and select broker3 from drop-down
      • Replace "multinode" with your own namespace. If you have used the default one, use the mentioned configurations.
      • mosquitto-0 has to be mapped to the mosquitto-1 node in the MMC UI and so on.
  • Click Save

Connect to cluster​

After the creation of the cluster, you can now select the cluster leader in the drop-down in the top right side of the MMC. This is needed, because only the leader is able to configure the cluster. The drop-down appears as soon as you are in one of the broker menus. Go to the "Client" menu and create a new client to connect from. Make sure to assign a role, like the default "client" role, to allow your client to publish and/or subscribe to topics.

Now you can connect to the Mosquitto cluster.

In this example command we use Mosquitto Sub to subscribe onto all topics: mosquitto_sub -h <external-public-ip-of-bastion-node> -p 31028 -u <username> -P <password> -t '#' Make sure to replace your IP, username and password.

Allow cluster access from outside the Bastion node​

Most of the time it is not practical to have the Mosquitto cluster only locally available. To be able to access it from the outside, you can do so by adding an additional HAProxy on the Bastion node.

Setup HA Proxy​

To access MMC and HAproxy connection over the internet, you can add an additional HAproxy service on Bastion node that would redirect traffic to your worker node which is running MMC and Haproxy (would again direct the traffic to Mosquitto brokers) pods. This is because the worker nodes do not have access over the internet. Below is an example how you can add extra frontend and backend to forward traffic to MMC and HA. Make sure to add the additional configuration of Haproxy on your own besides the below-mentioned config.

```
frontend localhost_31021
    bind *:31021
    option tcplog
    mode tcp
    default_backend servers_w0

backend servers_w0
    mode tcp
    server w0 10.1.20.2:31021

frontend localhost_31028
    bind *:31028
    option tcplog
    mode tcp
    default_backend servers_w1

backend servers_w1
    mode tcp
    server w1 10.1.20.2:31028  # Assuming 10.1.20.3 is the IP of server w1
```

You can then access the MMC console using external public ip of your bastion node on port 31021. Open using below url on your browser: http://<external-public-ip-of-bastion-node>:31021 (MMC)

You can also port-forward your local port to the external Bastion node external IP for accessing it on the localhost.

Port forwarding to your localhost​

By using a secure ssh connection it is possible to forward certain ports to your localhost. Use the following command:

 ssh -L 127.0.0.1:31021:127.0.0.1:31021 -L 127.0.0.1:31028:127.0.0.1:31028 <external-public-ip-of-bastion-node>

Port 31021 gives access to the Management Center, while port 31028 gives access to the cluster leader node.

It is now possible to access the Management Center via your browser: http://localhost:31021

Usage​

Once the installation is complete, you can start using the multi-node Mosquitto broker. Be sure to check the Mosquitto documentation for further details on configuring and using the broker.