Mosquitto High Availability Cluster

note

Premium feature

Cedalo High Availability Cluster Setup for Eclipse Mosquitto

Introduction

The Cedalo high availability feature provides the ability to run a cluster of Mosquitto nodes with a single leader and multiple followers, so that broker availability can be ensured even if a single node becomes unavailable through fault or for an upgrade. This document describes the recommended cluster architecture and how to configure the cluster for first use, and in the future.

Cluster architecture

The Mosquitto cluster comprises at least three nodes. A single node is available for use by MQTT clients at once, the other nodes operate as failover nodes. The cluster expects a minimum of two nodes to be available at once, to provide a leader and a fallback node. If the state of the cluster fails so that only a single node is available, clients will be unable to connect until the cluster is in a stable state again.

Figure 1 shows a suggested cluster architecture. There are three broker nodes operating Mosquitto, and a fourth node providing a Management Center for Mosquitto instance (MMC).

The nodes operate with public and private network communication. The private communication includes node-to-node synchronization data, which is not encrypted and must be kept private. In this example, all private communication happens on a separate private network in the 192.168.1.* address space. If private IP addresses are not available or appropriate, then the private connection can be made using a VPN. Do not use a publicly accessible network for the cluster communication, or your credentials and data will be exposed to the internet even if your main MQTT communication is encrypted.

The load balancers listen on port 8883 in this example. They carry out TLS termination and forward connections to the leader broker node on port 1884. Each broker node has port 1885 open to allow connections even when the node is not the leader. This port should only be used for cluster configuration and inspection. Any other use is not covered by HA.

It is strongly recommended that all public communication is encrypted. If required, using port 1883 for unencrypted connections can be done by exposing the port on the load balancer and forwarding it to the broker port 1884.

In this example there are three separate load balancers, and three separate broker nodes, with the broker nodes having no public networking. The purpose of the load balancer is threefold: to provide TLS/SSL termination, to route client connections to the currently available broker, and to provide separation between the public and private networks.

Other arrangements are possible, for example combining the load balancer and broker on nodes may be desirable in simple clusters where keeping node count low is required.

Docker image installation

The Docker image is at registry.cedalo.com/mosquitto/mosquitto:2.1. Cedalo Registry Credentials are needed to access this image.

Example docker-compose file showing the configuration described in the diagram.

version: '3.7'
 
services:
   mosquitto:
       image: registry.cedalo.com/mosquitto/mosquitto:2.1
       ports:
           - <private-network-ip>:1884:1884
           - <private-network-ip>:1885:1885
           - <private-network-ip>:7000:7000
       volumes:
           - ./mosquitto/config:/mosquitto/config
           - ./mosquitto/data:/mosquitto/data
       networks:
           - mosquitto
       environment:
           CEDALO_LICENSE_FILE: /mosquitto/data/license.lic
           CEDALO_HA_DATA_DIR: /mosquitto/data/ha
           MOSQUITTO_DYNSEC_PASSWORD: <admin password to use when generating config>
       restart: unless-stopped
networks:
   mosquitto:
       name: mosquitto
       driver: bridge

The suggested directory layout for the volume to be mounted on the Mosquitto Docker container is:

/mosquitto/docker-compose.yml
/mosquitto/mosquitto/config/mosquitto.conf
/mosquitto/mosquitto/data/license.lic
/mosquitto/mosquitto/data/ha/

Broker Configuration

The suggested broker configuration file for each node is given below:

 
global_plugin /usr/lib/cedalo_mosquitto_ha.so
enable_control_api true 
allow_anonymous false
 
# The listener for receiving incoming MQTT connections from the load balancer.
# This listener will be automatically closed when this node is not the cluster
# leader.
 
listener 1884
 
# Optional websockets listener, if used this must be exposed in the 
# docker-compose configuration
#listener 8080
#protocol websockets
 
# The listener for receiving the MMC connection / API control of a follower node.
# This listener will remain open regardless of the role of the node in the
# cluster. It should not be used by general purpose MQTT clients and is not
# HA supported.
listener 1885
admin_listener true
 
# Some sensible options - tweak as per your requirements
# Set max_keepalive to be reasonably higher than your expected max value
max_keepalive 1800
# max_packet_size can protect devices against unreasonably large payloads
max_packet_size 100000000
# Reduce network latency
set_tcp_nodelay true

In addition, environment variables must be set to configure the license file and HA data path, as demonstrated in the docker-compose example above:

The MOSQUITTO_DYNSEC_PASSWORD variable is not required, but allows the initial authentication/authorisation configuration to be generated with an admin user using this password, which is convenient when generating configurations for multiple nodes. If this is not set, then each node will generate its own admin password on startup and save a plain text copy at /mosquitto/data/dynamic-security.json.pw The cluster configuration is controlled through the HA MQTT topic API described below.

Load balancer configuration

The load balancer should be configured to listen on the public address for the node, to terminate TLS/SSL connections, and to forward the client connections to the private addresses of the cluster nodes. In this example the broker port is 1884.

If the load balancer has an idle timeout value that disconnects clients if no network traffic is observed for a given period, this should be configured to be greater than the MQTT keepalive value you intend to use with your clients, otherwise there will be frequent disconnections for your idle clients. Many MQTT clients use a default of 60 seconds keepalive. An example for HAProxy is given below

global
   user haproxy
   group haproxy
   daemon
   nbproc 4
   nbthread 1
   maxconn 4096
 
   # Default SSL material locations
   ca-base /etc/ssl/certs
   crt-base /etc/ssl/private
   # Default ciphers to use on SSL-enabled listening sockets.
   # For more information, see ciphers(1SSL). This list is from:
   #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
   # An alternative list with additional directives can be obtained from
   #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
   ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
   ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
   ssl-dh-param-file /etc/haproxy/dhparams.pem
 
defaults
   log global
   mode    tcp
   option  dontlognull
   timeout connect 5000
   timeout client 2m
   timeout server 2m
 
frontend mosquitto_frontend
   bind *:8883 ssl crt /etc/haproxy/server.comb.pem
   mode tcp
   option tcplog
   default_backend mosquitto_backend
 
backend mosquitto_backend
   mode tcp
   option tcplog
   option redispatch
   server node1 192.168.1.1:1884 check
   server node2 192.168.1.2:1884 check
   server node3 192.168.1.3:1884 check

Cluster configuration

The cluster can be configured dynamically using the MQTT topic API provided for this purpose. JSON messages are sent to the topic $CONTROL/cedalo/ha/v1 using the admin user and password configured in MOSQUITTO_DYNSEC_PASSWORD. Responses from the broker are published on the topic $CONTROL/cedalo/ha/v1/response

The Management Center for Mosquitto can be used to carry out the configuration, or the topic API can be used directly. The Mosquitto command line client mosquitto_rr can be used as a very basic control method:

mosquitto_rr -h node1 -p 1884 -u admin -P <password> -t ‘$CONTROL/cedalo/ha/v1’ -e ‘$CONTROL/cedalo/ha/v1/response’ -m ‘<command>’

Creating the Cluster

Configuring the first node

Send the createCluster command to the first node to initialize the cluster. The nodes array contains a list of at least three nodes to be used for the cluster. The address and port of each node are the private network address of that node and the port to be used for cluster communication. The nodeid is a unique integer for the node within the cluster, in the range 1-1023. The mynode boolean must be set to true on the details of the node where the command is being sent.

The command should be sent directly to the node from within the private network, rather than through the load balancer.

{
   "commands":[
{
  "command": "createCluster",
  "clustername": "name",
  "nodes": [
    {"address": "192.168.1.1", "port": 7000, "nodeid": 1, "mynode": true},
    {"address": "192.168.1.2", "port": 7000, "nodeid": 2},
    {"address": "192.168.1.3", "port": 7000, "nodeid": 3}
  ]
}
   ]
}

With the Cluster creation a leader is automatically elected. All nodes are synced and now work via the leader authentication.

Joining the nodes to the cluster

Once the first cluster node is configured, the nodes in the list must be told to join the cluster. Send the joinCluster command to each new cluster node. The nodes list must include details of the cluster as in the createCluster command, with the new node details having mynode set to true. All addresses and ports are for the private network and cluster port.

{
  "commands":[
    {
      "command": "joinCluster",
      "clustername": "name",
      "nodes":[
    {"address": "192.168.1.1", "port": 7000, "nodeid": 1},
    {"address": "192.168.1.2", "port": 7000, "nodeid": 2, "mynode": true},
    {"address": "192.168.1.3", "port": 7000, "nodeid": 3}
 
      ]
    }
  ]
}

Adding further nodes

To add further nodes at a later date requires two steps:

1. Informing the existing cluster of the new node
2. Telling the new node to join the cluster.

Informing the cluster

Send the addNode command to the existing cluster node. This must be sent to the leader node. Port 1883 will only be available for the leader node, port 1884 will be available on each node, but only the leader will accept cluster commands.

{
   "commands":[
       {
           "command":"addNode",
           "clustername":"name",
           "address":"192.168.1.4",
           "port":7000,
           "nodeid":4
       }
   ]
}

Joining the node to the cluster

Send the joinCluster command to the new cluster node. The nodes list must include details of the existing cluster nodes, with mynode omitted or set to false, and the new node details with mynode set to true. All addresses and ports are for the private network and cluster port.

info

It is strongly recommended adding nodes one at a time with the addNode and joinCluster commands, then verify the cluster is operating correctly before adding further nodes. If multiple nodes are added using addNode without those nodes joining the cluster, then it is possible that the cluster will be unable to maintain consensus which would adversely affect the HA capability.

Removing nodes from Cluster

Inform Cluster Leader

To remove a node from the cluster, send the removeNode command to the cluster leader.

{
   "commands":[
       {
           "command":"removeNode",
           "clustername":"name",
           "address":"192.168.1.2",
           "port":7000,
           "nodeid":2
       }
   ]
}

All details must match for the node to be removed.

The cluster will refuse to remove a node if it will result in the cluster having fewer than three nodes.

Leave Cluster

Once the Cluster leader is informed, remove the node from the cluster. The command should be sent to the node that is to leave the cluster, not the cluster leader. The clustername, address, port, and nodeid must match.

{
   "commands":[
       {
           "command":"leaveCluster",
           "clustername":"name",
           "address":"192.168.1.2",
           "port":7000,
           "nodeid":2
       }
   ]
}

Deleting the cluster

Send the deleteCluster command to the cluster leader. All nodes will retain their current settings, but will no longer be part of a cluster.

{
   "commands":[
       {
           "command":"deleteCluster",
           "clustername":"name"
       }
   ]
}

Getting cluster information

Send the getCluster command to the cluster leader.

{
   "commands":[
       {
           "command":"getCluster"
       }
   ]
}

Getting node information

Send the getNode command to any individual node. Nodes are always available for communication on port 1884.

   "commands":[
       {
           "command":"getNode"
       }
   ]
}

Cluster System Requirements

On each High-Availability node the components need the following system resources:

Server hardware

For the Load Balancer we recommend HAProxy. The recommended hardware components for the instance would be: Recent 4+ core CPU (AMD or Intel) and 8GB of RAM, if Load Balancers run separately, 8+ cores if load balancers run on same hosts as the Mosquitto nodes. As storage, 5 GB (R/W speed at least 500 MB/s) in a RAID is recommended.

	Mosquitto Node	Load balancer	Mosquitto Node + Loadbalancer
CPU:	4+ Cores	4+ Cores	8+ Cores
RAM:	8GB	8GB	8GB
Storage:	5GB (500Mb/s +)

In case stream processing is used, then the following recommendations apply for each node. While the CPU recommendations stay the same, it is proposed to use 24 GB of RAM and 50 GB (R/W speed at least 500 MB/s) of storage.

Stream Plugin usage:

	Mosquitto Node	Load balancer	Mosquitto Node + Loadbalancer
CPU:	4+ Cores	4+ Cores	8+ Cores
RAM:	24GB	24GB	24GB
Storage:	50GB (500Mb/s +)	50GB (500Mb/s +)	50GB (500Mb/s +)

Network

As shown in the cluster architecture, the following network system is required.

All nodes need to have access to a public network with public IPs (where MQTT clients and browser to view MMC information reside). A private network with private IPs (for communication between nodes, load balancers, and between nodes and Management Center). If this is not explicitly possible then a VPN can be pulled up to privately connect the components. Private network communication is done via TCP, while the Load Balancer are used for SSL termination.

All nodes are 100% replicas from each other and get synchronized in real-time via the private network.

Network speed: Depending on expected bandwidth, 100 Mbit or 1 Gbit network is recommended.

The Network speeds always need to be aligned with desired message bandwidth from your MQTT clients. The upper limit for bandwidth that the Mosquitto cluster can cope with is governed by a lot of factors, however 30 MB/sec or 240 Mbit is an ultimate upper limit what a Mosquitto node can cover under certain circumstances. Therefore, it is recommended that the network between the nodes is at least a Gbit network.