On premise Installation Guide Pro Mosquitto HA Cluster

Install Docker Environment

To run Cedalo products, you need an installed version of docker and docker compose. Docker is not system dependend and can also be installed on Mac or Linux.

Open the following link and install Docker. If you have not installed a version of docker desktop and only installed the docker engine, make sure to further install Docker Compose.

Figure 1 - Docker Desktop Installer

Make sure your system meets the minimum requirements.

● 64 - bit processor with Second Level Address Translation (SLAT)
● 4GB system RAM
● BIOS-level hardware virtualization support must be enabled in the BIOS settings. The status can be
checked in your Task Manager:

info

If you choose Hyper-V make sure virtualization is enabled. If virtualization is not enabled, you will have to turn it on in your BIOS. If you do not know how: Here is a very good step-by-step tutorial.

After installing make sure docker is running. On a typical installation the Docker daemon is started by a system utility, not manually by a user. This makes it easier to automatically start Docker when the machine reboots. The command to start Docker depends on your operating system. Check the correct page under Install Docker.

During the installation process, Docker requests from you that you set up an account with Docker. You need that account mainly for the setup/install process of Docker, but later you will be provided by Cedalo with a separate account to access the Cedalo registry.

Log into the Cedalo registry

To get your local premium subscription running you need to use the docker login we provided. You will not be able to pull the right images with your personal docker account.

After the docker installation you can use docker login to log into docker. (https://docs.docker.com/engine/reference/commandline/login/)

info

Under normal circumstances, running further docker containers should not influence existing processes. However, this cannot be guaranteed. Please make sure to bring all Docker based applications in a secured state or power them down before logging on with the Cedalo premium subscription account and starting the Mosquitto Pro HA Cluster Setup.

To login open your shell (e.g., cmd.exe under Windows) and enter following command:

docker login registry.cedalo.com -u [username provided by us] -p [password provided by us]

You should receive a message that the login succeeded, otherwise please let us know under getting.started@cedalo.com

info

If you are using Docker Desktop, logging in via the UI is not possible. Please use the command line login.

Start the Pro Mosquitto Setup

Start docker containers

Download the shared setup via the given link we provided you in the e-mail. This folder contains an example mosquitto 3-node cluster setup where each node has an own server (i.e., server1, server2, server3) directory. The server1 directory also contains the configuration for the Management Center. Enter the given license in the license folder in each server directory. The setup expects the file to be named license.lic.

The example configuration may simply be started using the provided startup scripts:

Linux/MacOS:

./start.sh up -d

Windows:

./start.bat up -d

The following containers will be created:

• mosquitto1, mosquitto2, mosquitto3: The three Mosquitto cluster nodes
• haproxy1, haproxy2, haproxy3: Three HAProxys forward MQTT client connections to the leader of the cluster, which is the active MQTT node.
• management-center: The Management Center (MMC) which will connect to the individual Mosquitto nodes and the HAProxys

Docker will create an internal network with IP range 172.20.0.0/16 used for the internal communication between the containers.

Each Mosquitto node will open the port 1885 on the internal network as an admin port for the reachability of the MMC. Please do not make this internal admin port reachable from outside the internal network. Additionally, the active cluster leader will open the Port 1883 in the dockers private network so that the HAProxy can forward request from his public 1883 port.

By default, the test setup will only be reachable on localhost (of the docker host) as the MQTT Port 1883 (form the HAProxy) and the MMC HTTP Port 8088 will only bound to localhost. If you need the setup to be reachable from the outside please adapt the port rules in the docker-compose.yml file accordingly.

Cluster setup in the MMC

The Management Center has six predefined connections:

1. To the three brokers via port 1885
2. To the three HAProxys via port 1883. We have included all three HAProxy connections, even though just one would suffice. This ensures all failover scenarios are covered. In a cluster, changes can only be amended to the cluster leader. The HAProxy connections allow for a direct connection to the leader via the 1883 port (which always chooses the leader). Port 1885 directs to each node and does not redirect.

A cluster is already created and running. Check the cluster state in the Cluster menu. The leader is indicated via a green icon.

This is an example of a cluster running three nodes on a single server.

How to run the setup on more than one server

We recommend recreating this setup on three separate servers. To adapt this example for a real-world implementation, follow these steps:

1. Navigate to the cluster menu in the Mosquitto Management Center and delete the existing cluster. This action will remove all entries from all folders associated with the current cluster.
2. Place the three folders named server1, server2, and server3 on three distinct servers.
3. Ensure that you update the IP addresses in both the docker-compose.yml files and the haproxy.cfg files to align with your local network configuration.
4. Change the connection details of the MMC in the config.json file. (server1/management-center/config)

For more information about the cluster configuration see here.

Create a Cluster

Open the Management Center via: http://localhost:8088/ The root credentials are the following and can be changed in the docker-compose.yml file.

username: cedalo
Password: mmcisawesome

Further users can be configured in the Management Center (User Management Menu).

Before creating a cluster please ensure that each Mosquitto node is connected to the Mosquitto Management Center. You can check this in the Broker Connections menu and establish the connection by clicking the connection switch of the deactivated connections.

A local example Cluster can directly be created without any further configuration. First, make sure all three brokers are connected (Broker Connections Menu). Then create a new cluster in the Cluster Management menu. The configuration is shown in the image below. You can choose between the cluster modes “Full Sync” and “Dynamic Security Sync”.

To add any DynamicSecurity (Clients/Roles/Groups) changes to the HA Cluster in the MMC, make sure to select the leader. To identify the leader, select the cluster by simply clicking on the entry in the “Cluster Management” menu. After that all associated nodes are displayed. The leader’s icon differs from the ones of the remaining nodes (see image below).

For more information about the cluster configuration see here.

Configuration

The configuration of the entire setup is managed through multiple files. Here are the details of each file:

• docker-compose.yml: This file controls various settings related to the docker environment. It manages configurations such as port mappings and environment variables.

Additionally, there are other configuration files:

• mosquitto.conf: This file is located in the \mosquitto\config directory. It contains specific configurations for the Mosquitto component.

• config.json: This file is located in the \management-center\config directory. It contains configurations specific to the Management Center component.

These files play a crucial role in customizing and fine-tuning the settings of the entire system.

Adjusting Docker Settings for Linux Environment

If you're using Linux for your Docker operations, there's one important setting you need to tweak. By default, Docker uses something called userland-proxy, but for smooth client communication, we need to turn this off.

Follow these steps:

1. Locate the Configuration File

Open your file explorer and go to the /etc/docker/ directory. Inside, you'll find a file named daemon.json.

2. Edit the Configuration

Open daemon.json with a text editor (like nano or VSCode) to make changes. If the file doesn't exist, you can create it.

3. Adjust the Settings

Replace the existing content with the following:

{
    "userland-proxy": false,
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m",
        "max-file": "10",
        "labels": "production_status",
        "env": "os,customer"
    }
}

This configuration ensures that the userland-proxy is turned off and sets up some useful logging options.

4. Save and Apply

Save the file and close the text editor.

5. Restart Docker

You'll need to restart Docker to apply these changes. You can do this by opening your terminal and running: sudo systemctl restart docker

Logs

Docker Compose logs provide valuable information about the status and behavior of containers running in a Docker Compose environment.

To access Docker Compose logs for a Mosquitto MQTT Broker service, follow these steps:

1. Open a terminal or command prompt.

2. Navigate to the directory containing your docker-compose.yml file.

3. Run the following command:

   docker-compose logs

   For all logs.

   docker-compose logs <service_name>

   Replace <service_name> with the actual name of your service as defined in your docker-compose.yml file to only access a specific service.

Mosquitto supports multiple log level to provide different levels of details for logs. To change the log level for the Mosquitto, update the mosquitto.conf file. Set the log_type to "all" to get access to the full log. Restart Mosquitto to apply changes.

```conf
log_type all
```