How to migrate from OS Mosquitto to Pro Mosquitto
In this guide you will learn how to migrate your current OS Mosquitto configuration in order to be able to fully utilize your new Pro Mosquitto broker without losing data. To simplify terminology we will use the terms OS Broker and Eclipse Mosquitto Broker synonym.
This guide is not about changing your current OS broker configuration such that you can connect it to the Cedalo MQTT Platform. For OS Connectivity please follow this guide.
General migration recommendation
Before you proceed with the following chapters please make sure to setup the Pro Mosquitto broker alongside the OS Mosquitto broker. The main reason is that, in case anything breaks during the migration you still have your old setup up and running. This means that you should simply copy your current configuration into the newly downloaded Pro Mosquitto setup. There are two ways of doing this:
- Deploy Pro Mosquitto broker on a different device (recommended): This way you can simply setup your ne Pro Mosquitto broker such that it can use the default MQTT port(s). Additionally, you do not have any interruption in your service compared to the second aaporach where you would have to shutdown you OS broker to enable default MQTT ports on your Pro Mosquitto broker. Once you are done testing the new setup you can simply switch the broker's address in your clients.
- Deploy Pro Mosquitto broker on the same device: This way you have to manage port as your OS Broker is already bound to the default MQTT port(s). Also, having two brokers running with production load (on the OS Broker) and test load (on Pro Mosquitto broker) may exceeds the capacity of your server.
But, before copying anything into the Pro Mosquitto setup you may need to migrate some configurations as per our recommendations in the following chapters.
mosquitto.conf
Good news first: The OS Mosquitto configuration options are fully compatible with the Pro Mosquitto broker. However, we recommend some options over others. For example:
- Prefer usage of Dynamic Security plugin over the combination of acl_file and password_file options
- Prefer usage of Persist SQLite plugin over usage of Snapshot persistence
In short, you can migrate your mosquitto.conf by simply copying the file into your new Pro Mosquitto broker setup. In case you want to migrate configurations for at least one of the suggested options in the above list, you can also go ahead and first run the migration scripts. These are also capable of migrating your mosquitto.conf.
Dynamic Security plugin
Having proper authentication and authorization enabled on a data service is mandatory in today's IIoT world. The Dynamic Security plugin of the Pro Mosquitto broker implements mechanisms for both auth tasks by providing Role Based Access Control (RBAC). Moreover, configuring it during runtime is also possible via its MQTT Control API. Last but not least, it is ready to be run in clustered environments without any further need for extra configuration.
Requirements
The Dynamic Security plugin must be enabled for the Cedalo MQTT Platform, which offers a straightforward way of configuring user authentication and authorization via a neat web UI.
Migration script
In case you have configured authentication and authorization via the acl_file and password_file options in the mosquitto.conf we highly recommend switching to Dynamic Security plugin as described previously. Setting up a dynamic-security.json properly from scratch can be as time-consuming as it is error-prone. Therefore, we created a migration script that helps you migrating your ACL file and password file into a single dynamic-security.json file. Optionally you can also directly edit the mosquitto.conf options using the script. It can be found in the OS Mosquitto repository. Having a recent python version installed on your machine is the only requirement to execute the script.
Please make sure to run the script in your current setup. Also you do not have to worry about backing up your current configuration files as the script will not delete anything on your disk. In case of mosquitto.conf migration the old file will simply be renamed after migration, resulting in the migrated version being named mosquitto.conf and your previous one named mosquitto.conf.old.dynsec.
The following is a complete list of the usage options:
- Print usage help:
python3 migrate_to_dynsec.py -h
- Migrate (without mosquitto.conf):
python3 migrate_to_dynsec.py --acl-file path/to/acl_file --pw-file path/to/pw_file
- Migrate (including mosquitto.conf):
python3 migrate_to_dynsec.py --acl-file path/to/acl_file --pw-file path/to/pw_file --conf path/to/mosquitto.conf --dynsec-lib path/to/mosquitto_dynamic_security.so/.dll
For each of our Pro Mosquitto deployment options, except for Windows and RPM packages, the Dynamic Security library (and other libraries) will be placed in the /usr/lib directory. For the Dynamic Security plugin this means the path you have to set for the --dynsec-lib command line argument is: /usr/lib/mosquitto_dynamic_security.so.
Merge setups
As a final step you need to merge the dynamic-security.json files of your setup and the new Pro Mosquitto setup. Please perform the following steps depending on your current configuration:
Scenario 1: No authentication and authorization mechanism is in place
As everything is setup by default, you can move on to the next section.
Setting the allow_anonymous option to true should be wisely considered. Without applying any authentication methods this can lead to uncontrolled data access by unauthenticated and/or unauthorized entities. It is strongly recommended to at least enable the Dynamic Security plugin for basic username/password authentication.
Scenario 2: ACL file and password file
If not already done, please run the migration script to retrieve a dynamic-security.json file that reflects your current settings of your ACL file and password file. Afterwards simply run the steps from Scenario 3.
Scenario 3: Dynamic Security plugin
For this scenario the only requirement for you is to copy the contents of the fields of your (migrated) dynamic-security.json into the one in the Pro Mosquitto broker setup. Also make sure that you do not override the admin client (see note below) and assert that there are no duplicates between your and the default dynamic-security.json that comes along with the Pro Mosquitto broker setup.
The admin client is used by the Cedalo MQTT Platform to perform MQTT Control requests to, e.g., create a Dynamic Security client, or to subscribe to $SYS/# topics to display statistics on its dashboard. Such operations require certain authorizations to publish to or subscribe on various topics. In the following possible scenarios are evaluated, which each require different handling in terms of the admin client.
A default version of the Dynamic Security plugin's configuration is contained in the pre-configured Pro Mosquitto setup. It includes the admin client as well, whose credentials are stored in the mosquitto_client_info.txt.
Persist SQLite plugin
Enabling persistence on a Mosquitto broker improves reliability by providing features like persistent sessions, which lets clients receive messages sent while they were offline, once they reconnect. Currently, the OS Broker Snapshot persistence and the Persist SQLite plugin exist. We strongly recommend to use the Persist SQLite plugin over Snapshot persistence. This is mainly due to the fact that the Snapshot persistence writes bulks per dump. At first this might sound sensible, but comes at the cost of possible several seconds of service interruption for largely scaled setups or brokers which are on higher load, which is likely not acceptable. The Persist SQLite plugin solves this issue by providing a solution that writes data continuosly, mitigating time consumption issues due to disk write operations. Moreover, in scenarios where data loss and downtimes of clients cannot be prevented (e.g., moving assets like a car fleet) Persistent Queueing increases the mechanism of message queueing in the broker. This feature can only be used with the Persist SQLite plugin.
Requirements
For other premium features like Persistent Queueing it is important to have a performant and reliable persistence plugin activated. The Persist SQLite plugin fulfills those needs and should therefore be enabled on the Pro Mosquitto broker setup instead of Snapshot persistence.
Migration script
In case you set the persistence option to true in your current setup we suggest to migrate to the Persist SQLite plugin. The migration script dumps out your current mosquitto.db (or whatever you specified as persist_file) into a JSON string and creates the mosquitto.sqlite3 database file using it, as if it has been using the Persist SQLite plugin all the time. Furthermore, it allows for automated migration of the mosquitto.conf file as well. In case you built the OS Mosquitto project yourself or the mosquitto_dump_tool is not part of your path for another reason you can also specify the path to that executable.
Please make sure to run the script in your current setup. Also you do not have to worry about backing up your current configuration files as the script will not delete anything on your disk. In case of mosquitto.conf migration the old file will simply be renamed after migration, resulting in the migrated version being named mosquitto.conf and your previous one named mosquitto.conf.old.persistence.
The following is a list of the usage options:
- Print usage help:
python3 migrate_to_persist_sqlite.py -h
- Migrate (without mosquitto.conf):
python3 migrate_to_persist_sqlite.py --persistence-db path/to/persistence_db
- Migrate (including mosquitto.conf):
python3 migrate_to_persist_sqlite.py --persistence-db path/to/persistence_db --conf path/to/mosquitto.conf --persist-sqlite-lib path/to/mosquitto_persist_sqlite.so/.dll
- Migrate (
mosquitto_dump_toolnot in PATH):
python3 migrate_to_persist_sqlite.py --persistence-db path/to/persistence_db --dump-tool path/to/mosquitto_dump_tool
Merge setups
As a final step you need to make sure that the Pro Mosquitto setup is using the correct SQLite3 database. Please perform the following steps depending on your current configuration:
Scenario 1: No persistence
The Persist SQLite plugin is activated by default in our Pro Mosquitto setups, hence, you can move on to the next section.
Scenario 2: Snapshot persistence
If not already done, please run the migration script to retrieve the mosquitto.sqlite3 database file. Afterwards simply run the steps from Scenario 3.
Scenario 3: Persist SQLite
In this case you simply have to copy the mosquitto.sqlite3 database file into the persistence_location path that is defined in the mosquitto.conf of your new Pro Mosquitto setup. If you assigned a custom name to the database, please copy the plugin_opt_db_file option as well.