Skip to main content
Version: Mosquitto 2.1

MQTT in a nutshell

The setup consists of two basic concepts:

  • PUBLISH / SUBSCRIBE

  • CLIENT / BROKER

And three basic functionalities:

  • CONNECT

  • PUBLISH

  • SUBSCRIBE

Also, several features help you to configure the messages. For example:

  • QoS (Quality of Service)
  • Retained Messages
  • Persistent Sessions
  • Last Will and Testament (LWT)
  • Keep Alive
  • and many more.

These features will be explained and dealt with subsequently.

Read more about Quality of Service: Here. And here. And here. And here. And here.

Read more about Retained Messages: Here. And here. And here. And here.

Read more about Persistent Sessions: Here. And here. And here.

Read more about Last Will and Testament.

Read more about Keep Alive: Here. And here. And here. And here.

The broker is the central entity.​

There is no alternative for clients (publisher and subscribers) than to connect to a broker when you are using MQTT protocol architecture.

All packets must pass the broker. Each getting checked for specific properties, also the broker filters the package in order to figure out whom (subscribers) to forward the package to.

brokerHandlingSubscriptions

The broker covers most of the workload. So the client device only has to do minimal processing, using minimal bandwidth.

Pro Mosquitto​

Brokers are being offered as so-called "Managed brokers", or "Self-hosted brokers".

The difference is whether a broker is operated/hosted by someone else, or the user is installing, hosting, and looking after the broker on its own server - on its own.

Cedalo is offering professional support to meet the customers expectations.

As usual, a "Managed broker" needs no server setup, as it runs on a cloud service. But as setups differ, a "Managed broker" might need a special server setup/configuration.

Self-hosted brokers require a VPS- or server-sided installation.

MQTT Model​

Broker-client connection​

If you look at the system without any security measurements, it's very simple.

To publish messages you must only know the hostname/IP and port of the broker.

To establish a connection there is always a client-initiated CONNECT packet needed, that is sent by the client to the broker.

CONNECT

The broker in return responds by sending a so-called CONNACK packet (Acknowledge connection request) and a status code.

CONNACK

After the CONNECT and CONNACK packet are exchanged, the connection is enabled.

connect

Connect​

To connect, the client sends a CONNECT MQTT-Packet to the broker.

CONNECT

clientId​

The clientId identifies each MQTT client that is connecting to an MQTT broker.

The clientId can come from the client, but it does not always have to.

If permitted, a clientId can be blank.

Whenever the client does not set a clientId, the broker sets one on behalf of the client.

Still, each client must have a unique clientId.

If a client connects to the broker and another client session already exists with the clientId that the new client has chosen, the old session will be kicked out and taken over.

The permission to be able to do so must be set in the configurations previously.

Read more about Mosquitto Configuration.

In this case, a clientId will be generated automatically by the broker.

For clients that are not persistent, there is no need to generate their clientId because they will not be using it. Unless it is used for authentication.

In that case, the client sends a blank clientId to the broker, and the broker generates a unique id for it whilst the session is open.

Automatically generated clientIds should not be used in combination with persistent sessions!

Read more about Mosquitto Configuration.

Whenever clientIds are used by different applications or users multiple times, each connection will have different conduct. Ensure that you request a new session whenever you connect with deviant operation matters.

cleanSession​

If the cleanSession is set true ( = true), the client does not want to store messages. Also, if set true, previous persistent sessions are dismissed.

Persistent sessions:​

Whenever the Persistent session is active (cleanSession flag=false):

All information a client has not been received from the broker since the last connection, will be saved on the broker, to ensure the client does not miss out on any messages whilst without connection.

The client must request a persistent session when connecting to the broker.

To tell the broker to create a persistent session you must set the clean Sessiom flag to "false" (cleanSession flag = false).

In return, whenever you wish a non-persistent session, you must set the clean Session flag to "true" (cleanSession flag = true).

Persistent sessions store following informations:

  • Whether the session exists. Even though a topic has no subscriptions and therefore the session would be without real use.

  • All subscriptions.

  • All QoS1 or QoS2 messages that haven't been confirmed by the client yet.

Read more about Quality of Service.

  • All new QoS1 or QoS2 messages the client missed whilst being offline.

Read more about Quality of Service.

  • All QoS2 messages the client has not completely acknowledged.

Read more about Quality of Service.

All clients must store a persistent session too, including:

  • Yet not confirmed messages by a broker in a QoS1 or QoS2 flow.

Read more about Quality of Service.

  • Not acknowledged QoS2 messages by the broker.

Read more about Quality of Service.

Whenever a connection between a broker and the client is disconnected, and the session is set as non-persistent, the client can not recreate the subscriptions to all topics it has described to.

A worst case scenario would be when the client can't remember all of its subscribtions.

The subscriber must manually subscribe to all topics again.

Whenever a persistent session is created by the client, and the broker already queued messages of an earlier persistent session, the broker delivers the queued messages using the same old session.

With MQTT 3.1.1 the CONNACK message contains a session present flag, telling the client whether a previously established session is still open.

Persistent or Clean Session - What kind of session do you need:​

 Persistent session:

  • The client must receive all messages.

  • The broker must queue the missed messages.

  • The client must resume all QoS1 and QoS2 messages after reconnecting to the broker.

Read more about Quality of Service: Quality of Service.

Clean Session:

  • The client only needs to publish messages to topics.

  • The client doesn't need to subscribe to topics.

  • The client must not resume with messages it missed.

username/password​

MQTT can forward and handle username and password within a sent packet.

It's important that whenever a packet is not encrypted or hashed (either by implementation or TSL), the password is sent as plain text.

Read more about Security: Security.

lastWill​

The lastWill message is sent by the broker on behalf of the client when a disconnect occurs without a DISCONNECT packet before the actual break off.

The lastWill notifies connected clients when this happens.

The lastWill can be configured by: lastWillTopic

  • The MQTT topic that clients subscribed to, who will receive the lastWill message.

lastWillMessage

  • Simply the payload of the lastWill message.
  • The broker distributes the lastWillMessage when one of these events happens:
  • There is no communication between client and broker within the defined keepAlive period. Read more about Keep Alive: Here. And here. And here. And here.
  • The client breaks off without sending a DISCONNECT packet.
  • The broker shuts down the network connection when an error occurs.

lastWeillRetain

  • Indicates whether the message will be a retained one or not.
  • lastWillRetain = true, the message will be retained. All consumers subscribing to the topic after on will get the retained messages.

lastWillQoS: If the lastWillQoS is set to 1 or 2, the message will be sent with the QoS Level.

  • QoS1: Every subscriber subscribing to the topic with QoS=1 will get the message at least once.
  • QoS2: Every subscriber subscribing to the topic with QoS=2 will get the message exactly once.

Read more about Quality of Service.

keepAlive​

No matter who is not answering or sending the PINGREQ or PINGRESP within the specific time interval plus another half of the very same interval, the broker or client must disconnect. When this happens, the broker sends the lastWill and testament message. If the client has created one.

Disabling the keepAlive feature is simply done by setting keepAlive=0.

As a client is online again whilst a half-open connection is still kept on by the broker, the broker simply performs a client take-over.

Meaning the connection is shut down and establishing a new connection.

Read more about Last Will and Testament.

In terms of data size, the PINGREQ, PINGRESP, and DISCONNECT packets require no variable header, nor a Payload. The packets, therefore, consume only 2 bytes each.

keepAlive contains two functions:

  • Network outage or peer death recognition.
  • Hold on to the connection in case no interaction took action for a specific time.

A maximum length of the time interval is defined for each client request to connect with the broker.

Within the interval, it's ok that the client and broker do not transmit messages.

In fact, after this interval, the client must send a so-called PING Request packet (PINGREQ).

PINGREQ

The broker must respond with a PING Response (PINGRESP).

PINGRESP

If the connection is out of sync, a participant is not responding - called a "half-open connection", keepAlive takes action.

The keepAlive feature makes sure an open connection between broker and client is still running. Additionally the client and broker are aware of the connection. Even though no messages are sent.

keepAlive

Publish / Subscribe model​

PUBLISH

Client-Server Model​

The PUBLISH/SUBSCRIBE model consists of the client-server model.

Any client communicates with an endpoint via the broker. Before a client wants to publish messages it first has to set up a connection with the broker.

Also, a subscribing client has to do the same.

Therefore the broker can decouple publisher and subscriber.

Read more about Decoupling.

Decpupling​

Since the broker decouples publisher and subscriber, the clients do not have to connect directly among themselves.

It follows that the connection count only grows linearly. If all clients had to connect to each other for communication, the connection count would grow exponentially.

All connections are solely with the broker. The broker is thus the central hub in MQTT communication. And with it the associated workload. This allows great flexibility in terms of the number and type of communication participants.

brokerHandlingSubscriptions

Decoupling impacts​

The publish-subscribe model is a decoupling solution of the traditional client/broker model.

Following: There is no need for the publisher and subscriber to run at the same time. Downtimes do not mean that messages are lost. Therefore publisher and subscriber don't exchange IP addresses and ports.

One of the most advantageous aspects MQTT and the pub/sub model offers is to finally be able to break data silos that represent sensitive walls and hurdles in a competitive surrounding more and more.

MQTT allows data interoperability and processing using only minimum bandwidth.

MQTT can be used in any industry 4.0 device system.

As the broker covers all the workload, the client device only has to do minimal processing using minimal bandwidth. Therefore it's also simple to implement within any technical surrounding.

The architecture can be scaled easily without affecting existing client devices. This makes it easy to work and change architecture.

MQTT can process messages event-driven.​

Whilst waiting for a message or publishing it, other tasks are not blocked.

Also, APIs can be used to create even more powerful systems.

In most cases, it's convenient to use the Management Center Cedalo offers. Lots of tasks you might need for your processes can be adjusted and incorporated there.

Read more about Management Center.

Most client libraries work asynchronously, based on callbacks and models similar to callbacks.

The Mosquitto MQTT Broker supports any client library that is supporting MQTT pro.

Here is an Overview.

Central Hub​

Direct contact between subscriber and publisher is never able. Only the broker enables transmitting messages. Therefore the publishing client has only one duty. To make sure that the message is sent to the broker. Everything else is dealt with by the broker.

Also, the subscribing client only has one duty. To subscribe to the topic it wants to receive messages from.

Read more about Publishing.

Read more about Subscribing.

The setup allows no other than the broker to connect the publisher and subscriber. There is no way around this setup. Once the PUBLISHER and SUBSCRIBER are authenticated by the broker. Backed on information the broker sent, the broker ensures the connected devices can send messages to other devices. And the other way round, to receive messages from other devices.

So the broker is the central hub every message must pass. The brokers job is to receive, filter, and forward the message.

The brokers' responsibility is to receive all messages that are sent by the publishing clients.

In the next step, the broker must filter the incoming message. Part of it is to check the configured rules the publishing client has set. Following the broker authorativates who have subscribed to the topic of the message, and finally forwarding the message to the subscribing clients.

broker

MQTT pub/sub model​

Publish​

PUBLISH

The broker simply delivers the message to all subscribers. It's the duty of the broker. There is no obstacle within the broker.

Either one publisher publishes messages to multiple Subscribers. Or multiple publisher publish messages to just one subscriber. Or any combination of multiple publishers/subscribers.

Also, consumers can publish as well. This is called "Bidirectional publishing".

bidirectionalPublishing

Any publisher can only be identified when information about the publisher is included in the topic or payload.

packetId​

Identifies a message.

The packetId is set by the client library and broker.

topicName​

The topic is set using a simple UTF-8 string.

Topics are treated hierarchically. To delimiter, a slash "/" symbolizes a separation. Allowing the client organizational structure, much like a common filesystem.

There is no need for creating a topic. You can subscribe to any topic, even though there might not be messages published. On the other hand, you must know about the topic names to be able to subscribe to them.

QoS​

The QoS (Quality of Service) defines a certain level of service that enables the publisher to make sure that the certainty and quantity of a subscriber receiving a sent message equal the importance to receive the message at all, plus whether to receive the message only once.

Also, a subscriber can set a QoS, defining the importance to receive every message, and whether to accept one message multiple times or not. There are three levels available:

QoS 0 - At Most Once​

QoS 0 is the minimum QoS level.

qos0

The broker/client is delivering the message only once, without confirmation. There is no guarantee that the message sent has been sent to the subscriber. The recipient on the other end does not acknowledge the receipt. Neither is the acknowledgment stored. The QoS0 - level is often referred to be the "fire and forget"-level.

Using the minimum level of quality can make sense, as MQTT requires to be lightweight, bandwidth-saving, and super fast delivery. QoS0 does not require as many resources on the broker.

A subscriber can also use QoS0. qos0+qos2

QoS 1 - At Least Once​

Level 1 QoS guarantees the delivery of a message to the receiver at least once.

qos1

However, "at least once" means that a single message can be transmitted to the receiver more than once. Thus, this QoS level ensures that a message reaches its subscribers but does not care about how many times that might be the case. And the message is stored until the broker receives an acknowledgment.

info

Digression: How the acknowledgment works

A subscriber must send a so-called PUBACK packet (publish acknowledge) to the broker when the broker demands acknowledgment.

If the subscriber does not send a PUBACK, the broker continues sending PUBLISH packets. If the broker sends the PUBLISH packet again, even the second time, it contains a duplicate flag (DUP).

The packetId enables the broker to match the PUBLISH packet and PUBACK packet. And the packetId is free and reusable again.

The broker must store:

  • The information that the session still exists.

  • All subscriptions a client subscribed to.

  • The QoS1 messages waiting to be sent to the client.

  • The QoS1 message, will message and will delay the interval that the client received, but without confirmation.

The publishing client must store all QoS1 messages sent to the broker without confirmation from the broker.

On the other hand, you really should think about the disadvantages as QoS1 occupies resources. Whether to use QoS1 or QoS0 you might want to make up your mind before setting up the broker.

A subscriber can also use QoS1.

qos0+qos1

QoS 2 - Exactly Once​

QoS level 2 guarantees each message is received exactly once.

qos2

To do as pleased the broker transmits the PUBLISH packet as usual. But the recipient must respond with a PUBREC Publish Release). After that happens, the broker sends another request (PUBREL), and the recipient must respond again.

The publishing client must store all QoS2 messages sent to the broker without confirmation from the broker.

The broker must store:

  • The information that the session still exists.

  • All subscriptions a client subscribed to.

  • The QoS2 messages waiting to be sent to the client.

  • The QoS2 message, will message and will delay the interval that the client received, but without confirmation.

To discard all stored states and answers a so-called PUBCOMP packet (Publish complete) is sent. That's the end of the QoS2 order of events.

Until a PUBCOMP is sent, the broker stores a reference of the original PUBLISH packets packetId. The packet and packetId become reusable again.

A subscriber can also use QoS2.

qos2

To define which level of Quality of Service you should use, this may help you:

  • QoS0: The connection between sender and receiver is stable. In addition, it's no problem to miss out on several sent messages if there might be problems with the connection stability. Also, message queueing is not needed.

  • QoS1: The receiver must receive every message sent. Also, your setup can handle the duplicate message. Still, QoS2 is too much overhead for your connection (also QoS1 delivers messages faster than QoS2).

caution

Attention

QoS1 needs way more resources than QoS0. (#).

  • QoS2: Your system must be fed with every message exactly once. Duplicate messages are not wanted or might even manipulate the system. Also, the loss of a message is crucial. Often QoS2 is set whenever the loss of messages may result in loss of life, e.g. clinical surgery, or property, e.g. stock trading. Some industries such as a bank, firefight, aviation, etc require high completeness of data and timeliness. QoS2 bears the possibility of overhead.

retainFlag​

A retained message safes the last known value of the topic. It's only possible to save up to one retained message, equal to one retained value.

info

Digression: Wildcards + retained messages

May the subscribing client include wildcards in the topic pattern they subscribe to, the client receives a retained message even if the topic of the retained message is not an exact match.

Read more about Wildcards.

In many cases, a device can not wait until the next time the publisher provides information sending a message.

note

Example A sensor of a car door tells whether the door is open or closed. As the door is used very infrequently, including a retained flag makes sense.

Without a retained flag set (retainFlag=false), you must wait for the next time the door is opened or closed to find out about the status.

With retained messages (retainFlag=true), you immediately find out, because the retained message gives you the last status. (#).

The retainFlag identifies a message to be a retained message. To identify the message itself the packetId is used.

Payload​

MQTT is not limited by its payload format. It delivers what the publisher is sending. The actual content of a message is the content of a payload.

  • This could be a value, string, a JSON file or even content of a jpg or a mp3 file.

  • Up to 256MB.

  • The payload does not encrypt the content.

Read more about Security.

  • The broker simply delivers what the publisher insert into the payload.

payload

Matter of fact the receiving clients don't even know about the data format they receive. Still, there's little chance that a client subscribes to a host and topic it doesn't know about in the first place. Following the client can forecast the receiving data format.

The subscriber can not reject a message. But as it can do with it as pleased, that could be doing nothing as well. Meaning a subscriber could receive a message, but simply don't do anything with it.

dupFlag​

A dupFlag indicates the duplicate of a message.

A message with dupFlag was resent.

Following a recipient didn't acknowledge the original message.

This could happen when the message contains QoS1 or QoS2.

Read more about Quality of Service.

Subscribe​

To receive messages on topics of interest, the client sends a SUBSCRIBE packet to the broker.

SUBSCRIBE

packetId​

Unique packet identifier.

subscriptions​

A SUBSCRIBE packet does not have to have only one subscription. Multiple can be added.

The limitation on subscriptions a subscriber can subscribe to within one SUBSCRIBE packet is negligible, as it's a high number.

Each subscription consists of a topic and a QoS level. When subscriptions overlap, the broker delivers the message that has the highest QoS level for the specific topic.

caution

Attention

To subscribe to a topic, the subscriber must also set a QoS level of the topic subscription. (#).

The brokers must make sure the subscribing client receives the messages that are permitted to be received by him in terms of security. But also concerning the QoS level that was set by the publishing client.

Read more about Quality of Service.

Subscribing to several topics simultaneously is an option that's possible. A so-called "wildcard" enables this feature.

There are two kinds of wildcards:​

Single level​

singleLevelWildcard

In addition to allowing clients to subscribe to specific topics, mosquitto also allows the use of two wildcards in subscriptions. + is the wildcard used to match a single level of hierarchy.

For example, for a topic of "a/b/c/d", the following example subscriptions will match:

  • a/b/c/d
  • +/b/c/d
  • a/+/c/d
  • a/+/+/d
  • +/+/+/+

The following subscriptions will not match:

  • a/b/c
  • b/+/c/d
  • +/+/+
Multi level​

multiLevelWildcard

A "#" is used, the client subscribes to each topic that's after the #.

There is no regulation for the length or hierarchical depth of the string.

The # must be placed as the last character in the topic and preceded by a forward slash. e.g.: "car/heating/#"

The second wildcard is # and is used to match all subsequent levels of hierarchy. With a topic of "a/b/c/d", the following example subscriptions will match:

  • a/b/c/d
  • a/#
  • a/b/#
  • a/b/c/#
  • +/b/c/#

The $SYS hierarchy does not match a subscription of "#". If you want to observe the entire $SYS hierarchy, subscribe to $SYS/#.

Note that the wildcards must be only ever used on their own, so a subscription of "a/b+/c" is not valid use of a wildcard. The # wildcard must only ever be used as the final character of a subscription.

The wildcard characters (+ and #) can be used in topic filters. But they can not be used in or as topic names. Therefore wildcards are only intended for subscribing to messages and not for publishing messages.

caution

Attention

Topic level separators (/) can be used in both cases! (#).

The broker checks the permission of the subscribing client and forwards the messages the subscriber is allowed to receive. Reversed this can mean that a subscribing client does not get the messages the client is not permitted to receive, even though the subscribing client may have subscribed to the topic using a wildcard.

Read more about Retained Messages: Here. And here. And here. And here.

May the subscribing client include wildcards in the topic pattern they subscribe to, the client receives a retained message even if the topic of the retained message is not an exact match.

Topics starting with $ are not hit by a wildcard according to the standard.

Read more about Broker Status.

The broker must acknowledge each SUBSCRIBE packet. Therefore the broker sends a SUBACK packet (subscribe acknowledge) to the client.

SUBACK

The broker sends one return code for each topic/QoS-pair that it receives in the SUBSCRIBE packet.

The return codes can be:

  • 0 - Success-Maximum QoS0
  • 1 - Success-Maximum QoS1
  • 2 - Success-Maximum QoS2
  • 128 - Failure
  • The Success-Maximum QoS0-2 shows the level of Quality of Service (QoS) that is set and can be supported.

Read more about Quality of Service.

Published messages, but no one subscribing, are discarded by the broker.

The broker filters the incoming messages relating to the topic.

In the next step, the filtered message gets forwarded to the subscribers that have subscribed to the topic.

That’s how the subscriber only receives messages of interest.

brokerHandlingSubscriptions

As there is no direct link between publisher and client, the publisher can never make sure that somebody is actually "listening" to the messages sent.

In MQTT v5, if a client publishes with QoS 1 or 2 and there is no subscriber, then the broker can tell the publisher this in the PUBACK/PUBREC reply.

PUBACK

Read more about Quality of Service.

caution

Attention

MQTT v.3.1.1:

Whenever you disconnect unintendedly and haven't set a cleanSession flag (cleanSession=false) all topic subscriptions are lost. You must resubscribe to every single one again.

In case you have set CleanFlag=false and reconnected with the same clientId, the broker remembers the topics you have subscribed to.

MQTT v.5.0:

The session duration is set by the session expiry interval. (#).

Read more about Clean Session: Here. And here. And here. And here.

Unsubscribe​

A client can always UNSUBSCRIBE from a topic.

UNSUBSCRIBE

packetId​

Uniquely identifies a message.

List of topics​

The UNSUBSCRIBE packet contains also the topics you want to unsubscribe from.

After the packet is sent, the broker unsubscribes the client from the topics.

$ Symbol​

Read more about Broker Status.

$ topics are just "reserved" and shouldn't be used by applications.