Skip to main content
Version: Mosquitto 2.5

Broker Introduction

Basics

Intro

MQTT is more and more becoming the standard messaging protocol for IoT messaging.

Since 2014 MQTT is an OASIS standard messaging protocol.

It was developed by IBM in 1999.

MQTT is very lightweight as all workload is done by the broker, while connected clients run on small hardware requirements.

Further, MQTT clients can be easily implemented in a large variety of use cases, as there are plenty of client code libraries.

note

Here's an example: Some Surroundings manufacturing items sensitive to temperature and humidity must make sure that temperature and humidity is stable throughout the process in the whole surrounding.(#).

To monitor temperature and humidity sensors are installed. In recurring intervals each sensor is transmitting information.

Creating workload.

The data is sent via MQTT protocol. The MQTT broker receives the data, filters the data-packet, and forwards it to the receiving clients.

In this scenario the receiving clients could be the radiator or humidifier.

This is possible because of the efficient protocol format, and because a broker is used to decouple the sender and receiver.

That way the broker bears most of the workload. And also the client-side can be easily embedded in a system as there is only a little code needed. Following there are plenty of MQTT client libraries for most programming languages.

bidirectionalPublishing

In fact a MQTT system enables receiving clients to become publishers as well.

A humidifier could transmit information to the broker. And the broker would forward the message to the sensors.

The next big strength is when it comes to sending messages via unstable networks.

As MQTT messages can be very small depending on the payload size, they can be sent easily even by small devices.

The MQTT messaging protocol is even suitable for special needs, like devices that are far off any other network.

A example for such a purpose can be a far off device that needs to be adjusted. A MQTT message can send the trigger to the device to adjust it.

MQTT can send messages with different so called "quality of service" levels. This means the publisher and subscriber can choose how reliably a message will be sent.

But a receiver will never be able to receive a message with a higher QoS level than the publisher has set when publishing the message.

qos

Quality of Service (QoS) level 0 means the message is sent without any confirmation from the receiver. This means it is technically possible for the message to be lost, given an unreliable connection.

QoS level 1 means that the receiver must send a confirmation, so that the sender knows that the message was received. However, with it is possible that the receiver gets a single message multiple times. This QoS level ensures that a message makes it from sender to receiver.

QoS level 2 uses a four step communication process to ensure a message is sent exactly once only, which can be important depending on the use case. For instance,

For a publisher the broker is the receiver. When the broker, then, forwards the message, the subscriber is the receiver.

Besides QoS there are several other features to help you to further configure your setup to your specific needs.

For example: Retained Messages, Persistent Sessions, Last Will and Testament (LWT), Keep Alive, and many more.

These features will be explained and dealt with subsequently.

MQTT Basis

MQTT = Message Queuing Telemetry Transport

MQTT is a lightweight PUBLISH/SUBSCRIBE messaging protocol.

MQTT is very advantageous when it comes to low bandwidth environments, as MQTT is designed especially for IoT - the Internet of things, or machine-to-machine messaging with low transmission capacity.

Comparing HTTP and MQTT it shows, e.g., that MQTT delivers messages many times faster than the HTTP protocol.

A typical MQTT setup includes one broker and as many clients as you want.

MQTT principle:

A Publisher (client) sends a message (including data) to a broker and there onto a topic. Other clients subscribe to this topic to receive the messages.

The broker filters the messages and checks whether the subscribers have the necessary rights. The broker then forwards the messages. Several publishers can send messages to one and the same topic. Multiple subscribers can subscribe to the same topic.

Important to know: A client can at the same time take up the role of a publisher as well as a subscriber. This is called bi-directional communication.

mqttProtocol

Since the broker decouples publisher and subscriber, the clients do not have to connect directly among themselves.

As a result, with an increasing number of clients the connection count only grows linearly. If all clients had to connect to each other for communication, the connection count would grow exponentially.

Further, all connections are solely with the broker. The broker is, thus, the central hub in MQTT communication. Following the associated workload is being processed by the broker as well. This allows great flexibility in terms of the number and type of communication participants/clients.

The load balancer is the main character that moves messages/packets within the broker.

The MQTT setup consists of two basic concepts:

  • PUBLISH / SUBSCRIBE
  • CLIENT / BROKER

And three basic functionalities:

  • CONNECT
  • PUBLISH
  • SUBSCRIBE

Also, several features help you to configure the messages. For example: QoS (Quality of Service), Retained Messages, Persistent Sessions, Last Will and Testament (LWT), Keep Alive, and many more. These features will be explained and dealt with subsequently.

info

Digression: Message vs. packet

The delivered subject is ether called "message" or "packet" by all participants working with MQTT.

Both - messages and packets - are the same and the terms can be used interchangeably.

There is no definition of when or why a delivery is called a packet or message.

We at Cedalo are using the term "packet" for any of the commands, and "message" for publish or subscribe messages.(#).

There is no alternative for clients (publisher and subscribers) to connect to MQTT than to use a broker. The broker is the central entity.

All packets go through the broker, get checked for specific properties, and are, then, sent by the broker to the subscribers.

The broker covers all the workload. Hence, the client device only has to do minimal processing using minimal bandwidth.

All MQTT versions are contributed by the OASIS Foundation. The recent version of the MQTT protocol is v.5 that has been presented in January 2018. Most brokers support MQTT v.5 protocol. Even though most client providers (client libraries) don't support v.5 yet.

Message expiry

Whenever the payload of a message has limited validity, e.g. real-time status messages, there is no need for the messages until a subscribing client is connected again.

The Message expiry allows the publisher to set a maximum time the message is valid. After that, the broker doesn't send the message to the subscribing clients anymore and the message gets discarded.

Session expiry

MQTT v3.1.1 does not stipulate when any persistent session expires. Meaning any persistent session, if not adjusted by the client, runs endlessly. Following the number of persistent sessions still running, but not in use, occupy the resource of the server more and more. To deal with that problem the system architectures usually did not follow as MQTT v3.1.1 protocol intended and configured expiration times for the use of persistent sessions.

In the MQTT 5.0, the field Clean Session is divided into field Clean Start and Session Expiry Interval.

The field Clean Start specifies whether a new session is needed. The field Session Expiry Interval specifies the session expiration time. They are specified when connecting.

The retainFlag can easily be implemented whenever the session disconnects ungracefully, as the Session Expiry Interval can be updated when an ungraceful disconnect happens. The session terminates when the client disconnects using a DISCONNECT packet.

Broker reference/broker redirect

Broker redirect tells clients where they should reconnect to.

This is very useful as manual reconnecting is a lot of work.

Metadata attachment

Metadata is enabled to be attached to the MQTT packet.

Extensions

MQTT can be mingled with many more extensions.

Client and server restrictions

Enables to set a maximum size of each message (bytes able to be transmitted).

Also, a maximum number of messages to be sent or received simultaneously by a client or broker.

Negative acknowledgments

Negative acknowledgments notify the client whenever an error occurs and what the error is about.

In MQTT v3.1.1 an error may lead to a disconnection of the client from the broker - in this case, the client has to subscribe to the topics again. Now with v.5, a client can react before it gets disconnected (best case scenario).

negativeAcknowledgement

Will delay interval

The will delay interval can be used by all clients. A so called Last Will Testament (LWT) is sent when the network connection closes without the client sending a DISCONNECT packet. The will delay interval provides a time window in which the client can reconnect, and if it does then the will message will not be sent. Whether the client is persistent or not has no bearing.

Shared subscriptions

Standard MQTT subscribers receive a duplicate of the messages that the subscriber has subscribed to. Each message load has to be beared by the subscriber itself.

In contrast shared subscription message workloads are distributed across all subscribers.

Subscription identifier

The subscription identifier helps the broker to work more efficient. That is when a message is published to the broker and the broker filters for the receivers of the message. Because the subscription identifier refers directly to the corresponding receivers. The broker does not need to match topics.

There are plenty of MQTT client libraries for most programming languages.

Here is an overview: Paho

MQTT further information

MQTT is very strong when you must transfer data and the de-facto standard in the IoT world.

In an MQT environment, the broker is the central entity and handles most of the workload. Therefore, client devices have to do minimal processing and use only minimal bandwidth. Also, MQTT is very easy to implement on the client-side.

That adds to why MQTT is very lightweight and performs even if used in unstable networks. It's a perfect fit for constrained devices with limited resources.

More information concerning the performance and transfer guarantee of messages (QoS) will follow within this learning.

The Mosquitto broker is the most efficient broker offered worldwide.

When considering how much a MQTT Broker is used and the work it has to perform, it is not so much about the number of clients that you should focus on but rather the amount of data sent per second, the "traffic". Suppose you have 1.5 million clients, but each client is sending only little data and, additionally, that even quite rarely, e.g. only every couple of minutes. Then, this leads to much less traffic and workload on the broker than merely a few thousand clients sending big data messages at high frequency.

A different scenario that can incur a lot of traffic and workload for the broker is, if you want to transfer pictures every millisecond using QoS level 2 (QoS2) - a Quality of Service level occupying the resources of a broker way more than other Quality of Service levels.

Additionally, the circumstances of the setup influence the performance. Thinking of remote devices, a very good latency is certainly harder to achieve than when the devices would be close together.

High Availability (HA) is the ability of a system to cope with the loss of a central component. This is typically achieved through redundancies. HA ensures that even if the (main) broker is lost, e.g. because of a hardware failure, the MQTT communication continuous to work properly as another broker seamlessly takes over. Following HA enables a system to work at optimal performance continuously for a long period. This means that the systems should work without failure. The MQTT Mosquitto broker is designed to be stable and consistently functional by clustering. Clustering provides prevention of disadvantages whenever a broker might break down, e.g. hardware failure. Therefore usually three or more brokers work together in a cluster. In the case of a breakdown of broker number one, the so-called "load balancer" shifts the workload to a passive broker number two. To be absolutely safe, there is another passive broker clustered.

In a clustered broker environment the load balancer detects which broker is currently available and send client connections to that broker.

The Payload each message carries must not be encoded nor decoded by the broker. The broker simply receives messages from the publisher, filters the messages, and delivers them to the subscribing clients. Following the workload is very little.

Overall it's possible to send up to 256MB in each Payload.

The Payload each message carries must not be encoded nor decoded by the broker. The broker simply receives messages from the publisher, filters the messages, and delivers them to the subscribing clients. Following the workload is very little.

Overall it's possible to send up to 256MB in each Payload.

broker

Self-hosted brokers require a VPS- or server-sided installation.

Not a Message Queue

By definition, MQTT is not a message queue -

caution

Attention It's possible to have message queues per client in MQTT. Even though a few characteristics seem to be the same, particular the publish-subscribe model ("pub-sub")(#).

MQTT deals with very different tasks. Most of them relate to the world of IoT. Enabling multiple device access, managing devices, and especially messaging is the center of the multiple usages of MQTT.

In contrast, message queues store messages, as the name tells, and forward these messages between server applications. The storage volume is often very large, but message queues only offer little access capacity. MQTT usually does not store data, but is able to forward messages to plenty receivers.

Security

Security measures are very important. The broker must be set up correct and safe. But also the clients (publisher and receiver) must set security measures.

For example, a MQTT client is located after a router, using the NAT (Network Address Translation) to transmit from a private network address to a public address.

NATcommunication

Clients can subscribe to any topic they want, when a broker is without enabled security.

In some cases, no level of security might be without risk. But it's always the end user that has to make up his mind about what level of security has to be chosen and how to implement it.

The use of ACLs (Access Control Lists) allows restriction of subscriptions and publishing of clients.

The information within a payload never gets restricted. But the permissions to publish and receive.

Usually, the broker supports common security measurements, e.g. TLS. Still, the end user has to make sure that the chosen measurements fit the security environment the broker itself is part of.

The default secured MQTT broker port is 8883.

The standard unsecure port is 1883.

tcpTLS

TCP is embedding TLS (Transport Layer Security), the successor of SSL (Secure Sockets Layer). Allowing MQTT packets to be transmitted via encrypted pipes.

Cryptology ensures privacy and integrity. Also authenticity with the use of certificates.

Implementing TLS and maintaining the structure, requires in-depth knowledge support.

Many more precautions can be set when using the SASL Framework (Simple authentication and security layer) for example. SASL provides authentication options, data integrity-checking, and encryption.

Again, MQTT only supports the implementation. Security must be set on the client-side.

The highest security levels can be achieved using client certificates (x509).

Again, implementing TLS and maintaining the structure, requires in-depth knowledge support.

TLS protects all parts of an MQTT packet, not only the payload.

Encrypting just the payload is also able. But again, encrypting a payload is done at the application level, not the broker. Following encrypted Payloads can be sent without broker configuration needed. The broker just delivers packets. Subscribing clients on the other end must be able to decrypt the Payload.

The TCP security and any kind of packet encryption are recommended to be dealt with by the Load Balancer, before handing off the incoming packet to the broker.

The broker can identify the client on three different pieces of information given. The clientId, a username, and password, or a certificate of the client.

Still, each client must have a unique clientId. The clientId can be set by the client, but it does not always have to. If a client connects to the broker and another client session already exists with the same clientId, the old session will be kicked out and taken over. The precise proceedings depend on the "clean start" value also provided by the client.

For clients that are not persistent, there is no need to generate their clientId because they will not be using it unless it is used for authentication. In that case, the client sends a blank clientId to the broker, so the broker generates a unique id for it whilst the session is open.

In MQTT v5 the broker can inform the client of the id that was chosen.

The clientId is useful for a few things. At first, for rejoining sessions that a client has been disconnected from earlier. The client can ask the broker to preserve its session when it disconnects, which includes the subscription information and possibly queued messages for it - this is controlled by the "clean session" flag in MQTT v3.1.1, and by the "session expiry interval" in MQTT v5.0.

Secondly, you may choose to use the clientId for authentication. You typically use either the username on its own for authentication, which does not need to be unique per client, or the username and clientId together. You may also use the clientId for controlling access to topics, but that's unusual.

Model

Publish / Subscribe

The PUBLISH/SUBSCRIBE model consists of the client-server model.

Any client communicates with an endpoint via the broker. Before a client wants to publish messages it first has to set up a connection with the broker.

Also, a subscribing client has to do the same.

Therefore the broker can decouple publisher and subscriber.

connect

The delivered subject is either called "message" or "packet" by all participants working with MQTT.

Both - messages and packets - are the same.

There is no definition of when a delivery is called a packet or message. Cedalo is using the term "packet" for any of the commands, and "message" for publishing or subscribing messages.

Direct contact between subscriber and publisher is never possible. Only the broker enables transmitting messages. Therefore, the publishing client has only one duty. To make sure that the message is sent to the broker. Everything else is dealt with by the broker.

Also, the subscribing client only has one duty. To subscribe to the topic it wants to receive messages from.

In fact, because of the decoupling of the publishing client and subscribing client, they have no chance to find out more about the other one. The subscriber only knows about the existence of the publisher as the client subscribes to the published topics.

Every exchange of messages is done by the broker.

The setup allows no other than the broker to connect the publisher and subscriber. There is no way around this setup but still using MQTT, no gap.

Once the PUBLISHER and SUBSCRIBER are authenticated by the broker. Backed on information the broker sent, the broker ensures the connected devices can send messages to other devices. And the other way round, to receive messages from other devices.

broker

The broker is the central hub every message sent must pass. The brokers job is to receive, filter, and forward the message. The so-called load balancer moves the message from process stage to the next.

The brokers' responsibility is to receive all messages that are sent by the publishing clients. In the next step, the broker must filter the incoming message. Part of it is to check the configured rules the publishing client has set. Following the broker authorativates who have subscribed to the topic of the message, and finally forwarding the message to the subscribing clients.

Whenever a subscribing client requests storage of not-sent messages, the broker does as pleased within the possibility the MQTT protocol sets.

Decoupling impact

The publish-subscribe model is a decoupling solution of the traditional client/broker model.

Therefore, publisher and subscriber don't exchange IP addresses and ports.

Following:

There is no need for the publisher and subscriber to run at the same time. Downtimes do not mean that messages are lost.

Even whilst publishing or receiving messages/packets the operations of the publisher and subscriber run fluent.

Any established connection is kept open by the broker until the client sends a DISCONNECT command or the connection breaks up.

One of the most advantageous aspects MQTT and the pub/sub model offers is to finally be able to break data silos that represent sensitive walls and hurdles in a competitive surrounding more and more.

MQTT allows data interoperability and processing using only minimum bandwidth.

MQTT can be used in any industry 4.0 device system.

As the broker covers all the workload, the client device only has to do minimal processing using minimal bandwidth. Therefore, it's also simple to implement within any technical surrounding.

The architecture can be scaled easily without affecting existing client devices. This makes it easy to work and change architecture.

To deal safely with a lot of connections use clustered broker nodes.

Clustering brokers is also important to make sure that connectivity is always given.

Of course, MQTT can process messages event-driven.

Most client libraries work asynchronously, based on callbacks and models similar to callbacks.

Whilst waiting for a message or publishing it, other tasks are not blocked.

Also, APIs can be used to create even more powerful systems.

In most cases, it's convenient to use the Management Center Cedalo offers. Lots of tasks you might need for your processes can be adjusted and incorporated there.

Broker-client connection

If you look at the system without any security measurements, it's very simple. To publish messages you must only know the hostname/IP and port of the broker.

To receive messages you must know the hostname/IP and port of the broker and the topic you want to subscribe to.

To establish a connection there is always a client-initiated CONNECT packet needed, that is sent by the client to the broker.

CONNECT

The broker in return responds by sending a so-called CONNACK packet (Acknowledge connection request) and a status code.

CONNACK

After the CONNECT and CONNACK packet are exchanged, the connection is enabled.

connect

To connect, the client sends a CONNECT MQTT-Packet to the broker. clientID, etc. are encoded as simple UTF-8 strings.

Note, that the packets are not secured by the broker. It's the responsibility of the clients to engage in security measurements.

clientId:

The clientId identifies each MQTT client that is connecting to an MQTT broker.

If permitted, a clientId can be blank. Still, each client must have a unique clientId. The clientId can come from the client, but it does not always have to. If a client connects to the broker and another client session already exists with the clientId that the new client has chosen, the old session will be kicked out and taken over. The permission to be able to do so must be set in the configurations previously. In this case, a clientId will be generated automatically by the broker.

For clients that are not persistent, there is no need to generate their clientId because they will not be using it unless it is used for authentication. In that case, the client sends a blank clientId to the broker, and the broker generates a unique id for it whilst the session is open. Automatically generated clientIds should not be used in combination with persistent sessions!

Whenever clientIds are used by different applications or users multiple times, each connection will have different conduct. Ensure that you request a new session whenever you connect with deviant operation matters.

cleanSession:

If cleanSession=true, the client does not want to store messages.

Also, previous persistent sessions are dismissed.

username/password:

MQTT can forward and handle username and password within a sent packet.

It's important that whenever a packet is not encrypted or hashed (either by implementation or TSL), the password is sent as plain text.

You can read about security matters in the Security chapter.

lastWill:

The lastWill message is sent by the broker on behalf of the client when a disconnect occurs without a DISCONNECT packet before the actual break off.

The lastWill notifies connected clients when this happens.

The lastWill can be configured by:

  • lastWillTopic: The MQTT topic that clients subscribed to, who will receive the lastWill message.
  • lastWillQoS: If the lastWillQoS is set to 1 or 2, the message will be sent with the QoS Level.

lastWill levels:

  • QoS1: Every subscriber subscribing to the topic with QoS=1 will get the message at least once.

  • QoS2: Every subscriber subscribing to the topic with QoS=2 will get the message exactly once.

  • lastWillMessage: Simply the payload of the lastWill message.

  • lastWillRetain: Indicates whether the message will be a retained one or not. lastWillRetain = true, the message will be retained. All consumers subscribing to the topic after on will get the retained messages.

The broker distributes the lastWillMessage when one of these events happens:

  • The broker detects an I/O error or network failure.
  • There is no communication between client and broker within the defined keepAlive period.
  • The client breaks off without sending a DISCONNECT packet.
  • The broker shuts down the network connection when an error occurs.

keepAlive:

caution

Attention The keepAlive feature is not the same as the TCP keepAlive, which is running on the TCP level.(#).

keepAlive contains two functions:

  • Network outage or peer death recognition.
  • Hold on to the connection in case no interaction took action for a specific time.

A maximum length of the time interval is defined for each client request to connect with the broker. Within the interval, it's ok that the client and broker do not transmit messages. In fact, after this interval, the client must send a so-called PING Request packet (PINGREQ).

PINGREQ

The broker must respond with a PING Response (PINGRESP).

PINGRESP

If the connection is out of sync, a participant is not responding - called a "half-open connection", keepAlive takes action. The keepAlive feature makes sure an open connection between broker and client is still running. Additionally the client and broker are aware of the connection. Even though no messages are sent.

keepAlive

No matter who is not answering or sending the PINGREQ or PINGRESP within the specific time interval plus another half of the very same interval, the broker or client must disconnect.

When this happens, the broker sends the lastWill and testament message. If the client has created one.

Disabling the keepAlive feature is simply done by setting keepAlive=0.

As a client is online again whilst a half-open connection is still kept on by the broker, the broker simply performs a client take-over.

Meaning the connection is shut down and establishing a new connection.

In terms of data size, the PINGREQ, PINGRESP, and DISCONNECT packets require no variable header, nor a Payload. The packets, therefore, consume only 2 bytes each.

keepAliveDisconnect

pub/sub

Publish

The broker simply delivers the message to all subscribers. It's the duty of the broker. There is no obstacle within the broker.

Either one publisher publishes messages to multiple Subscribers.

multipleSUBSCRIBERonePUBLISHER

Or multiple publisher publish messages to just one subscriber.

multiplePUBLISHERoneSUBSCRIBER

Or any combination of multiple publishers/subscribers.

Also, consumers can publish as well. This is called "Bidirectional publishing".

bidirectionalPublishing

Any publisher can only be identified when information about the publisher is included in the topic or payload.

PUBLISH

packetId:

Identifies a message.

The packetId is set by the client library and broker.

topicName:

The topic is set using a simple string.

Topics are treated hierarchically. To delimiter, a slash "/" symbolizes a separation. Allowing the client organizational structure, much like a common filesystem.

There is no need for creating a topic. You can subscribe to any topic, even though there might not be messages published. On the other hand, you must know about the topic names to be able to subscribe to them.

The topic is essentially a UTF-8 string. Filtering the topic of a sent message can be done by the broker by simply comparing the encoded UTF-8 bytes or decoded Unicode characters.

Be aware that an extra slash or slashes set by mistake are detected by the broker as a delimiter.

Quality of Service

QoS: The QoS (Quality of Service) enables the publisher to set a certain level of certainty that the message will reach the subscriber, and additional that the message may only reach the receiver once. This is an offer the subscriber can accept, but also can a subscriber choose a lower level of QoS. But not a higher level than the publisher has set.

There are three levels available:

QoS0 - at most once

QoS0 is the minimum QoS level. The broker/client is delivering the message only once, without confirmation. There is no guarantee that the message sent has been sent to the subscriber. The recipient on the other end does not acknowledge the receipt. Neither is the acknowledgment stored. The QoS0 - level is often referred to be the "fire and forget"-level.

Using the minimum level of quality can make sense, as MQTT requires to be lightweight, bandwidth-saving, and super fast delivery. QoS0 does not require as many resources on the broker.

qos0

A subscriber can also use QoS0.

qos0+qos0

QoS1 - At least once

Level 1 QoS guarantees the delivery of a message to the receiver at least once.

However, "at least once" means that a single message can be transmitted to the receiver more than once. Thus, this QoS level ensures that a message reaches its subscribers but does not care about how many times that might be the case. And the message is stored until the broker receives an acknowledgment.

On the other hand, you really should think about the disadvantages as QoS1 occupies resources. Whether to use QoS1 or QoS0 you might want to make up your mind before setting up the broker.

info

Digression: How the acknowledgment works

A subscriber must send a so-called PUBACK packet (publish acknowledge) to the broker when the broker demands acknowledgment.

If the subscriber does not send a PUBACK, the broker continues sending PUBLISH packets. If the broker sends the PUBLISH packet again, even the second time, it contains a duplicate flag (DUP).

The packetId enables the broker to match the PUBLISH packet and PUBACK packet. And the packetId is free and reusable again.

PUBACK

The publishing client must store all QoS1 messages sent to the broker without confirmation from the broker.

The broker must store:

  • The information that the session still exists.
  • All subscriptions a client subscribed to.
  • The QoS1 messages waiting to be sent to the client.
  • The QoS1 message, will message and will delay the interval that the client received, but without confirmation.

qos1

A subscriber can also use QoS1.

qos0+qos1

QoS2 - Exactly once

QoS level 2 guarantees each message is received exactly once.

To do as pleased the broker transmits the PUBLISH packet as usual. But the recipient must respond with a PUBREC Publish Release).

PUBREC

After that happens, the broker sends another request (PUBREL), and the recipient must respond again.

PUBREL

The publishing client must store all QoS2 messages sent to the broker without confirmation from the broker.

The broker must store:

  • The information that the session still exists.
  • All subscriptions a client subscribed to.
  • The QoS2 messages waiting to be sent to the client.
  • The QoS2 message, will message and will delay the interval that the client received, but without confirmation.

To discard all stored states and answers a so-called PUBCOMP packet (Publish complete) is sent. That's the end of the QoS2 order of events.

PUBCOMP

Until a PUBCOMP is sent, the broker stores a reference of the original PUBLISH packets packetId.

The packet and packetId become reusable again.

qos2

A subscriber can also use QoS2.

qos0+qos2

To define which level of Quality of Service you should use, this may help you:

  • QoS0:

The connection between sender and receiver is stable. In addition, it's no problem to miss out on several sent messages if there might be problems with the connection stability. Also, message queueing is not needed.

  • QoS1:

The receiver must receive every message sent. Also, your setup can handle the duplicate message.

Still, QoS2 is too much overhead for your connection (also QoS1 delivers messages faster than QoS2).

caution

Attention QoS1 needs way more resources than QoS0.(#).

  • QoS2:

Your system must be fed with every message exactly once. Duplicate messages are not wanted or might even manipulate the system.

Also, the loss of a message is crucial. Often QoS2 is set whenever the loss of messages may result in loss of life, e.g. clinical surgery, or property, e.g. stock trading.

Some industries such as a bank, firefight, aviation, etc require high completeness of data and timeliness.

QoS2 bears the possibility of overhead.

PUBLISH

retainFlag:

In many cases, a device can not wait until the next time the publisher provides information sending a message.

A sensor of a car door tells whether the door is open or closed. As the door is used very infrequently, including a retained flag makes sense.

Without a retained flag set (retainFlag=false), you must wait for the next time the door is opened or closed to find out about the status.

With retained messages (retainFlag=true), you immediately find out, because the retained message gives you the last status.

A retained message safes the last known value of the topic. It's only possible to save up to one retained message, equal to one retained value.

The retainFlag identifies a message to be a retained message. To identify the message the packetId is used.

info

Digression: Wildcards + retained messages

May the subscribing client include wildcards in the topic pattern they subscribe to, the client receives a retained message even if the topic of the retained message is not an exact match.

Payload:

The actual content of a message is the content of a payload. Whether it's a .mp3-file or .mp4-file, a picture, or a video. Up to 256MB. The payload does not encrypt the content. The broker simply delivers what the publisher inserted into the payload.

Matter of fact the receiving clients don't even know about the data format they receive. Still, there's little chance that a client subscribes to a host and topic it doesn't know about in the first place. Following the client can forecast the receiving data format.

The subscriber can not reject a message.

But as it can do with it as pleased, that could be nothing as well. Meaning a subscriber could receive a message, but simply don't do anything with it.

dupFlag

A dupFlag indicates the duplicate of a message.

A message with dupFlag was resent.

Following a recipient didn't acknowledge the original message.

Subscribe

To receive messages on topics of interest, the client sends a SUBSCRIBE packet to the broker.

caution

Attention To subscribe to a topic, the subscriber must also set a QoS level of the topic subscription. (#).

clientSubscribing

SUBSCRIBE

packetId:

Unique packet identifier.

subscriptions:

The limitation on subscriptions a subscriber can subscribe to within one SUBSCRIBE packet is huge.

A SUBSCRIBE packet does not have to have only one subscription. Multiple can be added.

Each subscription consists of a topic and a QoS level. When subscriptions overlap, the broker delivers the message that has the highest QoS level for the specific topic.

The brokers must make sure the subscribing client receives the messages that are permitted to be received by him in terms of security. But also concerning the QoS level that was set by the publishing client.

Subscribing to several topics simultaneously is an option that's possible. A so-called "wildcard" enables this feature.

There are two kinds of wildcards:

Single level

Single-level wildcards enable to set of a multi-option part of the topic. For example:

singleLevelWildcard

Multi level

A "#" is used, the client subscribes to each topic that's after the #. There is no regulation for the length or hierarchical depth of the string. The # must be placed as the last character in the topic and preceded by a forward slash. e.g.: "car/heating/#"

multiLevelWildcard

The wildcard characters (+ and #) can be used in topic filters. But they can not be used in or as topic names. Therefore wildcards are only intended for subscribing to messages and not for publishing messages.

caution

Attention Topic level separators (/) can be used in both cases! (#).

Wildcards + retained messages:

May the subscribing client include wildcards in the topic pattern they subscribe to, the client receives a retained message even if the topic of the retained message is not an exact match.

Topics starting with $ are not hit by a wildcard according to the standard.

caution

Attention The broker checks the permission of the subscribing client and forwards the messages the subscriber is allowed to receive. Reversed this can mean that a subscribing client does not get the messages the client is not permitted to receive, even though the subscribing client may have subscribed to the topic using a wildcard. (#).

The broker must acknowledge each SUBSCRIBE packet. Therefore the broker sends a SUBACK packet (subscribe acknowledge) to the client.

SUBACK

The broker sends one return code for each topic/QoS-pair that it receives in the SUBSCRIBE packet.

The return codes can be:

  • 0 - Success-Maximum QoS0
  • 1 - Success-Maximum QoS1
  • 2 - Success-Maximum QoS2
  • 128 - Failure

The Success-Maximum QoS0-2 shows the level of Quality of Service (QoS) that is set and can be supported.

Published messages, but no one subscribing, are discarded by the broker.

The broker filters the incoming messages. In the next step, the filtered message gets forwarded to the subscriber. That’s how the subscriber only receives messages of interest.

brokerHandlingSubscriptions

As there is no direct link between publisher and client, the publisher can never make sure that somebody is actually "listening" to the messages sent.

In MQTT v5, if a client publishes with QoS 1 or 2 and there is no subscriber, then the broker can tell the publisher this in the PUBACK/PUBREC reply.

caution

Attention

MQTT v.3.1.1:

Whenever you disconnect unintendedly and haven't set a cleanSession flag (cleanSession=false) all topic subscriptions are lost. You must resubscribe to every single one again.

In case you have set CleanFlag=false and reconnected with the same clientId, the broker remembers the topics you have subscribed to.

MQTT v.5.0:

The session duration is set by the session expiry interval. (#).

Unsubscribe

A client can always UNSUBSCRIBE from a topic.

UNSUBSCRIBE

packetId:

Uniquely identifies a message.

List of topics:

The UNSUBSCRIBE packet contains also the topics you want to unsubscribe from.

After the packet is sent, the broker unsubscribes the client from the topics.

$-Symbol

caution

Attention

$ topics are just "reserved" and shouldn't be used by applications.