Message Queue Telemetry Transport or MQTT is the de facto standard communications protocol in the IoT world right now. It is simple, lightweight, ideal for sensors and other devices with limited computational power and bandwidth and where low power consumption is a must. It uses a publish-subscribe messaging pattern where the clients send their messages to a broker to different topics. Consumers can subscribe to these topics to receive the messages.
Choosing the right MQTT broker is the key to be able to serve thousands of clients and scale well efficiently on the hardware available to you.
When we selected the MQTT broker for our SensorHUB IoT platform, we first gathered our requirements:
- Open source
- Support for TLS with client certificate based authentication
- Support for authorization using a database
- Scalable horizontally and vertically (clustering, multithreaded...)
- Full MQTT 3.1.1 implementation
The broker must be open-source and free to use. Security is very important to us so TLS support is a must, we choose client side TLS certs for the primary way of authentication so the broker must support this as well. Client side TLS certs remove the need for username and passowrd, however most authentication and authorization mechanisms on the broker side rely on these, or at least on the username. To solve this issue the brokers that support client side certs usually has an option to use the CN field of the client cert as a username. This feature enables us to use these certs instead of the username and password seamlessly.
We set topic access control rules dynamically from our web administration ui so the broker must be able to read these from some kind of a database and apply them accordingly. Scalability is also very important since we want to handle thousands of devices at once and we also like to be up to date and follow standards so we wanted full MQTT 3.1.1 support.
We have evaluated multiple brokers for the role of MQTT Gateway, in the following I will shortly summarize our findings.
Mosquitto is a really lightweight MQTT broker written in C. We have used it in the past so it came naturally to investigate its capabilities as a broker for a scalable IoT platform. It supports TLS and there are plugins for authorization using a database, but it has some downsides. Unfortunately Mosquitto does not support clustering, it makes scaling a bit difficult. It is using only a single thread so can't take advantage of multi core CPUs. An interesting issue that we had with it when we started to load test it is that above 10 client connections / sec over TLS, connection attempts started to fail due to some errors with the handling of SSL connections using OpenSSL. After searching on the internet for this issue we have found others facing the same and no real solution was found. I'm sure with some tuning on the OS and config levels Mosquitto can do more, but for us it was out of the game.
RabbitMQ is a very popular message broker written in Erlang that has support for MQTT among other protocols through a plugin. TLS support is there, clustering is fine, authorization cannot be done using a database directly but you can create an HTTP REST wrapper over your database and that can be used as an authorization backend. The problem with RabbitMQ is the MQTT support itself. This broker supports the AMQP protocol natively, the MQTT implementation is missing some important features such as QoS2. Quality of Service level 2 ensures that a message is received exactly once. This is very important in some cases, for example when commands are sent from the IoT platform to the devices or actuators. Lost or duplicate commands can cause serious problems in these scenarios so QoS2 is a must. We had to say goodbye to RabbitMQ.
EMQ is another Erlang based broker which was very promising. It is easy to configure and use, scales well, but unfortunately client certificates are not supported in the current version. In SensorHUB we issue x509 client certificates to the devices that we want to connect to the cluster. These certificates provide a secure, passwordless mode of authentication. This is a must for us, if this feature is available EMQ would be a very strong option.
VerneMQ is a relatively new MQTT broker written in Erlang (this language is very popular in the message broker world because its distributed and soft real-time capabilities). VerneMQ has all the features that we were looking for. Clustering, TLS 1.2, complete MQTT 3.1.1 implementation and authorization with database. It can be easily extended with Lua scripts, has a relatively nice documentation and a very helpful developer team on GitHub. We ran some performance tests on it and it could easily handle 400 joining clients/sec sending 3000 messages/sec on a 2 core 2 GHz server with 8 gigabytes of ram. At QoS level 1 on both consumer and producer side.
After checking out these and other brokers that fall out of the race even earlier, we went with VerneMQ. Another promising enterprise ready broker is HiveMQ but since it is not open source and free to use we couldn't evaluate it, however they have some very nice and detailed materials on the MQTT protocol, if someone is interested. https://www.hivemq.com/blog/mqtt-essentials/
The next steps for us was to configure the VerneMQ broker and integrate it into our environment. This includes setting up a custom Lua plugin for authorization, creating a Kubernetes deployment descriptor, generating certificates and so on. We have also concluded some performance tests. All these may be published in the future as a new blog post. Stay tuned.
UPDATE: Our experiences with the selected VerneMQ broker can be found in my new post here.
A list of the most popular MQTT brokers can be found here with some short descriptions:
MQTT Community on GitHub: