Some time ago I have written a post here about the process of choosing an MQTT broker that fit our requirements for an IoT project. Now I continue that article with some details and experiences about using the selected VerneMQ broker.
Running VerneMQ
You can easily download a VerneMQ binary package from their website and run it, there are builds available for multiple Linux distributions however running native binaries are not the hip thing to do nowadays. Lucky for us the developers of this broker make an official Docker image available. Besides the Docker image there are instructions and examples on how to run it using Kubernetes even with auto clustering enabled, so we can autoscale our VerneMQ cluster based on load with the help of Kubernetes.
In our projects we have used VerneMQ on plain old Docker, Docker Compose and Kubernetes as well.
Configuring VerneMQ
There is a well detailed documentation on the configuration options, feel free to use that. I'm going to show an example that we use in most of our setups. VerneMQ can be configured by passing it a vernemq.conf configuration file, but in the Docker world I like to use environment variables if possible and fortunately the VerneMQ Docker image supports those for all configuration options. So here is a Docker compose file with VerneMQ set up:
mqtt:
image: erlio/docker-vernemq:1.8.0
restart: always
environment:
DOCKER_VERNEMQ_PLUGINS__VMQ_ACL: 'off'
DOCKER_VERNEMQ_PLUGINS__VMQ_DIVERSITY: 'on'
DOCKER_VERNEMQ_VMQ_DIVERSITY__MYSQL__HOST: 'database'
DOCKER_VERNEMQ_VMQ_DIVERSITY__MYSQL__PORT: '3306'
DOCKER_VERNEMQ_VMQ_DIVERSITY__MYSQL__USER: 'root'
DOCKER_VERNEMQ_VMQ_DIVERSITY__MYSQL__PASSWORD: 'root'
DOCKER_VERNEMQ_VMQ_DIVERSITY__MYSQL__DATABASE: 'mydatabase'
DOCKER_VERNEMQ_VMQ_DIVERSITY__MYSQLACL__FILE: '/etc/scripts/mysqlacl.lua'
DOCKER_VERNEMQ_LISTENER__SSL__CAFILE: '/etc/ssl/ca.crt'
DOCKER_VERNEMQ_LISTENER__SSL__CERTFILE: '/etc/ssl/server.crt'
DOCKER_VERNEMQ_LISTENER__SSL__KEYFILE: '/etc/ssl/server.key'
DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT: '0.0.0.0:8884'
DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT__USE_IDENTITY_AS_USERNAME: 'off'
DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT__REQUIRE_CERTIFICATE: 'off'
ports:
- '8884:8884'
expose:
- '8884'
volumes:
- ./mqtt/certs:/etc/ssl
- ./mqtt/scripts:/etc/scripts
This configuration will produce an instance of VerneMQ with the following properties:
- Using an SSL certificate for TLS secured client connections
- Using a custom plugin to have a MySQL based authentication and authorization
Let's see how it is done!
With DOCKER_VERNEMQ_PLUGINS__VMQ_ACL: 'off' we turn of the built in access control list handling of VMQ to be able to use our custom one. With
DOCKER_VERNEMQ_PLUGINS__VMQ_DIVERSITY: 'on' we enable the Diversity plugin management subsystem of VerneMQ so we can use a custom MySQL based auth plugin. The DOCKER_VERNEMQ_VMQ_DIVERSITY__MYSQL_* properties configure the MySQL access for the Diversity plugins. The DOCKER_VERNEMQ_VMQ_DIVERSITY__MYSQLACL__FILE variable tells VerneMQ where it can get the lua script file containing the custom auth plugin. The following variables control the location of the certificates required for SSL secured connections. DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT: '0.0.0.0:8884' sets the default IP and port where the broker will listen. VerneMQ has the ability to authenticate clients with x.509 client certificates. If this is used we can tell VerneMQ to use the CN (Common Name) field of the client cert as username. By setting the DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT__REQUIRE_CERTIFICATE: 'off' and ```DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT__USE_IDENTITY_AS_USERNAME: 'off'`` variables we turn this feature off as in this setup we do not use client certificates. Lastly we attach a folder containing the certificates and another folder containing the Lua plugins to the Docker container so VerneMQ can access these files.
Plugins and hooks
VerneMQ is written in Erlang and plugin development is enabled with the Vmq Diversity plugin system that is able to run plugins written in Lua implementing various hooks provided by the flows in VerneMQ such as the subscribe flow or publish flow, the hooks in these flows one can implement a plugin for include
auth_on_subscribeon_subscribeon_unsubscribeauth_on_publishon_publishon_deliver
and so on. As you can see the opportunity is given to develop a custom authentication or authorization mechanism, gather statistics, notify other systems on some special incoming messages and many other. In the following example I show you a really simple authentication and authorization plugin that I have written.
require "auth/auth_commons"
function auth_on_register(reg)
local results
if reg.username ~= nil and reg.client_id ~= nil and reg.username == reg.client_id then
log.info("username based auth")
results = mysql.execute(pool,
[[SELECT publishAcl as publish_acl, subscribeAcl as subscribe_acl FROM MqttAcls WHERE clientId=? ;]], reg.client_id)
else
log.warning("bad connection setup")
return false
end
return check_result_add_to_cache(results, reg)
end
function check_result_add_to_cache(results, reg)
if #results == 1 then
row = results[1]
publish_acl = json.decode(row.publish_acl)
subscribe_acl = json.decode(row.subscribe_acl)
cache_insert(
reg.mountpoint,
reg.client_id,
reg.username,
publish_acl,
subscribe_acl
)
return true
else
log.warning("auth failure, client not found in ACL database, or wrong password")
return false
end
end
pool = "auth_mysql"
config = {
pool_id = pool
}
mysql.ensure_pool(config)
hooks = {
auth_on_register = auth_on_register,
auth_on_publish = auth_on_publish,
auth_on_subscribe = auth_on_subscribe,
on_unsubscribe = on_unsubscribe,
on_client_gone = on_client_gone,
on_client_offline = on_client_offline
}
This plugin implements the auth_on_register hook and will be called when a new client connects to the broker as described here. The plugin checks if the username and client id are the same for the client (it does't really makes sense, but illustrates what can you do with the plugin system) and if it is, it looks for a matching client registered in the database. All client related information are provided to our plugin method through the reg parameter. If there is a record in the result of the query the plugin caches the data, so the ACL-s won't be queried every single time this client sends a message. If the client id does not match the username or a match is not found in the database the plugin returns false so the connection will be refused.
The database connection, the client information, the cache are all provided to us by the diversity plugin system, we just have to use these and create any logic we want. It is a really easy and flexible way to extend or modify how the broker behaves.
Tips and tricks
Listeners
VerneMQ supports multiple listeners to be active at the same time for various types of clients. For example if you want to implement a system which allows MQTT username / password and SSL client cert based authentication for clients, you can easily define 2 listeners on 2 different ports, with these different settings. Clients connecting to any of these listeners will have access to the same topics. Here is an example:
listener.ssl.cafile = /etc/ssl/ca.crt
listener.ssl.certfile = /etc/ssl/server.crt
listener.ssl.keyfile = /etc/ssl/server.key
listener.ssl.certbased = 0.0.0.0:8883
listener.ssl.certbased.require_certificate = on
listener.ssl.certbased.use_identity_as_username = on
listener.ssl.pwbased = 0.0.0.0:8884
listener.ssl.pwbased.require_certificate = off
listener.ssl.pwbased.use_identity_as_username = off
We set up the secured SSL connection by providing the certificates for all SSL based lsiteners. After that we can define many using custom names such as certbased and pwdbased as in the example. Websocket based listeners are also supported.
VMQ Admin
VerneMQ comes with an admin CLI which provides many useful features for developers and administrators. You can stop, start and even add new listeners in runtime to the VMQ broker. It is also possible to trace the activity of clients by client id. This feature is really useful while developing plugins and struggling with an issue. More info can be found here
ACLs
We can easily define Acces Control Lists for publish and subscribe activities as well. In our use cases we have these custom ACLs stored with the clients in a database, and during authentication our plugin such as the example above will retrieve this information with the client. In these ACLs we can not only specify whether a client can publish or subscribe to some topics but apply other restrictions and even modify the properties of a message in the case of a publish ACL. Let's see an example:
{
"pattern": "a/+/c",
"max_qos": 2,
"max_payload_size": 128,
"allowed_retain": true,
"modifiers": {
"topic": "new/topic",
"payload": "new payload",
"qos": 2,
"retain": true,
"mountpoint": "other-mountpoint"
}
}
The ACL is given as a JSON where we can limit the topics, QoS level, payload size and many more. We can modify the message as well, in the example above in the modifiers section we define that we want a message to be added to the topic new/topic with the payload new payload with some other options.
MySQL 8 support
MySQL starting with version 8 replaced the default password hashing mechanism it uses. This change breaks VerneMQ MySQL support, the broker no longer will be able to connect to the database using a user created with the new defaults. Fortunately it is easy to fix by altering the user ALTER USER 'alma'@'%' IDENTIFIED WITH mysql_native_password BY 'password'; The VerneMQ team already has plans to update the MySQL client they use, so hopefully this solution won't be needed in the near future.
Final thoughts
We have chosen VerneMQ more than a year ago to serve as an MQTT broker in our IoT platform. Since then we have used it in many other projects to our satisfaction. The documentation is well maintained and easy to use, the broker itself performs well. The VerneMQ developers are quite active on GitHub, feel free to submit issues if you face any, or contribute if you find something you can fix.
Useful links:
Follow us on social media
