COMP3310/ENGN3539 Computer Networks - Assignment 3 - Testing MQTT
COMP3310/ENGN3539/etc Assignment 3 – Testing MQTT
Introduction :
• This assignment is worth 1 5% of the final mark • It is due by Tuesday 23 May 23.55 AEST (Canberra time) • Late submissions will not be accepted, except in special circumstances . • Any e xtensions should be requested well before the due date, with appropriate evidence. Please use the extension -request link on wattle rather than direct emails.
Assignment 3
MQTT is the most common open IoT protocol being deployed today. It provides a publisher/subscriber model, using a broker or server. It allows for an almost -unbounded number of sources to publish information , each at their own rate, and subscribers to receive information (and act on it). As such, it is designed to provide high -performance communication mechanism s, with minimal delays and excellent scaling performance. We’ll use it to monitor the performance of some imaginary system : say counting the total kilograms of minerals rushing by on a conveyor belt , that you can control . This assignment will look at the fu nctionality and performance of the publishers, brokers , the network and the subscribers.
This is a coding, analysis and writing assignment. You may code in C/Java/Python or any programming language that a tutor can assess (hope that’s enough for everyone), and yes, you may use MQTT and other help er libraries . The assessment will not rely solely on running on your code, but more on the data gathering and your analysis . However , we will review the code and may briefly test it against our own broker running in the usual lab -type environments or simil ar. You will need to identify in your README/code any libraries you are using.
Please note, you will be working in pairs or small groups for a key part of this assignment, to test each other’ s systems across the Internet . If you don’t have somebody to work with, ask your tutors NOW to help identify somebody in your tutorial group, or post to the Wattle Forum under the pinned Find-a- Partner topic. You do not need to be in the same location at any time.
Submitting
You will be submitting two things: your code and your analysis report . Note that there will be two submission links on the Wattle course -site:
- Your code must be submi tted to wattle as a zip file, with README instructions on how to compile/run the components as appropriate .
- Your analysis report (pdf) must be submitted via TurnItIn on the wattle site, so ensure you quot e
and cite sources properly.
Each question has an indication of how much ( rough maximum) you should write .
Outcomes
We’re assessing your understanding of MQTT, your code’s functionality in subscribing/publishing to a broker , dealing with high message rates from a number of sources , measuring message performance, and your insight to interpret what you’re seeing. You will be explor ing MQTT functionality, the quality -of-service (QoS) levels , describ ing how they work , why you would use different levels, and how they perform in real- world deployments given various publishers .
Resources
You will need to set up your own MQTT serv er/broker for you to connect to as per the specifications below. You and your partner can share the same broker , if that suits . The Mosquitto broker is perhaps the best choice to run on your own computer, it has binaries and/or packages for most operating systems, but if you find a better one you can use that. It must be at least MQTT v3.1.1 compliant ( MQTT v5 is not necessary ). A key consideration is that you will need to allow connect ion to it over the Internet for part of the assignment . If running it at home where you have a home modem/router that does NAT you will need to work out how to configure it to allow remote connect ions (think about the port(s) you need to forward, and how to manage any changing IP addresses ). If you have a firewall running on your home - network/ computer, you may need to unblock the appropriate ports there too . There are a few other ways to solve this, e.g. a VPN s ervice on your modem or a local computer , or tools like Cloudflare -tunnels or Twin gate -proxy . If it proves too difficult you might consider setting up a cloud server with a public IP address on e.g. Amazon, Azure, Alibaba, etc. and installing a broker there. Yet another option is to set up a (free) fully -hosted broker through a provider such as EMQX .io. Public brokers may have performance vary non- deterministically due to other pe ople hammering at random times.
Your broker should be configured for some basic authentication and access control , so try to use:
• username student and password 33102023
• require the client -id to start with 3310 -
(Public brokers may limit your ability to configure this , so note your approach in your code/README .)
You can test the broker works by subscribing to the $SYS topics, which describe the server, and help get you
familiar with the information presented there - you will be using them for your analysis later.
Assignment programming:
You need to write three small programs1. The diagram at the end might help to clarify the following.
• A Publisher :
o You will have 1, 2 or 3 instances of the Publisher running2. This will help stress the broker
and the network. Each publisher needs its own unique client -id.
o Each Publisher will send a simple message to the broker, namely an incrementing counter
(0, 1, 2, 3, …). It will send those messages at a specific MQTT QoS level (0, 1 or 2), and with
a specific delay between messages (0 ms, 1ms, 2ms, 4ms , 8ms, 16 ms, 32ms).
o Each will publish to the topic counter/
• A Controller : This will listen to the broker on three topic s: o request/ qos o request/delay o request/instancecount
1 The Controller and Publisher could be in the same program, but then it needs to be efficiently listening and sending potentially at high da ta rates at the same ti me. 2 It could also be a single mult i-threaded program , though that may have some additional complexities and perfor mance limitations . Page 3 of 5 Whenever the Controller gets a new message on any of the se topics it needs to modify the Publisher’s b ehaviour accordingly. All 1-3 publisher instances will run the same qos/delay parameters . One way is to stop any current publisher process es (e.g. send a KILL signal, or at worst put a 1.5-minute time limit on it running, see below ) and start a new one with the new parameters, say via command line arguments. Another approach might be through event - handling and multi -threading within a combined publisher/controller, but it is up to you. Please explain in your README file or code comments what approach you are taking.
• An Analyser : This will listen to the counter (s) on the broker and take measurements (below) to report statistics on the performance of the publisher/ broker/ network combination. Those measurements should be taken across the range of qos, delay and instance -count values as above, so that you can compare them. Your Analyser will publish to the request/ qos, request/delay and request/instancecount topics , asking the Controller to get some number of Publisher s to deliver what you r Analyser need s at that time. For the QoS, subscribe at the same QoS as requested from the Publisher (i.e. if P ublisher ->broker qos=2 then Analyser ->broker should have qos=2 as well). The Analyser does not need to take all (7 delay 3 qos 3 instances ) measurements sequentially in one run, but the closer in time the measurements occur the better, as network (and broker) conditions do change all the time.
You can test all of these at home on a local broker. It is only in the final measurements that your broker will allow another student’s Analyser to access your accessible broker /Controller , and vice -versa.
Analysis and Reporting In your report, please address the following :
-
Subscribe to some broker , you can use any mqtt client you like for this , to retrieve some $SYS/# value . Wireshark the handshake for one example of each of the differing QoS -level s (0,1,2), include screenshots in your report that show the wireshark capture (filter for mqtt) , and briefly explain how each QoS transfer works , and wha t it implies for message duplication and message order. Discuss briefly in your report in which circumstances would you choose each QoS level , and offer an example of a sensor that would use each of them . [around 1 page]
-
Run your Analyser against your partner’s broker , have it tell the broker (and so their Controller) what you want their Publisher to send , and record information for 1 minute for each set of the qos/delay/instance -coun t parameters . (Tips: (i) only print to screen for debugging, otherwise it will slow your code down a lot. (ii) Check with the broker owner what delay there might be between your request and their publisher responding. (iii) Use the counter values to tell you what messages are arriving, or are not arriving as expected , and w hat values they start/stop at when you subscribe t o calculate the rates be low. (iv) try publishing at a very low rate (i.e. high- delay ) to start with , to check your code )
a. Collect statistics, for each instance -count, delay and QoS level, and measure over the 1- minute period : i. The overall average rate of messages you actually receive from all publishers across the period [messages/second]. ii. The rate of message loss you see [percentage ]. (how many messages did you see, versus how many should you have seen) iii. The rate of any out-of-order messages you see [percentage ] (i.e. how often do you get a smaller number after a larger number ) Page 4 of 5 iv. The mean and median inter -message -gap you see , compared to the requested delay [milliseconds]. Only measure for actually consecutive counter -value messages, ignore the gap if you miss any messages in between .
b. While measuring the above also i. Subscribe to and record the $SYS topic s, and identify w hat, if anything, on the broker do any loss/ misordered -rates correlate with . (Look at measurements under e.g. ‘load’, as well as the ‘heap’ and ‘active clients’ and ‘messages ’, anything that seems relevant . See https://mosquitto.org/man/mosquitto- 8.html though it may not quite match your broker ) ii. Record a traceroute between the Analyser and the broker, and between the Publisher and the broker (you’ll need the broker owner to provide the second one )
-
In your report : [around 2 -3 pages of text, plus any charts] a. Summarise your measurements . Explain what you expected to see , especially in relation to the different QoS levels , and whether your expectations were matched. b. Describe what correlations of measured rates with $SYS topics you expect to see and why, and whether you do, or do not. c. Report the traceroutes captured above.
-
Consider the broader end -to-end (internet -wide maybe) network environment, in a situation with millions of sensors publishing regularly/irregularly to thousands of subscribers. Explain in your report [around 1- 2 pages ] a. What (cpu/memory/network) performance challenges there might be when using MQTT in general, from the source publishing its messages, all the way through the network and broke r to your subscribing client application . If you lose messages, where might they be lost? b. How the different QoS levels may help, or not, in dealing with the challenge s.
Amendments and Assessment
Any amendments to the assignment specification will be published in the wattle forum. Your assignment will be assessed on • Your code clarity , and documentation /comments ( 15%) • your code subscribing and properly publishing/listening to the broker (15%), and • your analysis report addressing the questions above (70%)
