ESP8266 MQTT overflow

#1

If you are asking a question, please follow this template:

  1. My goal is: [determine hardware and software limitation]

  2. My actions are:
    [Hardware : ESP8266 (Fanstel BWG832FX)
    Hardware : nRF52 connect UART to ESP8266
    Software : Mongoose build using MQTT
    Server : [Eclipse Mosquitto in local Laptop ]

  3. The result I see is: [Using the nRF52 to send data (40 bytes) in UART every 140ms and this packet would then do MQTT.publish in ESP8266. After a few packet is sent, ESP will show “MQTT FIFO overflow” error. If changed the timing from 140ms to 180ms, then FIFO error disappear]

  4. My expectation & question is: [Will this be the limitation of this configuration ESP8266 + Mongoose ?? My hope for the project is sending BLE data (20bytes) packet at 20ms sampling rate from BLE sensors.
    Is the limitation inside ESP8266 chip or configuration for Mongoose ??]

#2

Increase mqtt.max_queue_length.

#3

But if MQTT output rate is lower than UART packet coming in, eventually it will overflow.

#4

This is not a pipe nor an X.25 PAD, MQTT was not designed for taking constant speed serial in but to decouple complex systems by using the publish-subscribe design paradigm.
One possibility here is to buffer several samples in order to maximize packet length to protocol overhead and minimize latency by doing only so many MQTT transactions: get 1K serial data and send 1 1K MQTT message, it takes very similar time to send a 1-byte MQTT message than a 1K-byte message because usually the handshake dominates the transaction time. BTW, you can try QoS=0 to just fire a message and hope, that will have more throughput if you can spare msgs lost.
Other is to process data and send only differences or important changes every now and then (I guess there is an underlying application besides “sending UART data”).
Eventually you will get in trouble and intermediate queues will mitigate that, but yes, it may eventually overflow because it was not designed to have a sustained throughput. Ever wonder no one sends voice nor music nor video over MQTT ?
If you need constant throughput you need to design that from the ground up and MQTT is not for that.
There are no hardware nor software limitations here, you need to understand what you are trying to do and learn how to properly do it.

#5

Thanks for info. I will give this a try.

On a related issue. Do you know how I could optimize when UART.setDispatcher is called ?
I try not enable WiFi router, so ESP is not connected.
The ProcessUartData() would just reply a [ACK] packet in UART.
I place a timer measurement and it takes 70ms to response. Any way I can change the Dispatcher to return per byte for quicker response ?

UART.setDispatcher(uartNo, function(uartNo)
{
ProcessUartData();
}, null);

#6

You seem to be trying to do something funky.
Usually one does not want to work that way. The UART buffer is there to decouple your application from the serial nuances, you get a fair amount of data and then process it in chunks, the bigger the chunk usually the more efficient you are, but that depends on what you are doing.
The dispatcher is usually triggered by a buffer full or enough time without data (or room for transmission), you can toy with that by observing the driver code, iirc the “lingering” time might help, but these systems are not for time sensitive serial operation, usually you process a frame which is delimited by a particular data value or carries a frame length indication; you sync to the stream first and then process data in frames.

#7

Not too funky.
“usually you process a frame which is delimited by a particular data value or carries a frame length indication; you sync to the stream first and then process data in frames.”

This is what I want to change.
Where I can locate this code and make changes & rebuild online ?

Since I am new Mongoose, any link to info be excellent help.

#8

The dispatcher callback is called at every poll in the main mOS task.

mgos_add_poll_cb

I don’t think you will be able to keep pace with a 20ms sampling rate in mJS. I’d suggest to move your code in C/C++.