ESP8266 MQTT overflow

dilbert168 · November 21, 2020, 7:06pm

If you are asking a question, please follow this template:

My goal is: [determine hardware and software limitation]
My actions are:
[Hardware : ESP8266 (Fanstel BWG832FX)
Hardware : nRF52 connect UART to ESP8266
Software : Mongoose build using MQTT
Server : [Eclipse Mosquitto in local Laptop ]
The result I see is: [Using the nRF52 to send data (40 bytes) in UART every 140ms and this packet would then do MQTT.publish in ESP8266. After a few packet is sent, ESP will show “MQTT FIFO overflow” error. If changed the timing from 140ms to 180ms, then FIFO error disappear]
My expectation & question is: [Will this be the limitation of this configuration ESP8266 + Mongoose ?? My hope for the project is sending BLE data (20bytes) packet at 20ms sampling rate from BLE sensors.
Is the limitation inside ESP8266 chip or configuration for Mongoose ??]

nliviu · November 22, 2020, 9:17am

dilbert168 · November 22, 2020, 9:53pm

But if MQTT output rate is lower than UART packet coming in, eventually it will overflow.

scaprile · November 22, 2020, 10:31pm

This is not a pipe nor an X.25 PAD, MQTT was not designed for taking constant speed serial in but to decouple complex systems by using the publish-subscribe design paradigm.
One possibility here is to buffer several samples in order to maximize packet length to protocol overhead and minimize latency by doing only so many MQTT transactions: get 1K serial data and send 1 1K MQTT message, it takes very similar time to send a 1-byte MQTT message than a 1K-byte message because usually the handshake dominates the transaction time. BTW, you can try QoS=0 to just fire a message and hope, that will have more throughput if you can spare msgs lost.
Other is to process data and send only differences or important changes every now and then (I guess there is an underlying application besides “sending UART data”).
Eventually you will get in trouble and intermediate queues will mitigate that, but yes, it may eventually overflow because it was not designed to have a sustained throughput. Ever wonder no one sends voice nor music nor video over MQTT ?
If you need constant throughput you need to design that from the ground up and MQTT is not for that.
There are no hardware nor software limitations here, you need to understand what you are trying to do and learn how to properly do it.

dilbert168 · November 23, 2020, 4:18pm

Thanks for info. I will give this a try.

On a related issue. Do you know how I could optimize when UART.setDispatcher is called ?
I try not enable WiFi router, so ESP is not connected.
The ProcessUartData() would just reply a [ACK] packet in UART.
I place a timer measurement and it takes 70ms to response. Any way I can change the Dispatcher to return per byte for quicker response ?

UART.setDispatcher(uartNo, function(uartNo)
{
ProcessUartData();
}, null);

scaprile · November 23, 2020, 5:54pm

You seem to be trying to do something funky.
Usually one does not want to work that way. The UART buffer is there to decouple your application from the serial nuances, you get a fair amount of data and then process it in chunks, the bigger the chunk usually the more efficient you are, but that depends on what you are doing.
The dispatcher is usually triggered by a buffer full or enough time without data (or room for transmission), you can toy with that by observing the driver code, iirc the “lingering” time might help, but these systems are not for time sensitive serial operation, usually you process a frame which is delimited by a particular data value or carries a frame length indication; you sync to the stream first and then process data in frames.

dilbert168 · November 23, 2020, 8:25pm

Not too funky.
“usually you process a frame which is delimited by a particular data value or carries a frame length indication; you sync to the stream first and then process data in frames.”

This is what I want to change.
Where I can locate this code and make changes & rebuild online ?

Since I am new Mongoose, any link to info be excellent help.

nliviu · November 24, 2020, 8:12am

The dispatcher callback is called at every poll in the main mOS task.

github.com

cesanta/mongoose-os/blob/7390c4fd290d87251ee4bd9e0bd2f132973a1b5b/src/mgos_uart.c#L188


  struct mgos_uart_state *us = s_uart_state[uart_no];
  if (us == NULL) {
    us = (struct mgos_uart_state *) calloc(1, sizeof(*us));
    if (us == NULL) return false;
    us->uart_no = uart_no;
    mbuf_init(&us->rx_buf, 0);
    mbuf_init(&us->tx_buf, 0);
    if (mgos_uart_hal_init(us)) {
      us->lock = mgos_rlock_create();
#ifndef MGOS_BOOT_BUILD
      mgos_add_poll_cb(mgos_uart_dispatcher, (void *) (intptr_t) uart_no);
#endif
      s_uart_state[uart_no] = us;
      res = true;
    } else {
      mbuf_free(&us->rx_buf);
      mbuf_free(&us->tx_buf);
      free(us);
      us = NULL;
    }
  }

github.com

cesanta/mongoose-os/blob/7390c4fd290d87251ee4bd9e0bd2f132973a1b5b/src/mgos_uart.c#L69




void mgos_uart_dispatcher(void *arg) {
  int uart_no = (intptr_t) arg;
  struct mgos_uart_state *us = s_uart_state[uart_no];
  if (us == NULL) return;
  uart_lock(us);
  if (us->rx_enabled) mgos_uart_hal_dispatch_rx_top(us);
  if (!us->xoff_recd) mgos_uart_hal_dispatch_tx_top(us);
  if (us->dispatcher_cb != NULL) {
    uart_unlock(us);
    us->dispatcher_cb(uart_no, us->dispatcher_data);
    uart_lock(us);
  }
  mgos_uart_hal_dispatch_bottom(us);
  if (us->xoff_sent && us->rx_enabled && mgos_uart_rxb_free(us) > 0) {
    char xon = MGOS_UART_XON_CHAR;
    /* We put it at the end of tx_buf, so antire TX fifo will need to drain
     * before remote transmitter will be re-enabled. */
    mbuf_append(&us->tx_buf, &xon, 1);
    us->xoff_sent = false;
  }

mgos_add_poll_cb

I don’t think you will be able to keep pace with a 20ms sampling rate in mJS. I’d suggest to move your code in C/C++.

dilbert168 · December 4, 2020, 9:37pm

I have written a C function to process UART data from JS and able to response within 15ms on received packet (40 bytes) compare to 70ms using JS to process data. Which is Good start.

However, when packet size increased to 90 bytes, the receiving packet is corrupted.
I have defined
UART.setConfig(uartNo,
{
baudRate: 115200,
rxBufSize: 256,
}

Any idea how I could debug this ?