We are using MQTT over PPPoS. We tried various configurations: send 4 KB messages (one per couple of seconds), but the same problem was with 1 KB messages (one cca every second) or 12 bytes (in this case cca 60 per second - this case was most memory unstable).
In our code there is no calloc or malloc (ok, there is one, but used only once on BT connection).
Also the memory doesn’t free if we stop sending messages.
Now we test another device (but with the same HW configuration and same software) and symptoms are little different. We see New heap free LWM: .... only at the beginning and then only like once per couple of minutes. But at the end it crashes with corrupted heap: https://pastebin.com/EvhDDKe4
We’ve analysed core dump by base64 decoding DRAM data and we see that is is “full” of the data we send over MQTT. Now the question is who is putting it to heap (our code doesn’t seems to do it - it only uses local variable which should be on the stack) but more importantly why doesn’t that thing free it.
mongoose send function spools data in a heap buffer.
Then the IO task drains it to the network.
Thus, if the network is not fast enough (or is down), the data gets spooled in a heap.
You can examine the size of the output buffer by looking at:
struct mg_connection *c = mgos_mqtt_get_global_conn();
c->send_mbuf; // output dynamic buffer, "size" is the allocated size.
c->recv_mbuf; // input dynamic buffer
Thanks @cpq you were right about filling up send_mbuf! Sometimes MQTT stops with publishing the messages and stores them in the buffer. After a minute or so MQTT connecton fails, reconnects and pending messages are sent which empties the buffer.
Now we need to find out why does it happen but it is a completly different story.
The MQTT implementation tries to queue the message and returns. Return code tells you if queuing was successful; you should check and log that.
If somehow for some reason it can’t send, it will keep queuing. You should sniff your network to see what is going on, or at a bare minimum check the Mosquitto logs for published messages.
Your serial might be doing something nasty. How are you handling buffers ?
