Esp8266 crashes due to out of memory while douing SSL Https post

General
#1

I was making a HTTPS POST request

HTTP.query({
  url: 'https://mdash.net/api/v2/devices/device2?access_token=xxxxxxxxxxx',
  headers: { 'Content-Type': 'application/json' },
  data: mdata,      // { shadow: { state: { desired: { built_in: boolean}}}}
  cert: 'cert.pem',  // client certificate // just left it if you dont have this file
  key: 'key.pem',  // client key  //just left if you dont has this file
  ca_cert: 'ca.pem',  // ca bundle to verify server with ""
  success: function (body, full_http_msg) { print("body"); },
  error: function (err) { print(err); }
  });

sometime my device show this in serial monitor and ESP8266 automatically restarted:

[Sep 10 12:24:50.827] E:M 264 (34 blocks)
[Sep 10 12:24:50.827] 
[Sep 10 12:24:50.827] Exception 29 @ 0x4000e1b2, vaddr 0x00000000
[Sep 10 12:24:50.828]  A0: 0x40297617  A1: 0x3ffff1c0  A2: 0x00000000  A3: 0x00000000
[Sep 10 12:24:50.828]  A4: 0x00000108  A5: 0x00000000  A6: 0x00000100  A7: 
0x00000010
[Sep 10 12:24:50.836]  A8: 0x3ffe83cc  A9: 0x00000190 A10: 0x00000000 A11: 
0x0000007c
[Sep 10 12:24:50.844] A12: 0x00000000 A13: 0x00000108 A14: 0x00000003 A15: 
0x3ffff250
[Sep 10 12:24:50.853] 
[Sep 10 12:24:50.853] (exc SP: 0x3ffff020)
[Sep 10 12:24:50.853] 
[Sep 10 12:24:50.853] --- BEGIN CORE DUMP ---
[Sep 10 12:24:50.853] mos: catching core dump
[Sep 10 12:24:53.688] ....
[Sep 10 12:25:02.396] ---- END CORE DUMP ----

it created a dump file and it look like this:

--- BEGIN CORE DUMP ---
{"app": "app2", "arch": "esp8266", "version": "1.0", "build_id": "20190910-065332", "build_image": "docker.io/mgos/esp8266-build:2.2.1-1.5.0-r5",
"REGS": {"addr": 1073737920, "data": "
F3YpQDDy/z8AAAAAAAAAABQCAAAAAAAAEAIAACEAAADMg/4/kAEAAAAAAAB8AAAAAAAAABQCAAAAAAAAnCb/P7LhAEAIAAAAAAAAAAAAAAAAAAAAMAAAAA==", "crc32": 2733548080},
"DRAM": {"addr": 1073643520, "data": "SOME_VERY_LONG_DATA", "crc32": 4074477429}}

---- END CORE DUMP ----

“SOME_VERY_LONG_DATA”==> it is a a base64 encoded and it is very long so i delete it and renamed to this,so that the message is shorter.

#3

E:M 264 (34 blocks) means the device goes out of memory.

#4

Yap… I figured it out right now… My ram is Becomming low

[Sep 10 15:02:44.809] sending request...  
[Sep 10 15:02:44.809] {"ram_total":24652} 
[Sep 10 15:02:49.787] sending request...   
[Sep 10 15:02:49.787] {"ram_total":22640} 
[Sep 10 15:02:54.731] sending request...  
[Sep 10 15:02:54.731] {"ram_total":21724} 
[Sep 10 15:02:59.766] sending request...   
[Sep 10 15:02:59.766] {"ram_total":19840} 
[Sep 10 15:03:04.726] sending request...   
[Sep 10 15:03:04.726] {"ram_total":18988} 
[Sep 10 15:03:05.436] mg_ssl_if_mbedtls.c:35  0x3fff5a6c ciphersuite: TLS-RSA-WITH-AES-128-GCM-SHA256
[Sep 10 15:03:05.833] E:M 524 (66 blocks)
[Sep 10 15:03:05.833] 
[Sep 10 15:03:05.833] Exception 29 @ 0x4000e1b2, vaddr 0x00000000
[Sep 10 15:03:05.833]  A0: 0x40297617  A1: 0x3ffff210  A2: 0x00000000  A3: 0x00000000
[Sep 10 15:03:05.833]  A4: 0x0000020c  A5: 0x00000000  A6: 0x00000200  A7: 0x00000020
[Sep 10 15:03:05.836]  A8: 0x3ffe83cc  A9: 0x00000190 A10: 0x00000000 A11: 0x0000007c
[Sep 10 15:03:05.844] A12: 0x00000000 A13: 0x0000020c A14: 0x00000041 A15: 0x0000105f
[Sep 10 15:03:05.847] 
[Sep 10 15:03:05.847] (exc SP: 0x3ffff070)
[Sep 10 15:03:05.847] 
[Sep 10 15:03:05.850] --- BEGIN CORE DUMP ---

dont know why my Ram is becoming smaller and smaller

#5

The following request decreaase the RAM memory but it doesnot release the memory after the task is finished. I dont know why? Actually i am calling this function every 5 seconds.
when i remove this function the RAM memory does not decrease(while running the sketch).

HTTP.query({
  url: 'https://mdash.net/api/v2/devices/device2?access_token=xxxxxxxxxxxxxxx',
  headers: { 'Content-Type': 'application/json' },
  data: mdata,      // Optional. If set, JSON-encoded and POST-ed
  cert: 'cert.pem',  // client certificate
  key: 'key.pem',  // client key
  ca_cert: 'ca.pem',  // ca bundle to verify server with
  success: function (body, full_http_msg) { print("success"); },
  error: function (err) { print("err"); }
});
#6

It is saying that some memory is leaked while doing POST request
Memory leak 184 bytes at a time #1923

This memory should be reclaimed after a couple of minutes, once TCP connection exists TIME_WAIT state.

But i want to reclaimed the memory after the “success” or “error”

The sollution of the memory leak is as follow:

// do this before including other stuff like wificlient.h
#include "lwip/tcp_impl.h"
void tcpCleanup()
{
  while(tcp_tw_pcbs!=NULL)
  {
    tcp_abort(tcp_tw_pcbs);
  }
}

Here:

/* HeapCheckV3.ino by Costas 12 June 2017 was originally WebUdater.ino Example
   Line 15 of ESP8266HTTPUpdateServer.cpp in 
   C:\arduino-1.8.3portable\arduino-1.8.3\hardware\esp8266com\esp8266\libraries\ESP8266HTTPUpdateServer\src
   changed to:      </body></html>)";  // was </body></html>\n)";
   To upload through terminal you can use: curl -F "image=@firmware.bin" esp8266-webupdate.local/update
*/
#define HEAPCHECKER 0        // set to 1 to test HEAP loss fix
#ifdef HEAPCHECKER
  #include "lwip/tcp_impl.h" // losing bytes work around
#endif

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
#include <ESP8266mDNS.h>
#include <ESP8266HTTPUpdateServer.h>

const char* host = "esp8266-webupdate";
const char* ssid = "xxxxxxxxxx";
const char* password = "xxxxxxxxxx";
uint32_t originalram;

ESP8266WebServer httpServer(80);
ESP8266HTTPUpdateServer httpUpdater;

void setup(void){

  Serial.begin(115200);
  Serial.println();
  Serial.println("Booting Sketch...");
  WiFi.mode(WIFI_AP_STA);
  WiFi.begin(ssid, password);

  while(WiFi.waitForConnectResult() != WL_CONNECTED){
    WiFi.begin(ssid, password);
    Serial.println("WiFi failed, retrying.");
  }

  MDNS.begin(host);

  httpUpdater.setup(&httpServer);
  httpServer.begin();

  MDNS.addService("http", "tcp", 80);
  Serial.printf("HTTPUpdateServer ready! Open http://%s.local/update in your browser\n", host);
  originalram = ESP.getFreeHeap();
}

//
void tcpCleanup()   // losing bytes work around
{
  while(tcp_tw_pcbs!=NULL)
  {
    tcp_abort(tcp_tw_pcbs);
  }
}
//

void loop(void){
  httpServer.handleClient();
  delay(1000);
  uint32_t ram = ESP.getFreeHeap();
  Serial.printf("RAM: %d  change %d\n", ram, (ram - originalram ));
  if(HEAPCHECKER){          // losing bytes work around
    tcpCleanup();           
    Serial.printf("tcpCleanup completed\n");
  }
}

So i want to call the simillar function after the success or error. How can i add this?

#7

Yes, the connection made by HTTP.query() is not closed and the memory is not freed. AFAIK, there is no way to do that in mJS.

I’d suggest to move your request in C and close the connection as soon as you get a MG_EV_HTTP_REPLY event.

#8

Okey… I migrated the HTTP.query() in C. I am calling the function “void request(const char *data)” to send the POST request… So How to close the connection??? is the “mg_connectin *nc” is the connection? How to call the close function?

#include "mgos.h"
#include "mgos_mqtt.h"
#ifdef MGOS_HAVE_WIFI
#include "mgos_wifi.h"
#endif

static const char *s_url = "https://mdash.net/api/v2/devices/device2?access_token=xxxxxxxxxxxxxxx";

static int s_exit_flag = 0;

static void ev_handler(struct mg_connection *nc, int ev, void *ev_data, void *ud)
{
  struct http_message *hm = (struct http_message *)ev_data;
  int connect_status;

  switch (ev)
  {
  case MG_EV_CONNECT:
    connect_status = *(int *)ev_data;
    if (connect_status != 0)
    {
      printf("Error connecting to %s: %s\n", s_url, strerror(connect_status));
      s_exit_flag = 1;
    }
    break;
  case MG_EV_HTTP_REPLY:
    // printf("Got reply:\n%.*s\n", (int)hm->body.len, hm->body.p);
    printf("Got reply:\n");
    nc->flags |= MG_F_SEND_AND_CLOSE;
    s_exit_flag = 1;
    break;
  case MG_EV_CLOSE:
    if (s_exit_flag == 0)
    {
      printf("Server closed connection\n");
      s_exit_flag = 1;
    };
    break;
  default:
    break;
  }
  (void)hm;
  (void)ud;
}

void request(const char *data)
{
  struct mg_connect_opts bind_opts;
  memset(&bind_opts, 0, sizeof(bind_opts));
  bind_opts.ssl_ca_cert = "ca.pem";
  mg_connect_http_opt(mgos_get_mgr(), MG_CB(ev_handler, NULL), bind_opts, s_url,
  "Content-Type: application/json\r\n"
  , data);
  printf("\nStarting RESTful client against %s\n", s_url);
}

enum mgos_app_init_result mgos_app_init(void)
{

  /* Simple repeating timer */
  //mgos_set_timer(1000, MGOS_TIMER_REPEAT, timer_cb, NULL);

  return MGOS_APP_INIT_SUCCESS;
}
#9

nc->flags |= MG_F_SEND_AND_CLOSE; will do it.

#10

I tried but it does not work(I mean s the memory was not cleared)…

https://forum.mongoose-os.com/discussion/258/large-memory-consumption

They are saying to make a “Makfile”… but i dont undurstand very much about this.

Now i am checking free ram and keep it more than 26000 to perform the request. At least it wwill not crash the ESP :smiley:

Cannot call IFTTT maker webhooks from device - SSL error
#11

Performing a http post every 10 seconds, the memory is recovered:

[Sep 11 15:28:00.644] main.c:132              {"shadow":{"state":{"desired":{"on":false}}}}
[Sep 11 15:28:00.773] mg_ssl_if_mbedtls.c:35  0x3fff32a4 ciphersuite: TLS-RSA-WITH-AES-128-GCM-SHA256
[Sep 11 15:28:01.100] main.c:197              Got mqtt message "{"state":{"on":false},"version":38336,"timestamp":1568204881}" on topic "device10/shadow/delta"
[Sep 11 15:28:01.115] main.c:48               set_led - LED on -> false
[Sep 11 15:28:01.122] main.c:78               report_state -  ({"state":{"reported":{"on": false, "uptime": 16541.056442, "ram_min_free": 15792, "ram_free": 35108}}}) -> success
[Sep 11 15:28:10.631] main.c:78               report_state -  ({"state":{"reported":{"on": false, "uptime": 16550.569071, "ram_min_free": 15792, "ram_free": 39840}}}) -> success
[Sep 11 15:28:10.645] main.c:132              {"shadow":{"state":{"desired":{"on":true}}}}
[Sep 11 15:28:10.934] mg_ssl_if_mbedtls.c:35  0x3fff32a4 ciphersuite: TLS-RSA-WITH-AES-128-GCM-SHA256
[Sep 11 15:28:11.209] main.c:197              Got mqtt message "{"state":{"on":true},"version":38339,"timestamp":1568204891}" on topic "device10/shadow/delta"
[Sep 11 15:28:11.224] main.c:48               set_led - LED on -> true
[Sep 11 15:28:11.230] main.c:78               report_state -  ({"state":{"reported":{"on": true, "uptime": 16551.164440, "ram_min_free": 15792, "ram_free": 34124}}}) -> success
[Sep 11 15:28:20.631] main.c:78               report_state -  ({"state":{"reported":{"on": true, "uptime": 16560.569058, "ram_min_free": 15792, "ram_free": 39844}}}) -> success
1 Like