(Solved) Using LFS in mongoose OS (C on ESP8266)

lawrencerb · August 16, 2019, 4:47pm

My goal is:
Correctly use lfs functions to save files in a external SPI Flash memory.
My actions are:
Using Fstab to mount the Flash in initialization:

config_schema:

[“spi.enable”, true]

[“devtab.dev0.name”, “spif0”]

[“devtab.dev0.type”, “spi_flash”]

[“devtab.dev0.opts”, ‘{“cs”: 0, “freq”: 10000000}’]

[“fstab.fs0.dev”, “spif0”]

[“fstab.fs0.type”, “LFS”]

[“fstab.fs0.opts”, ‘{“bs”: 4096}’]

[“fstab.fs0.path”, “/data”]

[“fstab.fs0.create”, true]

Initialization ok and returns on Log:

[Aug 16 13:36:01.999] bcn 100
[Aug 16 13:36:01.999] esp_wifi.c:344          WiFi AP IP: 192.168.4.1/255.255.255.0 gw 192.168.4.1, DHCP range 192.168.4.2 - 192.168.4.100
[Aug 16 13:36:02.011] mgos_http_server.c:282  HTTP server started on [80]
[Aug 16 13:36:02.022] mgos_spi_gpio.c:72      SPI GPIO init ok (MISO: 12, MOSI: 13, SCLK: 14; CS0/1/2: 15/0/0)
[Aug 16 13:36:02.032] mgos_vfs_dev.c:73       spif0: spi_flash ({"cs": 0, "freq": 10000000}), size 2097152
[Aug 16 13:36:02.038] mgos_vfs.c:147          /data: LFS @ spif0, opts {"bs": 4096}
[Aug 16 13:36:02.062] mgos_vfs.c:320          /data: size 2097152, used: 8192, free: 2088960
[Aug 16 13:36:02.068] mgos_provision_btn.c:75 Factory reset button: pin 0, pull up, hold_ms 0 (hold on boot)
[Aug 16 13:36:02.088] mgos_provision_stat:119 Max state: 0
[Aug 16 13:36:02.088] mgos_rpc_channel_mq:203 0x3fff1f04 esp8266_249943/rpc
[Aug 16 13:36:02.099] mgos_rpc_channel_ua:313 0x3fff254c UART0

My C function is the simplest as possible:

#include "mgos.h"
#include "lfs.h"
#include "lfs_util.h"


// variables used by the filesystem
lfs_t lfs;
lfs_file_t file;

enum mgos_app_init_result mgos_app_init(void) {

  lfs_file_open(&lfs, &file, "/data/test.txt", LFS_O_RDWR | LFS_O_CREAT);
  lfs_file_close(&lfs, &file);
  

  return MGOS_APP_INIT_SUCCESS;
}

The result I see is:

> [Aug 16 13:36:02.011] mgos_http_server.c:282  HTTP server started on [80]
> [Aug 16 13:36:02.022] mgos_spi_gpio.c:72      SPI GPIO init ok (MISO: 12, MOSI: 13, SCLK: 14; CS0/1/2: 15/0/0)
> [Aug 16 13:36:02.032] mgos_vfs_dev.c:73       spif0: spi_flash ({"cs": 0, "freq": 10000000}), size 2097152
> [Aug 16 13:36:02.038] mgos_vfs.c:147          /data: LFS @ spif0, opts {"bs": 4096}
> [Aug 16 13:36:02.062] mgos_vfs.c:320          /data: size 2097152, used: 8192, free: 2088960
> [Aug 16 13:36:02.068] mgos_provision_btn.c:75 Factory reset button: pin 0, pull up, hold_ms 0 (hold on boot)
> [Aug 16 13:36:02.088] mgos_provision_stat:119 Max state: 0
> [Aug 16 13:36:02.088] mgos_rpc_channel_mq:203 0x3fff1f04 esp8266_249943/rpc
> [Aug 16 13:36:02.099] mgos_rpc_channel_ua:313 0x3fff254c UART0
> [Aug 16 13:36:02.099] 
> [Aug 16 13:36:02.099] Exception 28 @ 0x40299b9b, vaddr 0x0000001c
> [Aug 16 13:36:02.105]  A0: 0x4029ac64  A1: 0x3ffff8f0  A2: 0x00000004  A3: 0x00000000
> [Aug 16 13:36:02.110]  A4: 0x3ffeeff0  A5: 0x00000004  A6: 0x00000000  A7: 0x00000000
> [Aug 16 13:36:02.129]  A8: 0x00000000  A9: 0x00000090 A10: 0x3ffee95c A11: 0xffff8000
> [Aug 16 13:36:02.134] A12: 0x00000000 A13: 0x00000004 A14: 0x3ffeeff0 A15: 0x00000000
> [Aug 16 13:36:02.134] 
> [Aug 16 13:36:02.134] (exc SP: 0x3ffff750)
> [Aug 16 13:36:02.134] 
> [Aug 16 13:36:02.134] --- BEGIN CORE DUMP ---
> [Aug 16 13:36:02.137] mos: catching core dump
> [Aug 16 13:36:04.975] ....
> [Aug 16 13:36:13.681] ---- END CORE DUMP ----
> [Aug 16 13:36:13.684] mos: wrote to C:\mos\MyMemmoryPlayer\core-MyMemmoryPlayer-esp8266-20190816-133613.948335361 (133119 bytes)
> [Aug 16 13:36:13.708] mos: analyzing core dump

My expectation & question is:
How or what do I need to do to save files correctly using lfs functions do the partition /data created?

cpq · August 16, 2019, 6:09pm

Did you try to use ISO C API for file operations (e.g. fopen/fread/frwrite/fclose) ?

lawrencerb · August 16, 2019, 6:19pm

Thanks for the rapid response

Yes, I’ve tried using them. It functioned properly up to the point the script created 76 files of 1328 bytes each

It complained memmory was full using only ~318.900 bytes of the 2.000.000 available. System crashed, failing to perform any operation further: load, read or write in the flash memory anymore.

So I’m trying out the proper LFS commands (Saw a fix using lfs_sync), and couldn’t get any result. I tought I was just making the mistake of not using the proper library, but even the example code in the LFS official documentation I’ve failed to do right…

Sooo, any help is greatly appreciated

nliviu · August 20, 2019, 10:41am

You can’t use the lfs_file_* functions without a properly initialized lfs_t structure.
The standard fopen, fwrite, etc functions are the way to go.

I couldn’t reproduce your issue, but found something else. After creating a number of files lfs_file_sync and/or lfs_file_opencfg will take more and more time to complete, finally generating a HW WDT exception.

[Aug 20 13:35:30.128] lfs_trace:2314: lfs_file_opencfg(0x3fff12d0, 0x3ffefd00, "file58", 502, 0x40201148 {.buffer=0x0, .attrs=0x0, .attr_count=0})
[Aug 20 13:35:40.524] lfs_trace:2450: lfs_file_opencfg -> 0
...
[Aug 20 13:35:48.485] lfs_trace:2659: lfs_file_sync(0x3fff12d0, 0x3ffefd00)
[Aug 20 13:36:02.955] lfs_trace:2710: lfs_file_sync -> 0
...
[Aug 20 13:36:07.345] lfs_trace:2659: lfs_file_sync(0x3fff12d0, 0x3ffefd00)
[Aug 20 13:36:33.957]
[Aug 20 13:36:33.957] HW WDT @ 0x40004cf9

lawrencerb · August 20, 2019, 2:07pm

Thanks for the response
I’ve cut the piece of my code that stores strings in files:

  static int file_counter = 1;
  static int four_objects = 1;
  FILE * pFile;

  //Buf to load a Json Object
  char buf[400];
  struct json_out out = JSON_OUT_BUF(buf, sizeof(buf));
  json_printf(&out, "{temp1: %d, temp2: NULL, hum1: NULL, hum2: NULL, luminosityis: 50, json_filler: 42, json_filler: TRUE, another_json_filler: FALSE, another_json_filler: yes, the_last_json_filler_object: No, this_is_the_last: YES}", file_counter);

  //Script to save a file
  char my_file_name[20] = "";
  sprintf(my_file_name, "/data/saved%d.txt", file_counter);
  pFile = fopen(my_file_name, "a");
  fputs(buf, pFile);
  fputs("\n", pFile);
  fclose(pFile);
  LOG(LL_INFO, ("saved file: %d", file_counter));

  //Logic counter;
  four_objects++;
  if (four_objects == 5){
    file_counter++;
    four_objects = 1;
  }

I’ve tried running a code that only keep files in the flash memmory and it turned out OK
But as I run it in my main project code, system crashes at the file 78:
Mongoose OS Tool
Mongoose OS Tool

[Aug 20 10:56:06.506] HW WDT @ 0x402a5490 
[Aug 20 10:56:06.514] A0: 0x402a59cf A1: 0x3ffff680 A2: 0x0000000e A3: 0x00000000 
[Aug 20 10:56:06.514] A4: 0x00000001 A5: 0x00000000 A6: 0x00000001 A7: 0x00000002 
[Aug 20 10:56:06.514] A8: 0x00000000 A9: 0x00000080 A10: 0x00000600 A11: 0x00000002
 [Aug 20 10:56:06.514] A12: 0x3fff2584 A13: 0x0000000e A14: 0x00000041 A15: 0x0000000f
[Aug 20 10:56:06.514] 
[Aug 20 10:56:06.514] (exc SP: 0x3ffff4c0)

Tried saving another file and other dog’s positions indicate failure:
HW WDT @ 0x401071d4
HW WDT @ 0x402a566d

Called: “mos ls /data -l”, after devices restores from dump:

Mongoose OS Tool

$ mos ls /data -l saved1.txt 1706 saved10.txt 1715 saved11.txt 1715 saved12.txt 1715 saved13.txt 1715 saved14.txt 1715 saved15.txt 1715 saved16.txt 1715 saved17.txt 1715 saved18.txt 1715 saved19.txt 1715 saved2.txt 1715 saved20.txt 1715 saved21.txt 1715 saved22.txt 1715 saved23.txt 1715 saved24.txt 1715 saved25.txt 1715 saved26.txt 1715 saved27.txt 1715 saved28.txt 1715 saved29.txt 1715 saved3.txt 1715 saved30.txt 1715 saved31.txt 1715 saved32.txt 1715 saved33.txt 1715 saved34.txt 1715 saved35.txt 1715 saved36.txt 1715 saved37.txt 1715 saved38.txt 1715 saved39.txt 1715 saved4.txt 1715 saved40.txt 1715 saved41.txt 1715 saved42.txt 1714 saved43.txt 1710 saved44.txt 1712 saved45.txt 1710 saved46.txt 1710 saved47.txt 1710 saved48.txt 1710 saved49.txt 1710 saved5.txt 1715 saved50.txt 1710 saved51.txt 1710 saved52.txt 1710 saved53.txt 1710 saved54.txt 1710 saved55.txt 1710 saved56.txt 1710 saved57.txt 1710 saved58.txt 1710 saved59.txt 1710 saved6.txt 1715 saved60.txt 1710 saved61.txt 1710 saved62.txt 1710 saved63.txt 1710 saved64.txt 1710 saved65.txt 1710 saved66.txt 1710 saved67.txt 1710 saved68.txt 1710 saved69.txt 1710 saved7.txt 1715 saved70.txt 1710 saved71.txt 1710 
saved72.txt 1710 
saved73.txt 1710 
saved74.txt 1710 
saved75.txt 1710 
saved76.txt 1710 
saved77.txt 1710
 saved78.txt 1710 
saved79.txt 0
Command completed.

I can see it manages to create File79, but fails to populate it with any content.

Console also indicates free space after boot:

[Aug 20 11:05:08.024] mgos_spi_gpio.c:72      SPI GPIO init ok (MISO: 12, MOSI: 13, SCLK: 14; CS0/1/2: 15/0/0)
[Aug 20 11:05:08.024] mgos_vfs_dev.c:73       spif0: spi_flash ({"cs": 0, "freq": 104000000}), size 2097152
[Aug 20 11:05:08.024] mgos_vfs.c:147          /data: LFS @ spif0, opts {"bs": 4096}
[Aug 20 11:05:11.989] mgos_vfs.c:320          /data: size 2097152, used: 327680, free: 1769472
[Aug 20 11:05:11.989] mgos_rpc_channel_mq:203 0x3fff3194 esp8266_CCDFD4/rpc
[Aug 20 11:05:11.989] mgos_rpc_channel_ua:313 0x3fff3714 UART0

Any insights?

nliviu · August 20, 2019, 4:59pm

According to what I saw, the file is created, the data written, but fclose fails in lfs_file_sync which is called by lfs_file_close.

Unfortunately, there is nothing to do to avoid this issue on ESP8266. Move to ESP32 where I didn’t see that. At least not before creating 100 files.

LE. Try spiffs, it shouldn’t trigger a WDT for 2M.

lawrencerb · August 20, 2019, 6:23pm

Found a solution that suits my needs:

(Side note: I’ve tried using SPIFFS, but it was somehow slower than LFS, like crash causing slow)

I needed a fail safe external memory for this project I’m working on.

I used LFS with 2 files: a “saved.txt” and a “counter.txt” to keep track of the position of the stream if devices reboots or anything.

Data keeps piling up to the point of saved.txt having around ~<1.7MB. LFS seems to function properly when all the data is written in a single file.

To which it points back to the start of the file, using functions fseek and ftell to keep track of the stream and openning the file with fopen("/data/saved.txt" “r+”);

To publish all the memory I’ve done in pieces, kinda the same way. I know it’s impossible to discharge 1.7MB of json objects all at once, so i used a nice mongoose os timer function to handle this .

So far it’s works properly, a little bit of a delay when “overwriting” if file reaches max size. If anyone needs the piece of C code, just let me know.

Thank you guys for helping