How to generate SPIFFS image externally?

Hello,

We would like to generate spiffs image for ESP8266 out of mos build. We tried to use spiffsgen.py from ESP-IDF and https://github.com/igrr/mkspiffs but in both casess mount failed.

We need it for device manufacturing - firmware will be the same for every device but filesystem must differ (will contain diffrent keys and configuration). How can we achieve that?

Thanks in advance.

1 Like

We have managed it with following command:

docker run -v $(pwd):/mnt/data -t mgos/esp8266-build:2.2.1-1.5.0-r4 mkspiffs -s 262144 /mnt/data/fs -f /mnt/data/fs.bin

Is this the correct approach? I am asking because I think it is common to build firmware and filesystem separatley (exclusively) in the production. Please correct me if I am wrong.

1 Like

@Martin_Zdila, Could I ask you how do you flash/mount externally generated filesystem image? I am also working with ESP8266 and found guide only for ESP32
Here is my assumption - is it correct:

  1. mkspiffs used to generate the image
  2. esptool write_flash 0x300000 to flash filesystem image
  3. mos mount - to mount the image

In my case the above steps do not work:

python spiffsgen.py --page-size 128 0x40000 data data.img

esptool --port COM7 --baud 921600 write_flash 0x300000 data.img

mos call FS.Mount '{"path":"/http","dev_name":"sfl0","dev_type":"sysflash","fs_type":"SPIFFS","fs_opts":"{\"addr\":3145728,\"size\":262144‬,\"bs\":4096,\"ps\":128,\"es\":4096}"}'
Error: /go/src/github.com/mongoose-os/mos/mos/dev/dev_conn_impl.go:171: remote error 500: mount failed

Thanks a loot for response

Try \"offset\":3145728 instead of \"addr\":3145728.

Here is a working example using auto mount (ESP8266):

  • add the files in extra-fs
  • create the image: docker run -v $(pwd):$(pwd) -t mgos/esp8266-build:2.2.1-1.5.0-r5 mkspiffs -s 262144 -p 128 $(pwd)/extra-fs -f $(pwd)/extra.bin (Thank you @Martin_Zdila )
  • copy the image in the sources: directory.
  • add in mos.yml
config_schema:
  - ["devtab.dev0.name", "data"]
  - ["devtab.dev0.type", "part"]
  - ["devtab.dev0.opts", '{"dev": "sfl0", "offset": 0x300000, "size": 0x40000}']
  - ["fstab.fs0.dev", "data"]
  - ["fstab.fs0.type", "SPIFFS"]
  - ["fstab.fs0.opts", '{"bs": 4096, "ps": 128, "es": 4096}']
  - ["fstab.fs0.path", "/html"]
  - ["fstab.fs0.created", true]

libs:
# other libs
  - origin: https://github.com/mongoose-os-libs/vfs-dev-part
  - origin: https://github.com/mongoose-os-libs/fstab
    

build_vars:
  APP_EXTRA_FW_PARTS: fs_data:type=fs,src=src/extra.bin,addr=0x300000
  • build and flash

Output:

mgos_vfs_dev.c:73       data: part ({"dev": "sfl0", "offset": 0x300000, "size": 0x40000}), size 262144
mgos_vfs.c:147          /html: SPIFFS @ data, opts {"bs": 4096, "ps": 128, "es": 4096}
mgos_vfs.c:320          /html: size 236529, used: 3198, free: 233331
1 Like

Hi @nliviu,

I have tried your example above, but the html content in the “extra-fs” never get to the esp8266 after flash.

Here are the step that I did with some modification from your exmaple.

  1. Create a folder “chi” in esp8266 project, add html files to the “chi” folder
  2. Execute docker to create bin image
docker run -v $(pwd):$(pwd) -t mgos/esp8266-build:2.2.1-1.5.0-r5 mkspiffs -s 524288 $(pwd)/chi -f $(pwd)/src/chi.bin

Output from docker

     FS params: size=524288, bs=4096, ps=256, es=4096
       Adding a.png: 12038
       Adding b.ico: 2238
       Adding c.png: 12785
       Adding index.html: 33285
     FS stats : space total=474641, used=61746, free=412895
  1. add the following from your mos.yml with some modification
config_schema:
  - ["devtab.dev0.name", "data"]
  - ["devtab.dev0.type", "part"]
  - ["devtab.dev0.opts", '{"dev": "sfl0", "offset": 0x300000, "size": 0x80000}']
  - ["fstab.fs0.dev", "data"]
  - ["fstab.fs0.type", "SPIFFS"]
  - ["fstab.fs0.opts", '{"bs": 4096, "ps": 128, "es": 4096}']
  - ["fstab.fs0.path", "/chi"]
  - ["fstab.fs0.create", true]

libs:
  - origin: https://github.com/mongoose-os-libs/fstab
  - origin: https://github.com/mongoose-os-libs/vfs-dev-part

build_vars:
  APP_EXTRA_FW_PARTS: fs_data:type=fs,src=src/chi.bin,addr=0x300000

  • build and flash via uart

Output :

mgos_vfs_dev.c:73       data: part ({"dev": "sfl0", "offset": 0x300000, "size": 0x80000}), size 524288
mgos_vfs.c:147          /chi: SPIFFS @ data, opts {"bs": 4096, "ps": 128, "es": 4096}
mgos_vfs.c:320          /chi: size 480561, used: 0, free: 480561

Obeservation:

  • The “used” is 0 byte!
  • Using ’ mos --port ws://ip/rpc ls chi’, the folder return empty.
  • Using mos put/get command to put some file in the chi folder and read back work ok.

System info:

esp_main.c:174          Mongoose OS 2.16.0 (20191204-145848/2.16.0-g9b779cd)
esp_main.c:178          CPU: ESP8266EX, 160 MHz, RAM: 50720 total, 48084 free
esp_main.c:180          SDK 2.2.1(1247cc5); flash: 4M
esp_exc.c:211           Reset cause: 4 (soft reset)
mg_lwip_ev_mgr.c:93     Mongoose 6.15, LwIP 1.4.1
mg_ssl_if_mbedtls.c:57  mbed TLS 2.16.3-cesanta2
mgos_vfs_dev.c:73       sfl0: sysflash (), size 4194304
mgos_vfs_dev.c:73       root: part ({"dev": "sfl0", "offset": 32768, "size": 262144}), size 262144

Did I miss any steps?

- ["fstab.fs0.create", true] tells the system to create the filesystem if it does not exist.
- ["fstab.fs0.created", true] will not try to create the filesystem.

Hello @nliviu

Thank you for the reply and hint, but now my code goes into infinite reset after changing create to created. I have use lfs lib with compat 1, not sure if that matters. I guess if the concept is correct, I don’t need the migrate and compat flag. At moment, need to work on some other code, will come back next week to review again on this.

cdefs:
  LFS_MIGRATE: 0
  MGOS_LFS1_COMPAT: 1

libs:
  - origin: https://github.com/mongoose-os-libs/vfs-dev-spi-flash
  - origin: https://github.com/mongoose-os-libs/vfs-fs-lfs

Infite reset partial log where reset happen:

mgos_deps_init.c:146    init fstab...
mgos_vfs_dev.c:203      sfl0 refs 3
mgos_vfs_dev.c:73       data: part ({"dev": "sfl0", "offset": 0x300000, "size": 0x80000}), size 524288
mgos_vfs_dev.c:203      data refs 1
mgos_vfs.c:147          /chi: SPIFFS @ data, opts {"bs": 4096, "ps": 128, "es": 4096}
esp_vfs_dev_sysflash:83 read 2 @ 0x30007c => 0
esp_vfs_dev_sysflash:83 read 2 @ 0x30007e => 0
esp_vfs_dev_sysflash:83 read 2 @ 0x30107c => 0
mgos_vfs_fs_spiffs.:155 addr 0x0 size 524288 bs 4096 ps 128 es 4096 nfd 10 encr 0 => -10025
mgos_vfs_fs_spiffs.:203 SPIFFS mount failed
mgos_vfs.c:155          FS SPIFFS {"bs": 4096, "ps": 128, "es": 4096}: mount failed
mgos_vfs_dev.c:203      data refs 1
mgos_deps_init.c:148    fstab init failed
esp_main.c:186          MG init error: -32
mgos_ota_core.c:969     0 0
esp_main.c:200          Init failed: -32
mgos_event.c:135        ev MOS2 triggered 0 handlers
mgos_vfs.c:990          Unmounting filesystems
mgos_vfs_dev.c:203      root refs 1
mgos_vfs_dev.c:203      data refs 0
mgos_vfs_dev.c:203      sfl0 refs 2
mgos_vfs_dev.c:203      root refs 0
mgos_vfs_dev.c:203      sfl0 refs 1
mgos_vfs_dev.c:203      sfl0 refs 0
bcn 0
del if1
scandone
del if0
usl
mode : null
mgos_system.c:42        Restarting

The error -10025 means not a filesystem. Check the command line parameters when you call mkspiffs. Looks like -ps 128 is missing.

Hi @nliviu

Following your info, the -p 128 was missing previously. Now, I have it corrected as below, but 2 files missing after build, flash and reset.

Is there a command to view the content in the spiffs bin?
What happen to the 2 missing files?

Building spiffs “chi.bin”

$ docker run -v $(pwd):$(pwd) -t mgos/esp8266-build:2.2.1-1.5.0-r5 mkspiffs -s 524288 -p 128 $(pwd)/chi -f $(pwd)/src/chi.bin
     FS params: size=524288, bs=4096, ps=128, es=4096
       Adding apple-icon-180x180.png: 12038
       Adding favicon.ico: 2238
       Adding android-icon-192x192.png: 12785
       Adding chi_smart_114x114.png: 8289
       Adding index.html: 33285
     FS stats : space total=480561, used=70479, free=410082
cdefs:
  LFS_MIGRATE: 0
  MGOS_LFS1_COMPAT: 1
  
config_schema:
  - ["devtab.dev0.name", "data"]
  - ["devtab.dev0.type", "part"]
  - ["devtab.dev0.opts", '{"dev": "sfl0", "offset": 0x300000, "size": 0x80000}']
  - ["fstab.fs0.dev", "data"]
  - ["fstab.fs0.type", "SPIFFS"]
  - ["fstab.fs0.opts", '{"bs": 4096, "ps": 128, "es": 4096}']
  - ["fstab.fs0.path", "/chi"]
  - ["fstab.fs0.created", true]

result after build, flash and reset

$ mos --port ws://192.168.0.2/rpc ls chi
android-icon-192x192.png
apple-icon-180x180.png
favicon.ico 

missing 2 files, chi_smart_114x114.png and index.html

I don’t know why using ps in config_schema and -p in the mkspiffs would miss some files with mos ls.
Using

- ["fstab.fs0.opts", '{"bs": 4096, "es": 4096}']

and

docker run -v $(pwd):$(pwd) -t mgos/esp8266-build:2.2.1-1.5.0-r5 mkspiffs -s 524288  $(pwd)/chi -f $(pwd)/src/chi.bin

will show all the files in

mos --port ws://IP/rpc ls -l /chi
1 Like

Hi @nliviu

Thank you for the solution. IT WORKS!!!

I guess someone need to put up an issues in github or API document regarding your finding.

Thank you

hm… I was happy too early… could not add the 6th file

Now I add “chi.hex” file size 148656 bytes in the “chi” folder and uses docker command to generate “chi.bin”

$ docker run -v $(pwd):$(pwd) -t mgos/esp8266-build:2.2.1-1.5.0-r5 mkspiffs -s 524288 -b 4096 $(pwd)/chi -f $(pwd)/src/chi.bin

     FS params: size=524288, bs=4096, ps=256, es=4096
       Adding apple-icon-180x180.png: 12038
       Adding favicon.ico: 2238
       Adding android-icon-192x192.png: 12785
       Adding chi_smart_114x114.png: 8289
       Adding chi.hex: 148656
       Adding index.html: 33536
     FS stats : space total=474641, used=220880, free=253761

But after build, flash via uart, reset

mgos_vfs.c:147          /chi: SPIFFS @ data, opts {"bs": 4096, "es": 4096}
mgos_vfs.c:320          /chi: size 474641, used: 213852, free: 260789

The used size differ!

using $ mos --port ws://ip/rpc ls -l chi

android-icon-192x192.png 12785
apple-icon-180x180.png 12038
chi_smart_114x114.png 8289
favicon.ico 2238
index.html 33536

The file “chi.hex” did not present. However, I could use the “mos put” command to put the chi.hex file in chi folder.

Where could be the issues?

Finally, after having a look here, all the files are there.
Just replace mkspiffs with mkspiffs8 in the docker command.

Hi

Just an update, the info provide works well. Thank you @nliviu

Thanks to your comments, I have found my answer. Thanks very much

In my opinion the best way to generate spiffs images for ESP32 is to use Visual Code with the plug in PlatformIO.

See this tutorial to set everything up

Look at this picture to see where to click to generate image it.
It is super quick and you can find the newly generated bin image in
.pio\build\esp32doit-devkit-v1\spiffs.bin

image