[testing phase] ESP8266, Migrating from firmware built with mos 2.17 and earlier with extra fs at 0x300000 to mos 2.18

hdmt-hock · August 26, 2020, 3:51pm

My goal is:
To build the working esp8266 firmware from mos 2.17 to the mos release 2.18
My actions are:
update mos from 2.17 to 2.18 and rebuild the firmware
The result I see is:
When the firmware flash over uart (active_slot 0), it works

mgos_vfs_dev.c:72       data: part ({"dev": "sfl0", "offset": 0x300000, "size": 0x80000}), size 524288
mgos_vfs.c:173          /chi: SPIFFS @ data, opts {"bs": 4096, "ps": 128, "es": 4096}
mgos_vfs.c:344          /chi: size 480561, used: 257070, free: 223491
mgos_http_server.c:336  HTTP server started on [80]

but when ota over curl with the same firmware goes into infinite reset,

mgos_vfs_dev.c:72       data: part ({"dev": "sfl0", "offset": 0x300000, "size": 0x80000}), size 524288
mgos_vfs.c:173          /chi: SPIFFS @ data, opts {"bs": 4096, "ps": 128, "es": 4096}
mgos_vfs_fs_spiffs.:151 addr 0x0 size 524288 bs 4096 ps 128 es 4096 nfd 10 encr 0 => -10025
mgos_vfs_fs_spiffs.:204 SPIFFS mount failed
mgos_vfs.c:182          FS SPIFFS {"bs": 4096, "ps": 128, "es": 4096}: mount failed
mgos_deps_init.c:130    fstab init failed
esp_main.c:192          MG init error: -32
esp_ota_backend.c:609   Update failed, reverting to ROM 0
esp_ota_backend.c:590   cur 0 prev 0 fwu 0
mgos_vfs.c:1026         Unmounting filesystems
esp_main.c:158          SDK: bcn 0
esp_main.c:158          SDK: del if1
esp_main.c:158          SDK: scandone
esp_main.c:158          SDK: del if0
esp_main.c:158          SDK: usl
esp_main.c:158          SDK: mode : null
mgos_system.c:42        Restarting

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

My expectation & question is:
to migrate from 2.17 to 2.18 to hopefully solve another issue where esp8266 constantly lost wifi connection and never reconnect. Quite similar an issues with Shelly homekit in mongoose os.

below is part of the mos.yml that was working in 2.17

cdefs:
  LFS_MIGRATE: 0
  MGOS_LFS1_COMPAT: 1

config_schema:
  - ["devtab.dev0.name", "data"]
  - ["devtab.dev0.type", "part"]
  - ["devtab.dev0.opts", '{"dev": "sfl0", "offset": 0x300000, "size": 0x80000}']
  - ["fstab.fs0.dev", "data"]
  - ["fstab.fs0.type", "SPIFFS"]
  - ["fstab.fs0.opts", '{"bs": 4096, "ps": 128, "es": 4096}']
  - ["fstab.fs0.path", "/chi"]
  - ["fstab.fs0.created", true]

libs:
  - origin: https://github.com/mongoose-os-libs/fstab

hdmt-hock · August 28, 2020, 2:35pm

Hello

Anyone to help or give some hint? Critical to have this version mos 2.18 running as the unit are out in field having problem connecting or reconnecting to wifi.

I was hoping that with mos 2.18, the wifi connectivity is resolved, but with the filesystem init error after OTA is a no go as system reset infinitely.

nliviu · August 31, 2020, 11:29am

Somewhere between 2.17.0 and 2.18.0 the layout of esp8266 changed

hdmt-hock · September 2, 2020, 12:24pm

thank you @nliviu

I see the difference.

Address location :

mos	2.17	2.18
FS0	0x008000	0x008000
FS1	0x048000	0x300000
free 0	0x880000	0x048000
free 1	0x200000 ??? should be 0x300000?	0x340000

Based on this info, I reassign some number in mos.yml:

config_schema:
  - ["devtab.dev0.name", "data"]
  - ["devtab.dev0.type", "part"]
  - ["devtab.dev0.opts", '{"dev": "sfl0", "offset": 0x340000, "size": 0x80000}']
  - ["fstab.fs0.dev", "data"]
  - ["fstab.fs0.type", "SPIFFS"]
  - ["fstab.fs0.opts", '{"bs": 4096, "ps": 128, "es": 4096}']
  - ["fstab.fs0.path", "/chi"]
  - ["fstab.fs0.created", true]

build_vars:
# https://community.mongoose-os.com/t/how-to-generate-spiffs-image-externally/258/11
  APP_EXTRA_FW_PARTS: fs_data:type=fs,src=src/chi.bin,addr=0x340000

and using the (docker run)[How to generate SPIFFS image externally?]

p=$(pwd)
docker run -v $p:$p -t mgos/esp8266-build:2.2.1-1.5.0-r5 mkspiffs8 -s 524288 -p 128 -b 4096 $p/chi -f $p/src/chi.bin

the path “mgos/esp8266-build:2.2.1-1.5.0-r5”, Is this the latest version?

This solution seems working after a quick test, Now I will need to deploy for full testing.
Please let me know if the above changes may interfere any current and/or future memory alignment.

Regards
hock

Update :
See post 7 below for the OTA update failure.

The solution only works if the filesystem is created before the OTA. The question is how? How to migrate FS from 0x300000 to 0x340000 before OTA? The Filesystem is 512kb. Current data in the filesystem is a bit less than the 256kb.

Any solution?

nliviu · September 2, 2020, 12:30pm

I don’t think it’s necessary to rebuild the image file, but FYI the current SDK version is here

I can’t guarantee that this layout will not change in the future. Watch the commits in this repo

PS. Don’t forget to use mkspiffs8 in the command to create the image file.

hdmt-hock · September 2, 2020, 1:53pm

Seems backward compatibility is broken.

With the firmware built with mos 2.18 in any active_slot, I tried to flash an older firmware that has the filesystem of mos 2.17 over ota. I can neither update to slot 1 nor slot 0, it revert to the firmware built with mos 2.18.

Based on this behaviour, should this be consider a major change?
Well, I should change my app major version to reflect the behaviour.

hdmt-hock · September 2, 2020, 4:13pm

FW1 : firmware built with mos 2.17 (user file system at addr 0x300000)
FW2 : firmware built with mos 2.18 (user file system at addr 0x340000)

During one test, we load FW1 via uart with FULL CHIP ERASE and reset

$ mos flash fw.zip --esp-erase-chip

My understanding is via uart the filesystem is created.

then we upload via OTA FW2, the firmware revert to the FW1 after reset.

It seems that the user file system is handle when flash via uart and not via ota.

HELP!!!

nliviu · September 4, 2020, 10:16am

It appears that OTA update writes only the fw and fs sections in manifest.json regardless the version.

Snippet from 1.0.1 to 1.0.1 update:

[Sep  4 12:14:45.636] mgos_ota_core.c:330     File name: extra_fw_parts-1.0.1/chi.bin
[Sep  4 12:14:45.642] mgos_ota_core.c:333     Method 0 cs 524288 ucs 524288
[Sep  4 12:14:45.647] mgos_ota_core.c:353     File name to use: chi.bin
[Sep  4 12:14:45.650] mgos_ota_core.c:362     File size: 524288
[Sep  4 12:14:45.656] mgos_ota_core.c:366     General flag=0x0
[Sep  4 12:14:45.658] mgos_ota_core.c:370     File CRC32: 0xd51bce0f
[Sep  4 12:14:45.667] esp_ota_backend.c:222   Not interesting: chi.bin
[Sep  4 12:14:45.670] mgos_ota_core.c:675     chi.bin not claimed, skipping

hdmt-hock · September 4, 2020, 11:50am

Reference : How to generate SPIFFS image externally?

Till mos 2.17, the solution on extra fs at addr 0x300000 works for us since the location does not change after OTA. We manually update the extra fs after OTA if there is something new since OTA does not covers the update.

with mos 2.18, we cannot use the location 0x300000 as fs1 (assuming the filesystem for app1) is assigned there. For that we need to shift the extra fs from 0x300000 to 0x340000.

Rebuilding the situation:
FW1.0 : built with mos 2.17 with extra fs in addr 0x300000
FW1.1 : built with mos 2.17 with extra fs in addr 0x300000
FW2.0 : built with mos 2.18 with extra fs in addr 0x340000
FW2.1 : built with mos 2.18 with extra fs in addr 0x340000

If FW1.0 is uart clean flash, then OTA FW1.1,
the FW1.1 can start and uses the same extra fs as in FW1.0
The same goes if FW2.0 is uart clean flash, then OTA FW2.1, the FW2.1 can start and uses the extra fs as in FW2.0

But, if FW1.1 need to update to FW2.0 via OTA as device are in the field, the OTA will failed and revert to FW1.1 due to an init error in the filesystem.
Could the error due to the addr 0x300000 already contain the extra fs format? or the addr 0x340000 is not define?

Based on the doucumentation, https://mongoose-os.com/docs/mongoose-os/howtos/fs.md,
we may need to use rpc to call FS.Mkfs at addr 0x340000 prior to OTA. I had no success in doing it.

nliviu · September 4, 2020, 12:32pm

Mounting the extra filesystem after OTA update fails with error -10025 (SPIFFS_ERR_NOT_A_FS) because OTA update does not write the filesystem image file.

FS.Mkfs apparently fails the first time, but running it again shows

[Sep  4 15:16:41.343] mg_rpc.c:293            FS.Mkfs via WS_in 192.168.0.99:30283
[Sep  4 15:16:41.350] mgos_vfs.c:109          Create SPIFFS (dev 0x3fff1984, opts {"addr":3407872,"size":524288})
[Sep  4 15:16:41.368] mgos_vfs_fs_spiffs.:172 There is a valid FS already, reformatting

After that OTA update with fw built with 2.18.0 will mount the filesystem at 0x340000 and we are able to add files.

hdmt-hock · September 4, 2020, 2:01pm

Redo the procedure

UART flash FW1 with full chip erase
$ mos flash fw1.zip --esp-erase-chip
Reset chip
call FS.Mkfs 2x
1st call
$ mos --port ws://10.10.10.1/rpc call FS.Mkfs '{"dev_type": "sysflash", "fs_type": "SPIFFS", "fs_opts": "{\"addr\": 3407872, \"size\": 524288"}'

mg_rpc.c:293            FS.Mkfs via WS_in 10.10.10.2:61340
mgos_vfs.c:109          Create SPIFFS (dev 0x3fff2b54, opts {"addr": 3407872, "size": 524288)
mgos_vfs_fs_spiffs.:156 addr 0x340000 size 524288 bs 4096 ps 256 es 4096 nfd 10 encr 0 => -10025

2nd call
$ mos --port ws://10.10.10.1/rpc call FS.Mkfs '{"dev_type": "sysflash", "fs_type": "SPIFFS", "fs_opts": "{\"addr\": 3407872, \"size\": 524288"}'

mg_rpc.c:293            FS.Mkfs via WS_in 10.10.10.2:61415
mgos_vfs.c:109          Create SPIFFS (dev 0x3fff2b54, opts {"addr": 3407872, "size": 524288)
mgos_vfs_fs_spiffs.:172 There is a valid FS already, reformatting

curl OTA update
$ curl -i -v -F filedata=@fw2.zip http://10.10.10.1/update

mgos_http_server.c:180  0x3fff29bc HTTP connection from 10.10.10.2:61518
mg_rpc.c:293            Config.Get via WS_in 10.10.10.2:61518
mgos_mongoose.c:66      New heap free LWM: 24856
mg_rpc.c:293            Config.Set via WS_in 10.10.10.2:61518
mgos_sys_config.c:174   Saved to conf9.json
mgos_utils.c:38         Rebooting in 500 ms
mgos_vfs.c:1026         Unmounting filesystems
station: dc:a4:ca:f2:f2:db leave, AID = 1
rm 1
bcn 0
del if1
del if0
usl
mode : null
mgos_system.c:42        Restarting

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

System restart immediately, seems no ota executed, also from here fw reset continuously

hdmt-hock · September 4, 2020, 2:52pm

Another try with OTA going via http server

UART flash FW1 with full chip erase
$ mos flash 464fw.zip --esp-erase-chip
Reset chip
connect to wifi sta
$ mos --port ws://$ap_ip/rpc wifi $ssid $pass
license the module
$ mos --port ws://$sta:ip/rpc license
call FS.Mkfs 2x
$ mos --port ws://$sta_ip/rpc call FS.Mkfs '{"dev_type": "sysflash", "fs_type": "SPIFFS", "fs_opts": "{\"addr\": 3407872, \"size\": 524288"}'

mg_rpc.c:293            FS.Mkfs via WS_in 192.168.0.7:56019
mgos_vfs.c:109          Create SPIFFS (dev 0x3fff619c, opts {"addr": 3407872, "size": 524288)
mgos_vfs_fs_spiffs.:156 addr 0x340000 size 524288 bs 4096 ps 256 es 4096 nfd 10 encr 0 => -10025
mgos_http_server.c:180  0x3fff5fec HTTP connection from 192.168.0.7:56050
mg_rpc.c:293            FS.Mkfs via WS_in 192.168.0.7:56050
mgos_vfs.c:109          Create SPIFFS (dev 0x3fff619c, opts {"addr": 3407872, "size": 524288)
mgos_vfs_fs_spiffs.:172 There is a valid FS already, reformatting

call OTA update
$ `mos --port ws://$sta_ip/rpc call OTA.update ‘{“url”:“http://some_server.com/…/500fw.zip”}’

mgos_http_server.c:180  0x3fff5fec HTTP connection from 192.168.0.7:56115
mg_rpc.c:293            OTA.Update via WS_in 192.168.0.7:56115
mgos_ota_http_clien:270 Update URL: http://some_server.com/.../500fw.zip
mgos_ota_core.c:253     Starting, timeout 600, commit timeout 0, mem 32544
mgos_ota_core.c:489     FW: chi_smart esp8266 5.00 20200902-135819/v4.62-13-ge84ffbc-master-dirty
esp_ota_backend.c:152   Slot 1, FW: chi_smart.bin -> 0x200000, FS fs.bin -> 0x48000
mgos_ota_core.c:506     0.16% total, chi.bin 512 of 524288
mgos_ota_core.c:506     9.35% total, chi.bin 136704 of 524288
mgos_ota_core.c:506     19.21% total, chi.bin 282624 of 524288
mgos_ota_core.c:506     29.09% total, chi.bin 429056 of 524288
esp_ota_backend.c:245   Start writing chi_smart.bin (689648) @ 0x200000
mgos_ota_core.c:506     35.56% total, chi_smart.bin 512 of 689648
mgos_ota_core.c:506     42.79% total, chi_smart.bin 107520 of 689648
mgos_ota_core.c:506     49.84% total, chi_smart.bin 211968 of 689648
mgos_ota_core.c:506     57.48% total, chi_smart.bin 325120 of 689648
mgos_ota_core.c:506     64.39% total, chi_smart.bin 427520 of 689648
mgos_ota_core.c:506     71.27% total, chi_smart.bin 529408 of 689648
mgos_ota_core.c:506     78.77% total, chi_smart.bin 640512 of 689648
esp_ota_backend.c:282   Write finished, checksum ok
mgos_ota_core.c:506     82.11% total, esp_init_data_default_v08.bin 128 of 128
esp_ota_backend.c:245   Start writing fs.bin (262144) @ 0x48000
mgos_ota_core.c:506     82.14% total, fs.bin 512 of 262144
mgos_ota_core.c:506     89.47% total, fs.bin 109056 of 262144
mgos_ota_core.c:506     96.73% total, fs.bin 216576 of 262144
esp_ota_backend.c:282   Write finished, checksum ok
mgos_ota_core.c:506     99.85% total, rboot.bin 512 of 2320
mgos_ota_core.c:629     Reached the end of archive
esp_ota_backend.c:330   New rboot config: prev_rom: 0, current_rom: 1 current_rom addr: 0x200000, current_rom size: 689648, current_fs addr: 0x48000, current_fs size: 262144
mgos_ota_core.c:811     Update finished, result 1 (Update applied, rebooting)
mgos_ota_core.c:852     Update requires reboot
mgos_utils.c:38         Rebooting in 500 ms
mgos_vfs.c:1026         Unmounting filesystems
state: 5 -> 0 (0)
rm 0
bcn 0
del if1
del if0
usl
mode : null
mgos_system.c:42        Restarting

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x40100000, len 1540, room 16 
tail 4
chksum 0x82
load 0x3ffe8000, len 748, room 4 
tail 8
chksum 0x65
csum 0x65

rBoot v1.2.1-cesanta1 - richardaburton@gmail.com
Flash Size:   32 Mbit
Flash Mode:   DIO
Flash Speed:  80 MHz
rBoot Option: Big flash

First boot, attempt 0
Boot is unconfirmed
Booting rom 1 (0x200000).
úbl`.rÚn.lphå‚.Ï.ÇnÏírÇÚn|Ï.l.lll`..bí...û|.rí.lún‡.Çn‚..l`.å‚rílål.å...˛[ãïÅ: softAP(82:7d:3a
esp_main.c:178          chi_smart 5.00 (20200902-135819/v4.62-13-ge84ffbc-master-dirty)
esp_main.c:180          Mongoose OS 2.18.0 (20200902-135819/2.18.0-g24c520d)
esp_main.c:181          CPU: ESP8266EX, 160 MHz, RAM: 51768 total, 48660 free
esp_main.c:185          SDK 2.2.2-dev(a58da79); flash: 4M
esp_exc.c:211           Reset cause: 4 (soft reset)
mg_lwip_ev_mgr.c:75     Mongoose 6.18, LwIP 2.1.2
mg_ssl_if_mbedtls.c:56  mbed TLS 2.16.6-cesanta1
mgos_vfs_dev.c:72       sfl0: sysflash (), size 4194304
mgos_vfs_dev.c:72       root: part ({"dev": "sfl0", "offset": 294912, "size": 262144}), size 262144
mgos_vfs.c:173          /: SPIFFS @ root, opts {"bs":4096,"ps":256,"es":4096}
mgos_vfs.c:344          /: size 233681, used: 33885, free: 199796
esp_ota_backend.c:480   Mounting old FS: 262144 @ 0x8000
mgos_vfs_dev.c:72       oldroot: part ({"dev": "sfl0", "offset": 32768, "size": 262144}), size 262144
mgos_vfs.c:173          /old: SPIFFS @ oldroot, opts {bs:4096,ps:256,es:4096}
mgos_vfs.c:344          /old: size 233681, used: 34889, free: 198792
mgos_file_utils.c:36    Copying /old/conf9.json -> //conf9.json
mgos_file_utils.c:36    Copying /old/conf1.json -> //conf1.json
mgos_sys_config.c:232   Loading conf1.json
mgos_sys_config.c:232   Loading conf9.json
mgos_sys_config.c:366   MAC: 82:7d:3a:3d:bf:1f
mgos_sys_config.c:374   WDT: 30 seconds
mgos_core.c:104         Setting TZ to 'AR3'
esp_main.c:158          SDK: bcn 0
esp_main.c:158          SDK: del if1
esp_main.c:158          SDK: usl
esp_main.c:158          SDK: mode : null
mgos_wifi.c:88          WiFi STA: Using config 0 (ssid)
mgos_wifi.c:464         WiFi mode: AP+STA
esp_wifi.c:118          WiFi mode: AP
esp_main.c:158          SDK: mode : softAP(82:7d:3a:3d:bf:1f)
esp_main.c:158          SDK: add if1
esp_main.c:158          SDK: dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw
esp_main.c:158          SDK: :192.168.4.1)
esp_main.c:158          SDK: bcn 100
esp_wifi.c:317          WiFi AP: SSID CHI_3DBF1F, channel 6
esp_main.c:158          SDK: bcn 0
esp_main.c:158          SDK: del if1
esp_main.c:158          SDK: usl
esp_main.c:158          SDK: add if1
esp_main.c:158          SDK: dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw
esp_main.c:158          SDK: :192.168.4.1)
esp_main.c:158          SDK: bcn 100
esp_wifi.c:363          WiFi AP IP: 10.10.10.1/255.255.255.0 gw 10.10.10.1, DHCP range 10.10.10.2 - 10.10.10.11
esp_wifi.c:118          WiFi mode: AP+STA
esp_main.c:158          SDK: mode : sta(80:7d:3a:3d:bf:1f) + softAP(82:7d:3a:3d:bf:1
esp_main.c:158          SDK: f)
esp_main.c:158          SDK: add if0
mgos_wifi.c:270         WiFi STA: Connecting to ssid
mgos_vfs_dev.c:72       data: part ({"dev": "sfl0", "offset": 0x340000, "size": 0x80000}), size 524288
mgos_vfs.c:173          /chi: SPIFFS @ data, opts {"bs": 4096, "ps": 128, "es": 4096}
mgos_vfs_fs_spiffs.:151 addr 0x0 size 524288 bs 4096 ps 128 es 4096 nfd 10 encr 0 => -10025
mgos_vfs_fs_spiffs.:204 SPIFFS mount failed
mgos_vfs.c:182          FS SPIFFS {"bs": 4096, "ps": 128, "es": 4096}: mount failed
mgos_deps_init.c:138    fstab init failed
esp_main.c:192          MG init error: -32
esp_ota_backend.c:609   Update failed, reverting to ROM 0
esp_ota_backend.c:590   cur 0 prev 0 fwu 0
mgos_vfs.c:1026         Unmounting filesystems
esp_main.c:158          SDK: bcn 0
esp_main.c:158          SDK: del if1
esp_main.c:158          SDK: scandone
esp_main.c:158          SDK: del if0
esp_main.c:158          SDK: usl
esp_main.c:158          SDK: mode : null
mgos_system.c:42        Restarting 
 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x40100000, len 1540, room 16

after OTA update, the new fw report SPIFFS mount failed, fstab init failed.
and revert to the old fw and reset continuously with the same SPIFFS mount failed, fstab init failed.

nliviu · September 4, 2020, 5:18pm

You create the filesystem with ps 256 and try to mount it with "ps": 128.
I think we’ve discussed the ps 128 issue in the other thread.

hdmt-hock · September 5, 2020, 1:33am

@nliviu, you are the best.

modifying post 11 step 5 above

call FS.Mkfs 2x
$ mos --port ws://$sta_ip/rpc call FS.Mkfs '{"dev_type": "sysflash", "fs_type": "SPIFFS", "fs_opts": "{\"addr\": 3407872, \"size\": 524288, \"ps\": 128}"}'

I redo the whole process and it works…

As regard to another thread about "ps": 128, I cannot remember as I was too focus with copy and paste from the https://mongoose-os.com/docs/mongoose-os/howtos/fs.md.

As a recommendation, please update the Mongoose OS Documentation to reflect the missing and new information and include the entry DATE and mos version of the documentation. Looking at the github commit is too complication. Example, I still could not get the gh-token and credentials to work by just looking at the commit.

I will do some more test. The main question for me now : How to deploy the mos 2.18 firmware with minimum interaction, i.e. “one click” solution?