Optimize Private Server Builds

Hi, we’re currently optimizing our Mongoose OS build system to allow the developers to have faster development cycles.

The current machines were used for web development and are equipped with a 4 Core Intel i5 4th Gen CPU running Docker on WLS2 on Windows. They are able to build a complete FW in about 8/10 minutes.

As you can guess this slowed our development and to optimize it we installed on our server (12 Core AMD Ryzen CPU - 64GB RAM) the mgos build server running on Debian. The builds were faster: about 1 and half minute for a complete FW.

Diving deep dive in the building process, we tried to see if it was possible to optimize it even more and we noticed that at every build (even for the same project) the server re-downloads all the libraries and re-compiles the whole firmware. This is not something that happens on local builds, where only the missing libraries of the updated files are re-compiled. (although a whole re-build is triggered if a file is added/removed or mos.yml is updated).

Based on rough calculations, the build time could be optimized to less than 20 seconds if it was able to skip the already downloaded libraries and compiled files.

We managed to skip the re-download by increasing the upload file limit to 50MB (by editing the docker compose and adding --payload-size-limit=52428800) and specifying the deps lib when calling the mos executable:
./mos.exe build --platform ESP32 --deps-dir deps/ --libs-dir deps/ --server http://X.X.X.X:8000 --verbose

Unfortunately this fails at the end (probably because of exec permissions):

...
       Adding api_bitbang.js: 834
       Adding api_dataview.js: 7655
       Adding api_bt_gatt.js: 922
     FS stats : space total=233681, used=105420, free=128261
  GEN   /var/tmp/fwbuild-volumes/2.20.0/apps/iot-demo-fw/esp32/build_contexts/build_ctx_247885459/build/objs/fw_temp/manifest.json
make: execvp: /mongoose-os/tools/mgos_fw_meta.py: Permission denied
/mongoose-os/tools/mk/mgos_fw_meta.mk:24: recipe for target '/var/tmp/fwbuild-volumes/2.20.0/apps/iot-demo-fw/esp32/build_contexts/build_ctx_247885459/build/objs/fw_temp/manifest.json' failed
make: *** [/var/tmp/fwbuild-volumes/2.20.0/apps/iot-demo-fw/esp32/build_contexts/build_ctx_247885459/build/objs/fw_temp/manifest.json] Error 127
make: Leaving directory '/app'
...

I’m not really familiar with Docker, is there a way to not destroy the container that gets created and re-use it for future builds of the same project unless a clean rebuild is triggered?

Could you share an exact command / procedure you’ve used to run your build server, please?

I followed the instructions on mongoose-os/mos - Setup a private build server ( Ubuntu ).

Then I installed Docker on Debian 11.

I edited the file mos/fwbuild/docker-compose.yml (adding the payload-size-limit flag):

version: '2'
services:
  fwbuild-manager:
    image: docker.io/mgos/fwbuild-manager
    volumes:
      - /var/tmp/fwbuild-volumes:/var/tmp/fwbuild-volumes
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/acme-challenge:/acme-challenge
    ports:
      - "8000:8000"
    command: >-
      --logtostderr --v=2
      --volumes-dir=/var/tmp/fwbuild-volumes
      --port=8000
      --payload-size-limit=52428800
      --acme-challenge-dir=/acme-challenge
      --image-pull-interval=0  # For using custom images during local development

Then run it using:

$ sudo docker compose up

My first thought is that, by uploading all the deps from Windows, the executable flag is not present and the script mgos_fw_meta.py cannot be executed. (It’s just a guess, not really sure even if this file gets uploaded from the host machine).

Update: Tried on Ubuntu Server 22.04.1 LTS with the same result.

Update 2: Deleting deps/modules/mongoose-os folder on the host machine allows the build to succeed. I suppose it gets downloaded on the server with the correct executable permissions.

Update 3: Adding to the file deps/modules/mongoose-os/tools/mk/mgos_fw_meta.mk the chmod commands make the process work:

...
$(FW_ZIP): $(FW_MANIFEST) $(FW_META_CMD)
	$(vecho) "ZIP   $@"
	$(Q) chmod +x $(FW_META_CMD) # <---- this makes the script executable
	$(Q) $(FW_META_CMD) create_fw \
...
$(FW_MANIFEST): $(FW_META_CMD)
	$(vecho) "GEN   $(FW_MANIFEST)"
	$(Q) chmod +x $(FW_META_CMD) # <---- this makes the script executable
	$(Q) $(FW_META_CMD) create_manifest \
...

This speeds up the process but the fw is still completely re-built.

Not sure how relevant this is to you, but my builds increased dramatically removing docker from local builds: Building locally without docker

Thanks for the suggestion! If it was for a single user this would have been perfect but we are looking for a centralized solution.


@cpq After digging into the code for the mos tool I found something that might provide a solution. As I understand it, each server build gets assigned a random build_ctx_xxxxxxx folder. In our use case we would like to keep the context the same.
It should be possible by placing a build_context.txt file with the context name in the build/gen folder. (See build_remote.go#L284) this will be added to the form POST data and the server will use said context as a build directory.
I’ve tried it but unfortunately without any success. Am I doing something wrong here?

Update: It works! It turns our, the whole build/gen folder needs to be in place.
Steps:

  1. Run a server build and wait until it completes.
  2. Copy the folder project_dir/build/gen to another place (e.g. project_dir/gen)
  3. Now we must get the timing correctly: run a server build and run a command to copy the folder project_dir/gen into project_dir/build/gen. Why? This needs to be after mos has been called because it empties the build folder.

Build times reduced from 8 minutes to:

real    0m11.853s
user    0m0.000s
sys     0m0.016s

Would it be possible to have a CLI flag to skip the deletion of the files the build folder? This would eliminate the need to get the timings right. See GitHub Issue #73