Hi, we’re currently optimizing our Mongoose OS build system to allow the developers to have faster development cycles.
The current machines were used for web development and are equipped with a 4 Core Intel i5 4th Gen CPU running Docker on WLS2 on Windows. They are able to build a complete FW in about 8/10 minutes.
As you can guess this slowed our development and to optimize it we installed on our server (12 Core AMD Ryzen CPU - 64GB RAM) the mgos build server running on Debian. The builds were faster: about 1 and half minute for a complete FW.
Diving deep dive in the building process, we tried to see if it was possible to optimize it even more and we noticed that at every build (even for the same project) the server re-downloads all the libraries and re-compiles the whole firmware. This is not something that happens on local builds, where only the missing libraries of the updated files are re-compiled. (although a whole re-build is triggered if a file is added/removed or mos.yml is updated).
Based on rough calculations, the build time could be optimized to less than 20 seconds if it was able to skip the already downloaded libraries and compiled files.
We managed to skip the re-download by increasing the upload file limit to 50MB (by editing the docker compose and adding --payload-size-limit=52428800) and specifying the deps lib when calling the mos executable: ./mos.exe build --platform ESP32 --deps-dir deps/ --libs-dir deps/ --server http://X.X.X.X:8000 --verbose
Unfortunately this fails at the end (probably because of exec permissions):
I’m not really familiar with Docker, is there a way to not destroy the container that gets created and re-use it for future builds of the same project unless a clean rebuild is triggered?
I edited the file mos/fwbuild/docker-compose.yml (adding the payload-size-limit flag):
version: '2'
services:
fwbuild-manager:
image: docker.io/mgos/fwbuild-manager
volumes:
- /var/tmp/fwbuild-volumes:/var/tmp/fwbuild-volumes
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/acme-challenge:/acme-challenge
ports:
- "8000:8000"
command: >-
--logtostderr --v=2
--volumes-dir=/var/tmp/fwbuild-volumes
--port=8000
--payload-size-limit=52428800
--acme-challenge-dir=/acme-challenge
--image-pull-interval=0 # For using custom images during local development
Then run it using:
$ sudo docker compose up
My first thought is that, by uploading all the deps from Windows, the executable flag is not present and the script mgos_fw_meta.py cannot be executed. (It’s just a guess, not really sure even if this file gets uploaded from the host machine).
Update: Tried on Ubuntu Server 22.04.1 LTS with the same result.
Update 2: Deleting deps/modules/mongoose-os folder on the host machine allows the build to succeed. I suppose it gets downloaded on the server with the correct executable permissions.
Update 3: Adding to the file deps/modules/mongoose-os/tools/mk/mgos_fw_meta.mk the chmod commands make the process work:
...
$(FW_ZIP): $(FW_MANIFEST) $(FW_META_CMD)
$(vecho) "ZIP $@"
$(Q) chmod +x $(FW_META_CMD) # <---- this makes the script executable
$(Q) $(FW_META_CMD) create_fw \
...
$(FW_MANIFEST): $(FW_META_CMD)
$(vecho) "GEN $(FW_MANIFEST)"
$(Q) chmod +x $(FW_META_CMD) # <---- this makes the script executable
$(Q) $(FW_META_CMD) create_manifest \
...
This speeds up the process but the fw is still completely re-built.
Thanks for the suggestion! If it was for a single user this would have been perfect but we are looking for a centralized solution.
@cpq After digging into the code for the mos tool I found something that might provide a solution. As I understand it, each server build gets assigned a random build_ctx_xxxxxxx folder. In our use case we would like to keep the context the same.
It should be possible by placing a build_context.txt file with the context name in the build/gen folder. (See build_remote.go#L284) this will be added to the form POST data and the server will use said context as a build directory.
I’ve tried it but unfortunately without any success. Am I doing something wrong here?
Update: It works! It turns our, the whole build/gen folder needs to be in place.
Steps:
Run a server build and wait until it completes.
Copy the folder project_dir/build/gen to another place (e.g. project_dir/gen)
Now we must get the timing correctly: run a server build and run a command to copy the folder project_dir/gen into project_dir/build/gen. Why? This needs to be after mos has been called because it empties the build folder.
Build times reduced from 8 minutes to:
real 0m11.853s
user 0m0.000s
sys 0m0.016s
Would it be possible to have a CLI flag to skip the deletion of the files the build folder? This would eliminate the need to get the timings right. See GitHub Issue #73