building from source task #65

pxp9 · 2025-12-27T17:58:46Z

the purpose from this task is to make easy to test Elixir AtomVM with main branch code from the AtomVM repository for ESP32.

this task requires idf.py installed or docker for compiling the AtomVM from source.

feel free to comment any issue in the implementation as well.

Thank you for your attention.

Workflow with custom build

mix atomvm.esp32.build --atomvm-path ./_build/atomvm_source/AtomVM/ --use-docker --chip esp32s3

mix atomvm.esp32.install --image ./_build/atomvm_source/AtomVM/src/platforms/esp32/build/atomvm-esp32s3.img

Using ESP-IDF Docker container

UncleGrumpy

This is a very cool feature! One that has been on my todo list to add to the atomvm_rebar3_plugin for a while. I work much more in Erlang, so my Elixir is not that great, but I did spot a few problems, and have some suggestions on improving usability.

lib/mix/tasks/esp32.build.ex

UncleGrumpy · 2025-12-28T02:23:58Z

It might also be good to consider allowing a ESP-IDF docker image. I suspect many more Elixir developers will already have docker installed (or at least be more familiar with it than ESP-IDF), so it would be extra cool to pull in a docker image of ESP-IDF if they don't already have it installed.

pxp9 · 2025-12-28T10:28:11Z

Hey @UncleGrumpy , when i try to build AtomVM , the build fails because we are using the the stat struct but the header <sys/stat.h> is missing, is there something i need to add in the args of cmake to build it correctly or is this a bug in AtomVM main branch build ?

[913/1054] Building C object esp-idf/avm_sys/CMakeFiles/__idf_avm_sys.dir/sys.c.obj
  FAILED: [code=1] esp-idf/avm_sys/CMakeFiles/__idf_avm_sys.dir/sys.c.obj

  /home/pxp9/Programming/try_nerves/atom_vm_firmware/esp32_s3/_build/atomvm_source/AtomVM/src/platforms/esp32/components/avm_sys/sys.c: In function 'sys_open_avm_from_file':

  /home/pxp9/Programming/try_nerves/atom_vm_firmware/esp32_s3/_build/atomvm_source/AtomVM/src/platforms/esp32/components/avm_sys/sys.c:413:21: error: storage size of 'file_stats' isn't known
    413 |         struct stat file_stats;
        |                     ^~~~~~~~~~

  /home/pxp9/Programming/try_nerves/atom_vm_firmware/esp32_s3/_build/atomvm_source/AtomVM/src/platforms/esp32/components/avm_sys/sys.c:414:22: error: implicit declaration of function 'stat' [-Wimplicit-function-declaration]
    414 |         if (UNLIKELY(stat(path, &file_stats) < 0)) {
        |                      ^~~~

  /home/pxp9/Programming/try_nerves/atom_vm_firmware/esp32_s3/_build/atomvm_source/AtomVM/src/platforms/esp32/components/avm_sys/sys.c:413:21: warning: unused variable 'file_stats' [-Wunused-variable]
    413 |         struct stat file_stats;
        |                     ^~~~~~~~~~

  Build stopped:
  ninja: build stopped: subcommand failed.

pxp9 · 2025-12-28T15:45:57Z

Added docker feat based on this

https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/tools/idf-docker-image.html#building-a-project-with-cmake

UncleGrumpy · 2025-12-28T21:06:52Z

Hey @UncleGrumpy , when i try to build AtomVM , the build fails because we are using the the stat struct but the header <sys/stat.h> is missing, is there something i need to add in the args of cmake to build it correctly or is this a bug in AtomVM main branch build ?

This is not something I have seen before, and I have been working almost exclusively with main branch for at least the last few months.

Was this testing a docker build, or did this happen with an installed ESP-IDF? It has been a few ESP-IDF releases since I last used an esp-idf docker image, although we do use them in the AtomVM CI, so I don’t think that would be the problem.

pxp9 · 2025-12-28T21:08:34Z

@UncleGrumpy , this issue from the missing header happened with both toolchains. Esp idf installed and docker

UncleGrumpy · 2025-12-28T21:18:10Z

Interesting. Which ESP-IDF version? I think I am a point release behind locally (still using 5.5.1). I will try updating my toolchain this evening and see if I can reproduce the problem, I believe the AtomVM CI may also need to be bumped to 5.5.2, I will look into that too. This may be a new issue that you were the first to discover. I think you should open an issue in the AtomVM repo, and be sure to include Host OS, and ESP-IDF version in the report, along with the error and any other details that might be relevant.

pxp9 · 2025-12-28T21:21:08Z

I tried with the last ESP version 6.X and then I tried with 5.4.1 which afaik is the last supported by AtomVM but not sure. It happens the same with both versions.

UncleGrumpy · 2025-12-28T23:44:50Z

I tried with the last ESP version 6.X and then I tried with 5.4.1 which afaik is the last supported by AtomVM but not sure.

I have not tried 6.0 yet, but I believe a PR was recently merged to allow ESP-IDF 6.0 builds. ESP-IDF 5.5.1 is definitely working for main branch, we might not have updated the main branch docs to reflect this, that is something we will need to make sure is up to date before a 0.7.0 release of AtomVM. Since you are hitting this error with ESP-IDF 5.4.1 and 6.0 then I don’t think just switching to 5.5.x will be any different.

pxp9 · 2025-12-29T06:35:55Z

Okay @UncleGrumpy , today I will post an issue in AtomVM with this error and how to reproduce it.

Meanwhile could you review the code and see if everything is fine ?

pxp9 · 2025-12-29T09:52:09Z

Hey @UncleGrumpy , I tried 4 times to compile the AtomVM cloning each time just in case and it seems to be fixed.

this command but with each version

mix atomvm.esp32.build --use-docker --chip esp32s3 --idf-version latest

I tried with v5.4.1 ESP-IDF container and it worked
I tried with v5.5.1 ESP-IDF container and it worked
I tried with latest (v6.1) ESP-IDF container and it did not work
I tried with (the tag release-v6.0) ESP-IDF container and it did not work

it seems i had something weird in the cache or it has been fixed.

by default --use-docker uses the version v5.4.1

UncleGrumpy · 2025-12-29T21:05:51Z

Hey @UncleGrumpy , I tried 4 times to compile the AtomVM cloning each time just in case and it seems to be fixed.

this command but with each version
mix atomvm.esp32.build --use-docker --chip esp32s3 --idf-version latest
I tried with v5.4.1 ESP-IDF container and it worked

I tried with v5.5.1 ESP-IDF container and it worked

I tried with latest (v6.1) ESP-IDF container and it did not work

I tried with (the tag release-v6.0) ESP-IDF container and it did not work

it seems i had something weird in the cache or it has been fixed.

by default --use-docker uses the version v5.4.1

That is great! I would not necessarily expect any of the 6.x IDF versions to work at the moment. I think AtomVM itself will need some updating. If it works with stable releases from 5.1 (or 5.2) to 5.5 that is all that would be expected at the moment. The AtomVM main branch uses 5.5.x for release builds so that should probably be the default here too. I believe 5.5.2 is the latest, we still need to bump that in the AtomVM CI, which I believe is still using 5.5.1. There shouldn’t be any breakage between these versions since the point releases are usually just bug fixes.

UncleGrumpy · 2025-12-29T21:16:03Z

Meanwhile could you review the code and see if everything is fine ?

I will take another look this evening, but I think you will get some much more useful feedback when some of devs that work more in Elixir have a chance to review this, especially @bettio and @petermm.

lib/mix/tasks/esp32.build.flash.ex

lib/mix/tasks/esp32.install.ex

petermm · 2025-12-29T22:49:22Z

lib/mix/tasks/esp32.install.ex

+         selected_device <- EsptoolHelper.select_device(),
+         :ok <- confirm_erase_and_flash(selected_device, image_path),
+         true <-
+           EsptoolHelper.erase_flash([


think we will need a --no_erase (or better name?) option both for release and custom_image, but we can always do that in a follow up PR..

this one erases as well, it has the same behavior as the release, what i wonder is why if i try the full workflow in a real board with a simple project it does not work.

think there is a bug on later elixir like 1.19 - where we need to add @requirements ["app.config"] eg:

use Mix.Task @requirements ["app.config"]

weird I am using Elixir 1.18.4 and Erlang 27

@petermm , do you think I should add this ?? I believe AtomVM officially supports Elixir 1.18

@petermm , do you think I should add this ?? I believe AtomVM officially supports Elixir 1.18

AtomVM on main branch officially supports (and runs CI tests) OTP28 and Elixir 1.19 too. I just noticed that the release notes support matrix needs to be updated to reflect that.

Hey @petermm, i have tried this task with Elixir 1.19.2 and Erlang 28 with no issue, it is the latest version available in the Nix packages. but i can try to make the effort to reproduce this error in 1.19.4

let's just focus on the building and docker integration here.. the bug I mentioned is with esp32.install and using req library (ca certs etc.) - will triage and PR a fix.

lib/mix/tasks/esp32.build.ex

pxp9 · 2025-12-30T10:23:38Z

@UncleGrumpy and @petermm , could you try to build it with docker ? i believe there is something weird but i am not sure why.

pxp9 · 2025-12-30T10:39:46Z

i believe i broke something in the build when i have changed to the elixir boot.

I (31) boot: ESP-IDF v5.4.1 2nd stage bootloader
I (31) boot: compile time Dec 30 2025 10:32:36
I (31) boot: Multicore bootloader
I (31) boot: chip revision: v0.2
I (34) boot: efuse block revision: v1.3
I (38) boot.esp32s3: Boot SPI Speed : 80MHz
I (42) boot.esp32s3: SPI Mode       : DIO
I (45) boot.esp32s3: SPI Flash Size : 4MB
I (49) boot: Enabling RNG early entropy source...
I (54) boot: Partition Table:
I (56) boot: ## Label            Usage          Type ST Offset   Length
I (63) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (69) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (76) boot:  2 factory          factory app      00 00 00010000 001c0000
I (82) boot:  3 boot.avm         RF data          01 01 001d0000 00040000
I (89) boot:  4 main.avm         RF data          01 01 00210000 00100000
I (95) boot: End of partition table
I (98) esp_image: segment 0: paddr=00010020 vaddr=3c140020 size=36980h (223616) map
E (145) esp_image: invalid segment length 0x69742064
E (145) boot: Factory app partition is not bootable
E (146) boot: No bootable app partitions in the partition table
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x3 (RTC_SW_SYS_RST),boot:0xa (SPI_FAST_FLASH_BOOT)
Saved PC:0x403cceb5
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce2810,len:0x15a0
load:0x403c8700,len:0x4
load:0x403c8704,len:0xd20
load:0x403cb700,len:0x2f00
entry 0x403c8928

output of boot in my esp-s3

EDIT: I managed to fix the build in the esp-idf natively but docker build is still buggy.

EDIT 2: I am trying to debug why Docker esp-idf container produces a different image than the Esp-idf installed natively.

UncleGrumpy · 2025-12-30T20:08:57Z

E (145) boot: Factory app partition is not bootable

This seems more like it was flashed to the wrong offset than a library problem. It isn’t finding the partition with AtomVM, so it hasn’t made it as far as trying to load the library partition.

UncleGrumpy · 2025-12-30T20:14:30Z

Actually the offset must have been correct, it did find the partition table, so the problem is with the compiled VM.

pxp9 · 2025-12-30T20:19:57Z

Actually the offset must have been correct, it did find the partition table, so the problem is with the compiled VM.

yup the result of the docker build is like this, but the native AtomVM build it is working. i believe it is bug with the docker flow in the tool, but it is hard to find.

UncleGrumpy · 2025-12-30T20:29:16Z

yup the result of the docker build is like this, but the native AtomVM build it is working. i believe it is bug with the docker flow in the tool, but it is hard to find.

So for debugging in the docker container you can start a shell (docker run sh) and then use the commands the same as you would for a local install of ESP-IDF.

My last 4 days have been brutal at work, but I will try building with docker this evening if I can stay awake long enough ;-)

UncleGrumpy · 2025-12-31T06:51:28Z

yup the result of the docker build is like this, but the native AtomVM build it is working. i believe it is bug with the docker flow in the tool, but it is hard to find.

I discovered the problem with the docker container. I am working on a fix for the AtomVM repo that will change the configuration of the mkimage.sh and mkimage.config. The problem is that the paths to the binary files are configured by cmake in the container and then need to be executed in the host environment to use OTP to execute the escript. I can change the mkimage.sh script to find the esp32/build dir at runtime (rather than have it set by cmake) and pass that directly to the escript to use as the base for finding the partition binaries.

We should fix this there so users, other tools, and IDEs can use docker builds.

pxp9 · 2025-12-31T07:19:11Z

Amazing I had a mini fix which was patching the paths for the container build but it seems that is not working.

pxp9 · 2025-12-31T23:18:22Z

Happy new year ! 🎆

UncleGrumpy · 2026-01-02T03:33:51Z

You can try using this fix atomvm/AtomVM#2052 for the AtomVM repo and see if that fixes your problems with the docker builds.

pxp9 · 2026-01-02T10:09:18Z

@UncleGrumpy just tried your changes about that PR and the docker build is working !!!

@petermm , could you try to build the atomvm with the task with the custom mbed tls path ??

lib/mix/tasks/esp32.build.ex

UncleGrumpy · 2026-01-02T17:18:00Z

@UncleGrumpy just tried your changes about that PR and the docker build is working !!!

Great! That is not a complete fix, though. I targeted that fix to main, I probably should have done that on release-0.6 branch (maybe we can back port it, or I can submit a separate fix for release-0.6)… but that still won’t cover previous release tags, like v0.6.6. There really isn’t a way to push the fix back to cover those, so you may need to work out the path rewrite for those older tags, or we can just say this feature is supported for main branch (and release-0.6 branch if we apply the fix there too) and tagged releases starting from 0.7.0 (or 0.6.7 if we back port… and do a 0.6.7 release).

Another option might be to use git and cherry-pick the fix on top of whatever older checkout that the user might building from.

pxp9 · 2026-01-02T17:25:23Z

@UncleGrumpy just tried your changes about that PR and the docker build is working !!!

Great! That is not a complete fix, though. I targeted that fix to main, I probably should have done that on release-0.6 branch (maybe we can back port it, or I can submit a separate fix for release-0.6)… but that still won’t cover previous release tags, like v0.6.6. There really isn’t a way to push the fix back to cover those, so you may need to work out the path rewrite for those older tags, or we can just say this feature is supported for main branch (and release-0.6 branch if we apply the fix there too) and tagged releases starting from 0.7.0 (or 0.6.7 if we back port… and do a 0.6.7 release).

Another option might be to use git and cherry-pick the fix on top of whatever older checkout that the user might building from.

I believe this feature is oriented to people who want to build and experiment with newer releases, anyways, if they want to build 0.6.6 they can always make the build with Esp IDF installed in the host instead of docker build.

I will document that docker build is supported since the moment you merged that PR in main branch.

I think adding this feature for older AtomVM builds doesn't give much more value.

UncleGrumpy · 2026-01-02T17:29:00Z

I agree. If they want older tagged releases they can just install the official release download. It will be devs wanting to test the newest features that will be using this task.

pxp9 · 2026-01-02T17:31:16Z

I agree. If they want older tagged releases they can just install the official release download. It will be devs wanting to test the newest features that will be using this task.

Amazing, I will commit soon with the docs comment and addressing Peter comment as well.

pxp9 · 2026-01-02T20:38:27Z

@UncleGrumpy , added Note about docker usage.

add docker usage note, and address build generic_unix options passing

i was thinking we can add a guide of how to install the ESP-IDF in nix flake since it is a reproduceable environment is cool. (for telling the users who want to build old releases how to make them).

lib/mix/tasks/esp32.build.ex

petermm · 2026-01-03T12:10:50Z

lib/mix/tasks/esp32.build.ex

+  @default_ref "main"
+  @default_atomvm_url "https://github.com/atomvm/AtomVM"
+  @default_idf_path "idf.py"
+  @default_idf_version "v5.4.1"


Let's do v5.5.2

petermm · 2026-01-03T12:12:24Z

lib/mix/tasks/esp32.build.ex

+  defp build_generic_unix(opts) do
+    atomvm_path = Keyword.fetch!(opts, :atomvm_path)
+    mbedtls_prefix = Keyword.get(opts, :mbedtls_prefix)
+    clean = Keyword.get(opts, :clean, false)


of course a matter of opinion/taste, but DRY, we already parsed the :clean option, let's only do that in one place.

Same in build_atomvm function.. less code lines, more DRY..

eg. no Keyword.get/fetch just use function args.

building from source task

20de0ab

UncleGrumpy suggested changes Dec 28, 2025

View reviewed changes

pxp9 added 2 commits December 28, 2025 12:01

fix most comments

b7234f5

add docker feature

af50277

pxp9 requested review from UncleGrumpy December 28, 2025 15:47

petermm reviewed Dec 29, 2025

View reviewed changes

lib/mix/tasks/esp32.build.flash.ex Outdated Show resolved Hide resolved

petermm reviewed Dec 29, 2025

View reviewed changes

lib/mix/tasks/esp32.build.flash.ex Outdated Show resolved Hide resolved

address comments from Peter

5f8ddf0

petermm reviewed Dec 29, 2025

View reviewed changes

lib/mix/tasks/esp32.build.flash.ex Outdated Show resolved Hide resolved

pxp9 commented Dec 29, 2025

View reviewed changes

lib/mix/tasks/esp32.install.ex Show resolved Hide resolved

pxp9 requested a review from petermm December 29, 2025 22:40

petermm reviewed Dec 29, 2025

View reviewed changes

lib/mix/tasks/esp32.build.ex Outdated Show resolved Hide resolved

UncleGrumpy reviewed Dec 30, 2025

View reviewed changes

lib/mix/tasks/esp32.build.ex Outdated Show resolved Hide resolved

pxp9 added 3 commits December 30, 2025 01:25

fix build only elixir esp32 boot

586ed3c

add requeriments in the host, prefer ninja as build tool

0b8f5f3

add custom mbedtls path

65473b5

pxp9 added 2 commits December 30, 2025 11:28

remove bug

8172eb2

change message and partition table

bc7c78e

fix build with IDF installed

5b74771

UncleGrumpy mentioned this pull request Jan 2, 2026

Determine paths for ESP32 mkimage.sh at runtime atomvm/AtomVM#2052

Merged

remove unecessary things since the path is taken by the script

54fd5ab

petermm reviewed Jan 2, 2026

View reviewed changes

lib/mix/tasks/esp32.build.ex Outdated Show resolved Hide resolved

add docker usage note, and address build generic_unix options passing

0363e93

UncleGrumpy reviewed Jan 2, 2026

View reviewed changes

lib/mix/tasks/esp32.build.ex Outdated Show resolved Hide resolved

put commit instead of PR

5395d70

pxp9 requested review from UncleGrumpy and petermm January 2, 2026 21:11

petermm reviewed Jan 3, 2026

View reviewed changes

building from source task #65

Are you sure you want to change the base?

building from source task #65

Conversation

pxp9 commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Workflow with custom build

Using ESP-IDF Docker container

Uh oh!

UncleGrumpy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

UncleGrumpy commented Dec 28, 2025

Uh oh!

pxp9 commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pxp9 commented Dec 28, 2025

Uh oh!

UncleGrumpy commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pxp9 commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

UncleGrumpy commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pxp9 commented Dec 28, 2025

Uh oh!

UncleGrumpy commented Dec 28, 2025

Uh oh!

pxp9 commented Dec 29, 2025

Uh oh!

pxp9 commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

UncleGrumpy commented Dec 29, 2025

Uh oh!

UncleGrumpy commented Dec 29, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

UncleGrumpy Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pxp9 Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pxp9 commented Dec 30, 2025

pxp9 commented Dec 27, 2025 •

edited

Loading

pxp9 commented Dec 28, 2025 •

edited

Loading

UncleGrumpy commented Dec 28, 2025 •

edited

Loading

pxp9 commented Dec 28, 2025 •

edited

Loading

UncleGrumpy commented Dec 28, 2025 •

edited

Loading

pxp9 commented Dec 29, 2025 •

edited

Loading

UncleGrumpy Jan 2, 2026 •

edited

Loading

pxp9 Jan 3, 2026 •

edited

Loading

pxp9 commented Dec 30, 2025 •

edited

Loading

UncleGrumpy commented Dec 31, 2025 •

edited

Loading

pxp9 commented Dec 31, 2025 •

edited

Loading

UncleGrumpy commented Jan 2, 2026 •

edited

Loading

pxp9 commented Jan 2, 2026 •

edited

Loading