expo publish always ends up in Javascript head out of memory on Ubuntu

I have a PC running Ubuntu 20.04.1 LTS with 16GB RAM. I am trying to build apk of a expo react app. Whenever I try to run expo publish --release-channel channel_name it always runs out of memory and gives me following stacktrace:

transform[stdout]: ==== JS stack trace =========================================
transform[stdout]:
transform[stdout]:     0: ExitFrame [pc: 0x13cf099]
transform[stdout]: Security context: 0x1ec3465408d1 <JSObject>
transform[stdout]:     1: token(aka token) [0x2ac37c202af1] [0x0acc333004b1 <undefined>:~1916] [pc=0x1ca847032d11](this=0x0acc333004b1 <undefined>,0x158b955c4321 <String[#4]: punc>,0x285cf488fc21 <String[#1]: :>,0x0acc333004b1 <undefined>)
transform[stdout]:     2: next_token [0x2ac37c200119] [0x0acc333004b1 <undefined>:~2287] [pc=0x1ca846a7bc0c](this=0x2ac37c200159 <Object map = 0x1cca2bdcb39>,0x0ac...
transform[stdout]:
transform[stderr]: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
transform[stderr]:  1: 0xa093f0 node::Abort() [/usr/bin/node]
transform[stderr]:  2: 0xa097fc node::OnFatalError(char const*, char const*) [/usr/bin/node]
transform[stderr]:  3: 0xb8431e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
transform[stderr]:  4: 0xb84699 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
transform[stderr]:  5: 0xd31055  [/usr/bin/node]
transform[stderr]:  6: 0xd316e6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/bin/node]
transform[stderr]:  7: 0xd3df65 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/bin/node]
transform[stderr]:  8: 0xd3ee15 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
transform[stderr]:  9: 0xd418cc v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
transform[stderr]: 10: 0xd0837b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/bin/node]
transform[stderr]: 11: 0x1049fbe v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
transform[stderr]: 12: 0x13cf099  [/usr/bin/node]

I tried setting up NODE_OPTIONS=–max_old_space_size=8192 and then on running v8.getHeapStatistics() I see following result:

{
  total_heap_size: 5173248,
  total_heap_size_executable: 524288,
  total_physical_size: 4460520,
  total_available_size: 8636951928,
  used_heap_size: 3019200,
  heap_size_limit: 8640266240,
  malloced_memory: 8192,
  peak_malloced_memory: 712992,
  does_zap_garbage: 0,
  number_of_native_contexts: 1,
  number_of_detached_contexts: 0
}

I still get same stacktrace even with max_old_space_size set to 8 GB.

To ensure if my node options are working I changed it to NODE_OPTIONS=–max_old_space_size=4096 and sourced it. Result for v8.getHeapStatistics() for 4GB is:

{
  total_heap_size: 5173248,
  total_heap_size_executable: 524288,
  total_physical_size: 4460744,
  total_available_size: 4342034920,     <----- this value changed frmo 8GB to 4GB
  used_heap_size: 3050272,
  heap_size_limit: 4345298944,          <----- this value changed frmo 8GB to 4GB
  malloced_memory: 8192,
  peak_malloced_memory: 712992,
  does_zap_garbage: 0,
  number_of_native_contexts: 1,
  number_of_detached_contexts: 0
}

I have a similarly configured MacBook which has no issue in building APK with –max_old_space_size=4096.

I tried comparing these values to the one on my MacBook and they look similar. I then run v8.getHeapSpaceStatistics() command. The full output here: https://paste.ubuntu.com/p/5nvdYVmFRf/. This command has new_space section with space_size lower than one on my MacBook:

Ubuntu
  {
    space_name: 'new_space',
    space_size: 1048576,
    space_used_size: 298808,
    space_available_size: 748648,
    physical_space_size: 1014408
  },

MacBook
  {
    space_name: 'new_space',
    space_size: 2097152,
    space_used_size: 919712,
    space_available_size: 127744,
    physical_space_size: 2060464
  },

I am out of any ideas to try, can anyone please help me with the problem?

Node version: v12.18.4
Expo cli version: 3.27.12
Expo sdk version: 38

Answer

It certainly smells of a Node or V8 bug, but there’s not enough evidence so far to be sure. Getting a repro case would be great, especially in order to figure out what’s going wrong with enough detail to be able to fix it 🙂

Lacking that, I can think of a few things you can try:

Try with Node 14. Maybe whatever it was has been fixed already.
Try running Node with the flag --nodetect-ineffective-gcs-near-heap-limit. From looking at the code, it seems this flag should disable the particular check that resulted in the OOM error being thrown. That said, if we assume that something is broken/buggy under the hood, then disabling one of the symptoms may just lead to some other symptom emerging a few steps down the road…
If you’re comfortable with C++ debugging, you could compile a Debug build of Node, run it in GDB, reproduce the crash, go to the v8::internal::Heap::RecomputeLimits frame and take a look around (e.g. values of local variables) to try to understand what the situation is.
The output of running Node with --trace-gc might also provide some insights.

Advertisement

Answer