[Python-Dev] cpython and parallel make

Hi,

what’s people’s experience with compiling cpython using multiple jobs
(e.g., make -j 8)?

In my case, I sometimes experience build errors that happen when using -j,
whereas the single-job build always works. I haven’t tracked this down in
detail, though… here’s an extract from a sample log:

17:42:56 make[1]: Entering directory `/path/to/cpython/build'
17:42:57 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -O3 -fsanitize=address
-Werror=declaration-after-statement   -I. -IInclude
-I../../cpython/Include    -DPy_BUILD_CORE -o Python/Python-ast.o
Python/Python-ast.c
17:42:57 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -O3 -fsanitize=address
-Werror=declaration-after-statement   -I. -IInclude
-I../../cpython/Include    -DPy_BUILD_CORE -o Python/ast.o
./../cpython/Python/ast.c
17:42:57 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -O3 -fsanitize=address
-Werror=declaration-after-statement   -I. -IInclude
-I../../cpython/Include    -DPy_BUILD_CORE -o Python/ast.o
./../cpython/Python/ast.c
17:42:57 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -O3 -fsanitize=address
-Werror=declaration-after-statement   -I. -IInclude
-I../../cpython/Include    -DPy_BUILD_CORE -o Python/ceval.o
./../cpython/Python/ceval.c
17:42:58 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -O3 -fsanitize=address
-Werror=declaration-after-statement   -I. -IInclude
-I../../cpython/Include    -DPy_BUILD_CORE -o Python/compile.o
./../cpython/Python/compile.c
17:43:07 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -O3 -fsanitize=address
-Werror=declaration-after-statement   -I. -IInclude
-I../../cpython/Include    -DPy_BUILD_CORE -o Python/graminit.o
Python/graminit.c
17:43:07 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -O3 -fsanitize=address
-Werror=declaration-after-statement   -I. -IInclude
-I../../cpython/Include    -DPy_BUILD_CORE -o Python/ceval.o
./../cpython/Python/ceval.c
17:43:07 clang -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -O3 -fsanitize=address
-Werror=declaration-after-statement   -I. -IInclude
-I../../cpython/Include    -DPy_BUILD_CORE -o Python/compile.o
./../cpython/Python/compile.c
17:43:08 make Include/graminit.h
17:43:08 make[1]: Entering directory `/path/to/cpython/build'
17:43:08 make[1]: `Include/graminit.h' is up to date.
17:43:08 make[1]: Leaving directory `/path/to/cpython/build'

17:43:24 clang -fsanitize=address  -o Modules/_freeze_importlib
Modules/_freeze_importlib.o Modules/getbuildinfo.o Parser/acceler.o
[...] Objects/abstract.o [...] Objects/weakrefobject.o
Python/_warnings.o [...] Python/ceval.o Python/compile.o
Python/codecs.o [...] Modules/faulthandler.o  Modules/_tracemalloc.o
Modules/hashtable.o  Modules/symtablemodule.o  Modules/xxsubtype.o
-ldl  -lutil   -lm
17:43:24 /usr/bin/ld: error: Python/ceval.o: file is empty
17:43:24 /usr/bin/ld: error: Python/compile.o: file is empty

For some reason, some files like Python/ceval.o seem to be built multiple
times. I reckon this causes race conditions that result in some empty
object files, which causes the linker to fail.

Any ideas about where this comes from, and how to fix it?

Cheers,
Jonas
​