Results of running the regression test suite on nodejs

[emmatyping]

After spending a lot of time going through all the tests, here is a summary of my findings towards running the regression test suite on Emscripten/nodejs:

The command line I used and exclude list I used are here: https://gist.github.com/ethanhs/656637de570cc2c239c8728c4aa1e731

The list of tests excluded also includes a brief description of each test's failure(s). A quick summary:

• almost half of the test either directly import subprocess (51 tests) or indirectly does so via e.g. unitest.mock (97 tests)
• despite pthread being compiled in, I seem unable to create threads under nodejs (6 tests), I am getting EAGAIN...
• there are some locale issues which I presume are Emscripten bugs
• it also requires a couple of small changes to CPython, to patch around an Emscripten bug and also to disable the faulthandler thread because it won't start and causes the test runner to crash after a successful run.

diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py
index fc3c2b9692..0879bb6baa 100644
--- a/Lib/test/libregrtest/main.py
+++ b/Lib/test/libregrtest/main.py
@@ -659,7 +659,9 @@ def main(self, tests=None, **kwargs):
        except SystemExit as exc:
            # bpo-38203: Python can hang at exit in Py_Finalize(), especially
            # on threading._shutdown() call: put a timeout
-            faulthandler.dump_traceback_later(EXIT_TIMEOUT, exit=True)
+            # Emscripten has issues starting threads
+            if sys.platform != 'emscripten':
+                faulthandler.dump_traceback_later(EXIT_TIMEOUT, exit=True)

            sys.exit(exc.code)

diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
index 89e93c5818..dbd7d4dc63 100644
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -5114,6 +5114,7 @@ sock_initobj(PyObject *self, PyObject *args, PyObject *kwds)
                                     &family, &type, &proto, &fdobj))
        return -1;

+
#ifdef MS_WINDOWS
    /* In this case, we don't use the family, type and proto args */
    if (fdobj == NULL || fdobj == Py_None)
@@ -7926,7 +7927,7 @@ PyInit__socket(void)
#ifdef  IPPROTO_VRRP
    PyModule_AddIntMacro(m, IPPROTO_VRRP);
#endif
-#ifdef  IPPROTO_SCTP
+#if defined(IPPROTO_SCTP) && !defined(__EMSCRIPTEN__)
    PyModule_AddIntMacro(m, IPPROTO_SCTP);
#endif
#ifdef  IPPROTO_BIP

Huge thanks to @tiran for helping figure out the IPPROTO_SCTP issue which was causing test_socket to crash the test runner!

With this I think the steps towards adding a build bot for Emscripten/nodejs:

• Have bpo-40280: Add build target for Emscripten/node.js python/cpython#30495 merged
• Figure out why threads aren't starting - EDIT: It seems that WASM_STANDALONE doesn't work with pthread? strange...
• Patch Python around the IPPROTO_SCTP for now
• Set up build bot

Want to run things yourself? Here are the steps:

1. Do a normal checkout of this repo, pull my above mentioned PR
2. Apply the above patches
3. Grab the tests.txt file in the above gist and put it in the top level of the checkout
4. Run the command in the gist to run the test suite!

[emmatyping]

I've fixed the threading issue, it seems we need to pass -pthread in a few more places and -s USE_PTHREADS at link time.

We also need to decide whether we should use -s PROXY_TO_PTHREAD, which runs the main thread in a webworker/thread or use -s PTHREAD_POOL_SIZE="_emscripten_num_logical_cores()" -s DEFAULT_LIBRARY_FUNCS_TO_INCLUDE="emscripten_num_logical_cores" which makes a threadpool for Emscripten to use with the number of cores on the system (this seems better)

[tiran]

_testcapi used in lots of tests. We might be able to turn it into a built-in module.

--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -12,6 +12,14 @@
    macro defined, but only the public C API must be tested here. */
 
 #undef Py_BUILD_CORE_MODULE
+
+/* On WASM shared extensions modules do not work. Allow building with
+   statically */
+
+#ifdef __wasm__
+#  undef Py_BUILD_CORE_BUILTIN
+#endif
+
 /* Always enable assertions */
 #undef NDEBUG
 

[tiran]

With python/cpython#30552 and --with-emscripten-target=browser the faulthandler workaround is no longer necessary. test_hashlib is passing, too.

# node --experimental-wasm-threads --experimental-wasm-bulk-memory ./python.js -m test test
0:00:00 load avg: 0.10 Run tests sequentially
0:00:00 load avg: 0.10 [1/1] test_hashlib
warning: unsupported syscall: __syscall_wait4

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2.7 sec
Tests result: SUCCESS

[emmatyping]

Yeah the faulthandler work around was only needed because threads were broken. I believe a couple of other tests like test_enum should now work to!

[tiran]

clock_nanosleep is broken, sleeps forever. I'm adding a workaround to my PR.

[brettcannon]

Now that we have nightly CI working, is this issue worth keeping open, or are the more targeted issues we should open about what to try and tackle?

[emmatyping]

I think this is pretty much superseded by #43 and we can open issues with anything else that isn't working (that needs fixing on the CPython end)