Missing symbols in libcomponentize_py_runtime.so

[hishamhm]

I followed all build steps in the README and they worked succesfully, apart from cargo test --release failing during build on the first try. The error message suggested me to run rustup component add rust-src --toolchain nightly-x86_64-unknown-linux-gnu. When running this then trying cargo test --release again, then the build succeeded, but tests then failed with unresolved symbols in libcomponentize_py_runtime.so, as shown in the log below:

cargo test --release
   Compiling componentize-py v0.11.0 (https://github.com/bytecodealliance/componentize-py?rev=071a9ba1#071a9ba1)
   Compiling sockets-server v2.0.0 (/home/hisham/work/kong/wasi-sockets-tests/server)
    Finished `release` profile [optimized] target(s) in 1m 12s
     Running unittests src/lib.rs (target/release/deps/sockets_server-774ede4b91e21fd5)

running 16 tests
test tests::python_redis_name ... FAILED
test tests::python_redis ... FAILED
test tests::python_name ... FAILED
test tests::python_ipv4 ... FAILED
test tests::python_ipv6 ... FAILED
test tests::tokio_postgres_name ... FAILED
test tests::tokio_ipv6 ... FAILED
test tests::direct_name ... FAILED
test tests::std_ipv6 ... FAILED
test tests::tokio_postgres ... FAILED
test tests::tokio_ipv4 ... FAILED
test tests::direct_ipv4 ... FAILED
test tests::direct_ipv6 ... FAILED
test tests::std_name ... FAILED
test tests::std_ipv4 ... FAILED
test tests::tokio_name ... FAILED

failures:

---- tests::python_redis_name stdout ----
Error: unresolved symbol(s):
	libcomponentize_py_runtime.so needs _PyUnicode_Ready (function [I32] -> [I32])
	libcomponentize_py_runtime.so needs _CLOCK_PROCESS_CPUTIME_ID (global I32)
	libcomponentize_py_runtime.so needs _CLOCK_THREAD_CPUTIME_ID (global I32)

---- tests::python_redis stdout ----
Error: unresolved symbol(s):
	libcomponentize_py_runtime.so needs _PyUnicode_Ready (function [I32] -> [I32])
	libcomponentize_py_runtime.so needs _CLOCK_PROCESS_CPUTIME_ID (global I32)
	libcomponentize_py_runtime.so needs _CLOCK_THREAD_CPUTIME_ID (global I32)

---- tests::python_name stdout ----
Error: unresolved symbol(s):
	libcomponentize_py_runtime.so needs _PyUnicode_Ready (function [I32] -> [I32])
	libcomponentize_py_runtime.so needs _CLOCK_PROCESS_CPUTIME_ID (global I32)
	libcomponentize_py_runtime.so needs _CLOCK_THREAD_CPUTIME_ID (global I32)

---- tests::python_ipv4 stdout ----
Error: unresolved symbol(s):
	libcomponentize_py_runtime.so needs _PyUnicode_Ready (function [I32] -> [I32])
	libcomponentize_py_runtime.so needs _CLOCK_PROCESS_CPUTIME_ID (global I32)
	libcomponentize_py_runtime.so needs _CLOCK_THREAD_CPUTIME_ID (global I32)

---- tests::python_ipv6 stdout ----
Error: unresolved symbol(s):
	libcomponentize_py_runtime.so needs _PyUnicode_Ready (function [I32] -> [I32])
	libcomponentize_py_runtime.so needs _CLOCK_PROCESS_CPUTIME_ID (global I32)
	libcomponentize_py_runtime.so needs _CLOCK_THREAD_CPUTIME_ID (global I32)

---- tests::tokio_postgres_name stdout ----
Error: No such file or directory (os error 2)

---- tests::tokio_ipv6 stdout ----
Error: No such file or directory (os error 2)

---- tests::direct_name stdout ----
Error: No such file or directory (os error 2)

---- tests::std_ipv6 stdout ----
Error: No such file or directory (os error 2)

---- tests::tokio_postgres stdout ----
Error: No such file or directory (os error 2)

---- tests::tokio_ipv4 stdout ----
Error: No such file or directory (os error 2)

---- tests::direct_ipv4 stdout ----
Error: No such file or directory (os error 2)

---- tests::direct_ipv6 stdout ----
Error: No such file or directory (os error 2)

---- tests::std_name stdout ----
Error: No such file or directory (os error 2)

---- tests::std_ipv4 stdout ----
Error: No such file or directory (os error 2)

---- tests::tokio_name stdout ----
Error: No such file or directory (os error 2)


failures:
    tests::direct_ipv4
    tests::direct_ipv6
    tests::direct_name
    tests::python_ipv4
    tests::python_ipv6
    tests::python_name
    tests::python_redis
    tests::python_redis_name
    tests::std_ipv4
    tests::std_ipv6
    tests::std_name
    tests::tokio_ipv4
    tests::tokio_ipv6
    tests::tokio_name
    tests::tokio_postgres
    tests::tokio_postgres_name

test result: FAILED. 0 passed; 16 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.94s

error: test failed, to rerun pass `--lib`

Hope that helps diagnosing! Happy to provide any more info.

[hishamhm]

Searching around I found this, which might be related?
python/cpython#115757

[dicej]

Thanks for reporting this, @hishamhm. I'll try to reproduce it locally and see what the problem is.

Note that this repo isn't really being maintained, and most of the dependencies are quite out-of-date at this point, so I don't recommend doing anything serious with it :)

[dicej]

I was able to reproduce the issue. The Cargo.toml points to an immutable commit of componentize-py, which in turn points to an immutable commit of CPython as a submodule, and we're building with the same fork of wasi-sdk as before, so I'm not sure what could have changed to break the build. I'm going to update both componentize-py and wasi-sdk to see if that helps.

[dicej]

I just pushed an update to the main branch. All the tests are now passing locally for me. Please let me know if you still have issues.

Note that I've updated the instructions in README.md to use a newer wasi-sdk build, as well as to configure and build the Rust fork a bit differently.

[hishamhm]

I just pushed an update to the main branch. All the tests are now passing locally for me. Please let me know if you still have issues.

@dicej Thank you! I'll do a full rebuild and report here ASAP!

Note that this repo isn't really being maintained, and most of the dependencies are quite out-of-date at this point, so I don't recommend doing anything serious with it :)

Noted! I just wanted a starting point to be able to play a bit and start getting acquainted with the preview2 stuff. (since clearly there are a lot of moving parts and things under development in the dependencies!) Thank you for the initiative of putting together this repo!

[dicej]

FYI, I accidentally committed some debug lines to server/Cargo.toml earlier and then fixed that, so please pull one more time to make sure you have the latest code.

[hishamhm]

Hi! Happy to report that I got (most) tests passing!

After your updates I was still getting most tests failing with a "file not found", but I managed to address it by fetching the command test adapter separately like this:

mkdir target
cd target
wget https://github.com/bytecodealliance/wasmtime/releases/download/v18.0.4/wasi_snapshot_preview1.command.wasm
cd ..
export WASI_SOCKETS_TEST_ADAPTER=$(pwd)/target/wasi_snapshot_preview1.command.wasm

(I tried just updating the URL in server/src/lib.rs but that alone didn't work, and using the environment variable did the trick.)

Also, for some reason localhost seems to only match IPv4 on Ubuntu 22.04 (it has ip6-localhost for IPv6), but adding that to my /etc/hosts made the *_name tests pass. I bring this up for the benefit of search of any other Linux users.

The only remaining test failures are a couple of python_redis* tests which fail with ModuleNotFoundError: No module named 'redis'. But with 14 out of 16 passing, that already gives me enough of an environment to start playing with wasi-sockets! Thank you!