Build Failure with UnicodeDecodeError

[nokotan]

Updating EMSDK (Python 2.x to Python 3.7) results build error (seems to be encoding problem)

OS: Windows 10 Home version 1909
Language: Japanese

Project Files:
https://github.com/nokotan/DxLibForHTML5-VSCode

Command:

em++ Main.cpp -o index.html -O1 -std=c++17 -g4 -IDxLibForHTML5 -LDxLibForHTML5 -lbullet -lfreetype -logg -lpng -lvorbis -lz -lDxDrawFunc -lDxUseCLib -lDxLib --emrun -s ASSERTIONS=1 -s MAIN_MODULE=1 -s FULL_ES3=1 -s ALLOW_MEMORY_GROWTH=1 --source-map-base http://localhost:8080/ --preload-file assets --shell-file template.html

Traceback:

Traceback (most recent call last):
  File "\*\*\*/emsdk/upstream/emscripten/emcc.py", line 3905, in <module>
    sys.exit(run(sys.argv))
  File "\*\*\*/emsdk/upstream/emscripten/emcc.py", line 2373, in run
    final = do_emscripten(final, shared.replace_or_append_suffix(target, '.mem'))
  File "\*\*\*/emsdk/upstream/emscripten/emcc.py", line 475, in do_emscripten
    emscripten.run(infile, outfile, memfile)
  File "\*\*\*/emsdk/upstream/emscripten/emscripten.py", line 2794, in run
    return temp_files.run_and_clean(lambda: emscripter(
  File "\*\*\*/emsdk/upstream/emscripten/tools/tempfiles.py", line 105, in run_and_clean
    return func()
  File "\*\*\*/emsdk/upstream/emscripten/emscripten.py", line 2795, in <lambda>
    infile, outfile_obj, memfile, temp_files, shared.DEBUG)
  File "\*\*\*/emsdk/upstream/emscripten/emscripten.py", line 2207, in emscript_wasm_backend
    glue, forwarded_data = compile_settings(temp_files)
  File "\*\*\*/emsdk/upstream/emscripten/emscripten.py", line 773, in compile_settings
    cwd=path_from_root('src'), env=env)
  File "\*\*\*/emsdk/upstream/emscripten/tools/jsrun.py", line 100, in run_js_tool
    return shared.check_call(command, *args, **kw).stdout
  File "\*\*\*/emsdk/upstream/emscripten/tools/shared.py", line 200, in check_call
    return run_process(cmd, *args, **kw)
  File "\*\*\*/emsdk/upstream/emscripten/tools/shared.py", line 181, in run_process
    ret = subprocess.run(cmd, check=check, input=input, *args, **kw)
  File "D:/obj/windows-release/37amd64_Release/msi_python/zip_amd64/subprocess.py", line 474, in run
  File "D:/obj/windows-release/37amd64_Release/msi_python/zip_amd64/subprocess.py", line 926, in communicate
UnicodeDecodeError: 'cp932' codec can't decode byte 0x86 in position 24221: illegal multibyte sequence

[sbc100]

Interesting. It looks like python3 is choosing to use the system default rather than utf8. I see a PEP out that fixes this: https://www.python.org/dev/peps/pep-0597/

In the mean time we should probably force utf8 somehow.

[sbc100]

Re-opening because fix got revertted.

[sbc100]

if you set PYTHONUTF8 in the environment does that solve the issue for you?

[nokotan]

Thank you for your comment.

after setting environment variable "PYTHONUTF8" to 1, compiling my project on python 3.7.7 has succeeded.

[jeromelaban]

I've having a similar issue, but setting PYTHONUTF8=1 does not seem to have an effect.

Is there something else may be changed to work around this issue ?

(I removed the stack trace it was the same as above)

[nokotan]

Does setting PYTHONIOENCODING=utf8 instead of PYTHONUTF8=1 resolve your problem?
PYTHONUTF8=1 should be available option for python 3.7 and later.

[jeromelaban]

@nokotan thanks for the help. It's not having any effect, unfortunately. It looks like the ascii decoder is still used in subprocess.

I'll update my running Python, must be related.