File system is inconsistent when paths aren't valid utf8

[hoodmane]

Version of emscripten/emsdk:
Tested against 4.0.8 but problem is in tip of tree.

a.c

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>


char x1[] = "@test_42_tmp\xe7w\xf0";
char x2[] = "./@test_42_tmp\xe7w\xf0";

int main() {
    printf("x: %s\n", x1);
    int fd1 = open(x1, O_CREAT | O_WRONLY, 0x777);
    if (fd1 == -1) {
        printf("open failed: %s\n", strerror(errno));
        return 1;
    }
    int w = write(fd1, "hi!", 4);
    if (w == -1) {
        printf("write failed: %s\n", strerror(errno));
        return 1;
    }
    printf("wrote: %d\n", w);
    if (close(fd1) == -1) {
        printf("close failed: %s\n", strerror(errno));
        return 1;
    }


    printf("x2: %s\n", x2);
    int fd2 = open(x2, O_RDONLY);
    if (fd2 == -1) {
        printf("open failed: %s\n", strerror(errno));
        return 1;
    }
    printf("fd2: %d\n", fd2);
    char out[10];
    int r = read(fd2, out, 10);
    if (r == -1) {
        printf("read failed: %s\n", strerror(errno));
        return 1;
    }
    printf("read: %d %s\n", r, out);
    unlink(x1);
    return 0;
}

Compile and run a.c with gcc:

$ gcc a.c && ./a.out
x: @test_42_tmp�w�
wrote: 4
x2: ./@test_42_tmp�w�
fd2: 3
read: 4 hi!

Compile and run with emcc:

$ emcc a.c && node a.out.js 
x: @test_42_tmp緰
wrote: 4
x2: ./@test_42_tmp緰
open failed: No such file or directory

The problem is that @test_42_tmp\xe7w\xf0 is 15 bytes long whereas ./@test_42_tmp\xe7w\xf0 is 17 bytes long. The longer string hits this codepath:
https://github.com/emscripten-core/emscripten/blob/main/src/lib/libstrings.js#L58
and is decode by TextDecoder() to ./@test_42_tmp�w�. The shorter string gets decoded in JS to ./@test_42_tmp緰. Then those don't match so we don't find the file.

[hoodmane]

More direct example:

#include "emscripten.h"

EM_JS_DEPS(test, "$UTF8ToString");
EM_JS(void, test, (char* a, char* b), {
    console.log(UTF8ToString(a));
    console.log(UTF8ToString(b));
})

int main() {
    test("@test_42_tmp\xe7w\xf0", "./@test_42_tmp\xe7w\xf0");
    return 0;
}

logs:

@test_42_tmp緰
./@test_42_tmp�w�

[RReverser]

char x1[] = "@test_42_tmp\xe7w\xf0";
char x2[] = "./@test_42_tmp\xe7w\xf0";

Those bytes are not valid UTF-8. I feel like it's fair to say that you're hitting UB here so any result is okay.

UTF-8 spec leaves handling of invalid sequences up to the implementer, and stricter runtimes like Rust would outright throw an error if trying to decode those as-if they were UTF-8.

[RReverser]

I guess the real problem is that filesystem implementation (open()) shouldn't assume UTF-8 either.

[hoodmane]

I changed the name to reflect that.

[hoodmane]

I guess this is fixed by passing -sTEXTDECODER=2

[RReverser]

If making them consistent isn't too much code, I guess it might be still worth handling. I wonder, is it enough to add replacement character in this branch?

emscripten/src/lib/libstrings.js

Line 89 in 4182f94

if ((u0 & 0xF8) != 0xF0) warnOnce('Invalid UTF-8 leading byte ' + ptrToString(u0) + ' encountered when deserializing a UTF-8 string in wasm memory to a JS string!');

[hoodmane]

I'll look into it.

[hoodmane]

Building with assertions shows it doesn't seem to hit that path.