Assertion '(c & LIT_UTF8_3_BYTE_MASK) == LIT_UTF8_3_BYTE_MARKER' failed in lit_read_code_unit_from_utf8

[renatahodovan]

Jerry version:

Checked revision: a9e7dd7
Build command: ./tools/build.py --clean --debug

OS:

Ubuntu 16.04.2 LTS

Test case:

// in hex:
\xf0\x90\xae\xa9\xe0\xad\x9f

Download test case

Backtrace:

ICE: Assertion '(c & LIT_UTF8_3_BYTE_MASK) == LIT_UTF8_3_BYTE_MARKER' failed at jerryscript/jerry-core/lit/lit-strings.c(lit_read_code_unit_from_utf8):425.
Error: ERR_FAILED_INTERNAL_ASSERTION
bt
#0  0xf7fd8be9 in __kernel_vsyscall ()
#1  0xf7898ea9 in raise () from /lib/i386-linux-gnu/libc.so.6
#2  0xf789a407 in abort () from /lib/i386-linux-gnu/libc.so.6
#3  0x08054cf9 in jerry_port_fatal (code=ERR_FAILED_INTERNAL_ASSERTION) at jerryscript/jerry-port/default/default-fatal.c:71
#4  0x0809c786 in jerry_fatal (code=ERR_FAILED_INTERNAL_ASSERTION) at jerryscript/jerry-core/jrt/jrt-fatals.c:58
#5  0x0809c7b9 in jerry_assert_fail (assertion=0x8144040 "(c & LIT_UTF8_3_BYTE_MASK) == LIT_UTF8_3_BYTE_MARKER", file=0x8143d00 "jerryscript/jerry-core/lit/lit-strings.c", function=0x812c300 <__func__.2027.lto_priv.350> "lit_read_code_unit_from_utf8", line=425) at jerryscript/jerry-core/jrt/jrt-fatals.c:82
#6  0x0809faed in lit_read_code_unit_from_utf8 (buf_p=0x8180b80 <buffer.lto_priv> "𐮩ୟ", code_point=0xffffbd40) at jerryscript/jerry-core/lit/lit-strings.c:425
#7  0x0809ffed in lit_utf8_peek_next (buf_p=0x8180b80 <buffer.lto_priv> "𐮩ୟ") at jerryscript/jerry-core/lit/lit-strings.c:500
#8  0x0809d8e0 in lit_char_is_identifier_start (src_p=0x8180b80 <buffer.lto_priv> "𐮩ୟ") at jerryscript/jerry-core/lit/lit-char-helpers.c:229
#9  0x0809a229 in lexer_next_token (context_p=0xffffbe60) at jerryscript/jerry-core/parser/js/js-lexer.c:988
#10 0x08091425 in parser_parse_source (source_p=0x8180b80 <buffer.lto_priv> "𐮩ୟ", size=7, strict_mode=0, error_location_p=0xffffc030) at jerryscript/jerry-core/parser/js/js-parser.c:2040
#11 0x0809374d in parser_parse_script (source_p=0x8180b80 <buffer.lto_priv> "𐮩ୟ", size=7, is_strict=false, bytecode_data_p=0xffffc0e0) at jerryscript/jerry-core/parser/js/js-parser.c:2496
#12 0x08116b8b in jerry_parse (source_p=0x8180b80 <buffer.lto_priv> "𐮩ୟ", source_size=7, is_strict=false) at jerryscript/jerry-core/api/jerry.c:326
#13 0x08116cd9 in jerry_parse_named_resource (name_p=0xffffc669 "/home/reni/data/.fuzzinator/jerry/test-3479-139991477544776.js", name_length=62, source_p=0x8180b80 <buffer.lto_priv> "𐮩ୟ", source_size=7, is_strict=false) at jerryscript/jerry-core/api/jerry.c:381
#14 0x08113c97 in main (argc=3, argv=0xffffc3a4) at jerryscript/jerry-main/main-unix.c:682

^{Found by Fuzzinator with Grammarinator}

[glistening]

@renatahodovan In my opinion, as @zherczeg commented in #1867, it is the user's responsibility of providing valid utf8 and cesu8 and the input seems invalid csesu8 string. (jerry_is_valid_cesu8_string returns false although jerry_is_valid_utf8_string returns true.)

[LaszloLango]

@glistening the problem here is that the current main implementation checks the input with jerry_is_valid_utf8_string and the test still crashes the engine. We must fix this.

[zherczeg]

This is a valid utf8 input, and should throw parse error, because characters > 0xffff are never valid starting identifier characters.

[glistening]

@zherczeg, @LaszloLango I've digged around this issue. From ECMAScript 2015, some characters > 0xFFFF are valid. Please refer to Annex E incompatibilities and here. Then, what is good solution for us?

option 1) follow the ECMAScript 2015 specification?
option 2) apply the new rule only for es2015-subset profile enabled, and for default profile, apply characters > 0xffff. It will take more time, and the precise checking may cost us a burden in binary size or/and performance.

[zherczeg]

For source codes, I think everybody uses ascii characters. So I wouldn't implement it unless somebody really needs it.

[glistening]

@zherczeg For source codes, I agree with you. But I think the assertion failure for valid utf8 input must be fixed. As far as I understand, the content is high likely to be utf8 encoded (unix and web), but we decode the content with cesu-8. So if the code point is valid utf8 and invalid cesu-8, it causes problem. Please correct me if I am wrong.

[zherczeg]

Of course, the assertion should be fixed. What I am talking about:

• In strings (regexes) the 4 byte UTF-8 codes are converted to two CESU-8 characters. That should be ok.
• Other than strings, characters should be a literal start or special characters. However there are no >0xffff special characters (outside of the ascii range only a few space and newline characters are special, that is all). They could be start of literals, but detecting that would take a lot of extra space and the advantage is questionable on low-end systems (I wouldn't even mind forcing people to use ascii literals in their source code). This is changed in ES2015, but I am not sure supporting it is useful at all.

Hence my proposal if we encounter a 4 byte character outside of strings, we can consider it an invalid character and throw a syntax error. The fix is a few lines of code (perhaps only 1 line).

[glistening]

@zherczeg Again, I agree with you. I am thinking of similar approach since I also want practical solution which suits low-end devices. If you don't mind, I would like to make the fix PR for this issue.

[zherczeg]

Feel free to do it!

[rerobika]

I'm sorry, I've made a typo during referencing to another issue!