NG is slower compared to Bellard version

[ABBAPOH]

So, I ran Valgrind with 2 engines and it seems that NG is about 3% slower and also consumes more memory

========== Performance data for Resolving ==========
    Old instruction count: 9396116795
    New instruction count: 9761057471
    Relative change: +3 %
    Old peak memory usage: 118112672 Bytes
    New peak memory usage: 120718720 Bytes
    Relative change: +2 %

Results seem to be consistent between different runs.

However, attempt to use sampling profiler on MacOS does not provide consistent results making hard to track down the reason of degradation.
Some suspects are (the numbers are sample counts in 3 different runs):
Old
js_create_function 32/28/42
__JS_EvalInternal 87/73/81

New
js_create_function 39/36/39
__JS_EvalInternal 105/84/83
It seems that both functions are a bit slower, but that could be natural variance between runs.
Bisecting this would be harder this time, but if there's no other way, I can try doing so.

In the meantime, maybe you have any hints what could cause regression?

[saghul]

The memory consumption could be explained by the poly IC and the number you showed are what, ms? They look reasonably close to margin or error?

What benchmark did you run?

[ABBAPOH]

The bottom numbers are "sample numbers" - the amount of samples profiler took while stack pointer was in this particular function. But yeah, it seems to get a sample every 1ms.
It does seem to be random, that's why I am not sure this is related to eval/js_create_function.
OTOH the overall instruction count is way bigger according to Valgrind - even 1% is concerning.

I ran our build system on itself - the typical benchmark we do when doing changes to the language. There's a lot of JS code involved so it's hard to pinpoint the exact place.

[bnoordhuis]

Are both binaries built with the (exact!) same compiler and compiler flags? Even flag order can matter.

Does your benchmark contain a lot of code that runs only once? The IC infrastructure is pure overhead for such code at the moment; you pay for bookkeeping but without any payoff. It's on my todo list to introduce a IC preinit mode that delays IC creation until the second or third time the bytecode is executed.

[ABBAPOH]

Are both binaries built with the (exact!) same compiler and compiler flags? Even flag order can matter.

Yes, the only difference was that the new code use c11 instead of c99; but I excluded it by compiling the old version with c11 too.

Does your benchmark contain a lot of code that runs only once?

Can be, yes.
Can you please give the SHA of the commit that introduced the IC stuff so I can try benchmarking before it was introduced? Otherwise I will do a full bisect search.
Or can I disable the feature somehow?

[saghul]

IC was introduced quite a while ago, in v0.2.0: #120

There is no way to disable it, currently.

[bnoordhuis]

See #883 and #884. I ran some benchmarks where I, ahem, surgically removed ICs and here are the results for web-tooling-benchmark (>1.0 means ICs are faster, <1.0 means slower):

       acorn: 1.024
       babel: 0.9386
babel-minify: 1.419
     babylon: 0.9375
       buble: 0.9521
        chai: 0.9507
      espree: 1.033
     esprima: 1.055
      jshint: 0.9683
       lebab: 0.9649
     postcss: 0.9714
     prepack: 0.9403
    prettier: 1.543
  source-map: 1.061
      terser: 1.0
  typescript: 0.8935
   uglify-js: 0.9759
=====================
        mean: 1.0367

babel-minify and prettier benefit a great deal but for everything else it's either a minor win or slower; a good deal slower in case of the typescript compiler benchmark.

The mean suggests they're still a net win on average, or at least, a net win when running benchmarks 🙈

I don't know if that means keep 'em, chuck 'em, or some third option. We're trading memory for CPU. Which is more important?

I also experimented with pre-init ICs. They regress the babel benchmark 5x (which I expected albeit not that dramatically) but otherwise don't move the needle much, which makes sense because they're benchmarks, they don't contain much code that only runs once.

[saghul]

I don't think I'm qualified enough to decide, so I'll ask some questions :-)

• Is there something left on the table that would make the IC work "shine" more?
• Does Add prototype inline cache #551 move the needle in any direction?
• Intuitively I guess we do want both (of course) best CPU and memory usage
  • CPU in cases like lambda functions
  • Memory in constrained environments like IoT?
• IIRC all "big" engines have this. What does it mean? Do they rely on JIT to make the performance leap?
• Would it be hard to have a way to tune it, so those contrained environments could tweak it to their needs? Does that even make sense?

[ammarahm-ed]

@saghul I have a feeling the benchmarks give different results based on what arch you are on, arm64 or x86. Before quickjs-ng came in existence I ran some benchmarks with ic/proto ic and noticed it made performance a lot better. But on my arm64 m4 macbook today, the difference shrinks. So i think it depends on where you are running quickjs?

On an average system, IC will make a big difference compared to No IC.

I can share benchmarks of different qjc variants here if that helps.

[saghul]

Good point about architecture!

[gschwind]

Hello,

It's look like we can remove it, maybe inline cache can be made optional with compile time option ?

Best regards

[ABBAPOH]

See #883 and #884

Thanks, I'll try these!

In the meantime, I tried applying our changes on top of #120, but recent version of changes gives too many conflicts, so I'd need to try a different approach.

[saghul]

In the meantime, I tried applying our changes on top of #120, but recent version of changes gives too many conflicts, so I'd need to try a different approach.

Applying #884 would be the equivalent to the state of the repo pre-120, but with all current changes.

[bnoordhuis]

For the record: the benchmark numbers are from an i7 Intel system.

IIRC all "big" engines have this. What does it mean? Do they rely on JIT to make the performance leap?

V8/SM/JSC:

1. use ICs to collect type feedback for their optimizing tiers; that's more important to them than optimizing property access at the lowest tier because hot code doesn't stay at that tier long

2. have {mono,poly,mega}morphic ICs whereas we only have mono/poly

For property accesses with a degree of polymorphism > 4, we scan the IC cache and perform a hash table lookup in case of a cache miss.

Megamorphic ICs are an end state that says "too much polymorphism so don't bother and do the hash table lookup right away."

They're a potential performance trap though because it's not uncommon for long-living code to have lots polymorphism at first, then stabilize when the program reaches steady state.

I think most optimizing JS engines have second-chance strategies to mitigate that but fine-tuning that is finicky Goldilocks stuff: don't want to flush too often, don't want to wait too long, has to be just right.

Would it be hard to have a way to tune it, so those contrained environments could tweak it to their needs?

At the moment the only thing you could tweak is the number of cache slots, currently 4. Lower that to 3 and the size of an IC on 64 bits systems drops from 56 to 44 bytes, 32 bytes when lowered to 2.

I just tried with N=3. Most benchmarks are unaffected but babel-minify and typescript slow down by 40% and 46% respectively.

Is there something left on the table that would make the IC work "shine" more?

Yes, but no low hanging fruit. This comment is already way too long so I'll save that for another time :p