[SystemZ] Large compile time regression in `SystemZTTIImpl::adjustInliningThreshold()`

[alexrp]

#106058 appears to have caused a large compile time regression in LLVM 20. zig-bootstrap used to be able to compile and link the zig binary for s390x-linux-(gnu,musl) in a matter of minutes when we were on LLVM 19. Now on LLVM 20, I decided to cancel the build after it had been running for about an hour.

Some investigation shows that a lot of time is being spent in SystemZTTIImpl::adjustInliningThreshold(), in particular in this section:

llvm-project/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp

Lines 98 to 105 in c1c0d55

	for (const GlobalVariable &Global : M->globals())
	for (const User *U : Global.users())
	if (const Instruction *User = dyn_cast<Instruction>(U)) {
	if (User->getParent()->getParent() == Callee)
	CalleeGlobals.insert(&Global);
	if (User->getParent()->getParent() == Caller)
	CallerGlobals.insert(&Global);
	}

Some thoughts:

• A module can have lots of globals.
  • This doesn't appear to be the problem here, however.
• A global can have lots of users (think common deduplicated/merged constant values).
  • This seems to be the case here, though I couldn't determine exactly how many users we're talking about because I was using a RelWithDebInfo build of LLVM where p Global.users() was optimized out.
    • Building a Debug compiler as I file this to hopefully find out...
• std::set seems less than ideal compared to std::unordered_set here.
  • I tried changing this to std::unordered_set but it didn't seem to have too much of an impact. Still seems desirable to do, though.

In general, all of this seems like a lot of work to be doing on ~every inlining decision.

cc @JonPsson1 @uweigand

[alexrp]

• A module can have lots of globals.
  • This doesn't appear to be the problem here, however.

Ok, I take that back:

(gdb) p M->GlobalList.size()
$24 = 61011

There doesn't actually appear to be any globals with a disproportionately high number of users, but this is still an awful lot of globals and users thereof to be iterating on ~every inlining decision.

[JonPsson1]

Thanks for the report, I will take a look. Any chance of a test case, like a single file that takes a lot longer to compile?

[alexrp]

I can try to get you a bitcode file tomorrow. It's going to be fairly large; do you have a preference for where I should upload it?

In the meantime, if you feel like it and have cores that need exercise (it's fine if you don't!), the issue is reproducible by cloning zig-bootstrap and running e.g. CMAKE_GENERATOR=Ninja CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) ./build s390x-linux-gnu baseline. That'll build host LLVM + Zig and then use those to build LLVM + Zig to run on s390x-linux-gnu. The latter step is where the slowdown kicks in.

[JonPsson1]

Would it work to upload it here if compressed? Otherwise I don't know one way or the other, I guess there should be some place for large files we could use.

[alexrp]

Would it work to upload it here if compressed?

I think the limit on here is 25M, but let's see...

[alexrp]

zig.txt (actually a .tar.xz containing zig.bc but GitHub is silly about file extensions)

Weirdly enough, I can't actually reproduce the issue when I run LLVM 20's llc on this file, whether it's with -O0 or -O3. I don't quite understand why that would be the case, because I don't think we do anything particularly crazy in setting up the optimization pipeline in Zig:

• https://github.com/ziglang/zig/blob/2d33cc2e42d40ce0ee798d4d56c2d7da93ead44a/src/zig_llvm.cpp#L83-L155
• https://github.com/ziglang/zig/blob/2d33cc2e42d40ce0ee798d4d56c2d7da93ead44a/src/zig_llvm.cpp#L216-L418

Maybe I'm missing some llc flags?

[JonPsson1]

ah, the inliner is run in the middle-end, while llc runs the backend passes. It should work if you run the 'opt' tool instead of llc (with the IR emitted by the front-end). If you use -save-temps with clang, you should find a .bc file you could use with e.g.

opt -O3 -mtriple=s390x-linux-gnu -mcpu=z16

[alexrp]

Ah, that'll do it. opt -O3 -mtriple s390x-linux-gnu zig.bc on the file uploaded above seems to exhibit the problem.