[CMake] Force bootstrapping to HOSTTOOLS if the new parser is enabled

[bnbarham]

We cannot mix building the Swift libs with a host and new compiler. If the new parser is enabled, force to host tools.

Resolves rdar://116686158.

[bnbarham]

@swift-ci please test

[bnbarham]

preset=buildbot_incremental_asan,tools=RDA,stdlib=RDA

@swift-ci please test with preset macOS platform

[eeckstein]

lgtm

[bnbarham]

preset=buildbot_incremental_asan,tools=RDA,stdlib=RDA

@swift-ci please test with preset macOS platform

[bnbarham]

@swift-ci please test

[finagolfin]

Since this does exactly the same as the above block, is there a reason you didn't just add this condition there, ie changing the above check to if(SWIFT_HOST_VARIANT_SDK MATCHES "LINUX|OPENBSD|FREEBSD" OR BOOTSTRAPPING_MODE MATCHES "BOOTSTRAPPING.*") instead?

[bnbarham]

Just that I wanted to separate the actual reason for them, though it's odd I didn't update the message in that case. I'll update to if(NOT BOOTSTRAPPING_MODE STREQUAL "OFF" AND NOT BOOTSTRAPPING_MODE STREQUAL "HOSTTOOLS") instead. Unless there was a reason we were forcing bootstrapping on for Linux, maybe there was 🤔

[finagolfin]

No, that was an oversight on linux. You can just drop the second part of your new check, as it doesn't matter if it simply sets hosttools again, but are you sure you want to override the CROSSCOMPILE* modes also on macOS?

[finagolfin]

I dropped Android from that list because I didn't want CROSSCOMPILE overridden for Android.

[bnbarham]

You can just drop the second part of your new check, as it doesn't matter if it simply sets hosttools again

We output a warning, which I don't want if it's already hosttools.

but are you sure you want to override the CROSSCOMPILE* modes also on macOS?

Yes, bootstrapping is being removed. First step to that is forcing hosttools, then merging SwiftCompilerSources + ASTGen, and once that's complete (or maybe at the same time), removing all the bootstrapping CMake.

I dropped Android from that list because I didn't want CROSSCOMPILE overridden for Android

As above, bootstrapping is being removed. Would you prefer I force it to OFF here (rather than HOSTTOOLS) for Android?

[finagolfin]

Yes, bootstrapping is being removed. First step to that is forcing hosttools, then merging SwiftCompilerSources + ASTGen, and once that's complete (or maybe at the same time), removing all the bootstrapping CMake.

OK, but this also overrides the cross-compilation modes to build the compiler itself, eg building the compiler for macOS x86_64 on a macOS arm64 host or Android AArch64 on a linux x86_64 host. Of course, those modes are currently broken, because cross-compiling the Swift portions of the Swift compiler is currently unconfigured, but I was planning on filing an issue for that.

As above, bootstrapping is being removed.

As above, CROSSCOMPILE has nothing to do with bootstrapping: it is a way to cross-compile the Swift compiler itself with an already built Swift compiler.

Would you prefer I force it to OFF here (rather than HOSTTOOLS) for Android?

Nah, something like if(NOT SWIFT_HOST_VARIANT_SDK STREQUAL "ANDROID" AND NOT BOOTSTRAPPING_MODE MATCHES "HOSTTOOLS|OFF") should be fine instead. Of course, that still overrides all the cross-compilation modes for non-Android platforms, so you'll probably want to exempt that too.

[bnbarham]

To be clear, BOOTSTRAPPING= is being removed. Even if I do if(NOT SWIFT_HOST_VARIANT_SDK STREQUAL "ANDROID" AND NOT BOOTSTRAPPING_MODE MATCHES "HOSTTOOLS|OFF"), it won't continue to work for long - the path forward is requiring a host compiler + setting the various CMake flags.

[finagolfin]

To be clear, BOOTSTRAPPING= is being removed.

It should be renamed, but I don't think it should be removed, as there is still a difference between HOSTTOOLS and CROSSCOMPILE-WITH-HOSTLIBS, that you would be overriding with this pull as currently written.

Even if I do if(NOT SWIFT_HOST_VARIANT_SDK STREQUAL "ANDROID" AND NOT BOOTSTRAPPING_MODE MATCHES "HOSTTOOLS|OFF"), it won't continue to work for long - the path forward is requiring a host compiler + setting the various CMake flags.

To be clear, that should be if(NOT SWIFT_HOST_VARIANT_SDK STREQUAL "ANDROID" AND NOT BOOTSTRAPPING_MODE MATCHES "HOSTTOOLS|CROSSCOMPILE|OFF") for now. Once you remove the two bootstrapping modes, you can also remove this hosttools override you are modifying here altogether, but we will still need to differentiate between host builds and cross-compiles of the Swift compiler.

[bnbarham]

It should be renamed, but I don't think it should be removed,

CROSSCOMPILE* is only used to build SwiftCompilerSources, which aren't necessary to build the compiler regardless - hence my suggestion of OFF above. SwiftCompilerSources will end up merged with ASTGen, which doesn't (and won't) use BOOTSTRAPPING at all.

[finagolfin]

CROSSCOMPILE* is only used to build SwiftCompilerSources, which aren't necessary to build the compiler regardless - hence my suggestion of OFF above.

I haven't looked into that, but we will still need to handle cross-compiling the new swift-syntax parser too once it becomes mandatory.

SwiftCompilerSources will end up merged with ASTGen, which doesn't (and won't) use BOOTSTRAPPING at all.

Of course, nobody wants the BOOTSTRAPPING* modes but there would be no reason to throw out this existing CROSSCOMPILE* support for SwiftCompilerSources/ASTGen then either. Rather, we should build on it to allow cross-compiling all parts of the Swift compiler that are written in Swift, instead of only the few we currently support.

[bnbarham]

@swift-ci please test

[eeckstein]

@bnbarham do we have a plan to resolve the blocking issues so that this can be merged?

[bnbarham]

@bnbarham do we have a plan to resolve the blocking issues so that this can be merged?

The TLDR it is that forcing to HOSTTOOLS isn't something we can do as BOOTSTRAPPING_MODE impacts both the compiler build itself + the stdlib. We either want to:

1. Split BOOTSTRAPPING_MODE into two separate variables, one for the compiler and one for the stdlib.
2. Move SwiftCompilerSources into ASTGen (probably want to rename ASTGen at that point).

But just outright forcing isn't going to work. I'm not sure which I prefer here, (2) was our eventual goal anyway, so maybe it's easier to just do it rather than try to hack around it?

[finagolfin]

Let me also note that the current iteration of this pull is to simply force hosttools on all platforms when swift-syntax is enabled, which I think would have broken build-toolchain on macOS as that currently uses CROSSCOMPILE-WITH-HOSTLIBS to cross-compile the compiler for arm64.

[bnbarham]

Let me also note that the current iteration of this pull is to simply force hosttools on all platforms when swift-syntax is enabled, which I think would have broken build-toolchain on macOS as that currently uses CROSSCOMPILE-WITH-HOSTLIBS to cross-compile the compiler for arm64.

This is mostly what I'm referring to above. Using the "host" (builder, whatever we want to call it) compiler works perfectly fine for cross-compiling the compiler itself, just not the stdlib - the stdlib needs to be built with the new compiler, but the just-built compiler for the arm64 slice obviously can't run on x86.

And sure, we could change to just not force to HOSTTOOLS when CROSSCOMPILE-WITH-HOSTLIBS, but that doesn't really help us at all - we want to build eg. the arm64 with the host compiler.

[finagolfin]

And sure, we could change to just not force to HOSTTOOLS when CROSSCOMPILE-WITH-HOSTLIBS, but that doesn't really help us at all - we want to build eg. the arm64 with the host compiler.

I don't understand how that last part, ie "build eg. the arm64 with the host compiler," is ever going to happen. The reason "the stdlib needs to be built with the new compiler" is because the HOSTTOOLS Swift compiler version is 5.8 or 5.9 and they usually cannot build the latest trunk 6.0 stdlib. Unless you plan on installing prebuilt trunk 6.0 snapshots on the CI as the HOSTTOOLS Swift compiler, the current CROSSCOMPILE-WITH-HOSTLIBS mode will need to stick around.

[bnbarham]

I don't understand how that last part, ie "build eg. the arm64 with the host compiler,"

That should have read "the arm64 Swift parts of the compiler with the host compiler". I'm not referring to the stdlib there.

The reason "the stdlib needs to be built with the new compiler" is because the HOSTTOOLS Swift compiler version is 5.8 or 5.9 and they usually cannot build the latest trunk 6.0 stdlib

That's partially the reason, but really it's that we only support building the stdlib with the just built compiler. Doesn't matter what version is on the host, it would always be insufficient.

Unless you plan on installing prebuilt trunk 6.0 snapshots on the CI as the HOSTTOOLS Swift compiler, the current CROSSCOMPILE-WITH-HOSTLIBS mode will need to stick around.

Like I was trying to explain above, there's two separate parts of the build that BOOTSTRAPPING_MODE is used for. One is the Swift parts of the compiler and the other is the stdlib. The former can be built with the host compiler. The latter, yes, needs to keep CROSSCOMPILE-WITH-HOSTLIBS today, at least until https://forums.swift.org/t/implementing-parts-of-the-swift-compiler-in-swift/59524/49 is in.