SymbolGraphGen: Unsafe{Mutable?}{Raw?}Pointer members have vanished

[tayloraswift]

Describe the bug
all APIs inherited from _Pointer have apparently vanished from standard library symbolgraphs as of DEVELOPMENT-SNAPSHOT-2022-07-06-a. many of these APIs are quite important, such as:

• UnsafePointer.successor()
• UnsafePointer.init(bitPattern:)

this only affects APIs inherited from underscored protocols, @available(*, unavailable) APIs (such as SIMD2.+(_:_:)) are unaffected.

Steps To Reproduce
Steps to reproduce the behavior:

1. generate a symbolgraph with an invocation like:

subprocess.run((swift_symbolgraph_extract, 
                '-skip-inherited-docs', 
                '-target', 'x86_64-unknown-linux-gnu', 
                '-output-dir', 'Swift', 
                '-module-name', 'Swift'), 

Expected behavior

_Pointer APIs should be present, as they are in symbolgraphs generated by the april toolchain.

[tayloraswift]

update: it looks like the regression took place between the 5/31/2022 nightly and the 6/02/2022 nightly

[QuietMisdreavus]

This might be fallout from #59531 - though the dates wouldn't line up, since that was only merged on 2022-06-23 (and its first pass was merged on 2022-06-16 and reverted that same day).

I would guess that, even though these symbols are "public API", the fact that their protocol is prefixed with an underscore might suggest that they weren't intended to be. I'll ask around and see whether that was the case.

[tayloraswift]

the date in my last comment was off by a couple of days, 6/2 actually contains the members. they disappeared after 6/2, but reappeared in the 6/20 nightly only. they disappeared again in the 7/06 nightly.

members of underscored protocols are absolutely public API. (arguably, so are the underscored protocols.) aside from the standard library, projects like SwiftNIO also depend on this.

[tayloraswift]

in general, i would urge you to not try to make assumptions about the "publicness" of an API based off of its spelling or @available(*, unavailable) attribute(s) at the symbol graph level.

many packages, like swift-system and swift-nio, do this kind of thing intentionally, and it should really be the responsibility of higher-level tooling (eg DocC, Biome, Jazzy, etc) to decide whether to show them or not.

[QuietMisdreavus]

Underscored symbols are being filtered out by language specification:

Treat identifiers that begin with an underscore as internal, even if their declaration has the public access-level modifier. This convention lets framework authors mark part of an API that clients must not interact with or depend on, even though some limitation requires the declaration to be public.

APIs that are marked as @available(*, unavailable) are hidden because their availability information implies that they are unusable on every platform. These are APIs that are explicitly not meant to be used - are there packages that are deliberately using these unavailable APIs?

[QuietMisdreavus]

I can picture adding a flag to swift-symbolgraph-extract and/or swift-frontend to disable the underscore filtering or to re-add unavailable APIs, but i wouldn't want it by default. It's a policy of Swift-DocC to leave symbol filtering to the source of the symbol graphs, for example.

[tayloraswift]

yes, i am aware of TSPL’s prescription regarding underscored symbols, but this simply isn’t realistic to apply as a blanket policy, given how many projects in the ecosystem flaunt this recommendation. for example, it is not really possible to work with SwiftNIO without interacting with _ChannelInboundHandler and its members.

@available(*, unavailable) is often used expressly for documentation purposes. for example, swift-system vends all of the C Errno cases as unconditionally unavailable API, in order to educate users about alternatives to that API. in fact, documentation is the only reason those declarations exist, since they cannot actually be referenced by code.

a feature flag to opt-out of this behavior sounds like the correct solution to me. it does not need to be on by default.

[QuietMisdreavus]

After chatting with some standard-library devs offline, i'm going to revert the PRs that implemented the "underscored protocol" behavior. I'll come back to add the flags to add the new behavior (showing underscored and unavailable symbols).

[tayloraswift]

thank you so much!

[QuietMisdreavus]

I've posted #60096 and #60100 to revert the underscored protocol behavior change.