Python 3.8+ features in typeshed stubs

[JukkaL]

If typeshed would start using Python 3.8 only features, such as positional-only arguments, tools such as mypy that use typed-ast when running under 3.7 or earlier would sometimes be unable to parse those stubs, since typed-ast won't get any post-3.7 features according to current policies. This seems to rule out using 3.8 features in typeshed. Similarly, in case Python 3.9 introduces a new syntax for optional types, for example, it looks like we wouldn't be able to use that syntax in typeshed (at least for several years).

I see a few options that could help with the issue:

1. Work towards establishing a typeshed policy that stubs should only use syntax that is supported by the oldest maintained Python 3+ release (currently that would be 3.5). We actually already use 3.6 syntax in typeshed, but we could make it an exception.
2. Revert to the original policy of backporting all new Python syntax to typed-ast.

Option 1 could become a problem in a few scenarios that don't seem totally implausible:

• An existing keyword stops being a keyword, so older Python parsers won't be able to represent identifiers with that name. This could mean that we can't represent some stdlib module stubs precisely.
• Some new type related syntax gets introduced for which we can't design a good fallback syntax for older Python releases.

@gvanrossum @ilevkivskyi @msullivan What do you think about this?

[srittau]

I'd strongly prefer option 2 or any other option that allows typeshed and stubs in general to use the latest syntax. Mainly for two reasons:

• New syntax makes stubs more readable. This is especially important in the case of typestub that is widely used.
• Since typeshed is so central, people will use it as example when writing their own stubs. Therefore it should preferably use best practices.

In the (currently somewhat stalled but still planned) typestub PEP we recommend to use the latest syntax features in stubs, even if the implementation supports older Python versions.

[gvanrossum]

That’s going to be a problem unless we get some more volunteers to add the new features to typed_ast (or do a whole new backport from Python 3.8).

[ilevkivskyi]

On one hand I like the second option more, but on the other hand I don't volunteer to do the backports :-P (so maybe counting my opinion is not really fair).

[msullivan]

I think the best approach might be to backport only the features necessary for stubs. So for 3.8, that would mean positional arguments but not the walrus operator, and not a full rebase. That is likely to be lower effort than the full rebase, I think.

[gvanrossum]

If it's just positional arguments, the simplest change might be to just ignore the / in argument lists. Then mypy can still check these when it's running under 3.8. (Though that also hasn't been implemented yet.)

[JukkaL]

Support for positional-only arguments was recently added to mypy, so supporting them in mypy is not going to be a problem as long as typed-ast supports them.

More generally, selective backporting seems feasible, at least until there are big changes in the CPython parser. Mypy would still have to refuse to run in full 3.8 mode on Python 3.7, since not all 3.8 syntax would be supported.

[LouisStAmour]

My suggestion:

Like TypeScript, we could have a way to version the syntax such that syntax checkers can specify which version of “typed Python” they support from the same “latest” stub package. (In a future where typeshed publishes packages.) We could automatically translate newer syntax to a compatible older syntax until everyone has caught up, for example. To that end though, I think typeshed's standard library with support for multiple platforms and CPython target versions should be based on typeshed's but should ship with each type checker as the type checkers themselves might support different implementations of Python and it gives something out-of-the-box for end users to use, but that third-party packaged typeshed modules should “target” different versions of “typed Python” and thus remain compatible with multiple versions of the “typed Python” language.

I’m not sure if my use of “typed Python” language is precise enough, but I’m arguing that Python is to JS what “typed Python” is to TypeScript and thus different versions of the TypeScript compiler support different language features. Taken to a logical conclusion, when publishing if the goal is to support older versions of “typed Python” alongside new ones, we need a similar versioning strategy for “stubs” such that folks can opt in to newer “typed Python” features as they become available in type checkers.

Further context for my thoughts on the above at: https://gitter.im/python/typing?at=5eafad1a22f9c45c2a6a6e97 and https://gitter.im/python/typing?at=5eafd0c9a9de3d01b1e85685

Examples from TypeScript:

https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/node/v12/ts3.7/base.d.ts (note the file paths reference both which Node version they're targeting and which TS language version.) The equivalent to Node for Python would be which version of Python standard library they're supporting, while the equivalent to TS language version would be an increment of supported functionality for "typed Python" by MyPy or other third-party type checkers. If these diverge too significantly, it could be the name of the type checker and some other checker-specific versioning. Alternatively, we could find a different prefix to use to clarify that python/3.8/tpy3.7 refers to the legacy MyPy approaches, while python/3.8/tpy3.8 refers to every PEP included in 3.8, etc.

End users would not be exposed to the complexity of knowing a "typed Python" version to load because in such a scenario, the correct version would ship with with the type checker. They would only have to specify then which Python standard library they would prefer to use.

[LouisStAmour]

Documentation on the above approach from TypeScript is officially at https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-1.html#version-selection-with-typesversions and you can see an example DefinitelyTyped/DefinitelyTyped#44117

This appears common for TypeScript -- rather than support macros, guards or conditionals, TS relies on Node's package.json to determine which files to load, however the primary use of package.json to determine whether a package is loading in "node" or in "browser" environment isn't supported by TypeScript's module resolution microsoft/TypeScript#7753 so "universal" packages aren't supported via this syntax, and instead the code must do detection at runtime to determine whether to load browser or node-specific code, and for efficiency, this gets removed later at compile time by replacing runtime checks with known constants for each target, and performing tree shaking to remove branches that always evaluate to false from the build output, though this last compile time step is a build optimization.

So part of the problem is we don't have a consistent version number to increment for "typed Python" features we'd like to support regardless of what Python syntax you're targeting. It's possible someone would prefer to annotate using newer language features or proprietary extensions and we should have something capable of "graceful degradation" or version selection such that you could use a modern syntax to annotate, well, Python 2.x for example, such as by adopting new features/guards while also providing older syntax compatible files too.

If I'm understanding how Python packaging generally works, packages that want to support multiple versions of Python either need to restrict themselves to only commonly available features (the lowest supported language version including in dependencies) or they would need to come up with their own method of packaging/downgrading code such that they could release multiple packages each of which target a different version of Python as necessary.

In this regard, it's very similar to Node and Ruby where there aren't macros to support different language versions, instead you could hypothetically statically scan code for language and SDK features by version and then build a "supported versions" list from common features, or at runtime you could attempt to import different versions of modules (but this is rarely well-supported) or as part of a build script, specify alternative module versions for specific builds (also rarely done).

So the common case, targeting the lowest version and maybe providing syntax for alternative versions at build-time, is thus straightfoward when you can compile "typed JS" to regular JS, but it's a bit trickier for "typed Python" because ideally we'd want to be able to convert from "typed Python" of some newer language variety to "runtime Python" of an older variety so as to adopt newer Typed Python features while offering maximum downstream compatibility for library users. The alternative is to restrict yourself to only typings within pyi interfaces, and that sounds like a hassle to maintain, because you can't easily have one single source of truth.

[gvanrossum]

This is occasionally proposed (hi, @viridia :-). But Python does not use transpilation technology and many users who are currently using type annotations would have to rethink how they actually run their code if it had to be transpiled to the syntax their interpreter version actually supports.

IOW, I think there's a fatal flaw in the argument that "typed Python" is to "Python" what TypeScript is to JS.

[LouisStAmour]

Thanks for your reply! As I said, I am new to Python, so my apologies if this has been considered before, or if I'm not quite using the right terminology here. I'd be happy to work out more of the details or prototype something.

But let me try to clarify: JS transpilation is itself only beneficial in limited uses, and eventually, by design, is an overhead to be eliminated -- but it's also a moving target by design because it allows for adopting newer language features sooner:

"Python does not use transpilation technology" -- technically, neither does JavaScript. And while TypeScript was defined as a superset of JS, it doesn't have to use transpilation either: https://github.com/denoland/deno for example.

That said, transpiling, or in other words, compiling for lower targets by providing "polyfills" or alternatives is something that's required on the web because of how many possible runtimes the browser enables. I agree if the goal is remaining up-to-date, then we should all target the latest version, or maybe n-2 version of python, but for library authors who want to adopt newer features but remain compatible potentially all the way down to Python 2, there's an argument that if a feature can be polyfilled or ignored, you could still do so.

I'm not suggesting that this should be a default. Like typing, I think this should be adopted by advanced users, perhaps the way "six" or "python-future" were adopted, where "if you know you need it, use it" otherwise simply ignore it. To that end, I'm not even sure I'm suggesting that "typed python" requires its own filename extension, as theoretically, it's still just python but at a different version.

The comparison I might then make is JSX, which is an optional syntax of React that by convention uses .jsx extension instead of .js though either would work based on your project config. To that end, I'm perhaps suggesting a .py38 convention for files in a Python 3.8 syntax that might in turn compile to .py, or alternatively it's something that could run as part of a build-and-package process with an output folder. Thus you could say or specify that the src code is typed Python 3.8 while the destination package, optionally, could be Python 3.6 with type annotations or .pyi files.

If this sounds confusing, the same confusion technically exists in JavaScript, but we've gotten used to it by now. When writing JS, you're allowed to use any syntax available in your runtime environment, so you have to specify "the latest" runtime environment. When you want to target older browsers or produce a module others can use, you will likely produce one or more such outputs, and at times, the API for picking between them is a simple filename. You could do the same in Python, for example https://nvbn.github.io/2017/05/31/import-different/

Again, this would be completely optional. Those who only need to support one runtime environment can continue to do so, even with types. Typeshed, on the other hand, would ideally support publishing packages with either multiple targets or the lowest common target. To that end, in DefinitelyTyped's example, they only make exceptions for early TS (2.x), TS 3.2 and TS 3.7 so you'd only need to adopt this kind of approach for when new features to specify types or interfaces are introduced, and only if they make your life easier.

I actually think that the .ts file extension made adopting TypeScript harder, because step 1 was always, "tell people to rename their files" and then step 2 was "incrementally add types!". But it has the advantage of allowing .ts and compiled .js versions of identical files to live side-by-side, so you could import the .ts source for a typescript project and the .js source for a JS project.

There are issues with this, however, primarily in that most if not all npm projects have unique build processes, and therefore importing raw source files to use always required setting up a compatible build tool chain. This is one of the other reasons why JS files, though executable, are compiled -- a compiled JS file can be imported regardless of build process, as long as it's outputs code for the module system in use (these days ECMAScript module syntax).

But that's a different kind of dependency management: an example in Python-land might be the idea that developers could themselves adopt a Python 3.8 module into their downstream Python 2 project, say, by creating a build process that imports compatible polyfills into the upstream library, runs tests, and produces a Python 2 compatible module compatible with their target environment. Then set up CI to try and run the process against every new upstream version, and abandon the process as you update to Python 3.

Very few to nobody actually does this in practice for TypeScript or JavaScript unless they're running a monorepo for their dependencies, but the approach is still sound, and is somewhat inspired by Bazel's hermetic builds. (Often folks don't realise they need to micro-manage their dependencies this way for "best results" because it's simply "good enough" for libraries to use a subset of syntax supported by older versions of the language, or to require library end users to upgrade to the latest version of the language in order to use the library.

I think maybe I've not stated my case simply enough. I hope the above helps clarify a few points, but I'd be happy to either discuss this further or maybe try and outline things using um, fewer words and more specific Python examples. It'll just take me more time to edit some of my thoughts into coherence and research and write enough Python to come up with a prototype or two. My expertise is more JS/TS, then any non-Lisp language except Python. Early on I'd picked Ruby, mostly because of Rails, and Python was just close enough for a few years that I couldn't see it as a different language. But a number of PEPs have changed my mind, modern Python syntax is very appealing, and so I figured contributing to typed Python would be a great way to learn more about the language and its common packages/modules. I just think there's a lot to be learned from the "great TypeScript experiment" that some of the JavaScript community has opted in to, and it's worth trying to take some "lessons learned" from its approach.

For specific code examples from TypeScript, have a look at python/typeshed#2491 (comment) and later conversation that clarifies a few misconceptions of how TypeScript works and how it supports JavaScript in IDEs.

[gvanrossum]

I definitely suggest that you try to present this with actual examples from Python -- your "wall of text" approach doesn't seem to reach my brain somehow. :-( Also I'm not sure that this is the right repo to discuss it.

[LouisStAmour]

Okay, I realise I went down a rabbit hole on compatibility-via-transpiling, that wasn't my original intent in this issue.

For the sake of this suggestion, the "typed Python" version refers to the supported syntax of type stubs, such as MyPi (double underscore or Python 3.6), Python 3.8 and Python 3.9, etc. Ideally these would be versioned or named in such a way that it's obvious which syntax file belongs to which parser version, and the ideal scenario is that we wouldn't have to maintain older versions once they EOL in shipping/in-use type checkers. We could use telemetry to help determine this if the privacy implications aren't too terrible.

What I said originally still stands, that types in typeshed are broken out into two parts:

1. Those that ship with MyPy and other type checkers for standard library or extensions/plugins, and
2. Those that follow specific, ideally versioned, increments of typed Python syntax supported by type checkers.

For 1: This should be automatically bundled with the type checker, I covered them earlier by saying:

https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/node/v12/ts3.7/base.d.ts (note the file paths reference both which Node version they're targeting and which TS language version.) The equivalent to Node for Python would be which version of Python standard library they're supporting, while the equivalent to TS language version would be an increment of supported functionality for "typed Python" by MyPy or other third-party type checkers. If the type checkers diverge too significantly in syntax, it could be the name of the type checker and some other checker-specific versioning. Alternatively, we could find a different prefix to use to clarify that python/3.8/tpy3.7 refers to the legacy MyPy approaches, while python/3.8/tpy3.8 refers to every PEP included in 3.8, etc.

End users would not be exposed to the complexity of knowing a "typed Python" version to load because in such a scenario, the correct version would ship with with the type checker. They would only have to specify then which Python standard library they would prefer to use.

For 2: What you'd end up with for a given python module is a stubs package, versioned identically to the upstream python package, built to include the source files for every typed Python syntax supported (3.6, 3.8, 3.9, etc.) and published automatically by typeshed. The package is then loaded by the type checker and contains metadata that type checkers could read to determine which files to load based on the most recent syntax the type checker supports. The setuptools metadata (or whatever the package format contains) would look something like https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-1.html#version-selection-with-typesversions or DefinitelyTyped/DefinitelyTyped#44117

Again, end users would not be aware that we're shipping and supporting potentially multiple syntaxes because in such a scenario, the correct version would load automatically from a universal package of type stubs.

Finally, we've the maintenance issue of supporting multiple versions of type checkers in the same typeshed repo. The more we can automate the transpiling of syntax from an unsupported version to a supported version, the easier it is to introduce the different syntax versions either as generated files potentially committed to the repo, or as packaging artifacts as part of a package build process.

A hypothetical file listing for one module, codecs, in 5 versions of python and 4 versions of spec syntax:

• typeshed/stdlib/2and3/stub_py39/codecs.pyi
• typeshed/stdlib/2and3/stub_py38/codecs.pyi
• typeshed/stdlib/2and3/stub_py36/codecs.pyi
• typeshed/stdlib/2and3/stub_mypy/codecs.pyi
• typeshed/stdlib/3.6/stub_py39/codecs.pyi
• typeshed/stdlib/3.6/stub_py38/codecs.pyi
• typeshed/stdlib/3.6/stub_py36/codecs.pyi
• typeshed/stdlib/3.6/stub_mypy/codecs.pyi
• typeshed/stdlib/3.7/stub_py39/codecs.pyi
• typeshed/stdlib/3.7/stub_py38/codecs.pyi
• typeshed/stdlib/3.7/stub_py36/codecs.pyi
• typeshed/stdlib/3.7/stub_mypy/codecs.pyi
• typeshed/stdlib/3.8/stub_py39/codecs.pyi
• typeshed/stdlib/3.8/stub_py38/codecs.pyi
• typeshed/stdlib/3.8/stub_py36/codecs.pyi
• typeshed/stdlib/3.8/stub_mypy/codecs.pyi
• typeshed/stdlib/3.9/stub_py39/codecs.pyi
• typeshed/stdlib/3.9/stub_py38/codecs.pyi
• typeshed/stdlib/3.9/stub_py36/codecs.pyi
• typeshed/stdlib/3.9/stub_mypy/codecs.pyi

To reduce effort, it would be ideal if we could re-use content from other stub spec versions via transpiling or inclusion/additional annotation syntax to dynamically modify previous specifications. Ideally there would be an incremental way to load interfaces with redefinitions, but if not, or if that's too complicated, the assumption is that the type checker will ship, and load, only the stubs valid for its format. So if it were MyPy in this hypothetical, it could ship:

• typeshed/stdlib/2and3/stub_mypy/codecs.pyi
• typeshed/stdlib/3.6/stub_mypy/codecs.pyi
• typeshed/stdlib/3.7/stub_mypy/codecs.pyi
• typeshed/stdlib/3.8/stub_mypy/codecs.pyi
• typeshed/stdlib/3.9/stub_mypy/codecs.pyi

If a future version of, say Pyright, supports Python 3.8 syntax, it could instead use:

• typeshed/stdlib/2and3/stub_py38/codecs.pyi
• typeshed/stdlib/3.6/stub_py38/codecs.pyi
• typeshed/stdlib/3.7/stub_py38/codecs.pyi
• typeshed/stdlib/3.8/stub_py38/codecs.pyi
• typeshed/stdlib/3.9/stub_py38/codecs.pyi

It's up to the type checker if they want to preserve the above folder paths or package the stdlib folder, and its variants, differently.

As to whether typescript is like typed Python, I'll leave that argument for another day, or another thread. :)