Is the protocol caching still needed?

[jorenham]

Since Python 3.12 runtime-checkable typing.Protocol's use caching and some other tricks to achieve a ~20x speedup: python/cpython#102433

For older Python versions, typing-extensions provides a Protocol variant that includes (most of) these speedups (docs).
For what it's worth: It is used in the optype library, which I use for several of my (other) open- and closed-source projects, but I haven't had any problems with it at all (even though I tend to break everything typing related (not a threat BTW)) 🤷🏻.

I haven't done a benchmark, but regardless of the outcome, this could perhaps be a nice opportunity to make numerary more focused towards what all the cool kids are talking about these days: typing in Python 😎. Even fusion-powered quantum-blockchain AI has got nothing on us!

~

Hoping this banal participial closing causes no offense, I am,

Typically,

Joren S. Hammudoglu,
Typer of Things

[posita]

Thanks, @jorenham! As one might surmise, updates from the various metropoli of performant runtime type checking are slow to reach these backwaters of Appalachia, so we very much appreciate it when kind travelers bring news as they pass through. And please do not worry yourself on the matter of provoking offense. I try to reserve that for much more serious matters which largely ^{but tragically not absolutely} exist outside of software development.

And now for something completely different totally predictable: A rant!

It is deliberate that numerary describes itself as a "codified rant". The fundamental problem is that official "development" ^{and I use that term liberally as applied here} at the intersection of operations and types is a hot mess beyond the most trivial cases ^{and sometimes even then}. Every time I've tried to make use of a structure or feature in the standard library whose documentation is simple, authoritative, and clear, I find that the reality is anything but. There is always some unfinished corner or undeclared edge case that frustrates fitness for the purpose explicitly envisioned by the feature's designer(s).

No matter the path down which I tread, I find myself in the middle a murky swamp impassible with flora, fauna, and developers, who declare my very presence as an intrusion ^{often without patience or explanation, as if under the assumption that non-residents of that bog should magically just know to temper our understanding of the official documentation and various accepted PEPs through the lens of whatever in-the-moment, jargon-laden sentiment or "reasoning" again, used liberally casually littered throughout developer forums, mailing lists, or buried in some comment in some bug report that was closed as "not planned" five years ago}. Through lands far and wide, I have tracked and pursued ABCs, various Supports* conventions, the numeric type hierarchy, generics (to the extent Python has them), protocols, operations and dunder methods, and other head spinning chicanery like that which you've discovered. At the end of every exploration, I encounter the same small group of wizards declaring that I "shall not pass" or that my way lies "back the way I came" and that I should either be satisfied to limit my use to ints and floats or commit to explicit support of a particular third party mathematics library ^{most of which are thankfully at least more forthcoming about what is and isn't supported}.

I have a fantasy that Python 4 will be the typing-finally-works-coherently-and-we-should-be-clear-what-is-deprecated-by-getting-rid-of-everything-Python-developers-no-longer-wish-to-maintain release, but that isn't based on anything in reality and I'm not holding my breath.

In other words, I wonder if your prompt should really read, "Is the protocol caching numerary still needed?"

I love what you've done with the place. (optype just might be the new numerary!)

I would love to see performance numbers comparing Beartype's CachingProtocolMeta to 3.12's improved library version, but it's possible that numerary is no longer needed (or perhaps, more accurately, a dead-end). What I was essentially trying to capture is a way for things to declare capabilities and then allow for other things to offer functionality around those capabilities.

Unless I've misunderstood, this is precisely what optype does!

I still have yet to figure out is how to constrain secondary functionality based on the capabilities of the things on which that functionality should operate. That's probably too vague to be coherent, so an example may be in order. Consider:

R = TypeVar("R")
T = TypeVar("T")
type CanTMulR[T, R] = CanMul[T, R] | CanRMul[R, T]
type CanTAddR[T, R] = CanAdd[T, R] | CanRAdd[R, T]

class MyThingThatDoesMathLikeThingsWithOtherThings(Generic[T]):
    def __init__(self, thing: T):
        self._thing = thing
    def do_the_multiplication_thing(self, other: CanTMulR[T, R]) -> ???:
        if isinstance(other, CanRMul):
            return self._thing * other
        else:
            return other * self._thing
    def do_the_addition_thing(self, other: CanTAddR[T, R]) -> ???:
        if isinstance(other, CanRAdd):
            return self._thing + other
        else:
            return other + self._thing

There are two specific problems I don't yet know how to solve. (To be fair, numerary doesn't solve these either.)

1. How to declare/detect ^{at least statically, but ideally at runtime as well} that MyThingThatDoesMathLikeThingsWithOtherThings.do_the_multiplication_thing is unavailable if CanTMulR[T, R] doesn't resolve, but other methods may still work.
2. How to infer the return type of (e.g.) MyThingThatDoesMathLikeThingsWithOtherThings.do_the_multiplication_thing based on what T and R are (i.e., reveal_type(MyThingThatDoesMathLikeThingsWithOtherThings(2).do_the_multiplication_thing("I") yields str). Maybe something like ResultTypeForMul[T, R] | ResultTypeForRMul[R, T]? Although that isn't quite right. What I really mean is ResultTypeForMulOrRMulIfMulNotAvailable[T, R].

I don't know whether either problem is solvable given the current state of the world? The second may require some kind of "registration" mechanism? As an aside, division (__div__ and __rdiv__) has always been my go-to case, because it violates the naive assumption that A __op__ B always yields one of A or B (e.g., int / int -> float).

What do we do, tho, bro?!

Complicating a simple reversion to the standard library version is that (much of) CachingProtocolMeta is actually part of beartype. It's true that numerary provides its own work-arounds for declaring/overriding inclusion/exclusion for runtime checking, but those rely heavily on the proprietary internals of Beartype's implementation. There are a couple options I can see, pending some performance validation:

1. Revert/remove beartype's _CachingProtocolMeta and pin numerary's reliance on beartype to an upper bound where it was still present (and possibly declare numerary as officially deprecated and use its README to advocate for adopting optype instead 🥳).
2. Revert/remove beartype's _CachingProtocolMeta and remove numerary's include/exclude mechanism altogether (which means runtime type checking would be ... erm ... less correct 😬).
3. Revert/remove beartype's _CachingProtocolMeta and re-implement numerary's include/exclude mechanism such that it is self-contained within numerary.

Assuming there's no longer any performance benefit to beartype's _CachingProtocolMeta, the last option feels like the "right" thing to do ^{assuming anyone is actually using numerary for anything useful, which is probably a stretch}.

[jorenham]

Wow, that must be the best response I've ever gotten; you have my thanks 🏆 .

Every time I've tried to make use of a structure or feature in the standard library whose documentation is simple, authoritative, and clear, I find that the reality is anything but.

This is my experience as well: Seemingly simple things like binary operators, or iterations, are ridiculously to correctly type, often resulting in a mess of type aliases, protocols, overloads, typeshed PR's that get rejected for fallacious reasons, and angry neighbors telling you to stop screaming at your screen at 4 a.m...

I encounter the same small group of wizards declaring that I "shall not pass" or that my way lies "back the way I came"

I think that I have encountered this buggy endboss a couple of times as well. It's one of their special attacks that gets me, usually its either "dogma beam" or "fallacy cannon", or when they're at 1HP, they exploit a glitch that replaces them with an another, almost identical, boss.

In other words, I wonder if your prompt should really read, "Is the protocol caching numerary still needed?"

Yes, it is!
The numbers stdlib creates more problems than it solves, collections.abc doesn't seem to care about numbers, and there's that thing where at runtime you have that bool <: int, but whilst type checking it's treated as if complex <: float <: int <: bool (type-checkers allow it if you e.g. pass a bool to some def f(x: int), which is just rude to those with a bool-allergy).

Plus, there are more that one type of number; e.g. fractions.Fraction, decimal.Decimal, or any of the subtypes of numpy.number[Any].
And even when you dump you integer-like input into an int(x), you can't just accept everything that implements __int__ or __index__; e.g. numpy.ndarray implements both, but unless it is 0-dimensional, it'll raise a TypeError; which is impossible to type from within numpy itself (or at least until numpy fully supports shape typing).
I'm not sure whether this can all be covered by numerary, but I know that optype is too low-level do even come close.

Relying on typeshed alone won't do you much good either, as they're just typed versions of the runtime.
But what's needed is a bridge between the world of runtime and the world of typing.
That bridge will have to be shaped like a bunch of typing protocols that are flexible, but not too loose: Stiff bridges break, loose bridges wobble.
And currently, many people attempt to use the numbers lib as a bridge, even though it's made of moldy cardboard and duct-tape.

There are two specific problems I don't yet know how to solve. (To be fair, numerary doesn't solve these either.)

1. How to declare/detect at least statically, but ideally at runtime as well that MyThingThatDoesMathLikeThingsWithOtherThings.do_the_multiplication_thing is unavailable if CanTMulR[T, R] doesn't resolve, but other methods may still work.
2. How to infer the return type of (e.g.) MyThingThatDoesMathLikeThingsWithOtherThings.do_the_multiplication_thing based on what T and R are (i.e., reveal_type(MyThingThatDoesMathLikeThingsWithOtherThings(2).do_the_multiplication_thing("I") yields str). Maybe something like ResultTypeForMul[T, R] | ResultTypeForRMul[R, T]? Although that isn't quite right. What I really mean is ResultTypeForMulOrRMulIfMulNotAvailable[T, R].

How about something like this?

from __future__ import annotations

from typing import Any, Generic, TypeVar, overload
from optype import CanMul, CanRMul

_T_co = TypeVar('_T_co', covariant=True)
_R = TypeVar('_R')
_RHS = TypeVar('_RHS')


class SomeThingThatDoesMathLikeThingsWithOtherThings(Generic[_T_co]):
    def __init__(self, thing: _T_co) -> None:
        self._thing = thing

    @overload
    def do_the_multiplication_thing(
        self,
        other: CanRMul[_T_co, _R],
    ) -> _R: ...
    @overload
    def do_the_multiplication_thing(
        self,
        other: CanMul[_T_co, _R],
    ) -> _R: ...
    @overload
    def do_the_multiplication_thing(
        self: SomeThingThatDoesMathLikeThingsWithOtherThings[CanMul[_RHS, _R]],
        other: _RHS,
    ) -> _R: ...
    def do_the_multiplication_thing(
        self,
        other: CanRMul[_T_co, _R] | CanMul[_T_co, _R] | Any,
    ) -> _R:
        if isinstance(other, CanRMul):
            return self._thing * other
        if isinstance(other, CanMul):
            return other * self._thing
        return self._thing * other

Complicating a simple reversion to the standard library version is that (much of) CachingProtocolMeta is actually part of beartype. It's true that numerary provides its own work-arounds for declaring/overriding inclusion/exclusion for runtime checking, but those rely heavily on the proprietary internals of Beartype's implementation. There are a couple options I can see, pending some performance validation:

Purely in terms of isinstance speed, the Python 3.12+ runtime-protocols are great.
But because of that, they are very imprecise, and often result in false-positives, of which many should be easily avoidable.

So consider the following standard boilerplate code for some regular business application:

from typing import SupportsIndex, Self


class SomethingPositiveYouCanCountOn:
    """
    >>> _ = SomethingPositiveYouCanCountOn()
    >>> +_
    +_
    >>> ++++_
    ++++_
    """
    def __init__(self, /, *, _start: SupportsIndex = 0) -> None:
        self._value = int(_start)

    def __pos__(self) -> Self:
        return type(self)(_start=self._value + 1)

    def __repr__(self) -> str:
        return '+' * self._value + '_'


class SomethingThatLooksPositiveButIsNot:
    __pos__ = NotImplemented

How can we want to predict whether some x can have a + in front of it, but without actually trying it (i.e. what you do in practice if you want to avoid side-effects, e.g. sloths taking over the world, forcing humans to act as their trees).

Runtime-protocols are the way to go here:

>>> from optype import CanPos
>>> isinstance(SomethingPositiveYouCanCountOn(), CanPos)
True

Now let's check SomethingThatLooksPositiveButIsNot:

>>> isinstance(SomethingThatLooksPositiveButIsNot(), CanPos)
True

Ouch.

So let's check what every 3-year old python developer knows: that even if __pos__ is implemented, +x can only be used on the instance, but not on the type itself

>>> isinstance(SomethingThatLooksPositiveButIsNot, CanPos)
True

So yes, a better typing.Protocol might be a good idea 🦥

[posita]

So this was kind of rad (cc @leycec; hat tip to @AlexWaygood for the beartype mention; apologies that I'm only just now getting caught up on all of this):

Given that behaviour, it could well be reasonable to slap a cache on the whole _ProtocolMeta.__instancecheck__ call, similar to the way beartype does it... .

I also missed this:

FYI, all the changes that have been made to typing_extensions.Protocol in v4.6.0 are also going to be included in typing.Protocol in Python 3.12. The changes (to both typing.Protocol and typing_extensions.Protocol) were mostly made in order to achieve a 20x speedup for isinstance() checks against runtime-checkable protocols.

As well as the fact that typing_extensions.Protocol is much faster than typing.Protocol on <3.12, typing_extensions also backports several significant bugfixes that have been made to typing.Protocol in recent Python versions, but haven't been backported as far as Python 3.8.

If I'm reading that right, using typing_extensions.Protocol for Python <3.12 may very well be good enough to sunset beartype.typing._typingpep544._CachingProtocolMeta. I still want to do some perf testing, though. This would have to be optional, meaning maybe beartype could use the typing_extensions.Protocol if available, but I don't think it would (or should) tolerate a dependency.

However, based on your examples, @jorenham, (which are unsurprising to those who know how __instancecheck__ is implemented), I think numerary's registration bit may still be needed. I would love it if this could be done via Python's existing machinery. However, while ABCMeta allows explicit registration to avoid false negatives, it does not afford explicit exclusion to avoid false positives like the ones you identify. Without realizing it, numerary ended up trying to implement what turned out to be an old idea (cf. this, this, and this).

I'll try to tinker some more once I can carve out some time.

[AlexWaygood]

Yeah, would definitely encourage y'all to do detailed perf testing before making any decisions. We made a ton of optimisations to isinstance() checks against typing.Protocol in 3.12 and they're way faster than they were in 3.11, but:

• They're still really slow compared to isinstance() checks against other classes
• For a runtime-checking protocol with many protocol members (15 or 20), I think they're actually slower than they were in 3.11
• We don't do caching to the same degree as beartype because we have to be backwards compatible with behaviour on previous Python versions
• Not all optimisations could be backported to typing_extensions: some of the optimisations were achieved by optimising inspect.getattr_static, and those couldn't be realistically backported to typing_extensions.

There's a benchmark script in the PR description for python/cpython#103280 (comment) that you might find useful

[leycec]

Since Lithuanian breakdancer Dominika Banevič just got robbed in the B-Girls Olympics gold medal Breaking battle, I must now console myself with some fast-acting GitHubbing. I drain my sorrows in keyboard warrioring. </ahem> What I meant to say was:

Thanks so much, @AlexWaygood! That PR benchmark script is especially awesome. If CPython's official typing.Protocol implementation is anywhere near @beartype's mostly unmaintainable beartype.typing.Protocol implementation, I'll delightedly kick the latter to the curb. While that cat is fast as lightning, @beartype's current approach is also:

• Badly broken for class methods. I noticed a year or two ago that beartype.typing.Protocol silently ignores class methods, whereas typing.Protocol appears to properly detect class methods. In theory, beartype.typing.Protocol could be fixed to properly detect class methods as well. In practice, it might be preferable to just amputate this gangrenous limb from the @beartype codebase and let somebody else who is smarter than me maintain this special madness.
• Unmaintainable. Let us never forget that.

Will @leycec actually do something? Will @posita save the bitter day? Stay tuned as we all just watch more caustic Olympics.