`__getattribute__` does not propagate `AttributeError` error-message if `__getattr__` fails as well

[randolf-scholz]

This problem only appears when __getattr__ is implemented:

EDIT: Updated MWE

import pandas as pd

class MWE:
    @property
    def foo(self):
        s = pd.Index([1,2,3])
        return s.iloc[0]   # actual bug! pd.Index has no .iloc attribute

MWE().foo  # AttributeError: 'Index' object has no attribute 'iloc'
# without `__getattr__`  implemented we get the right error message!


class MWE_with_getattr:
    def __getattr__(self, key):
        if key == "dummy":
            return "xyz"
        raise AttributeError(f"{self.__class__.__name__} has no attribute {key}!")

    @property
    def foo(self):
        s = pd.Index([1,2,3])
        return s.iloc[0]   # actual bug! pd.Index has no .iloc attribute


MWE_with_getattr().foo  # AttributeError: MWE_with_getattr has no attribute foo!
# Traceback never mentions "AttributeError: 'Index' object has no attribute 'iloc'"
# Implementing `__getattr__` gobbles up the error message!

Expectation

The traceback should include "AttributeError: 'Index' object has no attribute 'iloc". The issue seems to be that object.__getattribute__ does not re-raise the AttributeError in case the fallback __getattr__ fails as well. According to the documentation, one might expect attribute lookup to work like:

class object:
    def __getattribute__(self, key):
        try:
            attr = self._default_attribute_lookup(key)
        except AttributeError as E:
            try:  # alternative lookup via __getattr__
                attr = self.__getattr__(key)
            except Exception as F:
                raise F from E   # re-raise if __getattr__ fails as well
            else:
                return attr
        else:
            return attr

But instead, the original error gets swallowed if __getattr__ fails as well, i.e. it works more like:

class object:
    def __getattribute__(self, key):
        try:
            attr = self._default_attribute_lookup(key)
        except AttributeError as E:
            # if __getattr__ fails, we are left in the dark as to why default lookup failed.
            return self.__getattr__(key)  
        else:
            return attr

This can be problematic, because:

1. Attribute lookup can fail due to actual bugs that unintentionally raise AttributeError, for example within the code of a @property
2. Attributes/Properties might only exist conditionally, and with the current behavior, as the traceback does not include the original AttributeError in case the fallback __getattr__ fails as well, the error message explaining the conditional failure is not passed along.

Actual Output

The code above produces the following output:

Traceback (most recent call last):
  File "/home/rscholz/Projects/KIWI/tsdm/bugs/python/__getattr__attributeerror.py", line 25, in <module>
    MWE_with_getattr().foo  # AttributeError: MWE_with_getattr has no attribute foo!
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rscholz/Projects/KIWI/tsdm/bugs/python/__getattr__attributeerror.py", line 17, in __getattr__
    raise AttributeError(f"{self.__class__.__name__} has no attribute {key}!")
AttributeError: MWE_with_getattr has no attribute foo!

The problem with non-existent .iloc is never mentioned!

Old MWE

from functools import cached_property

class Foo:
    def __getattr__(self, key):
        if key == "secret":
            return "cake"
        raise AttributeError(f"{self.__class__.__name__} has no attribute {key}!")


class Bar(Foo):
    @property
    def prop(self) -> int:
        raise AttributeError("Invisible Error message")

    @cached_property
    def cached_prop(self) -> int:
        raise AttributeError("Invisible Error message")

    filler: str = "Lorem_ipsum"


obj = Bar()
obj.prop  # ✘ AttributeError: Bar has no attribute prop!
obj.cached_prop  # ✘ AttributeError: Bar has no attribute cached_prop!

Expectation

The traceback should include "Invisible Error message". If we replace the AttributeError with, say, ValueError the problem does not occur.

Actual Output

The code above produces the following output:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[11], line 25
     21     filler: str = "Lorem_ipsum"
     24 obj = Bar()
---> 25 obj.prop  # ✘ AttributeError: Bar has no attribute cached_prop!

Cell In[11], line 8, in Foo.__getattr__(self, key)
      6 if key == "secret":
      7     return "cake"
----> 8 raise AttributeError(f"{self.__class__.__name__} has no attribute {key}!")

AttributeError: Bar has no attribute prop!

Your environment

• python 3.11.3
• Ubuntu 22.04

[randolf-scholz]

To avoid this, I think property and cached_property need to wrap the function call in another try-except block and re-raise a different error code than AttributeError. e.g.

try:
    val = self.func(instance)
except Exception as E:
    raise RuntimeError("Property evaluation failed!") from E

instead of the plain

cpython/Lib/functools.py

Line 992 in a5308e1

val = self.func(instance)

[gaogaotiantian]

Not sure if this is a "bug". The doc clearly states that __getattr__ will be called when the default attribute access fails with an AttributeError. Yes, the AttributeError is raised in your code, but it's still an AttributeError and the correct behavior is to call __getattr__ -> which raised the error you saw.

[randolf-scholz]

@gaogaotiantian The problem is that the error message doesn't show in the traceback, which makes it difficult to debug. I wrote this after spending quite some time debugging, because I thought there was a problem with my __getattr__, when the real error occurred in the computation of the property. Tracebacks should always go back to the original source of the error, no?

[gaogaotiantian]

The AttributeError raised in your property is handled so it won't show in traceback. Think of it as:

try:
    obj.prop()
except AttributeError:
    Foo._getattr__(obj, "prop")

I believe the original design for this is for a "non-exist" attribute, not an AttributeError raised when evaluating the attribute. However, I think this is a well-documented expected behavior.

Your proposal is a breaking change, which would affect code that actually taking advantage of this behavior. For example, I might want property methods that make the attribute "appear to be non-existent" so that it could be handled as a missing attribute.

class A:
    @property
    def data(self):
        if <condition>:
            return ...
        else:
            raise AttributeError()

[sunmy2019]

This is a long-existing and fully documented behavior from before Python 3.5.

Do not raise AttributeError then.

[randolf-scholz]

@sunmy2019

Do not raise AttributeError then.

In my case, there was an actual bug in my code, I wasn't intentionally raising an AttributeError during the evaluation of the property (I accidentally used .loc on a pandas.Index object). My point is: if the attribute lookup fails, the traceback should include the error message!

But even when it is intentional, like in the example @gaogaotiantian gave:

class A:
    @property
    def data(self):
        if <condition>:
            return ...
        else:
            raise AttributeError("Failed to load property due to <condition>")

Wouldn't it make sense to include the message "Failed to load property due to <condition>" if __getattr__ fails? I would expect the following behavior:

• If the default default attribute access fails with an AttributeError(msg), then try __getattr__.
• If __getattr__ fails as well, include msg in the traceback.
• Instead, currently we only get the error message from __getattr__!

I believe both you and @gaogaotiantian have misunderstood the point of this issue. The issue is not about how Python's mechanism for looking up attributes works, the issue is that if that mechanism fails, not all relevant error messages are included in the traceback!

[randolf-scholz]

Here is an updated MWE to more clearly demonstrate the issue:

import pandas as pd

class MWE:
    @property
    def foo(self):
        s = pd.Index([1,2,3])
        return s.iloc[0]   # actual bug! pd.Index has no .iloc attribute

MWE().foo  # AttributeError: 'Index' object has no attribute 'iloc'
# without `__getattr__`  implemented we get the right error message!


class MWE_with_getattr:
    def __getattr__(self, key):
        if key == "dummy":
            return "xyz"
        raise AttributeError(f"{self.__class__.__name__} has no attribute {key}!")

    @property
    def foo(self):
        s = pd.Index([1,2,3])
        return s.iloc[0]   # actual bug! pd.Index has no .iloc attribute


MWE_with_getattr().foo  # AttributeError: MWE_with_getattr has no attribute foo!
# Traceback never mentions "AttributeError: 'Index' object has no attribute 'iloc'"
# Implementing `__getattr__` gobbles up the error message!

[sunmy2019]

• If __getattr__ fails as well

if it succeeded, then there is no way to indicate this failure of property access.

[randolf-scholz]

• If __getattr__ fails as well

if it succeeded, then there is no way to indicate this failure of property access.

That's outside the scope of this issue.

[gaogaotiantian]

@randolf-scholz , first of all, we should all agree that this is not a bug report anymore, this is a feature request now. The code behaves as expected and documented, what you want is something more - a better traceback message.

I agree that in your case, being able to know where the attribute error originated would be helpful. The original improvement of yours (reraising as a different exception) was a breaking change (the code I gave would behave differently) so I don't think that would happen.

Is it possible to make __getattr__ track the original AttributeError and raise from it when there's an exception happening in __getattr__? I believe theoretically yes. However, how would you treat the much more common use cases - the normal attribute missing and AttributeError in __getattr__? You would have two redundent tracebacks - the original AttributeError would not contain any useful information.

I don't think it's worth it to complicate a more common use case, just to make a rare one better.

In this specific case, I think the right path is:

    @property
    def foo(self):
        try:
            s = pd.Index([1,2,3])
            return s.iloc[0]   # actual bug! pd.Index has no .iloc attribute
        except AttributeError as exc:
            raise RuntimeError() from exc

Or, write your own wrapper for properties.

You may argue that the user would have to know that the code in the method could potentially raise AttributeError to be able to do this, that's right and that's what they need to consider when they use __getattr__ - it's documented clearly that the method will be called when there's an AttributeError, and the user is responsible to deal with the unexpected AttributeError.