Connection pool doesn't recover from aborts or resets

[Enchufa2]

Version: 4.0.2

Platform: Python 3.9.7 on Fedora 34

Description: Steps to reproduce:

1. Make redis establish a connection. For instance, in a local setting:

>>> from redis import Redis()
>>> redis = Redis()
>>> redis.get("something")

2. Now there's a connection in the pool. The connection gets killed somehow. For instance:

$ sudo ss -K dst [::1] dport = redis

3. Execute anything else:

>>> redis.get("something")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/***/.local/lib/python3.9/site-packages/redis/commands/core.py", line 1023, in get
    return self.execute_command('GET', name)
  File "/home/***/.local/lib/python3.9/site-packages/redis/client.py", line 1068, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/home/***/.local/lib/python3.9/site-packages/redis/connection.py", line 1179, in get_connection
    if connection.can_read():
  File "/home/***/.local/lib/python3.9/site-packages/redis/connection.py", line 743, in can_read
    return self._parser.can_read(timeout)
  File "/home/***/.local/lib/python3.9/site-packages/redis/connection.py", line 315, in can_read
    return self._buffer and self._buffer.can_read(timeout)
  File "/home/***/.local/lib/python3.9/site-packages/redis/connection.py", line 224, in can_read
    self._read_from_socket(timeout=timeout,
  File "/home/***/.local/lib/python3.9/site-packages/redis/connection.py", line 192, in _read_from_socket
    data = self._sock.recv(socket_read_size)
ConnectionAbortedError: [Errno 103] Software caused connection abort

The same happens if there's a firewall that resets the connection, but then a ConnectionResetError is obtained instead. This doesn't happen with v3.5.3, which automatically detects the loss of connection and sets up a new one.

[Andrew-Chen-Wang]

does running CLIENT KILL have the same effect as ss -K or is Redis doing something extra on CLIENT KILL that allows for cleanup on redis-py and aioredis's side? Because I'm not able to reproduce...

[Enchufa2]

CLIENT KILL closes the connection gracefully. redis 3.5.3 and 4.0.2 do not fail after a CLIENT KILL, aioredis does fail.

[Enchufa2]

The main problem here is that the built-in ConnectionError has BrokenPipeError, ConnectionAbortedError, ConnectionRefusedError and ConnectionResetError as subclasses. But redis and aioredis are overwriting ConnectionError, and these other errors are not subclasses anymore. Consequently, the pool's get_connection method, which expects a ConnectionError, cannot catch them.

One possible solution would be to catch the superclass OSError too.

[Andrew-Chen-Wang]

Ah good catch! We’re these part of socket.error before?

…

On Fri, Dec 3, 2021 at 5:16 AM Iñaki Ucar ***@***.***> wrote: The main problem here is that the built-in ConnectionError has BrokenPipeError, ConnectionAbortedError, ConnectionRefusedError and ConnectionResetError as subclasses. But redis and aioredis are overwriting ConnectionError, and these other errors are not subclasses anymore. Consequently, the pool's get_connection method, which expects a ConnectionError, cannot catch them. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1772 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOLG4VRWYYZV3UVBQMVBS6LUPCKGTANCNFSM5JID4SMQ> .

[Enchufa2]

Ah good catch! We’re these part of socket.error before?

I think so.

[Andrew-Chen-Wang]

Hm aioredis prior to the current latest commit (fix socket.error) had class ConnectionError inherit from RedisError and builtins.ConnectionError which would still lead to an exception being raised if we use the OSError approach.