Tests: Flakyness with "active servers" logic within test suite

[amotl]

Hi,

on [1] and [2], I just observed a flakyness on a single job when running the tests on #397 and #402.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209']

With kind regards,
Andreas.

[1] https://github.com/crate/crate-python/actions/runs/687059879
[2] https://github.com/crate/crate-python/runs/2195227267#step:5:199

[amotl]

Maybe this happens because the retry interval is set to a very low value when running the tests.

crate-python/src/crate/client/test_http.py

Lines 311 to 313 in afb3b0b

	def setUp(self):
	self.client = Client(self.servers)
	self.client.retry_interval = 0.1 # faster retry

So, the invalid_host might have gotten swapped out and back in again too quickly.

I actually recognized that this specific job was executing very slow, so it probably has been scheduled on a CI machine currently experiencing more load than others.

[amotl]

Observed this again when running the tests on behalf of #402. So, I added 89507a0 there.

[amotl]

The same thing happened again at https://github.com/crate/crate-python/runs/2297326550#step:6:184.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209']

[amotl]

Now also, slightly different but the same issue, at https://github.com/crate/crate-python/runs/2297736340#step:6:166.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 61, in http.txt
Failed example:
    http_client._get_server()
Expected:
    'http://invalid_host:9999'
Got:
    'http://even_more_invalid_host:9999'

[amotl]

Same thing happens again ¹²:

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209/']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209/']

Footnotes

1. https://github.com/crate/crate-python/runs/6210356011#step:6:283 ↩

2. https://github.com/crate/crate-python/runs/6300072011#step:5:287 ↩

[amotl]

I did not observe this flaw recently, so I am closing this issue.

[amotl]

It still happens when runners seem to be slow.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209/']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209/']

-- https://github.com/crate/crate-python/actions/runs/3198965176/jobs/5224159502#step:4:338