[API server] graceful upgrade with client retry

[aylei]

This PR introduce graceful upgrade support with a client-side retry mechanism, which break down the upgrade process to the following states:

1. Serving state: all requests are served
2. User initiate an rolling-upgrade, kubelet send SIGTERM to API server, API server enters Cordon state. Behavior:
   • For new requests: return 503 (ask for client retry)
   • For internal requests: cancel
   • For /logs requests: cancel and mark it should be retried, the client will get retry error when streaming the log or call /api/get on the logs request
   • For other requests: wait for them to complete
3. All requests finish or the wait timeout reaches, enter Terminating state:
   • Nginx returns 503 for all requests, which asks client to retry
4. New server Pod created, enter Serving state:
   • All the requests that are waiting for retry on the server will get through to the API server.

The key differences with our previous graceful rolling-upgrade:

• The entire process is now fully automated;
• Client retry ensures there is no error occurs if the graceful rolling upgrade get completed within the timeout, where timeout is configurable;

Known issues:

• There is still a downtime since during the rolling-upgrade, new requests have to wait. This will be addressed in the future by introducing multi-replicas support for API server;
• Nginx sends 503 to client whenever there is no live backends (e.g. the API server is crashed and cannot be recovered), which may cause the client retries for a very long time (now the default retry timeout is 10 minutes).

UX:

Message when log tailing is interrupted by server upgrade:

Recovered after the upgrade complete:

Tested (run the relevant ones):

• Code formatting: install pre-commit (auto-check on commit) or bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
• Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
• Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

[Michaelvll]

Thanks @aylei! The PR looks awesome! Left some questions below. Also, is it possible to have a README for how to test this?

[aylei]

/quicktest-core

https://buildkite.com/skypilot-1/quicktest-core/builds/547#0197a255-5d5b-40de-8ebb-b4157713d6e8 passed

[aylei]

/smoke-test

https://buildkite.com/skypilot-1/smoke-tests/builds/1588#0197a4a9-a0e6-42db-8dfb-69e7f19a2318 passed

[Michaelvll]

Thanks for making this possible @aylei! LGTM once the tests passed.

We need the following two things to be added as well:

1. Docs for the behavior of the client side retry
2. Test script to avoid regression

[aylei]

/smoke-test

https://buildkite.com/skypilot-1/smoke-tests/builds/1590 succeed

[aylei]

/quicktest-core