Graceful Recovery test needs to be improved #2091
Description
Describe the bug
Graceful recovery test possibly have a race condition leading to unexpected failures in pipeline. The behavior is intermittent.
To Reproduce
Steps to reproduce the behavior:
- Follow this guide to run functional tests locally.
Expected behavior
No intermittent failures for graceful recovery tests
Your environment
- Version of the NGINX Gateway Fabric - release version 1.2
- Version of Kubernetes
Client Version: version.Info{Major:"1", Minor:"26+", GitVersion:"v1.26.13-dispatcher", GitCommit:"eb237aa977a4a0cf4fcec65fc730b4d96af37ccf", GitTreeState:"clean", BuildDate:"2024-02-08T00:19:39Z", GoVersion:"go1.20.13", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.0", GitCommit:"7c48c2bd72b9bf5c44d21d7338cc7bea77d0ad2a", GitTreeState:"clean", BuildDate:"2024-05-13T22:02:25Z", GoVersion:"go1.22.2", Compiler:"gc", Platform:"linux/arm64"}
```
- Kubernetes platform - Kind
* Details on how you expose the NGINX Gateway Fabric Pod - Read test setup
* Logs of NGINX container: `kubectl -n nginx-gateway logs -l app=nginx-gateway -c nginx` no logs
* NGINX Configuration: `kubectl -n nginx-gateway exec <gateway-pod> -c nginx -- nginx -T` no logs
**Additional context**
Add any other context about the problem here. Any log files you want to share.