gRPC via nginx ingress

[menvol3]

Hi,

I'm using a tool that utilizes gRPC-Java for communication between the client and server. The server is located in an AWS EKS cluster and is accessible externally via NGINX ingress.
To configure it, I followed this guide: https://kubernetes.github.io/ingress-nginx/examples/grpc/

After deploying all components, I tested it with grpcurl and received a successful response.

Then, I configured communication between the agent and the server, and it also worked. However, I encountered an issue when the agent lost its connection for a period of time during communication
Below is a log message

Agent log

Feb 03 14:24:11 test-agent-1 prometheus_proxy_agent[1042983]: 14:24:11.811 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [Agent test-agent-1]
Feb 03 14:24:11 test-agent-1 prometheus_proxy_agent[1042983]: 14:24:11.812 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Feb 03 14:24:11 test-agent-1 prometheus_proxy_agent[1042983]: 14:24:11.813 INFO  [Agent.kt:153] - Resetting agentId [Agent test-agent-1]
Feb 03 14:24:11 test-agent-1 prometheus_proxy_agent[1042983]: 14:24:11.813 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
Feb 03 14:24:12 test-agent-1 prometheus_proxy_agent[1042983]: 14:24:12.534 INFO  [AgentClientInterceptor.kt:58] - Assigned agentId: 2488 to Agent{agentId=2488, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [grpc-default-executor-5]
Feb 03 14:24:12 test-agent-1 prometheus_proxy_agent[1042983]: 14:24:12.535 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Feb 03 14:24:12 test-agent-1 prometheus_proxy_agent[1042983]: 14:24:12.900 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
Feb 03 14:24:12 test-agent-1 prometheus_proxy_agent[1042983]: 14:24:12.901 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 5s of inactivity [DefaultDispatcher-worker-9]
Feb 03 14:25:16 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:16.865 WARN  [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR [Agent test-agent-1]
Feb 03 14:25:16 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:16.866 INFO  [Agent.kt:216] - Waited 0s to reconnect [Agent test-agent-1]
Feb 03 14:25:16 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:16.866 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [Agent test-agent-1]
Feb 03 14:25:16 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:16.867 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Feb 03 14:25:16 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:16.868 INFO  [Agent.kt:153] - Resetting agentId [Agent test-agent-1]
Feb 03 14:25:16 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:16.868 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
Feb 03 14:25:17 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:17.699 INFO  [AgentClientInterceptor.kt:58] - Assigned agentId: 2492 to Agent{agentId=2492, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [grpc-default-executor-6]
Feb 03 14:25:17 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:17.700 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Feb 03 14:25:18 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:18.039 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
Feb 03 14:25:18 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:18.040 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 5s of inactivity [DefaultDispatcher-worker-6]
Feb 03 14:26:21 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:21.936 WARN  [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR [Agent test-agent-1]
Feb 03 14:26:21 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:21.937 INFO  [Agent.kt:216] - Waited 0s to reconnect [Agent test-agent-1]
Feb 03 14:26:21 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:21.937 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [Agent test-agent-1]
Feb 03 14:26:21 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:21.938 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Feb 03 14:26:21 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:21.939 INFO  [Agent.kt:153] - Resetting agentId [Agent test-agent-1]
Feb 03 14:26:21 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:21.940 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
Feb 03 14:26:22 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:22.664 INFO  [AgentClientInterceptor.kt:58] - Assigned agentId: 2495 to Agent{agentId=2495, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [grpc-default-executor-7]
Feb 03 14:26:22 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:22.665 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Feb 03 14:26:23 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:23.107 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
Feb 03 14:26:23 test-agent-1 prometheus_proxy_agent[1042983]: 14:26:23.108 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 5s of inactivity [DefaultDispatcher-worker-12]
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:25.801 WARN  [ScrapeResults.kt:120] - fetchScrapeUrl() java.util.concurrent.CancellationException: Parent job is Cancelling - http://127.0.0.1:9273/metrics [DefaultDispatcher-worker-11]
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]: java.util.concurrent.CancellationException: Parent job is Cancelling
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.engine.UtilsKt$attachToUserJob$cleanupHandler$1.invoke(Utils.kt:99)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.engine.UtilsKt$attachToUserJob$cleanupHandler$1.invoke(Utils.kt:97)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlinx.coroutines.InvokeOnCancelling.invoke(JobSupport.kt:1571)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlinx.coroutines.JobSupport.invokeOnCompletionInternal$kotlinx_coroutines_core(JobSupport.kt:500)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlinx.coroutines.JobSupport.invokeOnCompletion(JobSupport.kt:452)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlinx.coroutines.Job$DefaultImpls.invokeOnCompletion$default(Job.kt:313)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.engine.HttpClientEngineKt.createCallContext(HttpClientEngine.kt:166)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:91)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.engine.HttpClientEngine$DefaultImpls.access$executeWithinCallContext(HttpClientEngine.kt:24)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:70)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.execute$ktor_utils(DebugPipelineContext.kt:63)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:86)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpSend$DefaultSender.execute(HttpSend.kt:118)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.auth.AuthKt$Auth$2$2.invokeSuspend(Auth.kt:130)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.auth.AuthKt$Auth$2$2.invoke(Auth.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.auth.AuthKt$Auth$2$2.invoke(Auth.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpRequestRetryKt$HttpRequestRetry$2$1.invokeSuspend(HttpRequestRetry.kt:296)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpRequestRetryKt$HttpRequestRetry$2$1.invoke(HttpRequestRetry.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpRequestRetryKt$HttpRequestRetry$2$1.invoke(HttpRequestRetry.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpTimeoutKt$HttpTimeout$2$1.invokeSuspend(HttpTimeout.kt:175)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpTimeoutKt$HttpTimeout$2$1.invoke(HttpTimeout.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpTimeoutKt$HttpTimeout$2$1.invoke(HttpTimeout.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpRedirectKt$HttpRedirect$2$1.invokeSuspend(HttpRedirect.kt:97)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpRedirectKt$HttpRedirect$2$1.invoke(HttpRedirect.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpRedirectKt$HttpRedirect$2$1.invoke(HttpRedirect.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpCallValidatorKt$HttpCallValidator$2$2.invokeSuspend(HttpCallValidator.kt:112)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpCallValidatorKt$HttpCallValidator$2$2.invoke(HttpCallValidator.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpCallValidatorKt$HttpCallValidator$2$2.invoke(HttpCallValidator.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpSend$Plugin$install$1.invokeSuspend(HttpSend.kt:84)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpSend$Plugin$install$1.invoke(HttpSend.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpSend$Plugin$install$1.invoke(HttpSend.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.RequestError$install$1.invokeSuspend(HttpCallValidator.kt:134)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.RequestError$install$1.invoke(HttpCallValidator.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.RequestError$install$1.invoke(HttpCallValidator.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.SetupRequestContext$install$1.invokeSuspend$proceed(HttpRequestLifecycle.kt:40)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.SetupRequestContext$install$1.access$invokeSuspend$proceed(HttpRequestLifecycle.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.SetupRequestContext$install$1$1.invoke(HttpRequestLifecycle.kt:40)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.SetupRequestContext$install$1$1.invoke(HttpRequestLifecycle.kt:40)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpRequestLifecycleKt$HttpRequestLifecycle$1$1.invokeSuspend(HttpRequestLifecycle.kt:27)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpRequestLifecycleKt$HttpRequestLifecycle$1$1.invoke(HttpRequestLifecycle.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.HttpRequestLifecycleKt$HttpRequestLifecycle$1$1.invoke(HttpRequestLifecycle.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.SetupRequestContext$install$1.invokeSuspend(HttpRequestLifecycle.kt:40)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.SetupRequestContext$install$1.invoke(HttpRequestLifecycle.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.plugins.SetupRequestContext$install$1.invoke(HttpRequestLifecycle.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.DebugPipelineContext.execute$ktor_utils(DebugPipelineContext.kt:63)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:86)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.HttpClient.execute$ktor_client_core(HttpClient.kt:1393)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.statement.HttpStatement.fetchResponse(HttpStatement.kt:147)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.ktor.client.statement.HttpStatement.execute(HttpStatement.kt:68)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at com.github.pambrose.common.dsl.KtorDsl.get(KtorDsl.kt:85)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.prometheus.agent.AgentHttpService.fetchContent(AgentHttpService.kt:90)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.prometheus.agent.AgentHttpService.fetchContentFromUrl(AgentHttpService.kt:76)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.prometheus.agent.AgentHttpService.fetchScrapeUrl(AgentHttpService.kt:61)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.prometheus.agent.AgentGrpcService$readRequestsFromProxy$2$1$2.invokeSuspend(AgentGrpcService.kt:298)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.prometheus.agent.AgentGrpcService$readRequestsFromProxy$2$1$2.invoke(AgentGrpcService.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.prometheus.agent.AgentGrpcService$readRequestsFromProxy$2$1$2.invoke(AgentGrpcService.kt)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at io.prometheus.Agent$run$connectToProxy$3$4.invokeSuspend(Agent.kt:186)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:589)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:832)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:720)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]:         at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:707)
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:25.808 WARN  [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR [Agent test-agent-1]
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:25.808 INFO  [Agent.kt:216] - Waited 0s to reconnect [Agent test-agent-1]
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:25.808 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [Agent test-agent-1]
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:25.809 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:25.810 INFO  [Agent.kt:153] - Resetting agentId [Agent test-agent-1]
Feb 03 14:27:25 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:25.810 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
Feb 03 14:27:26 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:26.526 INFO  [AgentClientInterceptor.kt:58] - Assigned agentId: 2499 to Agent{agentId=2499, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [grpc-default-executor-8]
Feb 03 14:27:26 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:26.527 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Feb 03 14:27:26 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:26.892 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
Feb 03 14:27:26 test-agent-1 prometheus_proxy_agent[1042983]: 14:27:26.893 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 5s of inactivity [DefaultDispatcher-worker-1]

To rule out any potential errors with the app, I also tested it with a simpler configuration, where the server was deployed on a standard EC2 instance and made available to the web. In this setup, I didn’t encounter any problems; everything worked as expected. So, it seems that the issue lies somewhere in the configuration of the NLB used by NGINX ingress or with NGINX ingress itself

Below is a visualization of how often the connection is dropped

[kannanjgithub]

Feb 03 14:25:16 test-agent-1 prometheus_proxy_agent[1042983]: 14:25:16.865 WARN  [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR [Agent test-agent-1]

The AgentService gRPC client receives RST_STREAM from the server, the client doesn't know any other details, you should check the Prometheus server proxy side logs to see what happened.
I see a similar issue grpc/grpc-node#1747 reported in grpc-node and AWS, you can see this reply summarizing the diagnosis. The customer's issue was apparently fixed by increasing the heartbeat timeout to 10s. I see your logs show 5s as the time between pings. If the server getting overwhelmed by the heartbeat pings is the cause of the error (along with some bug in the Prometheus server's gRPC service code) you can may be try increasing the timeout also.

[menvol3]

Thanks for the reply @kannanjgithub

I've attached the server log

Server log

14:24:06.400 WARN  [ProxyPathManager.kt:139] - Missing agent context for agentId: 2483 (Termination) [grpc-nio-worker-ELG-3-3]
14:24:06.401 WARN  [AgentContextManager.kt:53] - Missing AgentContext for agentId: 2483 (Termination) [grpc-nio-worker-ELG-3-3]
14:24:06.401 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected with invalid agentId: 2483 [grpc-nio-worker-ELG-3-3]
14:24:08.070 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2486 (Termination) [grpc-nio-worker-ELG-3-1]
14:24:08.070 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2486, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=59.998507193s} for agentId: 2486 (Termination) [grpc-nio-worker-ELG-3-1]
14:24:08.070 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2486, launchId=Unassigned, consolidated=false, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=59.998573739s} [grpc-nio-worker-ELG-3-1]
14:24:11.897 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2485 (Termination) [grpc-nio-worker-ELG-3-2]
14:24:11.897 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2485, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 3.825870393s} for agentId: 2485 (Termination) [grpc-nio-worker-ELG-3-2]
14:24:11.897 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2485, launchId=Unassigned, consolidated=false, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 3.825934448s} [grpc-nio-worker-ELG-3-2]
14:24:11.898 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2484 (Termination) [grpc-nio-worker-ELG-3-4]
14:24:11.898 INFO  [ProxyPathManager.kt:150] - Removed path /test-agent-1 for AgentContextInfo(consolidated=false, labels={},agentContexts=[AgentContext{agentId=2484, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=15.745416063s}]) [grpc-nio-worker-ELG-3-4]
14:24:11.898 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2484, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=15.745473072s} for agentId: 2484 (Termination) [grpc-nio-worker-ELG-3-4]
14:24:11.898 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2484, launchId=CRudPRd53VSWxFS, consolidated=false, valid=false, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=15.745532372s} [grpc-nio-worker-ELG-3-4]
14:24:12.453 INFO  [AgentContextManager.kt:40] - Registering agentId: 2488 [grpc-nio-worker-ELG-3-4]
14:24:12.651 INFO  [ProxyServiceImpl.kt:96] - Connected to AgentContext{agentId=2488, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=197.759281ms} [DefaultDispatcher-worker-5]
14:24:12.819 INFO  [ProxyPathManager.kt:82] - Added path /test-agent-1 for AgentContext{agentId=2488, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=366.072715ms} [DefaultDispatcher-worker-6]
14:24:13.043 INFO  [AgentContextManager.kt:40] - Registering agentId: 2490 [grpc-nio-worker-ELG-3-2]
14:24:13.043 INFO  [AgentContextManager.kt:40] - Registering agentId: 2489 [grpc-nio-worker-ELG-3-1]
14:24:16.738 INFO  [AgentContextCleanupService.kt:50] - Evicting agentId 2487 after 1m 3.664075204s (max 1m) of inactivity: AgentContext{agentId=2487, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 3.664095634s} [AgentContextCleanupService]
14:24:16.738 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2487 (Eviction) [AgentContextCleanupService]
14:24:16.738 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2487, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 3.664272109s} for agentId: 2487 (Eviction) [AgentContextCleanupService]
14:24:18.003 INFO  [AgentContextManager.kt:40] - Registering agentId: 2491 [grpc-nio-worker-ELG-3-3]
14:24:26.111 INFO  [CallLogging.kt:45] - 200 OK: GET - /test-agent-1 - prometheus-kube-prometheus-stack-prometheus-0.prometheus-operated.monitoring.svc.cluster.local [DefaultDispatcher-worker-4]
14:24:56.140 INFO  [CallLogging.kt:45] - 200 OK: GET - /test-agent-1 - prometheus-kube-prometheus-stack-prometheus-0.prometheus-operated.monitoring.svc.cluster.local [DefaultDispatcher-worker-6]
14:25:11.935 WARN  [ProxyPathManager.kt:139] - Missing agent context for agentId: 2487 (Termination) [grpc-nio-worker-ELG-3-3]
14:25:11.935 WARN  [AgentContextManager.kt:53] - Missing AgentContext for agentId: 2487 (Termination) [grpc-nio-worker-ELG-3-3]
14:25:11.935 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected with invalid agentId: 2487 [grpc-nio-worker-ELG-3-3]
14:25:13.042 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2489 (Termination) [grpc-nio-worker-ELG-3-1]
14:25:13.042 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2489, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=59.998934897s} for agentId: 2489 (Termination) [grpc-nio-worker-ELG-3-1]
14:25:13.042 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2489, launchId=Unassigned, consolidated=false, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=59.999030482s} [grpc-nio-worker-ELG-3-1]
14:25:16.739 INFO  [AgentContextCleanupService.kt:50] - Evicting agentId 2490 after 1m 3.696316872s (max 1m) of inactivity: AgentContext{agentId=2490, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 3.696337839s} [AgentContextCleanupService]
14:25:16.739 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2490 (Eviction) [AgentContextCleanupService]
14:25:16.739 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2490, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 3.696527758s} for agentId: 2490 (Eviction) [AgentContextCleanupService]
14:25:17.007 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2488 (Termination) [grpc-nio-worker-ELG-3-4]
14:25:17.007 WARN  [ProxyPathManager.kt:139] - Missing agent context for agentId: 2490 (Termination) [grpc-nio-worker-ELG-3-2]
14:25:17.007 INFO  [ProxyPathManager.kt:150] - Removed path /test-agent-1 for AgentContextInfo(consolidated=false, labels={},agentContexts=[AgentContext{agentId=2488, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=20.878498248s}]) [grpc-nio-worker-ELG-3-4]
14:25:17.007 WARN  [AgentContextManager.kt:53] - Missing AgentContext for agentId: 2490 (Termination) [grpc-nio-worker-ELG-3-2]
14:25:17.008 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected with invalid agentId: 2490 [grpc-nio-worker-ELG-3-2]
14:25:17.008 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2488, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=20.878568563s} for agentId: 2488 (Termination) [grpc-nio-worker-ELG-3-4]
14:25:17.008 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2488, launchId=CRudPRd53VSWxFS, consolidated=false, valid=false, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=20.878878457s} [grpc-nio-worker-ELG-3-4]
14:25:17.617 INFO  [AgentContextManager.kt:40] - Registering agentId: 2492 [grpc-nio-worker-ELG-3-4]
14:25:17.786 INFO  [ProxyServiceImpl.kt:96] - Connected to AgentContext{agentId=2492, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=169.595291ms} [DefaultDispatcher-worker-4]
14:25:17.956 INFO  [ProxyPathManager.kt:82] - Added path /test-agent-1 for AgentContext{agentId=2492, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=339.060823ms} [DefaultDispatcher-worker-4]
14:25:18.127 INFO  [AgentContextManager.kt:40] - Registering agentId: 2493 [grpc-nio-worker-ELG-3-1]
14:25:18.127 INFO  [AgentContextManager.kt:40] - Registering agentId: 2494 [grpc-nio-worker-ELG-3-2]
14:25:23.133 INFO  [AgentContextManager.kt:40] - Registering agentId: 2495 [grpc-nio-worker-ELG-3-3]
14:25:26.269 INFO  [CallLogging.kt:45] - 200 OK: GET - /test-agent-1 - prometheus-kube-prometheus-stack-prometheus-0.prometheus-operated.monitoring.svc.cluster.local [DefaultDispatcher-worker-8]
14:25:26.740 INFO  [AgentContextCleanupService.kt:50] - Evicting agentId 2491 after 1m 8.736333250s (max 1m) of inactivity: AgentContext{agentId=2491, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 8.736353560s} [AgentContextCleanupService]
14:25:26.740 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2491 (Eviction) [AgentContextCleanupService]
14:25:26.740 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2491, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 8.736528364s} for agentId: 2491 (Eviction) [AgentContextCleanupService]
14:25:56.071 INFO  [CallLogging.kt:45] - 200 OK: GET - /test-agent-1 - prometheus-kube-prometheus-stack-prometheus-0.prometheus-operated.monitoring.svc.cluster.local [DefaultDispatcher-worker-2]
14:26:16.778 WARN  [ProxyPathManager.kt:139] - Missing agent context for agentId: 2491 (Termination) [grpc-nio-worker-ELG-3-3]
14:26:16.779 WARN  [AgentContextManager.kt:53] - Missing AgentContext for agentId: 2491 (Termination) [grpc-nio-worker-ELG-3-3]
14:26:16.779 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected with invalid agentId: 2491 [grpc-nio-worker-ELG-3-3]
14:26:18.126 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2494 (Termination) [grpc-nio-worker-ELG-3-2]
14:26:18.126 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2494, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=59.999069096s} for agentId: 2494 (Termination) [grpc-nio-worker-ELG-3-2]
14:26:18.126 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2494, launchId=Unassigned, consolidated=false, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=59.999183830s} [grpc-nio-worker-ELG-3-2]
14:26:22.017 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2493 (Termination) [grpc-nio-worker-ELG-3-1]
14:26:22.017 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2493, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 3.889918337s} for agentId: 2493 (Termination) [grpc-nio-worker-ELG-3-1]
14:26:22.017 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2493, launchId=Unassigned, consolidated=false, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 3.890007082s} [grpc-nio-worker-ELG-3-1]
14:26:22.018 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2492 (Termination) [grpc-nio-worker-ELG-3-4]
14:26:22.018 INFO  [ProxyPathManager.kt:150] - Removed path /test-agent-1 for AgentContextInfo(consolidated=false, labels={},agentContexts=[AgentContext{agentId=2492, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=25.957533718s}]) [grpc-nio-worker-ELG-3-4]
14:26:22.018 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2492, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=25.957604075s} for agentId: 2492 (Termination) [grpc-nio-worker-ELG-3-4]
14:26:22.018 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2492, launchId=CRudPRd53VSWxFS, consolidated=false, valid=false, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=25.957664659s} [grpc-nio-worker-ELG-3-4]
14:26:22.844 INFO  [ProxyServiceImpl.kt:96] - Connected to AgentContext{agentId=2495, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=59.711243346s} [DefaultDispatcher-worker-2]
14:26:23.022 INFO  [ProxyPathManager.kt:82] - Added path /test-agent-1 for AgentContext{agentId=2495, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=59.888860388s} [DefaultDispatcher-worker-2]
14:26:23.331 INFO  [AgentContextManager.kt:40] - Registering agentId: 2496 [grpc-nio-worker-ELG-3-1]
14:26:23.331 INFO  [AgentContextManager.kt:40] - Registering agentId: 2497 [grpc-nio-worker-ELG-3-4]
14:26:26.114 INFO  [CallLogging.kt:45] - 200 OK: GET - /test-agent-1 - prometheus-kube-prometheus-stack-prometheus-0.prometheus-operated.monitoring.svc.cluster.local [DefaultDispatcher-worker-5]
14:26:31.262 INFO  [AgentContextManager.kt:40] - Registering agentId: 2498 [grpc-nio-worker-ELG-3-2]
14:26:56.102 INFO  [CallLogging.kt:45] - 200 OK: GET - /test-agent-1 - prometheus-kube-prometheus-stack-prometheus-0.prometheus-operated.monitoring.svc.cluster.local [DefaultDispatcher-worker-9]
14:27:23.285 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2497 (Termination) [grpc-nio-worker-ELG-3-4]
14:27:23.285 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2497, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=59.954055439s} for agentId: 2497 (Termination) [grpc-nio-worker-ELG-3-4]
14:27:23.285 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2497, launchId=Unassigned, consolidated=false, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=59.954180971s} [grpc-nio-worker-ELG-3-4]
14:27:25.922 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2496 (Termination) [grpc-nio-worker-ELG-3-1]
14:27:25.922 INFO  [ProxyPathManager.kt:141] - Removing paths for agentId: 2495 (Termination) [grpc-nio-worker-ELG-3-3]
14:27:25.923 INFO  [ProxyPathManager.kt:150] - Removed path /test-agent-1 for AgentContextInfo(consolidated=false, labels={},agentContexts=[AgentContext{agentId=2495, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=29.831453486s}]) [grpc-nio-worker-ELG-3-3]
14:27:25.923 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2496, launchId=Unassigned, consolidated=false, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 2.592003977s} for agentId: 2496 (Termination) [grpc-nio-worker-ELG-3-1]
14:27:25.923 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2496, launchId=Unassigned, consolidated=false, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=1m 2.592294197s} [grpc-nio-worker-ELG-3-1]
14:27:25.923 INFO  [AgentContextManager.kt:56] - Removed AgentContext{agentId=2495, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=29.831954303s} for agentId: 2495 (Termination) [grpc-nio-worker-ELG-3-3]
14:27:25.923 INFO  [ProxyServerTransportFilter.kt:46] - Disconnected from AgentContext{agentId=2495, launchId=CRudPRd53VSWxFS, consolidated=false, valid=false, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=29.832124716s} [grpc-nio-worker-ELG-3-3]
14:27:26.204 INFO  [CallLogging.kt:45] - 503 Service Unavailable: GET - /test-agent-1 - prometheus-kube-prometheus-stack-prometheus-0.prometheus-operated.monitoring.svc.cluster.local [DefaultDispatcher-worker-1]
14:27:26.441 INFO  [AgentContextManager.kt:40] - Registering agentId: 2499 [grpc-nio-worker-ELG-3-3]
14:27:26.639 INFO  [ProxyServiceImpl.kt:96] - Connected to AgentContext{agentId=2499, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=198.444320ms} [DefaultDispatcher-worker-2]
14:27:26.807 INFO  [ProxyPathManager.kt:82] - Added path /test-agent-1 for AgentContext{agentId=2499, launchId=CRudPRd53VSWxFS, consolidated=false, valid=true, agentName=test-agent-1, hostName=prometheus-proxy.service.net, remoteAddr=Unknown, lastRequestDuration=366.150346ms} [DefaultDispatcher-worker-6]
14:27:26.977 INFO  [AgentContextManager.kt:40] - Registering agentId: 2500 [grpc-nio-worker-ELG-3-4]
14:27:26.977 INFO  [AgentContextManager.kt:40] - Registering agentId: 2501 [grpc-nio-worker-ELG-3-1]
14:27:32.034 INFO  [AgentContextManager.kt:40] - Registering agentId: 2502 [grpc-nio-worker-ELG-3-2]

[kannanjgithub]

Those are logs from your Prometheus proxy server that forwards to the gRPC backend service application. We would need the logs from the latter.

[menvol3]

These are all the logs available from the backend service in my first post.

Also, I've tried to increase the heartbeat timeout to 10s and even 30 seconds, unfortunately it didn't help

Agent log

12:47:06.581 INFO  [Agent.kt:307] - Version: unknown Release Date: unknown [main]
12:47:06.644 INFO  [AgentOptions.kt:98] - proxyHostname: https://prometheus-proxy.service.net:443 [main]
12:47:06.644 INFO  [AgentOptions.kt:102] - agentName: test-agent-1 [main]
12:47:06.646 INFO  [AgentOptions.kt:106] - consolidated: false [main]
12:47:06.649 INFO  [AgentOptions.kt:110] - scrapeTimeoutSecs: 15s [main]
12:47:06.650 INFO  [AgentOptions.kt:114] - scrapeMaxRetries: 0 [main]
12:47:06.650 INFO  [AgentOptions.kt:120] - chunkContentSizeKbs: 32768 [main]
12:47:06.650 INFO  [AgentOptions.kt:124] - minGzipSizeBytes: 512 [main]
12:47:06.651 INFO  [AgentOptions.kt:128] - overrideAuthority:  [main]
12:47:06.651 INFO  [AgentOptions.kt:133] - trustAllX509Certificates: false [main]
12:47:06.651 INFO  [BaseOptions.kt:146] - adminEnabled: false [main]
12:47:06.652 INFO  [BaseOptions.kt:152] - adminPort: 8093 [main]
12:47:06.652 INFO  [BaseOptions.kt:158] - metricsEnabled: false [main]
12:47:06.652 INFO  [BaseOptions.kt:170] - metricsPort: 8083 [main]
12:47:06.653 INFO  [BaseOptions.kt:176] - transportFilterDisabled: true [main]
12:47:06.653 INFO  [BaseOptions.kt:164] - debugEnabled: false [main]
12:47:06.653 INFO  [BaseOptions.kt:182] - certChainFilePath:  [main]
12:47:06.653 INFO  [BaseOptions.kt:188] - privateKeyFilePath:  [main]
12:47:06.654 INFO  [BaseOptions.kt:194] - trustCertCollectionFilePath: dev_cert.pem [main]
12:47:06.654 INFO  [AgentOptions.kt:146] - agent.scrapeTimeoutSecs: 15s [main]
12:47:06.654 INFO  [AgentOptions.kt:147] - agent.internal.cioTimeoutSecs: 1m 30s [main]
12:47:06.654 INFO  [AgentOptions.kt:148] - agent.internal.heartbeatCheckPauseMillis: 500 [main]
12:47:06.655 INFO  [AgentOptions.kt:151] - agent.internal.heartbeatMaxInactivitySecs: 30 [main]
12:47:06.664 INFO  [AgentPathManager.kt:52] - Proxy path /test-agent-1 will be assigned to http://127.0.0.1:9273/metrics with labels {} [main]
12:47:06.785 INFO  [TlsUtils.kt:83] - Reading trustCertCollectionFilePath: "dev_cert.pem" [main]
12:47:06.820 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [main]
12:47:06.825 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [main]
12:47:06.886 INFO  [Agent.kt:125] - Agent name: test-agent-1 [main]
12:47:06.886 INFO  [Agent.kt:126] - Proxy reconnect pause time: 3s [main]
12:47:06.886 INFO  [Agent.kt:127] - Scrape timeout time: 15s [main]
12:47:06.887 INFO  [GenericService.kt:121] - Metrics service disabled [main]
12:47:06.887 INFO  [GenericService.kt:129] - Zipkin reporter service disabled [main]
12:47:06.892 INFO  [GenericService.kt:188] - Adding service Agent{agentId=, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [main]
12:47:06.912 INFO  [GenericServiceListener.kt:29] - Starting Agent{agentId=, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [main]
12:47:06.913 INFO  [GenericServiceListener.kt:30] - Running Agent{agentId=, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [main]
12:47:06.914 INFO  [GenericService.kt:141] - All Agent services healthy [Agent test-agent-1]
12:47:06.929 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
12:47:07.934 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
12:47:08.307 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
12:47:08.323 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 30s of inactivity [DefaultDispatcher-worker-2]
12:48:12.701 WARN  [ScrapeResults.kt:120] - fetchScrapeUrl() io.grpc.StatusException: INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR - http://127.0.0.1:9273/metrics [DefaultDispatcher-worker-9]
io.grpc.StatusException: INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR
	at io.grpc.Status.asException(Status.java:547)
	at io.grpc.kotlin.ClientCalls$rpcImpl$1$1$1.onClose(ClientCalls.kt:300)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
12:48:12.708 WARN  [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR [Agent test-agent-1]
12:48:12.711 INFO  [Agent.kt:216] - Waited 0s to reconnect [Agent test-agent-1]
12:48:12.712 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [Agent test-agent-1]
12:48:12.717 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
12:48:12.719 INFO  [Agent.kt:153] - Resetting agentId [Agent test-agent-1]
12:48:12.719 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
12:48:13.483 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
12:48:13.833 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
12:48:13.833 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 30s of inactivity [DefaultDispatcher-worker-3]
12:49:42.688 WARN  [ScrapeResults.kt:120] - fetchScrapeUrl() java.util.concurrent.CancellationException: Parent job is Cancelling - http://127.0.0.1:9273/metrics [DefaultDispatcher-worker-6]
java.util.concurrent.CancellationException: Parent job is Cancelling
	at io.ktor.client.engine.UtilsKt$attachToUserJob$cleanupHandler$1.invoke(Utils.kt:99)
	at io.ktor.client.engine.UtilsKt$attachToUserJob$cleanupHandler$1.invoke(Utils.kt:97)
	at kotlinx.coroutines.InvokeOnCancelling.invoke(JobSupport.kt:1571)
	at kotlinx.coroutines.JobSupport.invokeOnCompletionInternal$kotlinx_coroutines_core(JobSupport.kt:500)
	at kotlinx.coroutines.JobSupport.invokeOnCompletion(JobSupport.kt:452)
	at kotlinx.coroutines.Job$DefaultImpls.invokeOnCompletion$default(Job.kt:313)
	at io.ktor.client.engine.HttpClientEngineKt.createCallContext(HttpClientEngine.kt:166)
	at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:91)
	at io.ktor.client.engine.HttpClientEngine$DefaultImpls.access$executeWithinCallContext(HttpClientEngine.kt:24)
	at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:70)
	at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
	at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
	at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
	at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
	at io.ktor.util.pipeline.DebugPipelineContext.execute$ktor_utils(DebugPipelineContext.kt:63)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:86)
	at io.ktor.client.plugins.HttpSend$DefaultSender.execute(HttpSend.kt:118)
	at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
	at io.ktor.client.plugins.auth.AuthKt$Auth$2$2.invokeSuspend(Auth.kt:130)
	at io.ktor.client.plugins.auth.AuthKt$Auth$2$2.invoke(Auth.kt)
	at io.ktor.client.plugins.auth.AuthKt$Auth$2$2.invoke(Auth.kt)
	at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
	at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
	at io.ktor.client.plugins.HttpRequestRetryKt$HttpRequestRetry$2$1.invokeSuspend(HttpRequestRetry.kt:296)
	at io.ktor.client.plugins.HttpRequestRetryKt$HttpRequestRetry$2$1.invoke(HttpRequestRetry.kt)
	at io.ktor.client.plugins.HttpRequestRetryKt$HttpRequestRetry$2$1.invoke(HttpRequestRetry.kt)
	at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
	at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
	at io.ktor.client.plugins.HttpTimeoutKt$HttpTimeout$2$1.invokeSuspend(HttpTimeout.kt:175)
	at io.ktor.client.plugins.HttpTimeoutKt$HttpTimeout$2$1.invoke(HttpTimeout.kt)
	at io.ktor.client.plugins.HttpTimeoutKt$HttpTimeout$2$1.invoke(HttpTimeout.kt)
	at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
	at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
	at io.ktor.client.plugins.HttpRedirectKt$HttpRedirect$2$1.invokeSuspend(HttpRedirect.kt:97)
	at io.ktor.client.plugins.HttpRedirectKt$HttpRedirect$2$1.invoke(HttpRedirect.kt)
	at io.ktor.client.plugins.HttpRedirectKt$HttpRedirect$2$1.invoke(HttpRedirect.kt)
	at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
	at io.ktor.client.plugins.api.Send$Sender.proceed(CommonHooks.kt:41)
	at io.ktor.client.plugins.HttpCallValidatorKt$HttpCallValidator$2$2.invokeSuspend(HttpCallValidator.kt:112)
	at io.ktor.client.plugins.HttpCallValidatorKt$HttpCallValidator$2$2.invoke(HttpCallValidator.kt)
	at io.ktor.client.plugins.HttpCallValidatorKt$HttpCallValidator$2$2.invoke(HttpCallValidator.kt)
	at io.ktor.client.plugins.api.Send$install$1.invokeSuspend(CommonHooks.kt:46)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.api.Send$install$1.invoke(CommonHooks.kt)
	at io.ktor.client.plugins.HttpSend$InterceptedSender.execute(HttpSend.kt:96)
	at io.ktor.client.plugins.HttpSend$Plugin$install$1.invokeSuspend(HttpSend.kt:84)
	at io.ktor.client.plugins.HttpSend$Plugin$install$1.invoke(HttpSend.kt)
	at io.ktor.client.plugins.HttpSend$Plugin$install$1.invoke(HttpSend.kt)
	at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
	at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
	at io.ktor.client.plugins.RequestError$install$1.invokeSuspend(HttpCallValidator.kt:134)
	at io.ktor.client.plugins.RequestError$install$1.invoke(HttpCallValidator.kt)
	at io.ktor.client.plugins.RequestError$install$1.invoke(HttpCallValidator.kt)
	at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
	at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
	at io.ktor.client.plugins.SetupRequestContext$install$1.invokeSuspend$proceed(HttpRequestLifecycle.kt:40)
	at io.ktor.client.plugins.SetupRequestContext$install$1.access$invokeSuspend$proceed(HttpRequestLifecycle.kt)
	at io.ktor.client.plugins.SetupRequestContext$install$1$1.invoke(HttpRequestLifecycle.kt:40)
	at io.ktor.client.plugins.SetupRequestContext$install$1$1.invoke(HttpRequestLifecycle.kt:40)
	at io.ktor.client.plugins.HttpRequestLifecycleKt$HttpRequestLifecycle$1$1.invokeSuspend(HttpRequestLifecycle.kt:27)
	at io.ktor.client.plugins.HttpRequestLifecycleKt$HttpRequestLifecycle$1$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.client.plugins.HttpRequestLifecycleKt$HttpRequestLifecycle$1$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.client.plugins.SetupRequestContext$install$1.invokeSuspend(HttpRequestLifecycle.kt:40)
	at io.ktor.client.plugins.SetupRequestContext$install$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.client.plugins.SetupRequestContext$install$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
	at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
	at io.ktor.util.pipeline.DebugPipelineContext.execute$ktor_utils(DebugPipelineContext.kt:63)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:86)
	at io.ktor.client.HttpClient.execute$ktor_client_core(HttpClient.kt:1393)
	at io.ktor.client.statement.HttpStatement.fetchResponse(HttpStatement.kt:147)
	at io.ktor.client.statement.HttpStatement.execute(HttpStatement.kt:68)
	at com.github.pambrose.common.dsl.KtorDsl.get(KtorDsl.kt:85)
	at io.prometheus.agent.AgentHttpService.fetchContent(AgentHttpService.kt:90)
	at io.prometheus.agent.AgentHttpService.fetchContentFromUrl(AgentHttpService.kt:76)
	at io.prometheus.agent.AgentHttpService.fetchScrapeUrl(AgentHttpService.kt:61)
	at io.prometheus.agent.AgentGrpcService$readRequestsFromProxy$2$1$2.invokeSuspend(AgentGrpcService.kt:298)
	at io.prometheus.agent.AgentGrpcService$readRequestsFromProxy$2$1$2.invoke(AgentGrpcService.kt)
	at io.prometheus.agent.AgentGrpcService$readRequestsFromProxy$2$1$2.invoke(AgentGrpcService.kt)
	at io.prometheus.Agent$run$connectToProxy$3$4.invokeSuspend(Agent.kt:186)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:589)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:832)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:720)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:707)
12:49:42.690 WARN  [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR [Agent test-agent-1]
12:49:42.691 INFO  [Agent.kt:216] - Waited 0s to reconnect [Agent test-agent-1]
12:49:42.691 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [Agent test-agent-1]
12:49:42.692 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
12:49:42.694 INFO  [Agent.kt:153] - Resetting agentId [Agent test-agent-1]
12:49:42.695 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
12:49:43.439 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
12:49:43.790 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
12:49:43.791 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 30s of inactivity [DefaultDispatcher-worker-7]

[kannanjgithub]

Those are logs from your Prometheus proxy server that forwards to the gRPC backend service application. We would need the logs from the latter.

A correction about my previous message. After reading the link https://github.com/pambrose/prometheus-proxy you gave it is clear that

backend application providing metrics behind firewall <----- Prometheus Agent behind firewall - - - uses gRPC- - -> Prometheus Proxy uses NGIX Ingress (outside firewall) <--- Prometheus server (not our worry here)

(Whenever you said server you referred to the Prometheus proxy and not the Prometheus server).

You have provided the Agent logs which is the gRPC client. You can see in its logs

Connecting to proxy at prometheus-proxy.service.net:443 using TLS

Does the Prometheus proxy service that is using nginx ingress not have logs? It runs a gRPC server so it should have them.

[panchenko]

nginx can close connection after the time specified by grpc_read_timeout, please try increasing that.

[kannanjgithub]

grpc_read_timeout is a client side setting for closing the connection when not receiving response from gRPC server so you are referring to the Prometheus Agent as the party that can close the connection? @panchenko

[panchenko]

As I understand the connection is like this grpc_client -> nginx -> grpc_server
if server does not send anything during grpc_read_timeout time, then connection is closed by nginx.
As I remember, such a situation is logged in nginx error.log.

So, grpc_read_timeout should be increased if either server-side process takes longer or there is a streaming call with long intervals between messages.

There is also grpc_send_timeout, but in our case there was no need to touch it.

[menvol3]

nginx can close connection after the time specified by grpc_read_timeout, please try increasing that.

According to the logs I see that the agent is able to establish the connection

16:23:18.976 INFO [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)

Then, after a minute RST_STREAM (HTTP/2 PROTOCOL_ERROR) error appears

16:24:23.383 WARN [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR

The agent immediately tries to reconnect
16:24:23.395 INFO [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... 16:24:24.144 INFO [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)
The error repeats after a certain period of time (~1 minute)

16:25:28.450 ERROR [AgentGrpcService.kt:275] - sendHeartBeat() failed Parent job is Cancelling 16:25:28.451 WARN [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR

So it looks like really problem is with either proxy-read-timeout or proxy-send-timeout, cuz their default values are equal to 60s

According to the doc if the backend protocol is set to GRPC or GRPCS, changing values for that annotation will change grpc_send_timeout and grpc_read_timeout

As @panchenko said I tried to increase grpc_read_timeout using nginx.ingress.kubernetes.io/proxy-read-timeout annotation, I've set value to 300s

So now ingress annotations looks like below

Annotations:                                     meta.helm.sh/release-name: prometheus-proxy
                                                 meta.helm.sh/release-namespace: monitoring
                                                 nginx.ingress.kubernetes.io/backend-protocol: GRPC
                                                 nginx.ingress.kubernetes.io/proxy-read-timeout: 300
                                                 nginx.ingress.kubernetes.io/ssl-redirect: true

[menvol3]

And unfortunately, it didn't help

The agent is still losing its connection to the server, and according to the log, it happens every 65 seconds

Agent log

Mar 24 16:25:53 test-agent-1 prometheus_proxy_agent[3421467]: 16:25:53.734 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Mar 24 16:25:54 test-agent-1 prometheus_proxy_agent[3421467]: 16:25:54.111 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
Mar 24 16:25:54 test-agent-1 prometheus_proxy_agent[3421467]: 16:25:54.130 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 5s of inactivity [DefaultDispatcher-worker-2]
Mar 24 16:26:58 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:58.640 WARN  [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR [Agent test-agent-1]
Mar 24 16:26:58 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:58.641 INFO  [Agent.kt:216] - Waited 0s to reconnect [Agent test-agent-1]
Mar 24 16:26:58 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:58.641 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [Agent test-agent-1]
Mar 24 16:26:58 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:58.642 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Mar 24 16:26:58 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:58.645 INFO  [Agent.kt:153] - Resetting agentId [Agent test-agent-1]
Mar 24 16:26:58 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:58.646 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
Mar 24 16:26:59 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:59.448 INFO  [AgentClientInterceptor.kt:58] - Assigned agentId: 108 to Agent{agentId=108, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [grpc-default-executor-3]
Mar 24 16:26:59 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:59.448 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Mar 24 16:26:59 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:59.797 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
Mar 24 16:26:59 test-agent-1 prometheus_proxy_agent[3421467]: 16:26:59.798 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 5s of inactivity [DefaultDispatcher-worker-2]
Mar 24 16:28:03 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:03.716 ERROR [AgentGrpcService.kt:275] - sendHeartBeat() failed Parent job is Cancelling [DefaultDispatcher-worker-5]
Mar 24 16:28:03 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:03.717 WARN  [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR [Agent test-agent-1]
Mar 24 16:28:03 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:03.717 INFO  [Agent.kt:216] - Waited 0s to reconnect [Agent test-agent-1]
Mar 24 16:28:03 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:03.717 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [Agent test-agent-1]
Mar 24 16:28:03 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:03.717 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Mar 24 16:28:03 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:03.717 INFO  [Agent.kt:153] - Resetting agentId [Agent test-agent-1]
Mar 24 16:28:03 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:03.717 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
Mar 24 16:28:04 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:04.493 INFO  [AgentClientInterceptor.kt:58] - Assigned agentId: 111 to Agent{agentId=111, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [grpc-default-executor-4]
Mar 24 16:28:04 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:04.494 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Mar 24 16:28:04 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:04.839 INFO  [AgentPathManager.kt:78] - Registered http://127.0.0.1:9273/metrics as /test-agent-1 with labels {} [Agent test-agent-1]
Mar 24 16:28:04 test-agent-1 prometheus_proxy_agent[3421467]: 16:28:04.841 INFO  [Agent.kt:244] - Heartbeat scheduled to fire after 5s of inactivity [DefaultDispatcher-worker-4]
Mar 24 16:29:08 test-agent-1 prometheus_proxy_agent[3421467]: 16:29:08.755 ERROR [AgentGrpcService.kt:275] - sendHeartBeat() failed Parent job is Cancelling [DefaultDispatcher-worker-12]
Mar 24 16:29:08 test-agent-1 prometheus_proxy_agent[3421467]: 16:29:08.755 WARN  [Agent.kt:209] - Cannot connect to proxy at prometheus-proxy.service.net:443 StatusException INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR [Agent test-agent-1]
Mar 24 16:29:08 test-agent-1 prometheus_proxy_agent[3421467]: 16:29:08.755 INFO  [Agent.kt:216] - Waited 0s to reconnect [Agent test-agent-1]
Mar 24 16:29:08 test-agent-1 prometheus_proxy_agent[3421467]: 16:29:08.756 INFO  [AgentGrpcService.kt:132] - Creating gRPC stubs [Agent test-agent-1]
Mar 24 16:29:08 test-agent-1 prometheus_proxy_agent[3421467]: 16:29:08.756 INFO  [GrpcDsl.kt:75] - Creating connection for gRPC server at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]
Mar 24 16:29:08 test-agent-1 prometheus_proxy_agent[3421467]: 16:29:08.756 INFO  [Agent.kt:153] - Resetting agentId [Agent test-agent-1]
Mar 24 16:29:08 test-agent-1 prometheus_proxy_agent[3421467]: 16:29:08.756 INFO  [AgentGrpcService.kt:163] - Connecting to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth)... [Agent test-agent-1]
Mar 24 16:29:09 test-agent-1 prometheus_proxy_agent[3421467]: 16:29:09.504 INFO  [AgentClientInterceptor.kt:58] - Assigned agentId: 115 to Agent{agentId=115, agentName=test-agent-1, proxyHost=prometheus-proxy.service.net:443, adminService=Disabled, metricsService=Disabled} [grpc-default-executor-5]
Mar 24 16:29:09 test-agent-1 prometheus_proxy_agent[3421467]: 16:29:09.505 INFO  [AgentGrpcService.kt:169] - Connected to proxy at prometheus-proxy.service.net:443 using TLS (no mutual auth) [Agent test-agent-1]

[panchenko]

@menvol3 Can you check nginx error.log?

[menvol3]

@menvol3 Can you check nginx error.log?

@panchenko Sure, below is a 10-minute log of the Ingress Controller, including the time range mentioned in my previous message

Ingress Nginx log

14.225.121.26 - - [24/Mar/2025:16:25:53 +0000] "POST /ProxyService/connectAgent HTTP/2.0" 200 5 "-" "grpc-java-netty/1.68.2" 168 0.005 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 38 0.005 200 4741348edbcf6ee1ca6279872f9c8107
14.225.121.26 - - [24/Mar/2025:16:25:53 +0000] "POST /ProxyService/registerAgent HTTP/2.0" 200 50 "-" "grpc-java-netty/1.68.2" 128 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 83 0.003 200 af47e4826583bf816b9777dec39fd9d0
14.225.121.26 - - [24/Mar/2025:16:25:54 +0000] "POST /ProxyService/registerPath HTTP/2.0" 200 48 "-" "grpc-java-netty/1.68.2" 67 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 81 0.003 200 e9feb1a8e59e85fd3df849f2bb9ec4aa
14.225.121.26 - - [24/Mar/2025:16:25:59 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 46 0.005 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.005 200 a8ca0f09fe70dba48ca539406ed8bcac
14.225.121.26 - - [24/Mar/2025:16:26:04 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 25bf107e11957a4f2d54e6fbf36e794d
14.225.121.26 - - [24/Mar/2025:16:26:09 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 952ef9cc56e545f6039782aa02764dd1
14.225.121.26 - - [24/Mar/2025:16:26:18 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.004 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.004 200 a2a0f79ee871c014a36a1154cf84ca62
14.225.121.26 - - [24/Mar/2025:16:26:23 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 42543e3a0623aeed38acded71f8b0b12
14.225.121.26 - - [24/Mar/2025:16:26:28 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 0de043813529ea6a391c869f4d9b1541
14.225.121.26 - - [24/Mar/2025:16:26:33 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 08932117a3a24c3b3bafad39bafe6bda
14.225.121.26 - - [24/Mar/2025:16:26:39 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 d74d8cb3e853fe68e26a43e9f168c00f
14.225.121.26 - - [24/Mar/2025:16:26:48 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 500edcd7f246e977a3e4c77481495b6c
14.225.121.26 - - [24/Mar/2025:16:26:53 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 441af14d0ce06ca2f8ab4e7f87b5f1d9
14.225.121.26 - - [24/Mar/2025:16:26:54 +0000] "POST /ProxyService/writeChunkedResponsesToProxy HTTP/2.0" 408 0 "-" "grpc-java-netty/1.68.2" 51 60.000 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 60.000 - 6bc66136c3588333b2631b9025303290
14.225.121.26 - - [24/Mar/2025:16:26:58 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.004 200 5e5cd09da3168f5bce7accf40686833e
14.225.121.26 - - [24/Mar/2025:16:26:58 +0000] "POST /ProxyService/writeResponsesToProxy HTTP/2.0" 400 0 "-" "grpc-java-netty/1.68.2" 59934 64.490 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 64.490 - 096194022e92ad03711384cc2c368287
14.225.121.26 - - [24/Mar/2025:16:26:58 +0000] "POST /ProxyService/readRequestsFromProxy HTTP/2.0" 200 572 "-" "grpc-java-netty/1.68.2" 54 64.495 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 590 64.495 200 8b6d727aae2aabaa71f45d1c3889af53
14.225.121.26 - - [24/Mar/2025:16:26:59 +0000] "POST /ProxyService/connectAgent HTTP/2.0" 200 5 "-" "grpc-java-netty/1.68.2" 168 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 38 0.002 200 10fac69d55a3e8a1f24994e30ccad1eb
14.225.121.26 - - [24/Mar/2025:16:26:59 +0000] "POST /ProxyService/registerAgent HTTP/2.0" 200 50 "-" "grpc-java-netty/1.68.2" 128 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 83 0.004 200 c11922ec922cd8e389988bb6b9c9d70b
14.225.121.26 - - [24/Mar/2025:16:26:59 +0000] "POST /ProxyService/registerPath HTTP/2.0" 200 48 "-" "grpc-java-netty/1.68.2" 67 0.004 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 81 0.004 200 6e085f64def6f0ab8a0c89dbe4d03bea
14.225.121.26 - - [24/Mar/2025:16:27:04 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 46 0.004 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.004 200 a8481dffeb28742f4eeb01cb2db27b5b
14.225.121.26 - - [24/Mar/2025:16:27:10 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.004 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.004 200 019a8d7787e7e68147b5b947afa62806
14.225.121.26 - - [24/Mar/2025:16:27:18 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 bf56269d3ce54daddd6c42b029e109c5
14.225.121.26 - - [24/Mar/2025:16:27:23 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 0fa9fd62ae0e53345912b17c1e1a1e75
14.225.121.26 - - [24/Mar/2025:16:27:28 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 322f27abb38ce6a89638bd5198d4514f
14.225.121.26 - - [24/Mar/2025:16:27:33 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 c694852206f5bac91c3a37ae96d48f76
14.225.121.26 - - [24/Mar/2025:16:27:38 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 9f05507fde52442581dae7904c94a428
14.225.121.26 - - [24/Mar/2025:16:27:48 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 637d5ea478de25f680db2e2442044747
14.225.121.26 - - [24/Mar/2025:16:27:53 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 caddc30c2d5665dd36a135d30fedd353
14.225.121.26 - - [24/Mar/2025:16:27:58 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 341728f4a762b5fd63d9557a2be03d20
14.225.121.26 - - [24/Mar/2025:16:27:59 +0000] "POST /ProxyService/writeChunkedResponsesToProxy HTTP/2.0" 408 0 "-" "grpc-java-netty/1.68.2" 51 60.000 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 60.000 - fd6aec11be88a8ed0c00add841330fb0
14.225.121.26 - - [24/Mar/2025:16:28:03 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 95f74fe77e29fcd0bfaab2712244ab14
14.225.121.26 - - [24/Mar/2025:16:28:03 +0000] "POST /ProxyService/writeResponsesToProxy HTTP/2.0" 400 0 "-" "grpc-java-netty/1.68.2" 59948 63.913 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 63.913 - 9e896b4350b6e22e77fec99281c521dd
14.225.121.26 - - [24/Mar/2025:16:28:03 +0000] "POST /ProxyService/readRequestsFromProxy HTTP/2.0" 200 572 "-" "grpc-java-netty/1.68.2" 54 63.914 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 590 63.914 200 4d87e327c1f41f3ebe60531201e86dd6
14.225.121.26 - - [24/Mar/2025:16:28:04 +0000] "POST /ProxyService/connectAgent HTTP/2.0" 200 5 "-" "grpc-java-netty/1.68.2" 168 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 38 0.002 200 134632a16f553aa3579df8cc14ae61f6
14.225.121.26 - - [24/Mar/2025:16:28:04 +0000] "POST /ProxyService/registerAgent HTTP/2.0" 200 50 "-" "grpc-java-netty/1.68.2" 128 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 83 0.002 200 2afa694edaff8f8f422c14ffaff88a4b
14.225.121.26 - - [24/Mar/2025:16:28:04 +0000] "POST /ProxyService/registerPath HTTP/2.0" 200 48 "-" "grpc-java-netty/1.68.2" 67 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 81 0.002 200 97f5df939310e0c2965c8740ba0e638a
14.225.121.26 - - [24/Mar/2025:16:28:09 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 46 0.006 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.006 200 45b6639cfb3e975a475544c6745cf21c
14.225.121.26 - - [24/Mar/2025:16:28:18 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 81044540477f28d2939bed913b594b23
14.225.121.26 - - [24/Mar/2025:16:28:23 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 428bbb705a94054d819890226f6838c3
14.225.121.26 - - [24/Mar/2025:16:28:28 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 66b150013e7260da7c45310f875fa236
14.225.121.26 - - [24/Mar/2025:16:28:33 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 6fb3965b4d46b5319e95b26963eda80e
14.225.121.26 - - [24/Mar/2025:16:28:38 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 ba5bd7c920b59371ecba190624dfadaf
14.225.121.26 - - [24/Mar/2025:16:28:47 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.004 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 1f195239f59a8c73941376b77ddc2936
14.225.121.26 - - [24/Mar/2025:16:28:53 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 50d66d0e41d15caa55cd826d5fcca7bc
14.225.121.26 - - [24/Mar/2025:16:28:58 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 24e8d303f676efd5134803e3b5eb0c39
14.225.121.26 - - [24/Mar/2025:16:29:03 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 71d3527fa892bfbe3b8d38d616e504b6
14.225.121.26 - - [24/Mar/2025:16:29:04 +0000] "POST /ProxyService/writeChunkedResponsesToProxy HTTP/2.0" 408 0 "-" "grpc-java-netty/1.68.2" 51 60.001 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 60.000 - adea686b07cff68d8710d03269648e2e
14.225.121.26 - - [24/Mar/2025:16:29:08 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 6f87692aff19e24db524c3e8a3d7a647
14.225.121.26 - - [24/Mar/2025:16:29:08 +0000] "POST /ProxyService/writeResponsesToProxy HTTP/2.0" 400 0 "-" "grpc-java-netty/1.68.2" 59798 63.911 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 63.910 - 9e65ed2333af19da4d55adf5938d5328
14.225.121.26 - - [24/Mar/2025:16:29:08 +0000] "POST /ProxyService/readRequestsFromProxy HTTP/2.0" 200 572 "-" "grpc-java-netty/1.68.2" 54 63.911 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 590 63.910 200 3daf398741574877b6f39fae2475dd57
14.225.121.26 - - [24/Mar/2025:16:29:09 +0000] "POST /ProxyService/connectAgent HTTP/2.0" 200 5 "-" "grpc-java-netty/1.68.2" 168 0.004 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 38 0.004 200 732478c77de932b8af806b30511981e7
14.225.121.26 - - [24/Mar/2025:16:29:09 +0000] "POST /ProxyService/registerAgent HTTP/2.0" 200 50 "-" "grpc-java-netty/1.68.2" 128 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 83 0.003 200 3b88dd0fefacfe3a712839b802d66f54
14.225.121.26 - - [24/Mar/2025:16:29:09 +0000] "POST /ProxyService/registerPath HTTP/2.0" 200 48 "-" "grpc-java-netty/1.68.2" 67 0.004 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 81 0.003 200 0566d5c2033e7fee124b877928761106
14.225.121.26 - - [24/Mar/2025:16:29:17 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 46 0.006 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.005 200 ca4e43454494d49a5bf4be3867cc9c01
14.225.121.26 - - [24/Mar/2025:16:29:23 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 6f18549c3116d996ab2603dcac52889a
14.225.121.26 - - [24/Mar/2025:16:29:28 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 a077a3d72c2695752dbfde7a3750e26d
14.225.121.26 - - [24/Mar/2025:16:29:33 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 7609f16a45568136e91f082e08242209
14.225.121.26 - - [24/Mar/2025:16:29:38 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 c83a258bf152efed22d199883b4df93a
14.225.121.26 - - [24/Mar/2025:16:29:48 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 b1bf8ac92b66504de38c38d272d9fe3c
14.225.121.26 - - [24/Mar/2025:16:29:53 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 61dbc2e09ca140e03faaf4e95eb7f01c
14.225.121.26 - - [24/Mar/2025:16:29:58 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 687600df6da3dac8bb4f1d2231dc3e6d
14.225.121.26 - - [24/Mar/2025:16:30:03 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 cbb536b5931f5ff5fdd989274c38e5c9
14.225.121.26 - - [24/Mar/2025:16:30:09 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 8845aaef0c612c7e190adb56f6b86f26
14.225.121.26 - - [24/Mar/2025:16:30:09 +0000] "POST /ProxyService/writeChunkedResponsesToProxy HTTP/2.0" 408 0 "-" "grpc-java-netty/1.68.2" 51 60.000 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 60.000 - ea6c425e733d85a676148ec51f2b956c
14.225.121.26 - - [24/Mar/2025:16:30:12 +0000] "POST /ProxyService/writeResponsesToProxy HTTP/2.0" 400 0 "-" "grpc-java-netty/1.68.2" 59981 62.816 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 62.816 - 4008c20010460a531ee2e08d28f28a1d
14.225.121.26 - - [24/Mar/2025:16:30:12 +0000] "POST /ProxyService/readRequestsFromProxy HTTP/2.0" 200 858 "-" "grpc-java-netty/1.68.2" 54 62.816 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 885 62.816 200 113d12d8d50ac2e7a82a81e1a4f7c7b0
14.225.121.26 - - [24/Mar/2025:16:32:23 +0000] "POST /ProxyService/connectAgent HTTP/2.0" 200 5 "-" "grpc-java-netty/1.68.2" 168 0.005 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 38 0.005 200 c86db60f2a9cbfc2066d83f118fadd33
14.225.121.26 - - [24/Mar/2025:16:32:23 +0000] "POST /ProxyService/registerAgent HTTP/2.0" 200 50 "-" "grpc-java-netty/1.68.2" 128 0.004 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 83 0.004 200 fb9d356e3dc6b7c7f03fd2ccdc6a1f56
14.225.121.26 - - [24/Mar/2025:16:32:24 +0000] "POST /ProxyService/registerPath HTTP/2.0" 200 48 "-" "grpc-java-netty/1.68.2" 67 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 81 0.003 200 0708762645f77281d0b8e37b08ac0883
14.225.121.26 - - [24/Mar/2025:16:32:29 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 46 0.005 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.006 200 c52c638ab042b6e04d75d8daea194cdd
14.225.121.26 - - [24/Mar/2025:16:32:34 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 0bb584853a40b6746df3fee282b8e6f0
14.225.121.26 - - [24/Mar/2025:16:32:39 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 cb32f24c515c8540a62fbd45ae2323fc
14.225.121.26 - - [24/Mar/2025:16:32:47 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 edf9bbcfd7d69332f263e087278a8d26
14.225.121.26 - - [24/Mar/2025:16:32:53 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 9ab10b8e1d1162578cd4677b95e8beff
14.225.121.26 - - [24/Mar/2025:16:32:58 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 aab5182045a96c2606d97a164b6b0240
14.225.121.26 - - [24/Mar/2025:16:33:03 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 cbb5ecfa1bda310674482befc2113b87
14.225.121.26 - - [24/Mar/2025:16:33:08 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.112 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.113 200 4a5d1e47b5d3f84c1acc5c8b2144b5eb
14.225.121.26 - - [24/Mar/2025:16:33:17 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 7bc7992f0936acce1c7cebc5427476c9
14.225.121.26 - - [24/Mar/2025:16:33:23 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 d1a8fc2782c22311651efd581be193d0
14.225.121.26 - - [24/Mar/2025:16:33:24 +0000] "POST /ProxyService/writeChunkedResponsesToProxy HTTP/2.0" 408 0 "-" "grpc-java-netty/1.68.2" 51 60.000 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 60.000 - 3db066a7a3773a8630f7e4e1271d4360
14.225.121.26 - - [24/Mar/2025:16:33:24 +0000] "POST /ProxyService/readRequestsFromProxy HTTP/2.0" 200 572 "-" "grpc-java-netty/1.68.2" 54 60.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 614 60.003 200 636c6de304b0fa24651bb5359032b5b5
14.225.121.26 - - [24/Mar/2025:16:33:24 +0000] "POST /ProxyService/writeResponsesToProxy HTTP/2.0" 499 0 "-" "grpc-java-netty/1.68.2" 59888 60.169 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 60.170 - d36544350fa0a2828d267f0ad89c17ee
14.225.121.26 - - [24/Mar/2025:16:34:29 +0000] "POST /ProxyService/connectAgent HTTP/2.0" 200 5 "-" "grpc-java-netty/1.68.2" 168 0.006 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 38 0.006 200 6f2d2964c78cbe7b7c1b8fa817d99903
14.225.121.26 - - [24/Mar/2025:16:34:29 +0000] "POST /ProxyService/registerAgent HTTP/2.0" 200 50 "-" "grpc-java-netty/1.68.2" 128 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 83 0.003 200 6c9b3984d95961f2d3e0e172b81a1593
14.225.121.26 - - [24/Mar/2025:16:34:29 +0000] "POST /ProxyService/registerPath HTTP/2.0" 200 48 "-" "grpc-java-netty/1.68.2" 67 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 81 0.003 200 0f13c135351d7758499770dbc3226118
14.225.121.26 - - [24/Mar/2025:16:34:34 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 46 0.006 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.005 200 6200a526189e367afaa59a27e3e38881
14.225.121.26 - - [24/Mar/2025:16:34:40 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 28ba1adac67abf0bc36a05036d056f64
14.225.121.26 - - [24/Mar/2025:16:34:48 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 ca06d8e917f446ddbb85d9ed5bf7d7c1
14.225.121.26 - - [24/Mar/2025:16:34:53 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.015 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.015 200 83f0c68d38ca45ebd3518df079e8f592
14.225.121.26 - - [24/Mar/2025:16:34:58 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.004 200 6bc6577055cba6fac5a715da07c51a20
14.225.121.26 - - [24/Mar/2025:16:35:03 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 edab8a395ba5664844f88b1dc73a29df
14.225.121.26 - - [24/Mar/2025:16:35:08 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 add24b02df2b9441259a5bb5362aa801
14.225.121.26 - - [24/Mar/2025:16:35:18 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 5da2efbbcd2fd364d48239935496dcf7
14.225.121.26 - - [24/Mar/2025:16:35:23 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 0420282540febe09521d926eed652847
14.225.121.26 - - [24/Mar/2025:16:35:28 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.003 200 86eda137cb3aac9391029cbaeefebbcf
14.225.121.26 - - [24/Mar/2025:16:35:29 +0000] "POST /ProxyService/writeChunkedResponsesToProxy HTTP/2.0" 408 0 "-" "grpc-java-netty/1.68.2" 51 60.000 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 60.000 - 6b350eabbc5dff04ebcc6f51c87518a7
14.225.121.26 - - [24/Mar/2025:16:35:33 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 20d27e8d9637de2117ca1593e7b79397
14.225.121.26 - - [24/Mar/2025:16:35:33 +0000] "POST /ProxyService/writeResponsesToProxy HTTP/2.0" 400 0 "-" "grpc-java-netty/1.68.2" 59705 63.893 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 0 63.892 - bd9164db6856011b5f8e9012d56cd110
14.225.121.26 - - [24/Mar/2025:16:35:33 +0000] "POST /ProxyService/readRequestsFromProxy HTTP/2.0" 200 572 "-" "grpc-java-netty/1.68.2" 54 63.894 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 590 63.894 200 579d55258dd7176ead2959d74538aaa8
14.225.121.26 - - [24/Mar/2025:16:35:34 +0000] "POST /ProxyService/connectAgent HTTP/2.0" 200 5 "-" "grpc-java-netty/1.68.2" 168 0.004 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 38 0.004 200 0fa23d5d21517cdc31a2b1363e7ce161
14.225.121.26 - - [24/Mar/2025:16:35:34 +0000] "POST /ProxyService/registerAgent HTTP/2.0" 200 50 "-" "grpc-java-netty/1.68.2" 128 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 83 0.002 200 7b97da6ad290bb309fa4b828f51c2148
14.225.121.26 - - [24/Mar/2025:16:35:34 +0000] "POST /ProxyService/registerPath HTTP/2.0" 200 48 "-" "grpc-java-netty/1.68.2" 67 0.119 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 81 0.120 200 34b924e4f10aeaaa0b2823b745e3fb70
14.225.121.26 - - [24/Mar/2025:16:35:40 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 46 0.021 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.021 200 0a2d72e0c3f883afe9f0926c3d618386
14.225.121.26 - - [24/Mar/2025:16:35:48 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.003 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 f783830dda8763b686c6b0102495dde1
14.225.121.26 - - [24/Mar/2025:16:35:53 +0000] "POST /ProxyService/sendHeartBeat HTTP/2.0" 200 45 "-" "grpc-java-netty/1.68.2" 18 0.002 [monitoring-prometheus-proxy-50051] [] 10.10.13.133:50051 78 0.002 200 e556025c3aa09b2fb60da988c0f24c52

[panchenko]

@menvol3 That's not error.log, but access.log, which has some lines with status 408. Error log should contain more error details.
Seems like we are slightly beyond the gRPC topic.
In AWS there are also load balancer properties which could affect long-running connections.
I would suggest searching in SO.