Description
Problem description
What I'm doing?
I have a basic grpc js client and a grpc python server.
The grpc client it does 60 iterations, in each iteration it fires 300 grpc requests in parallel to the grpc server. The server processing is pretty simple, it sleeps 5 seconds and it respond success.
What I expected to happen?
I expect to finish all 60 iterations.
What is actually happening?
The script does not reach the 60 iterations. It starts to complete iterations with all responses success, but approx at the iteration 9 It fails with the following error:
grpc-client {"group":"mcaruso-poc:grpc-client","error":{"kind":"Error","message":"Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: Session closed with error code 2",
"stack":"Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: Session closed with error code 2\n
at callErrorFromStatus (/app/node _modules/@grpc/grpc-js/build/src/call.js:31:19)\n
at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:192:76)\n
at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)\n
at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)\n
at /app/node_modules/@grpc/grpc-js/build/src/resolving-call.js:94:78\n
at process.processTicksAndRejections (node:internal/process/task_queues:77:11)\nfor call at\n
at ServiceClientImpl.makeUnaryRequest (/app/node_modules/@grpc/grpc-js/build/src/client.js:160:32)\n
at ServiceClientImpl.GetBookList (/app/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)\n
at /app/inventoryServiceClient.js:39:28\n
at new Promise (<anonymous>)\n
at Client.getBookList (/app/inventoryServiceClient.js:37:12)\n
at /app/index.js:28:9\n
at new Promise (<anonymous>)\n
at GetBookList (/app/index.js:26:12)\n
at /app/node_modules/async/dist/async.js:151:38\n
at /app/node_modules/async/dist/async.js:2588:44\n
at replenish (/app/node_modules/async/dist/async.js:446:21)\n
at /app/node_modules/async/dist/async.js:451:13\n
at Object.eachLimit$1 (/app/node_modules/async/dist/async.js:2718:34)\n
at Object.awaitable (/app/node_modules/async/dist/async.js:211:32)\n
at /app/index.js:48:15\n
at new Promise (<anonymous>)","code":13},
"message":"eachSeries ERROR","status":"Error","timestamp":"2024-01-22T02:48:17.546Z"}
grpc-client HTTP2 12: Http2Session client: finishSessionClose socket end undefined Error [ERR_HTTP2_SESSION_ERROR]: Session closed with error code 2
grpc-client at new NodeError (node:internal/errors:399:5)
grpc-client at Http2Session.onGoawayData (node:internal/http2/core:687:21)
grpc-client at Http2Session.callbackTrampoline (node:internal/async_hooks:130:17) {
grpc-client code: 'ERR_HTTP2_SESSION_ERROR'
grpc-client }
grpc-client D 2024-01-22T02:48:17.548Z | transport | (3) 172.20.93.63:8081 connection closed with error Session closed with error code 2
grpc-client D 2024-01-22T02:48:17.548Z | transport | (3) 172.20.93.63:8081 session closed
Environment
-
Docker image: node:18.15.0-slim
-
Package name and version: "@grpc/grpc-js": "1.8.21"
-
Eks 1.25 - Istio 1.19.4
-
This error is also happening with node 18.16.0, 20.11.0, 21.6.0, and also with grpc-js 1.9.14
Additional context
If I do 100 requests in parallel, it finishes all the 60 iterations, the problem starts when I increase the number of requests in parallel. If I do 600 requests in parallel instead of 300, it throws the error even earlier in the iteration 4 approx. If I do 1000 it fails with the same error in the iteration 1.