Disposed websocket after portforward

[dotnetwizard]

After creating 10+ pods and calling WebSocketNamespacedPodPortForwardAsync, passing the result to a demuxer, some of the websockets are disposed after a minute or two. Specifically, I get a websocket disposed exception. Is there a proper way to reconnect or to keep alive? If I call WebSocketNamespacedPodPortForwardAsync again sometimes it works.

[tg123]

could you please paste the code to repo?

[dotnetwizard]

Sorry, I can't. Do you have any examples of port forwarding working with several pods?

[tg123]

you can just paste the example of your port forwarding part, so it is easier for me to repro and figure out why it got disposed

[dotnetwizard]

public async Task<string> GetPodHostnameAsync(string podName, string podNamespace, ushort port, bool reconnect = true, CancellationToken token = default) {
	token.ThrowIfCancellationRequested();

	var podResponse = await _kubClient.CoreV1.ReadNamespacedPodWithHttpMessagesAsync(podName, podNamespace, cancellationToken: token);
	var pod = podResponse.Body;
	var podIp = pod.Status.PodIP;

	if (!IPAddress.TryParse(podIp, out var host))
		throw new ArgumentException($"Could not parse pod id: {podIp}");

	var canConnectDirect = await TcpConnectionTester.CanConnectAsync(pod.Status.PodIP, port);
		
	if (!canConnectDirect) {
		var demuxer = await setupPortForwardAsync(podName, podNamespace, port, token);
		//lines eliminated for brevity, however it stores the demuxer in memory

		host = IPAddress.Loopback;
		port = PORT;
	}

	return $"{host}:{port}";

}
private async Task<StreamDemuxer> setupPortForwardAsync(string podName, string podNamespace, ushort port, CancellationToken token = default) {
	token.ThrowIfCancellationRequested();

	var webSocket = await _kubClient.WebSocketNamespacedPodPortForwardAsync(podName, podNamespace, new int[] { port }, cancellationToken: token);
	var demux = new StreamDemuxer(webSocket, StreamType.PortForward, ownsSocket: true);
	demux.Start();
	return demux;
}

[null-d3v]

It could be that the server is including a Content-Length header with the 101 responses when the connection is established.

With System.Net.WebSockets.ClientWebSocket, this causes the connection to be terminated after 100 seconds.

Content-Length headers with 100-series response codes is technically not compliant with RFC 7230 Section 3.3.2.

[null-d3v]

This might be the case for some, but I am also experiencing a socket connection loss and my kube server is not including a Content-Length header:

Status: 101
Connection: upgrade
Strict-Transport-Security: max-age=15724800; includeSubDomains
Cache-Control: no-store, must-revalidate, no-cache
Date: Mon, 05 Dec 2022 22:09:28 GMT
X-Api-Cattle-Auth: true
Upgrade: websocket
Sec-WebSocket-Accept: YAbG4Mv+VSvv0Iw+fcDBvhpbhW8=
X-Content-Type-Options: nosniff
Sec-WebSocket-Protocol: v4.channel.k8s.io

[null-d3v]

After more investigation, I am definitely seeing the socket closed after just over 2 minutes when set up for port forwarding in a way similar to the example. This occurs even if no user-input data is sent or received.

Simple use of a ClientWebSocket with StreamDemuxer.Start() is causing the socket to be closed with a NormalClosure status after about 2 minutes:

var clientWebSocket = new ClientWebSocket();
clientWebSocket.Options.SetRequestHeader(
    "Authorization",
    "Bearer ***");
clientWebSocket.Options.AddSubProtocol("v4.channel.k8s.io");
await clientWebSocket.ConnectAsync(
    new Uri(
        "wss://{cluster}/api/v1/namespaces/{namespace}/pods/{pod}/portforward?ports=80"),
        CancellationToken.None);

var streamDemuxer = new StreamDemuxer(clientWebSocket, StreamType.PortForward);
streamDemuxer.Start();

while (clientWebSocket.State == WebSocketState.Open) { }

Console.WriteLine($"Disconnected {clientWebSocket.CloseStatus}");

return 0;

This could be possibly the apiserver not behaving properly with HTTP/2, as it appears the connection is closed by the server. I am definitely not in violation of streaming-connection-idle-timeout and SPDY socket connections from kubectl maintain integrity.