Replay using kafka

[bb-prasunasarana]

Hi,

We used Goreplay to capture and replay traffic using KAFKA for our system, which gets a throughput of 1.6 lakh RPM (equivalent to 2700 RPS).

We successfully captured data over a 2-hour run, pushing 2 crore messages during that period. However, when we attempt to replay the captured traffic, we observe a throughput of only ~300 RPS, significantly lower than the captured rate of 2700 RPS.

./gor --input-kafka-host broker-ip --input-kafka-topic topic-name --input-kafka-offset 0 --output-http prod_url --stats --output-http-stats --verbose 1
[DEBUG][elapsed 472.957µs]:
output_http:latest,mean,max,count,count/second,gcount
[DEBUG][elapsed 68.516µs]:
output_http:0,0,0,0,0,5
2025/01/04 23:30:13 [PPID 14882 and PID 15550] Version:2.0.0
[DEBUG][elapsed 5.000066334s]:
output_http:0,0,50,1427,285,99
[DEBUG][elapsed 5.000299959s]:
output_http:33,0,53,1390,278,12
[DEBUG][elapsed 1.387815704s]:
output_http:0,1,55,1404,280,112
[DEBUG][elapsed 5.000939106s]:
output_http:50,0,50,1085,217,118
[DEBUG][elapsed 5.000809986s]:
output_http:13,1,43,1043,208,42
[DEBUG][elapsed 5.000092741s]:
output_http:16,1,60,1404,280,136

[dosubot]

Hey @bb-prasunasarana 👋, I'm here to help you out with your bug or answer any questions you might have. Let's work together to solve this!

The issue of replay throughput using Kafka with GoReplay being significantly lower than the captured rate is a known challenge. Here are some steps you can take to improve the throughput:

1. Check Bandwidth: Ensure that the network bandwidth is sufficient to handle the replay traffic. If the network adapter is overwhelmed, it can cause the output-http queue to fill up, leading to reduced throughput.

2. Adjust Output Workers: Consider adjusting the number of HTTP output workers. You can use dynamic worker scaling by setting --output-http-workers=-1, or calculate the optimal number of workers using the formula:

   output-workers = (Average number of requests per second) / (Average target response time per second)
   

3. System Tuning: Adjust system limits on the source server for optimal performance. Here are some recommended settings:

   net.ipv4.tcp_max_tw_buckets = 65536
   net.ipv4.tcp_tw_recycle = 1
   net.ipv4.tcp_tw_reuse = 0
   net.ipv4.tcp_max_syn_backlog = 131072
   net.ipv4.tcp_syn_retries = 3
   net.ipv4.tcp_synack_retries = 3
   net.ipv4.tcp_retries1 = 3
   net.ipv4.tcp_retries2 = 8
   net.ipv4.tcp_rmem = 16384 174760 349520
   net.ipv4.tcp_wmem = 16384 131072 262144
   net.ipv4.tcp_mem = 262144 524288 1048576
   net.ipv4.tcp_max_orphans = 65536
   net.ipv4.tcp_fin_timeout = 10
   net.ipv4.tcp_low_latency = 1
   net.ipv4.tcp_syncookies = 0

By addressing these potential bottlenecks and tuning the system, you should be able to improve the replay throughput ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[bb-prasunasarana]

@buger ,
I tried all the above mentioned suggestions, still i am not able to see the RPS it replayed is not match with captured traffic.

[bb-prasunasarana]

Hi @dosu, I tried all the above mentioned suggestions, still there is no improvement in RPS while replay.

Apart from this, i tried tunning below Sarama Consumer setting,

Consumer.Fetch.Default = 10 * 1024 * 1024 // 10MB
Consumer.Fetch.Max = 50 * 1024 * 1024 // 50 MB
Consumer.MaxWaitTime = 10 // Reduce wait time for responsiveness
ChannelBufferSize = 1024

Still I dont see any improvement in RPS

[dosubot]

If you've already tried the previous suggestions and adjusted the Sarama consumer settings without seeing improvement, consider the following additional steps:

1. Increase Kafka Consumer Concurrency: Ensure that you have multiple consumers running in parallel to increase the throughput. This can be done by increasing the number of partitions in your Kafka topic and ensuring that each partition has a dedicated consumer.

2. Optimize Kafka Configuration: Check Kafka broker configurations such as num.network.threads, num.io.threads, and socket.send.buffer.bytes to ensure they are set to handle high throughput.

3. Monitor and Adjust GoReplay's Internal Queues: The internal message queue in GoReplay might be a bottleneck. You can try increasing the buffer size of the messages channel in the KafkaInput struct, which is currently set to 256 by default ^[1].

4. Profile and Analyze Performance: Use profiling tools to identify any bottlenecks in the GoReplay process itself. This can help pinpoint if there are specific areas in the code that are causing delays.

5. Check for System Resource Limits: Ensure that your system is not hitting any resource limits such as CPU, memory, or disk I/O, which could affect performance.

These steps should help in diagnosing and potentially resolving the throughput issue. If you're still facing challenges, it might be helpful to loop in a maintainer for further assistance. This is a tough nut to crack, could really use your help on this, @buger.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[bb-prasunasarana]

Hi @dosu, How to use GoReplay for a topic to consume from more than one partition.

[dosubot]

GoReplay automatically handles consuming from multiple partitions of a Kafka topic by creating a separate PartitionConsumer for each partition. When you set up a KafkaInput, it retrieves all partitions for the specified topic and starts a PartitionConsumer for each one. Each consumer runs in its own goroutine, allowing messages from all partitions to be consumed concurrently and processed by GoReplay ^[1].

You don't need to do anything special to enable this; it's part of GoReplay's default behavior when consuming from Kafka. Just ensure that your Kafka topic is configured with multiple partitions, and GoReplay will handle the rest.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[kaluza]

There's a solution for capturing traffic and replaying that's available and has active development. I'm not sure if it meets your needs because it is designed for Elasticsearch/OpenSearch but it captures HTTP traffic generically so I can see how one could modify the solution fairly easily to support an alternative HTTP destination.

https://github.com/opensearch-project/opensearch-migrations