High CPU and RAM consumption

[drmrbrewer]

Self-Hosted Version

22.9.0 (on Ubuntu 22.04)

CPU Architecture

x86_64

Docker Version

20.10.18, build b40c2f6

Docker Compose Version

2.10.2

Steps to Reproduce

Install Sentry-self hosted, leave it running for a while.

Expected Result

Normal CPU and RAM consumption.

Actual Result

When I first installed Sentry self-hosted, and started sending it data, all was going well. The self-hosted dashboard was reporting performance and error data as expected, and RAM/CPU usage was completely under control.

But then overnight the CPU and RAM usage on the server went crazy, with CPU maxing out, high RAM and swap usage. This appeared to be down to sentry, because running docker compose down restored things back to normal (except that sentry was no longer running).

This seems very similar to this issue from @umgbhalla but there wasn't any apparent resolution there, except maybe to re-install sentry from fresh. I also have 32 GB RAM and 8 cores. It's also similar to this issue from @wodCZ, but I'm not seeing the little CPU spikes.

I tried installing sentry from fresh, but CPU was still maxing out after:

The short period maxing out at 100% was when I set it running again, followed soon after by shutting it down again.

However, I don't think that it was a completely fresh install, because despite removing the self-hosted folder completely and re-installing, when running docker compose run --rm web createuser after installing to add a user (I install with ./install.sh --skip-user-creation to avoid any prompts during installation), it said User: me@mydomain.com exists, use --force-update to force... so it still had some retained state from before?

In any case, it shouldn't really be necessary to re-install in order to get CPU/RAM back under control. Any ideas?

(And an aside... is it possible to script the docker compose run --rm web createuser step, i.e. with no user prompts and instead pass the info by args or env vars?)

[wodCZ]

I remember there were some issues after upgrades, which caused a process to hang in restart loop due to some incompatibility.
You could see if some service keeps spamming the docker compose logs -f output.

Dangerous, you will lose data: For a fresh install you might need to drop the volumes too (docker compose down -v), then check that there's no leftover sentry volume using docker system df -v.

Checking which container is causing a high CPU usage could be helpful too: docker stats -a

[drmrbrewer]

@wodCZ I really appreciate the reply.

There does seem to be something spamming the docker compose logs -f output... small sample below...

sentry-self-hosted-post-process-forwarder-1                    | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64993, 64993]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 10ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64994, 64994]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 8ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64995, 64995]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 9ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64996, 64996]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-snuba-transactions-consumer-1               | 2022-09-23 11:43:51,295 Completed processing <Batch: 1 message, open for 0.82 seconds>.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:51.509446 [ 95 ] {0a61de27-eaae-4301-b955-3a29f9b5fc6b} <Information> executeQuery: Read 19 rows, 72.86 KiB in 0.020 sec., 938 rows/sec., 3.51 MiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:51.512513 [ 95 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:51.529926 [ 96 ] {1603a3d4-a4d0-4ba1-a304-c673f0dce9b4} <Information> executeQuery: Read 19 rows, 1.00 KiB in 0.018 sec., 1040 rows/sec., 54.80 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:51.537604 [ 96 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1                   | 2022-09-23 11:43:51,540 Completed processing <Batch: 19 messages, open for 0.83 seconds>.
sentry-self-hosted-snuba-consumer-1                            | 2022-09-23 11:43:51,514 Completed processing <Batch: 19 messages, open for 0.80 seconds>.
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64997, 64997]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64998, 64998]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64999, 64999]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 9ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65000, 65000]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65001, 65001]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65002, 65002]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65003, 65003]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65004, 65004]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65005, 65005]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65006, 65006]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65007, 65007]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 10ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65008, 65008]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65009, 65009]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65010, 65010]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65011, 65011]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65012, 65012]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65013, 65013]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65014, 65014]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65015, 65015]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:52.316671 [ 96 ] {b3256d45-320f-42e0-88be-8a73868e6844} <Information> executeQuery: Read 17 rows, 919.00 B in 0.022 sec., 762 rows/sec., 40.26 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:52.321504 [ 95 ] {873acf29-5f64-4c92-a6fe-d6495dd41239} <Information> executeQuery: Read 17 rows, 65.73 KiB in 0.027 sec., 636 rows/sec., 2.40 MiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:52.324163 [ 95 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:52.326841 [ 96 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1                   | 2022-09-23 11:43:52,328 Completed processing <Batch: 17 messages, open for 0.82 seconds>.
sentry-self-hosted-snuba-consumer-1                            | 2022-09-23 11:43:52,328 Completed processing <Batch: 17 messages, open for 0.84 seconds>.
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65016, 65016]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65017, 65017]}): forced:False size:True time:False
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:53.070237 [ 96 ] {26cccaf0-7fb3-4037-8b9f-a7e63b1902cd} <Information> executeQuery: Read 16 rows, 866.00 B in 0.006 sec., 2649 rows/sec., 140.02 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:53.070653 [ 96 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:53.082137 [ 95 ] {2655494c-39d8-4ed9-ae56-5d1fb5960562} <Information> executeQuery: Read 16 rows, 56.80 KiB in 0.018 sec., 877 rows/sec., 3.04 MiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 11:43:53.082556 [ 95 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1                   | 2022-09-23 11:43:53,073 Completed processing <Batch: 16 messages, open for 0.78 seconds>.
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 11ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65018, 65018]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65019, 65019]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 10ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65020, 65020]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65021, 65021]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65022, 65022]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65023, 65023]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65024, 65024]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65025, 65025]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65026, 65026]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65027, 65027]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65028, 65028]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65029, 65029]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65030, 65030]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65031, 65031]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65032, 65032]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65033, 65033]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 12ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65034, 65034]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 8ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65035, 65035]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65036, 65036]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65037, 65037]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 9ms
sentry-self-hosted-snuba-consumer-1                            | 2022-09-23 11:43:53,084 Completed processing <Batch: 16 messages, open for 0.79 seconds>.
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65038, 65038]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65039, 65039]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 8ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65040, 65040]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 7ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65041, 65041]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65042, 65042]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65043, 65043]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65044, 65044]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65045, 65045]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65046, 65046]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65047, 65047]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65048, 65048]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1                    | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms

[drmrbrewer]

As for docker stats -a, there are a few major CPU hogs:

CONTAINER ID   NAME                                                            CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
c2b394f89938   sentry-self-hosted-nginx-1                                      0.00%     7.332MiB / 30.61GiB   0.02%     1.38kB / 0B       5.65MB / 4.1kB    9
f145b7803578   sentry-self-hosted-relay-1                                      0.31%     34.12MiB / 30.61GiB   0.11%     6.97kB / 6.37kB   12MB / 0B         38
c1249a215531   sentry-self-hosted-worker-1                                     233.27%   874.8MiB / 30.61GiB   2.79%     830MB / 1.03GB    6.16MB / 34.9MB   68
710a423a3849   sentry-self-hosted-sentry-cleanup-1                             0.00%     1.16MiB / 30.61GiB    0.00%     3.29kB / 0B       164kB / 217kB     1
772c5d6bc99d   sentry-self-hosted-snuba-cleanup-1                              0.00%     13.21MiB / 30.61GiB   0.04%     4.81kB / 1.97kB   11.4MB / 143kB    1
597010fda5ed   sentry-self-hosted-web-1                                        0.01%     415.8MiB / 30.61GiB   1.33%     34.2kB / 33.4kB   6.05MB / 270kB    19
073198558a64   sentry-self-hosted-subscription-consumer-transactions-1         0.23%     123.4MiB / 30.61GiB   0.39%     215kB / 143kB     36.9kB / 205kB    6
3b0ef276d52f   sentry-self-hosted-ingest-consumer-1                            0.24%     128.3MiB / 30.61GiB   0.41%     381kB / 200kB     32.8kB / 205kB    6
923052b6610c   sentry-self-hosted-snuba-transactions-cleanup-1                 0.00%     10.26MiB / 30.61GiB   0.03%     4.81kB / 1.98kB   8.27MB / 131kB    1
15f98ab07e28   sentry-self-hosted-subscription-consumer-events-1               0.29%     123.8MiB / 30.61GiB   0.39%     213kB / 141kB     508kB / 201kB     6
88164011a5ec   sentry-self-hosted-post-process-forwarder-1                     12.64%    136.1MiB / 30.61GiB   0.43%     350MB / 16.6MB    16.4kB / 205kB    13
457087d09979   sentry-self-hosted-cron-1                                       0.01%     148.7MiB / 30.61GiB   0.47%     32kB / 155kB      819kB / 270kB     3
6712f96c98c3   sentry-self-hosted-geoipupdate-1                                0.00%     0B / 0B               0.00%     0B / 0B           0B / 0B           0
6ee727622535   sentry-self-hosted-symbolicator-cleanup-1                       0.00%     1.559MiB / 30.61GiB   0.00%     3.6kB / 0B        938kB / 12.3kB    1
722467cd7c17   sentry-self-hosted-postgres-1                                   57.74%    98.79MiB / 30.61GiB   0.32%     107MB / 47MB      12MB / 133MB      20
87e1ab6a15a5   sentry-self-hosted-snuba-transactions-consumer-1                0.84%     80.23MiB / 30.61GiB   0.26%     225MB / 1.6MB     827kB / 0B        9
59efe315bd55   sentry-self-hosted-memcached-1                                  6.72%     16.31MiB / 30.61GiB   0.05%     36.3MB / 84.6MB   995kB / 41kB      10
09f64b0d0348   sentry-self-hosted-snuba-api-1                                  0.01%     85.38MiB / 30.61GiB   0.27%     15.8kB / 33.2kB   5.02MB / 65.5kB   7
792a3c51352e   sentry-self-hosted-snuba-subscription-consumer-transactions-1   0.47%     78.24MiB / 30.61GiB   0.25%     339kB / 178kB     205kB / 0B        9
86955e5f1c57   sentry-self-hosted-smtp-1                                       0.00%     1.621MiB / 30.61GiB   0.01%     3.67kB / 0B       258kB / 4.1kB     2
286670644c63   sentry-self-hosted-snuba-outcomes-consumer-1                    1.48%     78.29MiB / 30.61GiB   0.25%     3.59MB / 2.21MB   73.7kB / 0B       10
ea23c1a0e793   sentry-self-hosted-snuba-subscription-consumer-events-1         0.37%     78.23MiB / 30.61GiB   0.25%     340kB / 177kB     160kB / 0B        9
0255012d5077   sentry-self-hosted-snuba-replacer-1                             0.27%     77.82MiB / 30.61GiB   0.25%     206kB / 134kB     20.5kB / 0B       5
d5872f0aab61   sentry-self-hosted-snuba-consumer-1                             2.88%     79.8MiB / 30.61GiB    0.25%     219MB / 20MB      553kB / 0B        10
64e10abfea02   sentry-self-hosted-snuba-sessions-consumer-1                    0.31%     78.05MiB / 30.61GiB   0.25%     203kB / 131kB     0B / 0B           8
1c36a15d6a76   sentry-self-hosted-symbolicator-1                               0.01%     9.203MiB / 30.61GiB   0.03%     3.67kB / 0B       2.2MB / 0B        26
2d115aeef1c2   sentry-self-hosted-kafka-1                                      6.81%     608.7MiB / 30.61GiB   1.94%     227MB / 798MB     3.54MB / 212MB    83
e4681ebf6c6d   sentry-self-hosted-zookeeper-1                                  0.14%     78.37MiB / 30.61GiB   0.25%     43.6kB / 49.1kB   3.13MB / 422kB    67
354b3064f3e8   sentry-self-hosted-redis-1                                      8.96%     113.2MiB / 30.61GiB   0.36%     683MB / 700MB     6.25MB / 219MB    5
5b5d617cbe77   sentry-self-hosted-clickhouse-1                                 17.80%    841.7MiB / 30.61GiB   2.68%     19.3MB / 458kB    14.4MB / 822MB    62

Note that this server isn't dedicated to sentry... there are some of my own containers running, the worst of which is probably this one:

d29a327d8ceb   srv-hsjdkfk--jdheudnfu.2.od2sau9y2xkun6qt9olpkh719              272.72%   1.551GiB / 30.61GiB   5.07%     214MB / 641MB     65.3MB / 23.9MB   129

[drmrbrewer]

And there does seem to be quite a bit left over even after running docker compose down -v:

~/sentry/self-hosted# docker compose down -v
[+] Running 7/0
 ⠿ Volume sentry-self-hosted_sentry-secrets         Removed                                                                              0.0s
 ⠿ Volume sentry-self-hosted_sentry-nginx-cache     Removed                                                                              0.0s
 ⠿ Volume sentry-self-hosted_sentry-kafka-log       Removed                                                                              0.0s
 ⠿ Volume sentry-self-hosted_sentry-clickhouse-log  Removed                                                                              0.0s
 ⠿ Volume sentry-self-hosted_sentry-smtp-log        Removed                                                                              0.0s
 ⠿ Volume sentry-self-hosted_sentry-smtp            Removed                                                                              0.0s
 ⠿ Volume sentry-self-hosted_sentry-zookeeper-log   Removed                                                                              0.0s

~/sentry/self-hosted# docker system df -v | grep sentry
sentry-cleanup-self-hosted-local         latest          e7ede052ba7f   34 minutes ago      848.2MB   846.9MB       1.386MB       0
sentry-self-hosted-local                 latest          ca31283ff33a   34 minutes ago      846.9MB   846.9MB       0B            0
getsentry/relay                          nightly         f9338e4e54e9   2 hours ago         238.7MB   85.19MB       153.5MB       0
getsentry/sentry                         nightly         7e6a40d43a81   2 hours ago         846.8MB   846.8MB       0B            0
getsentry/snuba                          nightly         97d532d10472   15 hours ago        427.7MB   427.7MB       0B            0
getsentry/symbolicator                   nightly         9ad1d9ba1684   3 days ago          151.3MB   151.3MB       0B            0
sentry-clickhouse                                                  0         20.96MB
sentry-data                                                        0         0B
sentry-symbolicator                                                0         0B
sentry-zookeeper                                                   0         341.4kB
sentry-kafka                                                       0         7.178GB
sentry-postgres                                                    0         766.9MB
sentry-redis                                                       0         30.24MB

[drmrbrewer]

I cleared out all unused volumes with docker volume prune, then did a fresh install, and this time indeed it didn't have any memory of any previous user.

And sure enough, now when I start sentry, the docker compose logs -f output is completely clean before I start sending data, and CPU is completely fine even once I start sending data from my production system, with a nice, clean log showing that it's just calming receiving and processing data:

sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:04.028748 [ 927 ] {6ec623a5-39b4-451f-a2cd-e6c4ec170f0b} <Information> executeQuery: Read 3 rows, 1.45 KiB in 0.002 sec., 1338 rows/sec., 646.28 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:04.028984 [ 927 ] {} <Information> TCPHandler: Processed in 0.003 sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:04.182565 [ 928 ] {} <Information> TCPHandler: Done processing connection.
sentry-self-hosted-snuba-transactions-cleanup-1                | 2022-09-23 13:40:04,030 Dropped 0 partitions on clickhouse:9000
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:04.219153 [ 927 ] {} <Information> TCPHandler: Done processing connection.
sentry-self-hosted-nginx-1                                     | 2022/09/23 13:40:11 [warn] 23#23: *277 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000018, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1                                     | 10.0.1.32 - - [23/Sep/2022:13:40:11 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "172.18.0.1, 10.0.1.5"
sentry-self-hosted-nginx-1                                     | 2022/09/23 13:40:12 [warn] 23#23: *279 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000019, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1                                     | 10.0.1.32 - - [23/Sep/2022:13:40:12 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "95.217.18.255, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1                           | 13:40:13 [INFO] batching-kafka-consumer: Flushing 2 items (from {('ingest-transactions', 0): [24, 25]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1                           | 13:40:13 [INFO] batching-kafka-consumer: Worker flush took 43ms
sentry-self-hosted-snuba-transactions-consumer-1               | 2022-09-23 13:40:14,597 Completed processing <Batch: 2 messages, open for 1.05 seconds>.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:14.576825 [ 86 ] {52a7981a-e29b-4d3d-874e-3bea4b56a768} <Information> executeQuery: Read 2 rows, 124.00 B in 0.013 sec., 153 rows/sec., 9.27 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:14.577499 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:14.590924 [ 871 ] {0321dcfc-8f99-418b-81b8-38aee257a35e} <Information> executeQuery: Read 2 rows, 3.85 KiB in 0.024 sec., 81 rows/sec., 157.42 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:14.591532 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1                   | 2022-09-23 13:40:14,583 Completed processing <Batch: 2 messages, open for 1.03 seconds>.
sentry-self-hosted-nginx-1                                     | 2022/09/23 13:40:19 [warn] 23#23: *281 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000020, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1                                     | 10.0.1.32 - - [23/Sep/2022:13:40:19 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "95.217.18.255, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1                           | 13:40:20 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [26, 26]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1                           | 13:40:20 [INFO] batching-kafka-consumer: Worker flush took 15ms
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:21.427031 [ 871 ] {429e4f5e-8b43-450f-813c-b988ece75c4f} <Information> executeQuery: Read 1 rows, 71.00 B in 0.011 sec., 94 rows/sec., 6.55 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:21.427716 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:21.433547 [ 86 ] {ba60fced-5f24-4cfd-81c7-cdf5000c5fb8} <Information> executeQuery: Read 1 rows, 2.04 KiB in 0.021 sec., 48 rows/sec., 99.37 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:21.434016 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1                   | 2022-09-23 13:40:21,431 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-snuba-transactions-consumer-1               | 2022-09-23 13:40:21,437 Completed processing <Batch: 1 message, open for 1.03 seconds>.
sentry-self-hosted-nginx-1                                     | 2022/09/23 13:40:28 [warn] 23#23: *283 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000021, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1                                     | 10.0.1.32 - - [23/Sep/2022:13:40:28 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "172.18.0.1, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1                           | 13:40:29 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [27, 27]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1                           | 13:40:29 [INFO] batching-kafka-consumer: Worker flush took 11ms
sentry-self-hosted-snuba-transactions-consumer-1               | 2022-09-23 13:40:30,582 Completed processing <Batch: 1 message, open for 1.03 seconds>.
sentry-self-hosted-snuba-outcomes-consumer-1                   | 2022-09-23 13:40:30,567 Completed processing <Batch: 1 message, open for 1.01 seconds>.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:30.564002 [ 86 ] {28b6de30-aa77-45b3-96ec-d95d8e3b4f09} <Information> executeQuery: Read 1 rows, 71.00 B in 0.005 sec., 183 rows/sec., 12.75 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:30.565143 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:30.574041 [ 88 ] {6c1e39a1-b496-46c1-b34c-e42f0689f6c0} <Information> executeQuery: Read 1 rows, 2.08 KiB in 0.015 sec., 68 rows/sec., 142.74 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:30.574501 [ 88 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-nginx-1                                     | 10.0.1.32 - - [23/Sep/2022:13:40:49 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "172.18.0.1, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1                           | 13:40:50 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [28, 28]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1                           | 13:40:50 [INFO] batching-kafka-consumer: Worker flush took 19ms
sentry-self-hosted-nginx-1                                     | 2022/09/23 13:40:50 [warn] 23#23: *287 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000022, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1                                     | 10.0.1.32 - - [23/Sep/2022:13:40:50 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "95.217.18.255, 10.0.1.5"
sentry-self-hosted-snuba-transactions-consumer-1               | 2022-09-23 13:40:51,278 Completed processing <Batch: 1 message, open for 1.04 seconds>.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:51.252905 [ 871 ] {cc51e6dc-e066-452e-a2d8-2073d4a57fda} <Information> executeQuery: Read 1 rows, 71.00 B in 0.011 sec., 92 rows/sec., 6.40 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:51.253389 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:51.274097 [ 86 ] {6e07f3e6-4d94-4e83-9779-48695a9d3157} <Information> executeQuery: Read 1 rows, 2.14 KiB in 0.028 sec., 36 rows/sec., 77.68 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:51.274580 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1                   | 2022-09-23 13:40:51,256 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-ingest-consumer-1                           | 13:40:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [29, 29]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1                           | 13:40:51 [INFO] batching-kafka-consumer: Worker flush took 20ms
sentry-self-hosted-snuba-outcomes-consumer-1                   | 2022-09-23 13:40:52,572 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:52.567509 [ 871 ] {cd393bc3-29e9-4554-ad3b-9bd370ed12bc} <Information> executeQuery: Read 1 rows, 71.00 B in 0.009 sec., 117 rows/sec., 8.12 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:52.568424 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:52.573708 [ 86 ] {a963468b-41f3-4b0b-b5e1-e99b95f9ae6c} <Information> executeQuery: Read 1 rows, 2.05 KiB in 0.016 sec., 64 rows/sec., 131.65 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:40:52.573999 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-transactions-consumer-1               | 2022-09-23 13:40:52,578 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-nginx-1                                     | 2022/09/23 13:41:02 [warn] 23#23: *289 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000023, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1                                     | 10.0.1.32 - - [23/Sep/2022:13:41:02 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "95.217.18.255, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1                           | 13:41:03 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [30, 30]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1                           | 13:41:03 [INFO] batching-kafka-consumer: Worker flush took 20ms
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:41:04.212943 [ 88 ] {85e86ebc-d71b-48ba-8a45-be7c1e263815} <Information> executeQuery: Read 1 rows, 71.00 B in 0.012 sec., 83 rows/sec., 5.81 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:41:04.213436 [ 88 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:41:04.225864 [ 871 ] {63aebc9c-4066-4d7d-94d2-f7b6ea4e7829} <Information> executeQuery: Read 1 rows, 2.00 KiB in 0.023 sec., 43 rows/sec., 86.27 KiB/sec.
sentry-self-hosted-clickhouse-1                                | 2022.09.23 13:41:04.226333 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1                   | 2022-09-23 13:41:04,218 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-snuba-transactions-consumer-1               | 2022-09-23 13:41:04,232 Completed processing <Batch: 1 message, open for 1.04 seconds>.

So it seems that sentry somehow gets stuck in some horrible loop (as @wodCZ suspected), with log output as per the post above... causing CPU to max out and the whole thing grinds to a halt... any ideas?

[hubertdeng123]

Thanks for bringing this up to us...looks like this will require some additional investigation. I'll keep this thread updated if I can find something meaningful

[aminvakil]

@drmrbrewer How many requests per second are you handling at your self-hosted instance at the time of cpu spike?

I mean maybe there is another reason which sentry keeps getting hit more and more and therefore higher cpu usage.

[drmrbrewer]

@aminvakil it's a fair question but I don't think it could just be due to a surge in traffic. Traffic is normally around 5 requests per second, and wouldn't go above around 15 per second, but I have actually set tracesSampleRate to 0.01 so Sentry is normally pretty quiet. This does seem like something triggers Sentry to go haywire. I should add that, since re-installing Sentry, I haven't seen this again, so it is probably quite rare?

[aminvakil]

@drmrbrewer Sorry for late response.

It's good to see your issue has got solved, but I had not seen this in my experience prior to your instance, but as reinstalling self-hosted has fixed the issue, I think maybe there was a problem with installation at the first place, not sure what though.

[afaianswq]

I experienced a similar issue recently: Sentry consumed all available RAM and caused system to cripple with endless OOM-killer events. Restarting Sentry helped for several hours before the issue came back. From docker container stats output it was clear that Redis is the problem. The solution was to change Redis configuration, we added two lines in Sentry startup script:

docker compose exec redis redis-cli CONFIG SET maxmemory-policy volatile-ttl
docker compose exec redis redis-cli CONFIG SET maxmemory 4G

[drmrbrewer]

@afaianswq interesting... I don't know enough about redis to understand how far this changes the redis setup from its default? Is this setup something that the Sentry team could safely add to the default Sentry startup script for all?