Thread parking for Kotlin/Common #498

bbrockbernd · 2024-12-19T11:08:06Z

Allows to pause and resume thread execution. On native platforms it is based on pthread_cond_wait, on JVM it uses LockSupport.

Threads can be pre unparked (calling unpark before park cancels the parking operation). And thread can be parker with a timeout.

Suspend the current thread by calling:

ParkingSupport.park(timeout: kotlin.time.Duration)
// or
ParkingSupport.parkUntil(deadline: TimeMark)
// or park without timout
ParkingSupport.park(kotlin.time.Duration.INFINITE)

Resume a thread by calling:

ParkingSupport.unpark(handle: ParkingHandle)

Get current thread reference by calling:

val handle = ParkingSupport.currentThreadHandle()

atomicfu/src/jvmMain/kotlin/kotlinx/atomicfu/parking/KThread.kt

atomicfu/src/jvmMain/kotlin/kotlinx/atomicfu/parking/JvmParkingDelegator.kt

atomicfu/src/androidNative32BitMain/kotlin/kotlinx/atomicfu/parking/PosixParkingDelegator.kt

atomicfu/src/commonMain/kotlin/kotlinx/atomicfu/parking/ParkingDelegator.kt

atomicfu/src/jvmMain/kotlin/kotlinx/atomicfu/parking/JvmParkingDelegator.kt

atomicfu/src/androidNative32BitMain/kotlin/kotlinx/atomicfu/parking/PosixParkingDelegator.kt

atomicfu/src/androidNative64BitMain/kotlin/kotlinx/atomicfu/parking/PosixParkingDelegator.kt

atomicfu/src/androidNative32BitMain/kotlin/kotlinx/atomicfu/parking/PosixParkingDelegator.kt

atomicfu/src/jsAndWasmSharedMain/kotlin/kotlinx/atomicfu/parking/KThread.kt

…t current

… Commonizing is difficult due to absence of common threading api.

…arking behaviour on jvm and native.

…ow exceptions

dkhalanskyjb

The implementation is almost ready to go, I believe. Good work! Only some documentation fixes are left, and there's maybe potential to simplify the code.

I haven't looked at the tests yet, though.

atomicfu/build.gradle.kts

atomicfu/src/concurrentMain/kotlin/kotlinx/atomicfu/locks/ParkingSupport.kt

dkhalanskyjb · 2025-04-03T10:37:08Z

atomicfu/src/nativeMain/kotlin/kotlinx/atomicfu/locks/ParkingSupport.kt

+        if (timout == Duration.INFINITE) threadLocalParkingHandle.parker.park()
+        else threadLocalParkingHandle.parker.parkNanos(timout.toLong(DurationUnit.NANOSECONDS))
+    }
+    actual fun parkUntil(deadline: TimeMark) = park(deadline.elapsedNow() * -1)


Question: can the code be simplified if we take parkUntil as the fundamental operation and then implement parkNanos on top of it? The idea is that POSIX exposes something that's similar to parkUntil already, so waitWithDeadline may be easier to write than timedWait, and LockSupport.parkUntil may be somewhat simpler than LockSupport.parkNanos (due to the simpler "is it time to exit yet?" check).

AFAIK, a TimeMark is not directly convertible to nanos from epoch. The only way I have found so far is by somehow getting the current time from epoch and subtracting TimeMark.elapsedNow (which is converting to duration.)

Additionally, to get current time in nanos from epoch (on native) we either need kotlinx-datetime which doesn't seem to be more granular than milliseconds. Or rely on clock_gettime from POSIX and convert TimeMark to duration, which is what we currently do.

But I might be missing some api?

You're right, TimeMark does not expose its relationship to absolute time.

kotlin.time.Instant (which used to be in kotlinx-datetime) does provide nanosecond precision, and in fact, delegates to clock_gettime. So, with Kotlin 2.1.20, which introduced Instant, it's possible to do this (pseudocode):

fun parkUntil(deadline: TimeMark) { val estimatedTargetTime: Instant = Clock.System.now() - deadline.elapsedNow() sleepUntil(instant.epochSeconds, instant.nanosecondsOfSecond) }

The problem is, we're not on Kotlin 2.1.20, so we indeed don't have access to Instant.

But! This gets me thinking: why do we need an Instant at all? It's semantically wrong. In fact, we may have a subtle bug.

See, CLOCK_REALTIME (https://man7.org/linux/man-pages/man2/clock_gettime.2.html), kotlin.time.Instant, and by default, pthread_cond_timedwait (https://docs.oracle.com/cd/E19120-01/open.solaris/816-5137/gfvid/index.html) all have to do with system time, that is, what your clock in the corner of the screen says. This clock may get out of sync, and then get back in sync when your system queries the Internet for the up-to-date time.

It seems like the following may happen:

At 12:13:59, we park for three seconds. We are waiting until 12:14:02.

One second later, at 12:14:00, we query the Internet and see that our clock is too fast and is into the future by two seconds.

One second later, it's 12:13:59 again.

Then, three seconds later, it's finally 12:14:02... but five seconds have passed.

TimeSource.Monotonic, though, works with elapsed time, which is what we want.

Proposed fix: https://stackoverflow.com/a/14397906. With this, we don't need to extract epoch time at all.

What do you think?

That is quite the edge case! However, the fix proposed relies on pthread_condattr_setclock which is not available on darwin.

Also, I am wondering, is the kotlin TimeSource.Monotonic synced to the posix CLOCK_MONOTONIC?

That is quite the edge case!

Indeed, but I'd estimate that every day, someone somewhere encounters this. Having clocks drift by a couple of seconds per day is typical.

not available on darwin.

Oh, ok. If you also can't think of a way to fix this there, we need to at least document that on Native, the system clock is used and is susceptible to such issues.

Also, I am wondering, is the kotlin TimeSource.Monotonic synced to the posix CLOCK_MONOTONIC?

Kotlin/Native delegates to C++'s std::conditional<high_resolution_clock::is_steady, high_resolution_clock, steady_clock>. I don't know how our C++ standard library implements those, but it's not necessarily a wrapper around POSIX.

atomicfu/src/concurrentMain/kotlin/kotlinx/atomicfu/locks/ParkingSupport.kt

… in build.gradle.kts.

atomicfu/build.gradle.kts

…s is unsupported.

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/BarrierTest.kt

dkhalanskyjb · 2025-04-14T11:13:37Z

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/CyclicBarrierTest.kt

+            val threadSetSize = (iteration + 1) * 5
+            val bar = CyclicBarrier(threadSetSize)
+            val syncBar = CyclicBarrier(threadSetSize * 5)
+            val ar = Arrs(threadSetSize * 5)
+            repeat(threadSetSize * 5) { tId ->
+                val t = Fut {
+                    repeat(50) { internalIteration ->
+                        sleepMills(Random.nextLong(100))
+                        bar.await()
+                        sleepMills(Random.nextLong(100))
+                        val newN = ar.before[tId].value!! + 1
+                        ar.before[tId].value = newN
+                        syncBar.await()
+                        repeat(ar.before.size) { otherThread ->
+                            if (ar.before[otherThread].value!! < newN) {
+                                fail("Thread $tId (value: $newN, id: ${ParkingSupport.currentThreadHandle()}) continued too early: $otherThread had value ${ar.before[otherThread].value!!}")
+                            }
+                            if (ar.before[otherThread].value!! > newN + 1) {
+                                fail("Thread $tId (value: $newN, id: ${ParkingSupport.currentThreadHandle()}) too far behind: $otherThread had value ${ar.before[otherThread].value!!}")
+                            }
+                        }
+                    }
+                }
+                threads.add(t)
+            }
+            Fut.waitAllAndThrow(threads)


There are quite a few constants mentioned here: 5, 50, 100. Could you give them clear names? This goes for other tests, too.

dkhalanskyjb · 2025-04-14T11:14:24Z

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/CyclicBarrierTest.kt

+            repeat(threadSetSize * 5) { tId ->
+                val t = Fut {
+                    repeat(50) { internalIteration ->
+                        sleepMills(Random.nextLong(100))


In total, these sleepMills affect test execution time considerably. Are they necessary?

They are not necessary for the test to pass no. However, I believe that adding random waits gives us more variantions in thread arrival time at the barrier relative to each other. Which could increase the chance of catching bugs.

But 100 milliseconds might be a bit excessive, I have decreased it to 5 wdyt?

dkhalanskyjb · 2025-04-14T11:18:13Z

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/ThreadParkerTest.kt

+            ParkingSupport.park(Duration.INFINITE)
+        }
+
+        assertTrue(time.inWholeMilliseconds > 900)


A more reliable (and still less time-consuming) way to test this would be to spin for several thousands of times and see if park ever finishes earlier than the corresponding unpark (basically, that happens-before is established between the code before unpark and the code after park).

Also, can't this test be flaky due to spurious wakeups? The thread being parked should at least be a separate one and not the test thread, I believe; otherwise, the tests can interfere with one another.

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/TestThread.kt

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/ThreadParkerTest.kt

- in ThreadParkerTest.kt move parking behaviour to sub thread to prevent potential interference with other tests. - Create consts for numbers - Removed simple cyclic barrier test (as this should be covered byt the stress variant) - Decreased wait times

atomicfu/src/androidNativeMain/kotlin/kotlinx/atomicfu/locks/PosixParkingDelegator.kt

atomicfu/src/mingwMain/kotlin/kotlinx/atomicfu/locks/PosixParkingDelegator.kt

atomicfu/src/nativeMain/kotlin/kotlinx/atomicfu/locks/NativeParkingUtils.kt

dkhalanskyjb · 2025-04-25T09:51:21Z

atomicfu/src/androidNativeMain/kotlin/kotlinx/atomicfu/locks/PosixParkingDelegator.kt

+    actual fun createRef(): ParkingData {
+        val mut = nativeHeap.alloc<pthread_mutex_t>().ptr
+        val cond = nativeHeap.alloc<pthread_cond_t>().ptr
+        val attr = nativeHeap.alloc<pthread_condattr_tVar>().ptr


Note: this is irritatingly close to pthread_condattr_t, which is used by the other Unix-like platforms, but after several attempts, I did not manage to find an incantation that would allow avoiding the code duplication for androidNativeMain.

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/ExchangerTest.kt

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/ThreadParkerTest.kt

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/TimedParkingTest.kt

- move destruction of condattr to `createRef` - common callAndVerify + happy paths - timout typos

… redundant null checks, refactor waitAllAndThrow

fzhinkin · 2025-04-28T18:13:40Z

Please update the target branch to develop.

fzhinkin · 2025-04-28T19:56:03Z

There are small formatting issues here and there (like trailing spaces), please run the "Code/Reformat Code" in IJ (you can perform a bulk cleanup by selecting all the necessary files and then running the reformatting).

fzhinkin

Nicely done! I left a few comments regarding tests and documentation, but overall it looks good.

fzhinkin · 2025-04-28T18:36:49Z

atomicfu/src/concurrentMain/kotlin/kotlinx/atomicfu/locks/ThreadParker.kt

+    }
+}
+
+private interface ParkingState


Does it make sense to make it sealed?

fzhinkin · 2025-04-28T18:42:57Z

atomicfu/src/concurrentMain/kotlin/kotlinx/atomicfu/locks/ParkingSupport.kt

+     * - (Only on JVM) The thread was interrupted. The interrupted flag stays set after wakeup.
+     * A future call to [park] this thread will return immediately, unless the `Thread.interrupted` flag is cleared.
+     */
+    fun parkUntil(deadline: TimeMark)


There are no tests for this function.

fzhinkin · 2025-04-28T18:51:03Z

atomicfu/src/concurrentMain/kotlin/kotlinx/atomicfu/locks/ParkingSupport.kt

+
+    /**
+     * Unparks the thread corresponding to [handle].
+     * If [unpark] is called while the corresponding thread is not parked, the next [park] call will return immediately


Could you please add a test covering this part of the contract?

fzhinkin · 2025-04-28T18:59:59Z

atomicfu/src/concurrentMain/kotlin/kotlinx/atomicfu/locks/ParkingSupport.kt

+    fun unpark(handle: ParkingHandle)
+
+    /**
+     * Returns the [ParkingHandle] that can be used to [unpark] the current thread.


Missing parts:

does this function return the same instance for the same thread on each call?

is it safe to cache returned value?

is it safe to share it with other threads and call unpark concurrently?

fzhinkin · 2025-04-28T19:03:23Z

atomicfu/src/concurrentMain/kotlin/kotlinx/atomicfu/locks/ParkingSupport.kt

+ * Parking and unparking support for threads on Kotlin/Native and Kotlin/JVM. 
+ * Can be used as a building block to create locks and other synchronization primitives.
+ *
+ * A call to [ParkingSupport.park] or [ParkingSupport.parkUntil] will suspend the current thread.


It might be worth using the word "block" instead of the "suspend" to avoid potential confusion with coroutines suspension.

fzhinkin · 2025-04-28T19:59:04Z

atomicfu/src/concurrentMain/kotlin/kotlinx/atomicfu/locks/ParkingDelegator.kt

+internal fun Long.addNanosToSeconds(nanos: Long): Long {
+


Suggested change

internal fun Long.addNanosToSeconds(nanos: Long): Long {

/**

* Adds nano seconds to current time in seconds.

*/

internal fun Long.addNanosToSeconds(nanos: Long): Long {

fzhinkin · 2025-04-28T20:10:38Z

atomicfu/src/concurrentMain/kotlin/kotlinx/atomicfu/locks/ParkingSupport.kt

+ * Is used to unpark a thread.
+ * Can be obtained by calling [ParkingSupport.currentThreadHandle].
+ * Is required by [ParkingSupport.unpark].


Maybe something like?

Suggested change

* Is used to unpark a thread.

* Can be obtained by calling [ParkingSupport.currentThreadHandle].

* Is required by [ParkingSupport.unpark].

* A handle allowing to unpark a thread of execution using [ParkingSupport.unpark].

* There is a one-to-one mapping between threads and parking handles.

* A handle can be obtained by calling [ParkingSupport.currentThreadHandle].

* Refer to [ParkingSupport] documentation for more details on how to use [ParkingHandle] and how parking works in general.

fzhinkin · 2025-04-28T20:24:59Z

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/TimedParkingTest.kt

+class TimedParkingTest {
+
+    @Test
+    fun testNanosFirstUnpark400() = retry(3) {


If test cases differ only by constants, maybe we can create a single function that accepts corresponding parameters and call it from each test method?

fzhinkin · 2025-04-28T20:29:41Z

atomicfu/src/nativeUnixLikeMain/kotlin/kotlinx/atomicfu/locks/PosixParkingDelegator.kt

+internal expect val clockId: Int
+internal expect fun setClock(attr: CPointer<pthread_condattr_t>): Int


I would recommend either using more specific names, or scoping the property and the function somehow. Right now they are too generic and are quite confusing because of that.

fzhinkin · 2025-04-28T21:08:45Z

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/TimedParkingTest.kt

+class TimedParkingTest {
+
+    @Test
+    fun testNanosFirstUnpark400() = retry(3) {


Reliability of these tests worries me. 100ms slack feels huge, but in practice we can accidentally exceed it and got a test failure (and I guess we're already doing that given the test failures on Windows).

For deadline-specific scenarios, I don't think we need to unpark a future from the main thread at all, and we should only verify that it was parked for at least timeout-time-units.

For first-unpark-specific scenarios, we can't get away from verifying that the park was actually "interrupted" by the unpark call, but maybe we can increase the slack to a few minutes, instead of 100ms?

bbrockbernd requested review from ndkoval and dkhalanskyjb December 19, 2024 11:08

bbrockbernd marked this pull request as ready for review December 19, 2024 11:10

dkhalanskyjb reviewed Jan 16, 2025

View reviewed changes

atomicfu/src/jvmMain/kotlin/kotlinx/atomicfu/parking/KThread.kt Outdated Show resolved Hide resolved

bbrockbernd requested a review from dkhalanskyjb January 21, 2025 08:58

ndkoval reviewed Jan 29, 2025

View reviewed changes

atomicfu/src/jvmMain/kotlin/kotlinx/atomicfu/parking/JvmParkingDelegator.kt Outdated Show resolved Hide resolved

ndkoval reviewed Jan 29, 2025

View reviewed changes

atomicfu/src/jvmMain/kotlin/kotlinx/atomicfu/parking/JvmParkingDelegator.kt Outdated Show resolved Hide resolved

dkhalanskyjb reviewed Jan 29, 2025

View reviewed changes

atomicfu/src/androidNative32BitMain/kotlin/kotlinx/atomicfu/parking/PosixParkingDelegator.kt Outdated Show resolved Hide resolved

dkhalanskyjb reviewed Jan 29, 2025

View reviewed changes

atomicfu/src/androidNative64BitMain/kotlin/kotlinx/atomicfu/parking/PosixParkingDelegator.kt Outdated Show resolved Hide resolved

dkhalanskyjb reviewed Jan 29, 2025

View reviewed changes

atomicfu/src/androidNative32BitMain/kotlin/kotlinx/atomicfu/parking/PosixParkingDelegator.kt Outdated Show resolved Hide resolved

atomicfu/src/androidNative32BitMain/kotlin/kotlinx/atomicfu/parking/PosixParkingDelegator.kt Outdated Show resolved Hide resolved

dkhalanskyjb reviewed Jan 29, 2025

View reviewed changes

atomicfu/src/jsAndWasmSharedMain/kotlin/kotlinx/atomicfu/parking/KThread.kt Outdated Show resolved Hide resolved

bbrockbernd added 18 commits January 30, 2025 12:01

Thread parking for kotlin/common

6cd373c

Comments

ab7fcba

Bugfix for jvm: use ThreadLocal.withInitial, unpark passed kThread no…

7574618

…t current

Added Barriertests, and ported native parking tests to jvm sourceset.…

d7318f2

… Commonizing is difficult due to absence of common threading api.

Bugfix parkNanos on JVM

352678a

Make barriertest not time dependent

c69cf64

Change modifier order back to test failing build on K1

3bb31ad

Explicitly define return types

fc003a1

K1 compatibility

03f2acc

Solve flaky test by giving timed parking tests more room for deviation.

9011bd6

Added CountDownLatch tests

2bd1f73

Added CyclicBarrier synchronization primitive to further stresstest p…

d28045d

…arking behaviour on jvm and native.

Retry flaky timing tests

70994b6

Catch and rethrow assertion errros in multithreaded tests on jvm

cc2c8c9

Prevent potential deadlock in tests by waiting non-blocking and rethr…

87cf4a2

…ow exceptions

Bugfixes in cyclicBarrierTest

3c25e16

Remove excessive printing in tests

8500459

Fix 32bit timedWait

35101b7

Added usage example for docs

40e11bf

dkhalanskyjb requested changes Apr 3, 2025

View reviewed changes

dkhalanskyjb self-requested a review April 3, 2025 10:42

bbrockbernd added 2 commits April 4, 2025 12:29

Improved docs according to review. Changed sourceSet structure syntax…

7867549

… in build.gradle.kts.

Use monotonic clock where possible

f0af600

dkhalanskyjb reviewed Apr 8, 2025

View reviewed changes

atomicfu/build.gradle.kts Outdated Show resolved Hide resolved

bbrockbernd added 3 commits April 8, 2025 17:56

Remove date-time dependency

5aaa89f

Remove unused param. Remove setting monotonic clock for mingw, as thi…

fb3e697

…s is unsupported.

Change back to realtime clock for clock_gettime on mingw

73872ca

bbrockbernd requested a review from dkhalanskyjb April 9, 2025 12:33

dkhalanskyjb requested a review from fzhinkin April 14, 2025 11:09

dkhalanskyjb reviewed Apr 14, 2025

View reviewed changes

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/BarrierTest.kt Outdated Show resolved Hide resolved

dkhalanskyjb reviewed Apr 14, 2025

View reviewed changes

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/TestThread.kt Outdated Show resolved Hide resolved

dkhalanskyjb reviewed Apr 14, 2025

View reviewed changes

atomicfu/src/concurrentTest/kotlin/kotlinx/atomicfu/locks/ThreadParkerTest.kt Outdated Show resolved Hide resolved

bbrockbernd added 3 commits April 14, 2025 18:12

- sleepMills -> sleepMillis

f797b55

- in ThreadParkerTest.kt move parking behaviour to sub thread to prevent potential interference with other tests. - Create consts for numbers - Removed simple cyclic barrier test (as this should be covered byt the stress variant) - Decreased wait times

Fix transform issue by removing unnecessary atomic

8bf7939

Increase time for slow windows target

4a950f0

bbrockbernd requested a review from dkhalanskyjb April 15, 2025 15:44

dkhalanskyjb requested changes Apr 25, 2025

View reviewed changes

bbrockbernd added 5 commits April 25, 2025 15:18

- include condattr on android native

a0f1beb

- move destruction of condattr to `createRef` - common callAndVerify + happy paths - timout typos

Apply minor suggestions: rename constants, use AtomicIntArray, remove…

f6fccac

… redundant null checks, refactor waitAllAndThrow

Use Fut instead of testThread

389fc74

Remove unused imports

a3e958a

Wait for parking handle to be published

519e59d

fzhinkin requested changes Apr 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Thread parking for Kotlin/Common #498

Thread parking for Kotlin/Common #498

bbrockbernd commented Dec 19, 2024 •

edited

Loading

dkhalanskyjb left a comment

dkhalanskyjb Apr 3, 2025

bbrockbernd Apr 4, 2025

dkhalanskyjb Apr 4, 2025

bbrockbernd Apr 4, 2025

dkhalanskyjb Apr 4, 2025

dkhalanskyjb Apr 14, 2025

dkhalanskyjb Apr 14, 2025 •

edited

Loading

bbrockbernd Apr 14, 2025

dkhalanskyjb Apr 14, 2025 •

edited

Loading

dkhalanskyjb Apr 25, 2025

fzhinkin commented Apr 28, 2025

fzhinkin commented Apr 28, 2025

fzhinkin left a comment

fzhinkin Apr 28, 2025

fzhinkin Apr 28, 2025

fzhinkin Apr 28, 2025

fzhinkin Apr 28, 2025

fzhinkin Apr 28, 2025

fzhinkin Apr 28, 2025

fzhinkin Apr 28, 2025

fzhinkin Apr 28, 2025

fzhinkin Apr 28, 2025

fzhinkin Apr 28, 2025

- * Is used to unpark a thread.
- * Can be obtained by calling [ParkingSupport.currentThreadHandle].
- * Is required by [ParkingSupport.unpark].
+ * A handle allowing to unpark a thread of execution using [ParkingSupport.unpark].
+ * There is a one-to-one mapping between threads and parking handles.
+ * A handle can be obtained by calling [ParkingSupport.currentThreadHandle].
+ * Refer to [ParkingSupport] documentation for more details on how to use [ParkingHandle] and how parking works in general.

		internal expect val clockId: Int
		internal expect fun setClock(attr: CPointer<pthread_condattr_t>): Int

Thread parking for Kotlin/Common #498

Are you sure you want to change the base?

Thread parking for Kotlin/Common #498

Conversation

bbrockbernd commented Dec 19, 2024 • edited Loading

dkhalanskyjb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkhalanskyjb Apr 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkhalanskyjb Apr 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fzhinkin commented Apr 28, 2025

fzhinkin commented Apr 28, 2025

fzhinkin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbrockbernd commented Dec 19, 2024 •

edited

Loading

dkhalanskyjb Apr 14, 2025 •

edited

Loading

dkhalanskyjb Apr 14, 2025 •

edited

Loading