feat(backend): the utilities and services required for the pow feature #5784

DecentAgeCoder · 2025-04-10T10:25:07Z

Motivation

This PR introduces utilities and tests to improve the worker communication framework, ensuring robust validation, seamless message handling, and secure hashing functionality. The changes enhance developer confidence in worker-related operations and establish a foundation for future scalability.

Changes

Introduced routeWorkerResponse and sendMessageRequest utilities in worker.utils.ts for handling and sending worker messages with type-safe validation using Zod.
Added worker.utils.spec.ts test suite to thoroughly validate the behavior of routeWorkerResponse and sendMessageRequest.
Created crypto.utils.ts to provide utility functions for secure hashing (e.g., sha256, bufferToHex, hashToHex).
Enhanced post-message.ts and post-message.schema.ts to centralize and type-check worker message schemas and definitions.

Tests

Mocking and Setup for tests
Tests for worker.utils.ts
Tests for crypto.utils.ts

Simplifying optional param handling using toNullable() Co-authored-by: Antonio Ventilii <antonio.ventilii@dfinity.org>

…roducing-pow-worker-1

…w-worker-1' into feat/frontend/pow/introducing-pow-worker-1

…roducing-pow-worker-1

…roducing-pow-worker-2

…rontend/pow/introducing-pow-worker-2

…w-worker-2' into feat/frontend/pow/introducing-pow-worker-2

…rontend/pow/introducing-pow-worker-2 # Conflicts: # src/frontend/src/tests/lib/services/loader.services.spec.ts

…roducing-pow-worker-2

AntonioVentilii · 2025-04-11T11:17:35Z

src/frontend/src/lib/schema/post-message.schema.ts

+ */
+export const PostMessageBaseSchema = z.object({
+	msg: z.string(), // Message type
+	requestId: z.string() // Unique identifier for tracking requests and responses


we have a pre-existing base PostMessageSchema, can we expand that?

I do not want to risk the to refactor the other messages since this structure is specifically designed for implementing the short-lived Request-Response Messaging pattern.

AntonioVentilii · 2025-04-11T12:48:03Z

src/frontend/src/lib/schema/post-message.schema.ts

+ */
+export const PostMessageRequestBaseSchema = PostMessageBaseSchema.extend({
+	type: z.literal('request'), // Specifies this is a request
+	data: z.unknown().optional() // Optional data payload


why unknown? don't we know what the data are?

i'm using a hierarchical inheritance structure for schema. The data is overridden by the actual type.

AntonioVentilii · 2025-04-11T12:50:02Z

src/frontend/src/lib/schema/post-message.schema.ts

@@ -189,3 +189,92 @@ export const inferPostMessageSchema = <T extends z.ZodTypeAny>(dataSchema: T) =>
 		msg: z.union([PostMessageRequestSchema, PostMessageResponseSchema]),
 		data: dataSchema.optional()
 	});
+
+// -----------------------------------------------------------------------------------------------
+// The generic data structures which are required to implement the Short Polling (request-response)


In general, we already have quite a lot PostMessage types and schema, we use them for all the workers/schedulers. Is it not possible to use the same? I imagine you are using the same logic, no?

AntonioVentilii · 2025-04-11T12:52:04Z

src/frontend/src/lib/types/post-message.ts

+
+// -----------------------------------------------------------------------------------------------
+// The post message types used for short polling between:
+// pow.worker.ts <--->  worker.pow.services.ts
+// -----------------------------------------------------------------------------------------------
+// Base Types
+export type PostMessageBase = z.infer<typeof PostMessageBaseSchema>;
+export type PostMessageRequestBase = z.infer<typeof PostMessageRequestBaseSchema>;
+export type PostMessageResponseBase = z.infer<typeof PostMessageResponseBaseSchema>;
+export type PostMessageCreatePowChallengeRequest = z.infer<
+	typeof PostMessageCreatePowChallengeRequestSchema
+>;
+export type PostMessageCreatePowChallengeResponse = z.infer<
+	typeof PostMessageCreatePowChallengeResponseSchema
+>;
+export type PostMessageAllowSigningRequest = z.infer<typeof PostMessageAllowSigningRequestSchema>;
+export type PostMessageAllowSigningResponse = z.infer<typeof PostMessageAllowSigningResponseSchema>;


which one of these types is used in this PR? If they are not needed here, let's remove them (and their schemas if not needed)

AntonioVentilii · 2025-04-11T12:55:22Z

src/frontend/src/lib/utils/crypto.utils.ts

+export async function sha256(input: string): Promise<ArrayBuffer> {
+	return await crypto.subtle.digest('SHA-256', textEncoder.encode(input));
+}
+
+/**
+ * Converts an ArrayBuffer to its hexadecimal string representation.
+ */
+export function bufferToHex(buffer: ArrayBuffer): string {
+	return Array.from(new Uint8Array(buffer))
+		.map((byte) => byte.toString(16).padStart(2, '0'))
+		.join('');
+}
+
+/**
+ * Combines the hashing and hex conversion of a string into a single function.
+ */
+export async function hashToHex(input: string): Promise<string> {
+	const hashBuffer = await sha256(input);
+	return bufferToHex(hashBuffer);
+}


don't these functions already exists in some library that we have?

P.S. can you use arrow convention for them? for example: export const sha256 = ...

AntonioVentilii · 2025-04-11T12:59:46Z

src/frontend/src/lib/utils/worker.utils.ts

+
+const responseHandlers = new Map<string, (data: unknown) => void>();
+
+export function routeWorkerResponse(event: MessageEvent): boolean {


why the return is a boolean? WDYT of using ResultSuccess?

AntonioVentilii · 2025-04-11T13:00:28Z

src/frontend/src/lib/utils/worker.utils.ts

+import type { PostMessageRequestBase } from '$lib/types/post-message';
+import type { z } from 'zod';
+
+const responseHandlers = new Map<string, (data: unknown) => void>();


What is this constant? why is it used? why is it not everything handled inside the worker?

This is part of the short-lived Request-Response Messaging pattern. It is used to automatically route the request back to the sender. It is private and constant. If I move responseHandlers to pow.worker.ts I would also have to move routeWorkerResponse and sendMessageRequest to pow.worker.ts.

AntonioVentilii · 2025-04-11T13:00:41Z

src/frontend/src/lib/utils/worker.utils.ts

+		responseHandlers.delete(requestId);
+		return true;
+	}
+	console.warn('No handler found for event', requestId);


let's avoid warnings

How do we usually troubleshoot? Is there I dedicated debug logger that can be activated if needed? Should I remove all warnings?

AntonioVentilii · 2025-04-11T13:01:13Z

src/frontend/src/lib/utils/worker.utils.ts

+	data: object;
+	schema: z.ZodType<T>;
+}): Promise<T> {
+	const requestId = crypto.randomUUID();


this entire function should be inside the worker. How the other workers handle this?

AntonioVentilii · 2025-04-11T13:03:01Z

src/frontend/src/lib/utils/worker.utils.ts

+			resolve(parsed.data);
+		});
+
+		worker.postMessage(payload); // Send the typed payload


i am not really sure this pertains here... usually the workers that we have handle the messaging in code. For example, have a look at class Scheduler

AntonioVentilii · 2025-04-11T13:33:46Z