Skip to content

feat(backend): the utilities and services required for the pow feature #5784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 58 commits into from
Closed
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
249a09e
The canister changes required for the Pow Worker
DecentAgeCoder Apr 10, 2025
d1b36e2
Merge branch 'main' into feat/frontend/pow/introducing-pow-worker-1
DecentAgeCoder Apr 10, 2025
71a07ac
🤖 Apply formatting changes
github-actions[bot] Apr 10, 2025
0c86f2a
The utilities required for the Pow Worker
DecentAgeCoder Apr 10, 2025
a464686
The services required for the pow feature
DecentAgeCoder Apr 10, 2025
228ed56
Merge branch 'main' into feat/frontend/pow/introducing-pow-worker-2
DecentAgeCoder Apr 10, 2025
3b9364e
🤖 Apply formatting changes
github-actions[bot] Apr 10, 2025
6db98e4
Update src/frontend/src/lib/canisters/backend.canister.ts
DecentAgeCoder Apr 10, 2025
4962c80
Update src/frontend/src/lib/api/backend.api.ts
DecentAgeCoder Apr 10, 2025
8792a03
Update src/frontend/src/lib/api/backend.api.ts
DecentAgeCoder Apr 10, 2025
c78d546
Fixed the raised issues
DecentAgeCoder Apr 10, 2025
5e28068
Merge remote-tracking branch 'origin/main' into feat/frontend/pow/int…
DecentAgeCoder Apr 10, 2025
820cdff
🤖 Apply formatting changes
github-actions[bot] Apr 10, 2025
5f7e7f9
Inverted result handling for allowSigning
DecentAgeCoder Apr 10, 2025
ed1d241
Merge remote-tracking branch 'origin/feat/frontend/pow/introducing-po…
DecentAgeCoder Apr 10, 2025
e24dd02
Final changes for review
DecentAgeCoder Apr 10, 2025
be5ffed
Final changes for review
DecentAgeCoder Apr 10, 2025
1409ab4
🤖 Apply formatting changes
github-actions[bot] Apr 10, 2025
ae82919
Merge branch 'main' into feat/frontend/pow/introducing-pow-worker-2
DecentAgeCoder Apr 10, 2025
3b2370e
Final changes for review
DecentAgeCoder Apr 10, 2025
2997b61
🤖 Apply formatting changes
github-actions[bot] Apr 10, 2025
9078f2c
Fixed loader tests
DecentAgeCoder Apr 10, 2025
a004a9e
Merge remote-tracking branch 'origin/feat/frontend/pow/introducing-po…
DecentAgeCoder Apr 10, 2025
c223181
Merge remote-tracking branch 'origin/main' into feat/frontend/pow/int…
DecentAgeCoder Apr 10, 2025
4fd268a
Merge remote-tracking branch 'origin/main' into feat/frontend/pow/int…
DecentAgeCoder Apr 10, 2025
8504980
Merge branch 'feat/frontend/pow/introducing-pow-worker-1' into feat/f…
DecentAgeCoder Apr 10, 2025
49ef283
🤖 Apply formatting changes
github-actions[bot] Apr 10, 2025
2fedbd0
Merge remote-tracking branch 'origin/feat/frontend/pow/introducing-po…
DecentAgeCoder Apr 10, 2025
64c0da1
🤖 Apply formatting changes
github-actions[bot] Apr 10, 2025
5ce2ad7
Added post message types and schema. Added tests for worker.utils.ts
DecentAgeCoder Apr 10, 2025
2ea2384
Added tests for crypto utils
DecentAgeCoder Apr 10, 2025
ccdde45
Merge branch 'main' into feat/frontend/pow/introducing-pow-worker-1
DecentAgeCoder Apr 11, 2025
20761a1
Refactored and fixed the tests for the worker.utils.ts and crypto.uti…
DecentAgeCoder Apr 11, 2025
4a192a7
🤖 Apply formatting changes
github-actions[bot] Apr 11, 2025
f16304a
Fixed the crypto utils tests
DecentAgeCoder Apr 11, 2025
03306f0
Merge remote-tracking branch 'origin/feat/frontend/pow/introducing-po…
DecentAgeCoder Apr 11, 2025
543b545
🤖 Apply formatting changes
github-actions[bot] Apr 11, 2025
ceaa96b
revert type
AntonioVentilii Apr 11, 2025
7b2a247
revert usage of type
AntonioVentilii Apr 11, 2025
3092ca0
revert usage of type
AntonioVentilii Apr 11, 2025
957e714
revert
AntonioVentilii Apr 11, 2025
90b0aa8
refactor
AntonioVentilii Apr 11, 2025
c33932a
revert
AntonioVentilii Apr 11, 2025
cd2a95b
refactor
AntonioVentilii Apr 11, 2025
18b45aa
refactor
AntonioVentilii Apr 11, 2025
392f889
refactor
AntonioVentilii Apr 11, 2025
4a3632c
test
AntonioVentilii Apr 11, 2025
ebbc8d2
tests
AntonioVentilii Apr 11, 2025
899db80
clean
AntonioVentilii Apr 11, 2025
c055b79
clean
AntonioVentilii Apr 11, 2025
cb729c4
🤖 Apply formatting changes
github-actions[bot] Apr 11, 2025
de27d22
Merge branch 'feat/frontend/pow/introducing-pow-worker-1' into feat/f…
DecentAgeCoder Apr 11, 2025
71cd3e8
Merge branch 'main' into feat/frontend/pow/introducing-pow-worker-2
DecentAgeCoder Apr 11, 2025
6efbfcb
Reverted unnecessary change
DecentAgeCoder Apr 11, 2025
66af348
Merge remote-tracking branch 'origin/main' into feat/frontend/pow/int…
DecentAgeCoder Apr 11, 2025
950067f
🤖 Apply formatting changes
github-actions[bot] Apr 11, 2025
bbd73cb
Merge branch 'main' into feat/frontend/pow/introducing-pow-worker-2
DecentAgeCoder Apr 11, 2025
4a29858
Merge branch 'main' into feat/frontend/pow/introducing-pow-worker-2
DecentAgeCoder Apr 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 89 additions & 0 deletions src/frontend/src/lib/schema/post-message.schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -189,3 +189,92 @@ export const inferPostMessageSchema = <T extends z.ZodTypeAny>(dataSchema: T) =>
msg: z.union([PostMessageRequestSchema, PostMessageResponseSchema]),
data: dataSchema.optional()
});

// -----------------------------------------------------------------------------------------------
// The generic data structures which are required to implement the Short Polling (request-response)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, we already have quite a lot PostMessage types and schema, we use them for all the workers/schedulers. Is it not possible to use the same? I imagine you are using the same logic, no?

// design pattern used for a simplified communication between the worker(s) and the web application
// The data structure for a request or response to/from a canister call is more or less 1:1 propagated
// by the data structure defined by the schema.
// -----------------------------------------------------------------------------------------------
/**
* Base schema for all post messages
*/
export const PostMessageBaseSchema = z.object({
msg: z.string(), // Message type
requestId: z.string() // Unique identifier for tracking requests and responses
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have a pre-existing base PostMessageSchema, can we expand that?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not want to risk the to refactor the other messages since this structure is specifically designed for implementing the short-lived Request-Response Messaging pattern.

});

/**
* Base schema for all post message requests
*/
export const PostMessageRequestBaseSchema = PostMessageBaseSchema.extend({
type: z.literal('request'), // Specifies this is a request
data: z.unknown().optional() // Optional data payload
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why unknown? don't we know what the data are?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm using a hierarchical inheritance structure for schema. The data is overridden by the actual type.

});

export const PostMessageErrorSchema = z.string();

/**
* Base schema for all post message responses
*/
export const PostMessageResponseBaseSchema = PostMessageBaseSchema.extend({
type: z.literal('response'),
result: z.union([z.object({ Ok: z.unknown() }), z.object({ Err: PostMessageErrorSchema })])
});

export const CreatePowChallengeRequestDataSchema = z.object({});

export const PostMessageCreatePowChallengeRequestSchema = PostMessageRequestBaseSchema.extend({
msg: z.literal('CreatePowChallengeRequest'),
data: CreatePowChallengeRequestDataSchema.optional()
});

export const CreatePowChallengeResponseResultSchema = z.object({
difficulty: z.number(),
start_timestamp_ms: z.bigint(),
expiry_timestamp_ms: z.bigint()
});

export const PostMessageCreatePowChallengeResponseSchema = PostMessageResponseBaseSchema.extend({
msg: z.literal('CreatePowChallengeResponse'),
result: z.union([
z.object({ Ok: CreatePowChallengeResponseResultSchema }),
z.object({ Err: PostMessageErrorSchema })
])
});

export const AllowSigningRequestDataSchema = z.object({
nonce: z.bigint()
});

export const PostMessageAllowSigningRequestSchema = PostMessageRequestBaseSchema.extend({
msg: z.literal('AllowSigningRequest'),
data: AllowSigningRequestDataSchema
});

export const ChallengeCompletionSchema = z.object({
solved_duration_ms: z.bigint(),
next_allowance_ms: z.bigint(),
next_difficulty: z.number(),
current_difficulty: z.number()
});

const AllowSigningStatusSchema = z.union([
z.object({ Skipped: z.null() }),
z.object({ Failed: z.null() }),
z.object({ Executed: z.null() })
]);

export const AllowSigningResponseResultSchema = z.object({
status: AllowSigningStatusSchema, // Use the corrected schema for status
challenge_completion: z.array(z.any()).optional(), // Assuming ChallengeCompletion is modeled correctly elsewhere
allowed_cycles: z.bigint()
});

export const PostMessageAllowSigningResponseSchema = PostMessageResponseBaseSchema.extend({
msg: z.literal('AllowSigningResponse'),
result: z.union([
z.object({ Ok: AllowSigningResponseResultSchema }),
z.object({ Err: PostMessageErrorSchema })
])
});
79 changes: 79 additions & 0 deletions src/frontend/src/lib/services/pow.services.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
import type {
AllowSigningRequest,
Result_2 as AllowSigningResult,
Result_6 as CreateChallengeResult
} from '$declarations/backend/backend.did';
import { allowSigningResult, createPowChallenge } from '$lib/api/backend.api';
import type { OptionIdentity } from '$lib/types/identity';
import { hashToHex } from '$lib/utils/crypto.utils';

function getTimestampNowMs(): bigint {
return BigInt(Date.now());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is nowInBigIntNanoSeconds maybe you can create nowInBigInt there too

}

export const solvePowChallenge = async ({
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests for this one?

suggestion: put this in another PR

timestamp,
difficulty
}: {
timestamp: bigint;
difficulty: number;
}): Promise<number> => {
if (difficulty <= 0) {
throw new Error('Difficulty must be greater than zero');
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i18n?

}

let nonce = 0;
const target = Math.floor(0xffffffff / difficulty);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe some comment here


// is only used to measure the effective execution time
const startTimestampMs = getTimestampNowMs();

while (true) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make it functional

const challengeStr = `${timestamp}.${nonce}`;
const hashHex = await hashToHex(challengeStr); // Using the new hashToHex function
const prefix = parseInt(hashHex.slice(0, 8), 16); // Only consider the first 4 bytes
if (prefix <= target) {
break;
}
nonce++;
}

const solveTimestampMs = getTimestampNowMs() - startTimestampMs;
console.error(`Pow Challenge solved in ${Number(solveTimestampMs) / 1e3} seconds.`);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why printing an error?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was used for debug purpose. I will remove it!

return nonce;
};

export const _createPowChallenge = async ({
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

identity
}: {
identity: OptionIdentity;
}): Promise<CreateChallengeResult> => {
try {
return await createPowChallenge({ identity });
} catch (error) {
return {
Err: {
Other: `UnexpectedError: ${error}`
}
};
}
};

export const _allowSigning = async ({
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this hidden method?

identity,
request
}: {
identity: OptionIdentity;
request?: AllowSigningRequest;
}): Promise<AllowSigningResult> => {
try {
return await allowSigningResult({ identity, request });
} catch (error) {
// Ensure the `Err` matches the `CreateChallengeError` type
return {
Err: {
Other: `UnexpectedError: ${error}`
}
};
}
};
28 changes: 26 additions & 2 deletions src/frontend/src/lib/types/post-message.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
import type {
inferPostMessageSchema,
PostMessageAllowSigningRequestSchema,
PostMessageAllowSigningResponseSchema,
PostMessageBaseSchema,
PostMessageCreatePowChallengeRequestSchema,
PostMessageCreatePowChallengeResponseSchema,
PostMessageDataRequestBtcSchema,
PostMessageDataRequestExchangeTimerSchema,
PostMessageDataRequestIcCkBTCUpdateBalanceSchema,
Expand All @@ -16,10 +22,11 @@ import type {
PostMessageDataResponseWalletCleanUpSchema,
PostMessageDataResponseWalletSchema,
PostMessageJsonDataResponseSchema,
PostMessageRequestBaseSchema,
PostMessageResponseBaseSchema,
PostMessageResponseSchema,
PostMessageResponseStatusSchema,
PostMessageSyncStateSchema,
inferPostMessageSchema
PostMessageSyncStateSchema
} from '$lib/schema/post-message.schema';
import type * as z from 'zod';
import type { ZodType } from 'zod';
Expand Down Expand Up @@ -78,3 +85,20 @@ export type PostMessageDataResponseBTCAddress = z.infer<
export type PostMessage<T extends PostMessageDataRequest | PostMessageDataResponse> = z.infer<
ReturnType<typeof inferPostMessageSchema<ZodType<T>>>
>;

// -----------------------------------------------------------------------------------------------
// The post message types used for short polling between:
// pow.worker.ts <---> worker.pow.services.ts
// -----------------------------------------------------------------------------------------------
// Base Types
export type PostMessageBase = z.infer<typeof PostMessageBaseSchema>;
export type PostMessageRequestBase = z.infer<typeof PostMessageRequestBaseSchema>;
export type PostMessageResponseBase = z.infer<typeof PostMessageResponseBaseSchema>;
export type PostMessageCreatePowChallengeRequest = z.infer<
typeof PostMessageCreatePowChallengeRequestSchema
>;
export type PostMessageCreatePowChallengeResponse = z.infer<
typeof PostMessageCreatePowChallengeResponseSchema
>;
export type PostMessageAllowSigningRequest = z.infer<typeof PostMessageAllowSigningRequestSchema>;
export type PostMessageAllowSigningResponse = z.infer<typeof PostMessageAllowSigningResponseSchema>;
Comment on lines +88 to +104
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which one of these types is used in this PR? If they are not needed here, let's remove them (and their schemas if not needed)

25 changes: 25 additions & 0 deletions src/frontend/src/lib/utils/crypto.utils.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
const textEncoder = new TextEncoder();

/**
* Hashes the given input string using the SHA-256 algorithm.
*/
export async function sha256(input: string): Promise<ArrayBuffer> {
return await crypto.subtle.digest('SHA-256', textEncoder.encode(input));
}

/**
* Converts an ArrayBuffer to its hexadecimal string representation.
*/
export function bufferToHex(buffer: ArrayBuffer): string {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't Buffer.from(...).toString('hex') the same?

return Array.from(new Uint8Array(buffer))
.map((byte) => byte.toString(16).padStart(2, '0'))
.join('');
}

/**
* Combines the hashing and hex conversion of a string into a single function.
*/
export async function hashToHex(input: string): Promise<string> {
const hashBuffer = await sha256(input);
return bufferToHex(hashBuffer);
}
Comment on lines +6 to +25
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't these functions already exists in some library that we have?

P.S. can you use arrow convention for them? for example: export const sha256 = ...

68 changes: 68 additions & 0 deletions src/frontend/src/lib/utils/worker.utils.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
import type { PostMessageRequestBase } from '$lib/types/post-message';
import type { z } from 'zod';

const responseHandlers = new Map<string, (data: unknown) => void>();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why data is unknown?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this constant? why is it used? why is it not everything handled inside the worker?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is part of the short-lived Request-Response Messaging pattern. It is used to automatically route the request back to the sender. It is private and constant. If I move responseHandlers to pow.worker.ts I would also have to move routeWorkerResponse and sendMessageRequest to pow.worker.ts.


export function routeWorkerResponse(event: MessageEvent): boolean {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's use arrow convention pls

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the return is a boolean? WDYT of using ResultSuccess?

//const { type } = event.data;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this comment?

const { type, requestId } = event.data;

// Exit immediately if 'type' is missing or not equal to 'response'
if (!type || type !== 'response') {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nonNullish(type) || ...

console.error("Invalid message type. Expected 'response', but got:", event.data);
return false;
}

console.warn('Valid data received: ', event.data);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's avoid printing warning to console

is it necessary?


const handler = responseHandlers.get(requestId);
if (handler) {
handler(event.data);
responseHandlers.delete(requestId);
return true;
}
console.warn('No handler found for event', requestId);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's avoid warnings

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we usually troubleshoot? Is there I dedicated debug logger that can be activated if needed? Should I remove all warnings?

return false;
}

/**
* Sends a typed request to the worker and awaits the fully typed response envelope (T).
*/
export function sendMessageRequest<T>({
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<T extends ... >

worker,
msg,
data,
schema
}: {
worker: Worker;
msg: string;
data: object;
schema: z.ZodType<T>;
}): Promise<T> {
const requestId = crypto.randomUUID();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this entire function should be inside the worker. How the other workers handle this?


// Explicitly type the payload as PostMessageRequestBase
const payload: PostMessageRequestBase = {
msg,
requestId,
type: 'request',
data
};

return new Promise((resolve, reject) => {
responseHandlers.set(requestId, (rawResponse: unknown) => {
const parsed = schema.safeParse(rawResponse as PostMessageRequestBase);
if (!parsed.success) {
console.error(
`Invalid response for message '${msg}' (Request ID: ${requestId}):`,
parsed.error.format()
);
reject(parsed.error);
return;
}
resolve(parsed.data);
});

worker.postMessage(payload); // Send the typed payload
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am not really sure this pertains here... usually the workers that we have handle the messaging in code. For example, have a look at class Scheduler

});
}
24 changes: 24 additions & 0 deletions src/frontend/src/tests/lib/utils/crypto.utils.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import { bufferToHex, hashToHex } from '$lib/utils/crypto.utils';
import { describe, expect, it } from 'vitest';

describe('crypto.utils', () => {
describe('bufferToHex', () => {
it('should convert an ArrayBuffer to a hexadecimal string', () => {
const buffer = new Uint8Array([1, 255, 16, 32]).buffer;
const hex = bufferToHex(buffer);

expect(hex).toBe('01ff1020');
});
});

describe('hashToHex', () => {
it('should hash a string and return its hex representation', async () => {
const input = 'hash-to-hex-test';
const hex = await hashToHex(input);

expect(typeof hex).toBe('string');

expect(hex.length).toBe(64);
});
});
});
Loading