code processors, object processors

[moodymudskipper]

The traditional flow is :

graph TD;
    Code-in-scripts --> |source| Objects-in-package

Loading

But we can have:

graph TD;
    Code-in-scripts --> |code-processor| Processed-code --> |source| Unprocessed-Objects --> |object-processor| Objects-in-package

Loading

Funnily enough the code doesn't need to be syntactic, we can easily extend R by forking from https://github.com/REditorSupport/vscode-R/blob/master/syntax/r.json

Maybe the code processor should handle the code selection too (right now, pick all R files, recursive or not, and exclude some prefixes).

And I believe the natural input for a code processor is a list of files + some simple arguments, and the natural input for an object processor is a list of objects + some simple arguments.

With the code processor we can do #10 and everything discussed in https://x.com/jdatap/status/1910275123652763699

With the object processor (and a bit of code processing too) we can do #3 (the file name is in attr(attr(fun, "srcref"), "srcfile")$filename, but we need other annotations)

So this fits really well the YAML format.

We can easily chain processors with YAML too, so we can modularize all the steps.

[moodymudskipper]

4 things to consider

1

I think instead of object processor we might talk about state processor.

The code processor builds state from flat files. Then the state processor builds the state that we actually work with.
For instance i might want to modify my state using library(strict).

Only objects however go to sysdata.rda, but objects don't have to be built only from objects, so "object processor" is still misleading.

2

Code can be processed in place, or can be processed to memory, this can be seen as 2 distinct steps

3

The objects that end up in our package don't have to be the same, so technically we have 2 pathways

4

We should be able to run the tests again both pkg and dev state, but pkg state is mandatory

updated diagram

graph TD;
    Code-in-scripts --> |in-place-code-processor| Updated-code-in-scripts
    Updated-code-in-scripts--> |in-memory-code-processor-for-pkg| Code-in-memory-for-pkg
    Updated-code-in-scripts--> |in-memory-code-processor-for-dev| Code-in-memory-for-dev
    Code-in-memory-for-pkg --> |source| Unprocessed-state-for-pkg
    Code-in-memory-for-pkg --> |source| Other-unprocessed-shared-state 
    Code-in-memory-for-dev --> |source| Other-unprocessed-shared-state 
    Code-in-memory-for-dev --> |source| Unprocessed-state-for-dev
    Other-unprocessed-shared-state --> |state-processor-for-pkg| Objects-in-package
    Unprocessed-state-for-pkg --> |state-processor-for-pkg| Objects-in-package
    Other-unprocessed-shared-state --> |state-processor-for-pkg| Dev-state
    Unprocessed-state-for-dev --> |state-processor-for-dev| Dev-state

Loading

formalization

I'm not sure how this needs to be formalized in the package.

Maybe from the package's point of view it's just a sequence of transformation functions, and that the above only helps for vision and conventions.

We still need some amount of formalization though because we do need to end with a list of objects to

[moodymudskipper]

It's not even "Code" at the top, it's any file, doesn't even have to flat files.

We need better names. And examples for every step.

We don't have to provide steps for both dev and pkg when they are the same, we can have an inheritance system.

We use a temp location to source preprocessed code rather than memory, will work better with existing tools.

• file_processor takes only a fun (no arg, no return value)
  *virtual_file_processor takes a fun (mandatory, no arg, no return value), ascope (optional), and a path (optional). The path is set as a global value to be used by the source step these are the same:

- virtual_file_processor: identity_file_preprocessor 
- virtual_file_processor:
    - scope: both
    - fun: identity_file_preprocessor

• source takes only a scope, sets an env as a global var in the ns
• state_processor takes a fun (1 arg: the env defined by source, and returns this modified env)and a scope (optional)

These can be chained in the chosen order, the order above makes more sense in most scenarios but they work independently so we stay flexible.

A no op but explicit, the yaml would be:

- file_processor: identity_file_preprocessor 
- virtual_file_processor : identity_file_preprocessor
- source: both
- state_processor : identity_state_processor

I don't know if we can afford to put nothing by default, the order would be ambiguous if we provided only state_processor for instance. So maybe better have these steps by default, and mandatory one time each, except for virtual_file_processor skipped by default