Skip to content

Ink Scripted Importer #205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open

Ink Scripted Importer #205

wants to merge 24 commits into from

Conversation

SHiLLySiT
Copy link

This aims to implement #53, which IMO, has some huge benefits:

  1. There is no longer separate source .INK files and compiled .JSON files that can get out of sync. Which means designers don't need to remember commit both .INK and JSON files, which can happen if they're testing/writing outside Unity with Inky.
  2. No need to deal with managing a ink file compilation queue, and fighting with the Editor. Ink files are compiled when imported, either when added or changed.
  3. You can expose InkFile references instead of generic TextAsset references in game code.
  4. This codebase becomes significantly less complex, and less prone to weird errors that have to deal with compiling assets outside of Unity's normal import process.

This PR is currently in a proof of concept stage: it hasn't been fully tested, but loading and running ink files at runtime and in the Editor is working.

The original implementation was written before Unity supported custom scripted importers, so many of the systems are now obsolete. Since this is a pretty big change, I wanted to open a PR to get an initial review to get some initial thoughts from the maintainers.

Interested in all feedback, but the question at the top of my mind: Do we need to expose includes and master files in the editor? While Ink at a language level still has these concepts, I'm not sure how important it is to expose these in Unity if its no longer important for compilation?

@SHiLLySiT SHiLLySiT marked this pull request as draft September 2, 2024 02:12
@tomkail
Copy link
Collaborator

tomkail commented Sep 2, 2024

Oh wow thanks for looking into this! I'll take a look this week if I find time. At a glance it looks lovely (so much less code!!)
I've pinged the Unity dev community on Discord (https://discord.gg/inkle) to take a look too, since this is a big change and I'd like as many devs as possible to confirm that it doesn't affect them negatively in any way.

@SHiLLySiT
Copy link
Author

Totally! I see your message in #ink-unity-integration-dev. I'll keep watch for questions/comments there too. Thanks!

@SHiLLySiT SHiLLySiT marked this pull request as ready for review October 6, 2024 23:45
@SHiLLySiT
Copy link
Author

@tomkail Okay I'm moving this out of draft as I've cleaned up the known things I needed to do, but I suspect there may be things I've missed since this was such a huge change. Feel free to tag me on any changes you want after doing a review. I'll loop back when I have the time 🤘

@tomkail
Copy link
Collaborator

tomkail commented Oct 7, 2024 via email

@SHiLLySiT
Copy link
Author

SHiLLySiT commented Oct 8, 2024

Sounds good to me! I'd rather give more people more time to test it out before its merged into main 😅

I don't appear to have the option to change the base branch to a new (non-existing) branch tho. If you make a beta branch I can update the PR!

@SHiLLySiT
Copy link
Author

I forgot to update the docs, so I snuck another commit in 👍

Comment on lines -114 to +115
DefaultAsset asset = AssetDatabase.LoadAssetAtPath<DefaultAsset>(path);
DrawInkFile(InkLibrary.GetInkFileWithFile(asset), rect);
InkFile asset = AssetDatabase.LoadAssetAtPath<InkFile>(path);
DrawInkFile(asset, rect);
Copy link

@ErnSur ErnSur Nov 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

InkFile is now a Scriptable Object.
No OnDrawProjectWindowItem would be needed if we assign a custom icon to InkFile.cs from the inspector




public class InkFile : ScriptableObject {
Copy link

@ErnSur ErnSur Nov 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with the way the project is maintained but this is a breaking change so it would be nice to bump the major version in the package manifest.

{
  "name": "com.inkle.ink-unity-integration",
  "version": "2.0.0",
  ...
}

Copy link

@ErnSur ErnSur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some general comments, moving to scripted importers is a natural step forward for the library but it looks like it will require breaking changes.
This could be a chance for some cleanups and other breaking changes around the codebase.
Let me know if you want more feedback, I'm interested in closer collaboration.

/// Automatically compiles .ink assets each time they are imported, and creates an InkFile asset.
/// </summary>
[ScriptedImporter(1, "ink")]
public class InkImporter : ScriptedImporter
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing namespace: Ink.UnityIntegration

Comment on lines +19 to +50
var compiler = new Compiler(inputString, new Compiler.Options
{
countAllVisits = true,
fileHandler = new UnityInkFileHandler(Path.GetDirectoryName(absolutePath)),
errorHandler = (string message, ErrorType type) => {
InkCompilerLog log;
if(InkCompilerLog.TryParse(message, out log)) {
if(string.IsNullOrEmpty(log.relativeFilePath)) log.relativeFilePath = Path.GetFileName(absolutePath);
switch (log.type)
{
case ErrorType.Error:
inkFile.errors.Add(log);
Debug.LogError("Ink "+log.type+" for "+Path.GetFileName(absolutePath)+": "+log.content + " (at "+log.relativeFilePath+":"+log.lineNumber+")", inkFile);
break;
case ErrorType.Warning:
inkFile.warnings.Add(log);
Debug.LogWarning("Ink "+log.type+" for "+Path.GetFileName(absolutePath)+": "+log.content + " (at "+log.relativeFilePath+" "+log.lineNumber+")", inkFile);
break;
case ErrorType.Author:
inkFile.todos.Add(log);
if (InkSettings.instance.printInkLogsInConsoleOnCompile)
{
Debug.Log("Ink Log for "+Path.GetFileName(absolutePath)+": "+log.content + " (at "+log.relativeFilePath+" "+log.lineNumber+")", inkFile);
}
break;
}

} else {
Debug.LogWarning("Couldn't parse log "+message);
}
}
});
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be extracted to a new method for readability, same with errorHandler inside it

Suggested change
var compiler = new Compiler(inputString, new Compiler.Options
{
countAllVisits = true,
fileHandler = new UnityInkFileHandler(Path.GetDirectoryName(absolutePath)),
errorHandler = (string message, ErrorType type) => {
InkCompilerLog log;
if(InkCompilerLog.TryParse(message, out log)) {
if(string.IsNullOrEmpty(log.relativeFilePath)) log.relativeFilePath = Path.GetFileName(absolutePath);
switch (log.type)
{
case ErrorType.Error:
inkFile.errors.Add(log);
Debug.LogError("Ink "+log.type+" for "+Path.GetFileName(absolutePath)+": "+log.content + " (at "+log.relativeFilePath+":"+log.lineNumber+")", inkFile);
break;
case ErrorType.Warning:
inkFile.warnings.Add(log);
Debug.LogWarning("Ink "+log.type+" for "+Path.GetFileName(absolutePath)+": "+log.content + " (at "+log.relativeFilePath+" "+log.lineNumber+")", inkFile);
break;
case ErrorType.Author:
inkFile.todos.Add(log);
if (InkSettings.instance.printInkLogsInConsoleOnCompile)
{
Debug.Log("Ink Log for "+Path.GetFileName(absolutePath)+": "+log.content + " (at "+log.relativeFilePath+" "+log.lineNumber+")", inkFile);
}
break;
}
} else {
Debug.LogWarning("Couldn't parse log "+message);
}
}
});
var compiler = CreateCompiler(inputString, absolutePath, inkFile);

Comment on lines +31 to +47
Debug.LogError("Ink "+log.type+" for "+Path.GetFileName(absolutePath)+": "+log.content + " (at "+log.relativeFilePath+":"+log.lineNumber+")", inkFile);
break;
case ErrorType.Warning:
inkFile.warnings.Add(log);
Debug.LogWarning("Ink "+log.type+" for "+Path.GetFileName(absolutePath)+": "+log.content + " (at "+log.relativeFilePath+" "+log.lineNumber+")", inkFile);
break;
case ErrorType.Author:
inkFile.todos.Add(log);
if (InkSettings.instance.printInkLogsInConsoleOnCompile)
{
Debug.Log("Ink Log for "+Path.GetFileName(absolutePath)+": "+log.content + " (at "+log.relativeFilePath+" "+log.lineNumber+")", inkFile);
}
break;
}

} else {
Debug.LogWarning("Couldn't parse log "+message);
Copy link

@ErnSur ErnSur Nov 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AssetImportContext.LogImportError() could be used for handling all of the importer errors.
Unity handles UI for displaying errors logged this way in the importer header. This in turn could eliminate the need for custom error UI in the Ink File Editor

@@ -68,17 +68,6 @@ public string templateFilePath {
else return AssetDatabase.GetAssetPath(templateFile);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the templateFile should probably change type to the ScriptableObject imported by the InkImporter

Comment on lines 16 to 17
public static System.Version inkVersionCurrent = new System.Version(1,2,0);
public static System.Version unityIntegrationVersionCurrent = new System.Version(1,2,1);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be best to put it somewhere else

// Helper class for ink files that maintains INCLUDE connections between ink files
/// <summary>
/// ScriptableObject that serves as the imported version of .ink files.
/// </summary>
[Serializable]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Serializable is redundant on a scriptable object

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the static nested class InkIncludeParser is still inside the type but its not used (could be removed).

[ScriptedImporter(1, "ink")]
public class InkImporter : ScriptedImporter
{
public override void OnImportAsset(AssetImportContext ctx)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If compiling output of one ink file depends on another ink file then the AssetImportContext.DependsOnSourceAsset should be used for proper importer behavior.

/// Automatically compiles .ink assets each time they are imported, and creates an InkFile asset.
/// </summary>
[ScriptedImporter(1, "ink")]
public class InkImporter : ScriptedImporter
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A complimentary InkImporterEditor : ScriptedImporterEditor could be created for better UX.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest writing new UI code with UI Toolkit.
it's impossible given that the package manifest says unity 2018 is still supported.
That's a separate topic, but maybe a poll on discord could be useful to see if users really need a support for very old editor versions.
IMO bumping minimal version to 2021.3+ would be a sensible decision.

@SHiLLySiT
Copy link
Author

@ErnSur

I left some general comments, moving to scripted importers is a natural step forward for the library but it looks like it will require breaking changes.

Hey, thank you for the review! I just wanted to drop a note here that I probably won't have time to review and make changes until around Feb 2025.

This could be a chance for some cleanups and other breaking changes around the codebase. Let me know if you want more feedback, I'm interested in closer collaboration.

While I'm in favor of cleaning up the code, I'm not a maintainer and this would probably be a better conversation to have with @tomkail. But I'll certainly circle back for a re-review once I've had a chance to process your feedback.

Cheers!

@tomkail
Copy link
Collaborator

tomkail commented Nov 25, 2024 via email

@SHiLLySiT
Copy link
Author

any big breaking changes should be tested on large scale projects before merging to the main branch (which is really the only reason this hasn’t been merged in, it’s fantastic stuff)

@tomkail I've been advocating for using Ink in our projects at work, so when the next one spins up, I'll use my fork with these changes to get them tested.

@SHiLLySiT
Copy link
Author

@tomkail @ErnSur I think I've stumbled on to one of reasons why the import process prior to this PR had been so complex - the Ink language's INCLUDE feature.

The Ink language supports including other Ink files, which allows them to be broken up into smaller chunks. I had overlooked this initially, so to support this I added a bit of code in 0b5980d (which ErnSur had also reccomended in this comment) so that master ink files were aware of their includes. This solved the problem of Ink files used as includes not triggering reimports of their master files.

However the problem I now have is that some Ink files used as includes cannot be compiled on their own outside of the context of the master file that includes them. For example, when a master file defines global variables, any include files that use the global variables fail to independently compile. This is problem for the changes introduced in this PR because the custom importer will flag such files as failed to compile, when they really should just be ignored.

And this is where I'm stuck.

I'm not quite sure if Unity has a way to express this kind of dependency with the ScriptedImporter API. While its possible to know if a master file has includes (we can parse it to look for INCLUDE keywords) there's no way to know if an Ink file is an include file. Do either of you (maybe more directed at @ErnSur) have a suggestion on how to handle this?

I think I'd like to avoid a system similar to the old InkLibrary but I feel like we may not be able to get around generating some sort of intermediary file to allow for a two-pass import?

@SHiLLySiT
Copy link
Author

Ok here's are two proposals, both centered around not trying to auto identify if files are used as includes:

  1. We allow Ink files to fail to compile. We also switch logging of compilation errors to warnings instead of errors. I think seeing red error messages in Unity's console indicate something you must fix for the game to work properly. In this case, an Ink script that fails to compile is intended behavior. We'd still keep the error icon on Ink assets so developers can easily identify which ones have errors.
  2. Use a ScriptedImporterEditor to introduce a SkipCompilation flag. This would be similar to the "Should also be Master File" flag, except in reverse. All files would be treated as a master file and compiled, but specific files can be opted out with SkipCompilation enabled.

@ErnSur
Copy link

ErnSur commented Mar 25, 2025

However the problem I now have is that some Ink files used as includes cannot be compiled on their own outside of the context of the master file that includes them.

the custom importer will flag such files as failed to compile, when they really should just be ignored.

there's no way to know if an Ink file is an include file

Some API context:

  • Any ScriptedImporter can declare serializable fields that can be modified in the inspector and act as importer settings for the file.
  • ScriptedImporter can create more than one asset from a file. One asset needs to be main asset and other the rest of them are so-called sub-assets. In editor it appears as scriptable object with a foldout that hides other scriptable objects.

So my idea for a solution to this issue would be to import ink variables as additional scriptable object (sub-asset). Then InkImporter could have a public InkScriptVariables[] ScriptVariables field which can be optionally populated.

  • When InkImporter is importing a script it will try to "compile" in the context of ScriptVariables variables if any are referenced.
  • When InkImporter is importing a script that has include files it will try to find them in the project and add its variables asset reference to their ScriptVariables collection.

I don't know how feasible is the second point, but I'd try this approach. I think it should be possible to load another InkImporter from OnImport callback.

Use a ScriptedImporterEditor to introduce a SkipCompilation flag. This would be similar to the "Should also be Master File" flag, except in reverse. All files would be treated as a master file and compiled, but specific files can be opted out with SkipCompilation enabled.

As I mentioned before, you don't really have to introduce any custom editors to add serializable state to the importer. Just add a serializable filed to importer the same way you would add it to a scriptable object and it'll work.

@SHiLLySiT
Copy link
Author

Thanks for your input!

Any ScriptedImporter can declare serializable fields that can be modified in the inspector and act as importer settings for the file.
As I mentioned before, you don't really have to introduce any custom editors to add serializable state to the importer. Just add a serializable filed to importer the same way you would add it to a scriptable object and it'll work.

Oops! I hadn't realized I could just defined them in the importer, thanks for calling this out 👍

So my idea for a solution to this issue would be to import ink variables as additional scriptable object (sub-asset). Then InkImporter could have a public InkScriptVariables[] ScriptVariables field which can be optionally populated.

  • When InkImporter is importing a script it will try to "compile" in the context of ScriptVariables variables if any are referenced.
  • When InkImporter is importing a script that has include files it will try to find them in the project and add its variables asset reference to their ScriptVariables collection.

I don't know how feasible is the second point, but I'd try this approach. I think it should be possible to load another InkImporter from OnImport callback.

Ohh interesting. I'm not sure how the separate ink variables will solve the core problem? As your second bullet is where the issue is, specifically determining a file is an include before the file that actually includes it is imported.

@ErnSur
Copy link

ErnSur commented Apr 1, 2025

Seems I didn't explain the idea that well...

specifically determining a file is an include before the file that actually includes it is imported.

importer wouldn't have to successfully import ink file if its missing variable declarations.

Let's say we have a project with following files:

  • File A - main file
    • Contains a reference (include) to File B
    • Defines global variable X
  • File B - companion file
    • Requires variable X definition

Let's say we import the project and assets are imported in unfavorable order for us where File B is being being imported before file A:

  • File B import: fails to import the asset since it cannot find variable X definition
  • File A import:
    • Generates the InkVariables object that contains the variable X definition
    • Goes over all ink files included in the main file and re-imports them with the newly created InkVariables object. With that they should be able to be imported correctly.

Importer design issue

Even if the above solution fixes the issue I would actually change the way importer works.
Ink Importer should not fail to import the file if its missing variable definitions.

We're trying to import File B but its missing variable X? No problem, it should report importer warning but still succeed at file import. The result will have "{variable name}" in places where the variable should be.
The moment you populate the InkVariables array with a reference to the InkVariables asset (the idea I described here) the file will reimport.

As a source of inspiration I recommend looking at Unity UXML and USS files, how they're imported and how they deal with references between each other.

@SHiLLySiT
Copy link
Author

Ahh thanks for the explanation!

I'm still not sure what the additional InkVariable objects get us, but I think the second half of your last post hits on what I've been trying to describe:

Importer design issue
Even if the above solution fixes the issue I would actually change the way importer works.
Ink Importer should not fail to import the file if its missing variable definitions.

We're trying to import File B but its missing variable X? No problem, it should report importer warning but still succeed at file import. The result will have "{variable name}" in places where the variable should be.
The moment you populate the InkVariables array with a reference to the InkVariables asset (the idea I described #205 (comment)) the file will reimport.

As a source of inspiration I recommend looking at Unity UXML and USS files, how they're imported and how they deal with references between each other.

If I'm understanding correctly, I think we're on the same page here. Specifically, if an Ink file fails to compile (and by compile I mean compile via the Ink compiler) it should still be successfully imported into Unity.

However, what I'm still unsure about is the developer experience for this. Going with your example, if File B is imported first and logs a warning regarding the compilation failure to the Unity console, how should we follow up once File A is imported? Additionally, importing File A still doesn't fix the compilation of File B because File B is not intended to be compiled as a main file.

I'm starting to feel more strongly for proposal #2 from my earlier comment as I think this is likely the same issue the original developers faced and why the master file toggle was added.

@tomkail Do you have any historical context you could share?

…s that aren't intended to be compiled standalone
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants