Support external JSON references

[czechboy0]

Support external JSON references to allow breaking up a large doc into smaller ones.

"External" can mean either:

1. another file on local disk
2. a remote file

To start, let's consider other local files first, as remote files might have other implications (security, privacy, etc).

[Fab1n]

Without this feature this tool is extremely limited. Most real world projects make use of external references.
Minimum requirement is local references, remotes would be a nice addon.

[czechboy0]

To help inform how this should work (with the considerations of sandboxing, security, usability, and so on), can you provide links to some real-world OpenAPI documents that are split across multiple local files? To see which parts are commonly outlined, how duplicates are handled, and so on.

And do you have some opinions on how you'd like it to work?

Thoughts, from anyone, appreciated below, as we first gather requirements, before someone proposes a plan.

[andrewse02]

Glad to see this issue addressed. Here are some of my thoughts below.

Examples

Unfortunately, I don't have real-world schemas to share, as I cannot share my company's, and I don't have any from personal projects. Here is one example I've come up with that illustrates why this feature, or lack thereof, is so important

With external refs

index.yaml

openapi: 3.0.0
info:
  title: Example API
  version: 1.0.0
paths:
  /users:
    get:
      summary: Get a list of users
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: './schemas/UserList.yaml'
  /users/{id}:
    get:
      summary: Get a user by ID
      parameters:
        - $ref: './parameters/UserId.yaml'
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: './schemas/User.yaml'

schemas/UserList.yaml

type: array
items:
  $ref: './User.yaml'

schemas/User.yaml

type: object
properties:
  id:
    type: string
  name:
    type: string

parameters/UserId.yaml

name: id
in: path
description: User ID
required: true
schema:
  type: string

Without external refs

index.yaml

openapi: 3.0.0
info:
  title: Example API
  version: 1.0.0
paths:
  /users:
    get:
      summary: Get a list of users
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/UserList'
  /users/{id}:
    get:
      summary: Get a user by ID
      parameters:
        - $ref: '#/components/parameters/UserId'
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/User'

components:
  schemas:
    User:
      type: object
      properties:
        id:
          type: string
        name:
          type: string
    UserList:
      type: array
      items:
        $ref: '#/components/schemas/User'
  parameters:
    UserId:
      name: id
      in: path
      description: User ID
      required: true
      schema:
        type: string

Results

While this may be a reasonably light example, we still have 350 fewer lines when using external refs in index.yaml. Here are some reasons why writing specs with external refs is practically a necessity.

Modularity and Reusability

External references allow you to define schemas and parameters in separate files that can be reused across multiple API endpoints or even different API specifications. This promotes modularity, reduces duplication, and improves maintainability.

Simplified Main File

By using external references, the main OpenAPI specification file remains concise and focused on defining the high-level structure and behavior of your API. The actual schema and parameter definitions reside in separate files, making the main file easier to read, understand, and update. You would know exactly where to look for everything. Eventually, you would have to call it "The Monolith", speaking from experience, that's really bad.

Easier Collaboration

External references facilitate collaboration among team members working on different aspects of the API. With separate schema and parameter files, different team members can work on those specific components independently without stepping on each other's toes. This leads to smoother collaboration and reduces merge conflicts.

Enhanced Reusability Across APIs

When you have a library of reusable schemas and parameters defined as external files, you can leverage them across multiple API projects. This promotes consistency in data models, validation rules, and API contract definitions, saving development time and effort.

How should this work?

And do you have some opinions on how you'd like it to work?

@czechboy0 I will be honest, I'm not quite sure from what perspective I should answer your question.

User

As a general developer of an API that wants to use this tool in my apps, I wouldn't even want to think about how to use it. I would expect to point to external files and for it to just work. It should be able to be completely seamless. If I had blindly copied a random spec into my project, I wouldn't want to be able to tell which models are external refs or not.

Project contributor

Speaking in terms of this project, I've looked into it on a surface level. From what I can tell (correct me if I'm wrong), we parse the openapi.yaml file, convert its contents to Data, and translate that data.
My idea would be to check for external references and replace "$ref" with the contents of the file it's pointing to recursively. Probably easier said than done, I haven't looked that far into it yet.
I did also look into TypeAssigner, and noticed that we use OpenAPIKit to check if the reference is internal or external. It's pretty hardcoded to avoid external references. It seems like that's not an OpenAPIKit problem and something we just simply don't support yet.

Conclusion

Overall, this is a much-needed feature. I'd be more than happy to help. I'd love to hear anyone's thoughts on this. If I could get some guidance just as to where to look to get started I'd feel confident in making a PR. Also please let me know if I need to reformat this as a proposal, or create one separately. Thanks!

[czechboy0]

Thank you @andrewse02 for the details, this is great.

A few questions that come to mind that we'll need to think about:

• what's the namespacing behavior of the generated types? Do we generate a namespace per external file, or do we transparently inline everything before passing it to the generator?
• if you use User.yaml from two APIs, and generate both into neighboring targets, would you expect that the generated User struct is only generated in one place, or two? That means, would you be able to pass the same User struct from one generated API to the other?

Let's figure out more of this behavior first, and then we'll see if a proposal would be useful to let others chime in on the design.

[andrewse02]

Great questions, @czechboy0!

• I wouldn't expect to generate separate namespaces for each externally referenced file. I think it makes more sense to essentially compile every reference, and only after that do we generate.
• But to your second question, what happens when we have identical schemas? My first thought would be that if we have shared objects, they would only be generated once, in some sort of "Common" or "Shared" collection. It also would imply that the user would know that that's going on. So maybe having that not be the default behavior, and some sort of a config value instead.

I'm not quite sure how feasible any of this is, these are just my thoughts from a user's perspective.

[czechboy0]

All of these are feasible, as far as I can tell, so it's really about us choosing how this should work.

I'd like to avoid configuration options if at all possible, and choose a solution that is deterministic and doesn't lose data, so that folks can build conveniences on top, in line with the general design principle of the project.

If later it genuinely seems necessary to support multiple common variants, we can consider a configuration option then, but for the first version, we should be able to do without it.

At this point it might make sense to draft a proposal and discuss the details in a PR. It might also help to try to do a prototype implementation and see what other questions come up.

But don't feel obligated, the information gathered above is already very helpful to whomever ends up implementing it, so all of this is part of the process - and is much appreciated 🙂

[andrewse02]

I think those are great points. I agree, it would be better to start somewhere and build on top of it later. Maybe introducing so much complexity all at once isn't a good idea. Thanks for that!

I'll start to dive deeper into how we can implement this. I would actually like to create a proposal, so I'll get started on preparing that.

[andrewse02]

I apologize, it looks like I was incorrect earlier. OpenAPIKit actually does not support external references. See the below function from OpenAPIKit's Components+JSONReference. How should we proceed from here?

    /// - Important: Looking up an external reference (i.e. one that points to another file)
    ///     is not currently supported by OpenAPIKit and will therefore always throw an error.
    ///
    /// - Throws: `ReferenceError.cannotLookupRemoteReference` or
    ///     `MissingReferenceError.referenceMissingOnLookup(name:)` depending
    ///     on whether the reference points to another file or just points to a component in
    ///     the same file that cannot be found in the Components Object.
    public func lookup<ReferenceType: ComponentDictionaryLocatable>(_ reference: JSONReference<ReferenceType>) throws -> ReferenceType {
        guard case .internal = reference else {
            throw ReferenceError.cannotLookupRemoteReference
        }
        guard let value = self[reference] else {
            throw ReferenceError.missingOnLookup(name: reference.name ?? "unnamed", key: ReferenceType.openAPIComponentsKey)
        }
        return value
    }

[czechboy0]

Let's summon @mattpolzin for thoughts here 🙂

[mattpolzin]

This topic is definitely layered. Some of that was peeled apart above, so I'll do my best to strike a balance between being clear but not repeating things already said.

First, I'll definitely agree that external references are incredibly useful. Where I work we maintain an OpenAPI document across more than a hundred YAML files easily.

Second, I'll point out a dereferencing consideration I saw touched on above: how do you avoid generating distinct types instead of one shared type where a reference would imply the latter behavior. I point this out because if I didn't miss anything the conversation above didn't quite mention something I think is important. Setting aside heuristics this project may use or configurations this project may support to determine when two parts of a document should become the same generated code, the OpenAPI document creator has one really useful tool for specifying this themselves as well: JSON references to the components object. In my eyes, a reference to a shared schema within the components object is at least potentially a clear indication of intent that the schema should remain shared conceptually if a tool generates API resources from it. I'll revisit this later because it plays into pros and cons of strategies I've used to work with OpenAPI documents.

By in large, I've been able to work with OpenAPI Documents by using a tool chain where the OpenAPI document is fully dereferenced early on. The advantage to this is later tools having to deal with less complexity and this has allowed me (and I assume others as well) to use OpenAPIKit very effectively without needing it to follow external references because those references can be inlined by a tool earlier in the tool chain. I won't mention the tools here because there is a plethora of them out there in various ecosystems, many of which are linked to by the OpenAPI website's list of implementations. This may not be as ideal as a one stop shop, but it does give the person using the tooling quite a bit of control, albeit with added effort. Still, I am not opposed to OpenAPIKit supporting external references, it's just quite a step up in complexity so I always figured I would wait for it to be properly motivated. With that, I'll mention one big drawback to dereferencing a document prior to passing it into OpenAPIKit based tooling: if you fully dereference then you lose the component references, whether they were local or not. This defeats attempts by the OpenAPI Document author to define shared schema's explicitly as mentioned above.

Let's say you could specifically combine an OpenAPI Document spread across multiple files but leave local references (e.g. to the components object) in place. Would it be especially more useful to do this within OpenAPIKit as opposed to prior to the OpenAPI Document being processed by OpenAPIKit? Of course it is useful not to need to add prior steps to a toolchain, but does a pipeline approach otherwise limit OpenAPIKit's information about the document (or tools like this code generator)? I've not had the need to look, but I wonder if a tool out there already does this selective inlining of references as a builtin option.

Those are my musings on behavior. My thoughts on implementation start with the fact that currently OpenAPIKit is able to make itself agnostic of the parsing details, letting the project using it decide between e.g. YAML and JSON or which decoder or encoder to use regardless of the file format. That makes it difficult for me to imagine OpenAPIKit using a simple "inline" strategy because OpenAPIKit doesn't work with the JSON or YAML at all so it cannot inline things without worrying about the semantic meaning of those things beyond the structure of the text. That abstraction over parsing details might be worth maintaining if this is brought into OpenAPIKit which makes the API a certain way (probably essentially supported by a function supplied by the project using OpenAPIKit to read new files and provide parsed data from them). I suspect a new decodable type is needed somehow because these individual files are not themselves OpenAPI Documents but rather "fragments" of documents that can only be validated as OpenAPI once combined into a single document. This is where I kind of get stuck in my head and end up walking back my statement that OpenAPIKit should handle this with a new type because it is by far simpler to have never left the JSON or YAML semantics since you really don't want to do anything but treat the individual files as arbitrary parts of the whole Document. [Future self, came back and added a bit of clarity] I may have already realized I was conflating two unrelated things in my head and each individual file possibly should be passable to some existing OpenAPIKit type so perhaps a handler approach is tenable.

Maybe I am overthinking this, though. I'll give it some space and mull on it more.

Here's what I'd want if I could snap my fingers and have it: I'd want to convert external references into internal references to the components object, therefore getting a single OpenAPI Document but retaining an explicit representation of the intention that some part of the document is shared. The components object doesn't support all parts of a document that can be referenced (I think that actually isn't true anymore in 3.1.x but I haven't checked that thoroughly) so you'd still get a mix of things thrown into the components object and things inlined entirely.

Maybe I'm starting to imagine a reasonable way to do this inside OpenAPIKit, though at the same time the approach I am imagining is not easier to do inside OpenAPIKit than outside of it, so it's more a matter of convenience that it be built within. Nothing wrong with that, it's a good argument for doing it inside OpenAPIKit and letting everyone benefit from the implementation.

I'd like to let the conversation here as to what ideally happens to the generated code play out a bit more before solidifying opinions on how OpenAPIKit might help because I don't want to see OpenAPIKit's approach accidentally throw away useful information found in the way that files are used. One example would be the file name of a referenced file -- is that important information for a dereferencing step to hold onto and pass to a code generator?

[czechboy0]

Thank you @mattpolzin for the thorough explanation – it really resonates with me.

this has allowed me (and I assume others as well) to use OpenAPIKit very effectively without needing it to follow external references because those references can be inlined by a tool earlier in the tool chain

We took the same approach of encouraging composability, explicitly documented here, to allow more flexibility for adopters, and to integrate, rather than replace, other OpenAPI tools (even written in other languages). So any features that can be achieved by modifying the OpenAPI document before passing it to the generator is considered out-of-scope (by default, with us being open to reasonable exceptions) – that's why there are very few configuration options, as the generator aims to maintain the fidelity of the OpenAPI document, without losing information, and allowing adopters to build convenience layers on top.

So if we are to support external references in Swift OpenAPI Generator (presumably through OpenAPIKit) will be influenced by whether we can do so without losing fidelity and throwing away potentially important information for the generated code.

I'll mention one big drawback to dereferencing a document prior to passing it into OpenAPIKit based tooling: if you fully dereference then you lose the component references, whether they were local or not.

💯, staying faithful to the source OpenAPI document is one of the design principles of Swift OpenAPI Generator, which is why it doesn't dereference anything, and instead respects reusable versus inlined types as accurately as possible. Supporting external references shouldn't change that, so any support for external references should, at most, make the OpenAPI document appear to have "copied" all the external references into #/components/..., while preserving the relationships between types. But even that might be throwing away the file boundary, which might be unacceptable. That's why I proposed an approach like this, that would preserve the information about which file a type came from:

/// Gets generated today
enum Operations {
  // ...
}
enum Components {
  enum Schemas {
    // ...
  }
  // Parameters, Headers, Responses, ...
}

/// Could be generated when external references are found.
/// First example shows a schema whose contents are the file my-external-object.yaml itself.
/// Second example shows referencing a schema from another OpenAPI document.
enum External {
  enum Components {
    enum Schemas {
      // - Remark: Generated from ./my-external-object.yaml
      struct MyExternalObject: ... {
        // ...
      }
    }
  }
  
  // - Remark: Generated from ./shared-api.yaml
  struct Files {
    enum shared_api_yaml {
      enum Components {
        enum Schemas {
          // - Remark: Generated from ./shared-api.yaml#/components/schemas/SharedError
          struct SharedError: ... {
            // ...
          }
        }
      }
    }
  }
}

perhaps a handler approach is tenable

I think so, I imagine the lowest level of resolving external references could be done as a (String) async throws -> Something function, which then could have an implementation for resolving using the local file system, but potentially even from memory in tests (and from the network in the future). I imagine the handler would continue to be agnostic to which JSON/YAML is used, so it'd both get the bytes and parse them into a type-safe OpenAPIKit object.

I don't want to see OpenAPIKit's approach accidentally throw away useful information found in the way that files are used

👍

One example would be the file name of a referenced file -- is that important information for a dereferencing step to hold onto and pass to a code generator?

I think so, as the generator would likely use the file name to inform the Swift type name. So for the generator, more than dereferencing (which isn't used today at all), we're interested in resolving (or, lookup, in OpenAPIKit's terminology) external references, to maintain fidelity.