[GSoC 2025] Add framework for adding new drivers at runtime, for instance for cloud

[afbjorklund]

The docker-machine program had a plugin system, so that you could provide new drivers besides the built-in ones.

This would be a good option to provide drivers for various proprietary hypervisors and cloud vendors, for instance.

They typically come with a big SDK, that might require certain hardware and large downloads when building.

So it would be nice if those were kept out of the Lima code base, but is still a nice addition to enable them...

---

docker-machine drivers:

https://github.com/machine-drivers/docker.github.io/blob/173d3c65f8e7df2a8c0323594419c18086fc3a30/machine/AVAILABLE_DRIVER_PLUGINS.md

https://github.com/machine-drivers/docker.github.io/blob/173d3c65f8e7df2a8c0323594419c18086fc3a30/machine/DRIVER_SPEC.md

Example of Lima drivers:

• VirtualBox driver, for legacy compatibility and portability #1287 (PR Support for VBox driver #1277)

• Libvirt driver, for hypervisors on remote machines  #2031 (PR Support for Virt driver #2054)

• vfkit driver (TBD)

• cloud drivers (TBD)

---

EDIT: Some general updates, after discussion below and feedback:

• The example drivers were just examples, and it might not be vfkit as a "new" driver and which cloud was not specified.
  Note that the actual driver implementation is not part of the summer project, which is about the RPC plugin framework.
• The main code should not call directly into the driver code. This is supposed to be cleaned up, before GSoC project starts.
  Note that this is more of a general rule for drivers (only SDK), but fixing other lima code is not in the project GSoC itself.
  • Decouple pkg/start (and others) from VM driver internals #1746
• Driver authors and implementations should not have to know about the RPC implementation, but only the Go interface.
• Drivers can be compiled in to the main lima applications or built as standalone executable, from the same driver code*.
• Changing which drivers are included in the main download, and which one is default for each os/arch, doesn't change.
• It should be possible to add new drivers after release, as long as they use the same SDK version (i.e. the interface code)

Other details can be discussed on Slack or in other GitHub issues.

---

* Here is another example, from a related project (containerd):
https://github.com/containerd/fuse-overlayfs-snapshotter/#option-1-embed-fuse-overlayfs-plugin-into-the-containerd-binary
https://github.com/containerd/fuse-overlayfs-snapshotter/#option-2-execute-fuse-overlayfs-plugin-as-a-separate-binary

Two installation options are supported:

1. Embed fuse-overlayfs plugin into the containerd binary
2. Execute fuse-overlayfs plugin as a separate binary

Choose 1 if you don't mind recompiling containerd, otherwise choose 2.

This is the same setup, as in docker-machine (mentioned above)

[jandubois]

Having pluggable drivers might be useful, but they should be maintained in separate repos (like socket_vmnet).

Random memory: I think docker-machine-driver-hyperkit must be run with as root and has several easily exploitable vulnerabilities (e.g. because it spawns subprocesses that also run as root). So this might be something to look out for as you implement other drivers.

[afbjorklund]

I was thinking to only add the API, but keep all (both?) supported drivers in-tree

[afbjorklund]

Also for me I think that qemu is plenty, it was mostly for the cloud...

[jandubois]

but keep all (both?) supported drivers in-tree

Yes, the supported drivers need to stay in, off course, but things like adding a hyperkit driver, or cloud drivers, should be external. I don't actually see that anyone would actively maintain them, if the history of docker-machine drivers is any indication.

[afbjorklund]

What I meant was that I would not write another xhyve/hyperkit driver (especially not with vz already being there), I think we do need a hyperv driver but that can be done by exec to some PowerShell scripts rather than linking with a big SDK and thus it can stay in.

[afbjorklund]

I will do some kind of proof-of-concept interface next month, with one private cloud and one public cloud implementation.

[afbjorklund]

A simplified approach, for drivers that are internal to the program but have a lot of dependencies and bloat the binary, is to use the plugin interface instead of the net/rpc. The two major drawbacks is that it needs to be compiled at the same time as the program, and that it currently does not work on Windows (no .dll support yet, only .so). But it makes it easier to include something like libvirt, that has a lot of dependencies but allows for remote communication (the qemu:///session doesn't really add anything).

• https://pkg.go.dev/plugin

• https://pkg.go.dev/net/rpc

It would not be a solution for random cloud vendor drivers, that would be shipped independent of the limactl program.

Those would need some kind of RPC solution, perhaps wrapped in a framework like https://github.com/hashicorp/go-plugin

[AkihiroSuda]

Can we use REST or gRPC over stdio?

[afbjorklund]

I think that go-plugin can use both rpc and grpc.

[afbjorklund]

Having to define a public interface and a protocol for something like REST or gRPC is overkill for this use case, I think...

That was the benefit of using a "plugin" wrapper, over doing it ourselves. Just doing function or remote calls, like above.

[afbjorklund]

My attempt to load libvirt dynamically failed, due to it calling pkg-config (to find libvirt.pc) already from the go.mod...

The building and loading of the .so worked fine, but it was hard to make the plugin optional without a separate module.

[balajiv113]

Yes, during the initial driver interface as well the idea was to provide support via external plugin mechanism on future.

We should be easily able to expose a external plugin via https://pkg.go.dev/plugin with current code. Will just need driverutil to load the external drivers.

[afbjorklund]

If we use plugin, then the driver is not "external" anymore. It needs to be built at the same time, or it won't load later.

So if we want external plugins, then they will need to use RPC. Also, they were unable to module library dependencies.

[afbjorklund]

When doing the RPC driver, we should also consider how to forward configuration. This needs to be communicated as well.

We should not do it like minikube, when they either compile with the driver config or duplicate the struct without an API.

Note that we might not know (in the lima main program) what configuration that the plugin is added (in the plugin program).

For docker-machine, there was a rather hacky framework based on CLI flags which "worked" but json/yaml would be better...

[afbjorklund]

We also might have to add some new functions to the interface (=API), for instance the ability to upload images.

And how to send credentials to the driver, for instance passing on a token. Monitoring might also need revising.

[saz97]

Hello everyone,

I came across this discussion through the GSoC project description (https://github.com/cncf/mentoring/blob/main/programs/summerofcode/2025.md#vm-plugin-subsystem) and found the idea very interesting. I would love to join the discussion and contribute to this effort.

To get started, I plan to create a small demo that separates the QEMU driver into an independent binary that communicates with the main Lima binary via RPC (probably gRPC).

I appreciate any guidance or suggestions from the maintainers and the community. Looking forward to contributing!

[saz97]

Thanks for the feedback!

I understand that net/rpc is frozen and not recommended for new applications. This is just an initial version of demo, and I will update my code based on your suggestions, including switching to UNIX socket or stdio instead of TCP.

I got the point about --vm-type=rpc not being a valid option. However, I’m a bit confused about the plugin lookup mechanism. Combining your guideline and what I understand, there should be multiple plugin drivers in lima/plugins/vm/, such as lima/plugins/vm/qemu, lima/plugins/vm/vz, etc. The RPC server should act as a bridge, handling client requests and delegating them to the appropriate plugin driver. My mistake was adding a static vmType=rpc, whereas the valid vmType should be dynamically determined based on the available plugin drivers in lima/plugins/vm/. Please let me know if I got this right!

For these code should be decoupled from pkg/start, pkg/limayaml, and other modules. No problem, but i prefer this is a work beyond initial demo, maybe until the develop duration in project. but I try to work on this now because this sounds interesting!

Again, thanks for the guidance! Looking forward to your thoughts.

[AkihiroSuda]

there should be multiple plugin drivers in lima/plugins/vm/, such as lima/plugins/vm/qemu, lima/plugins/vm/vz, etc.

Right.

The vmType should be specified via vmType in the YAML or --vm-type in the CLI.
When the vmType is not specified, Lima should check ok, err := getPlugin("vz").AcceptsTemplate(yaml) and then falls back to the "qemu" plugin.

[saz97]

Hi everyone, I am working on integrating an RPC client into the QEMU driver to address the issue:

Probably there should not be --vm-type=rpc. --vm-type=${PLUGIN} should look up a plugin in ${PREFIX}/libexec/lima/plugins/vm/${PLUGIN} (or maybe just lima-plugin-vm-${PLUGIN} in the ${PATH})

The updated PR link is https://github.com/saz97/lima/pull/1/files#diff-90164061f3572f8e2aca979d03cdd74187cbccfd16c2e9c28d349cbdf454155b. And the logs from both the RPC client and server are shown in the attached figure:

I am currently trying to understand the architecture of the RPC implementation. Specifically, I am unsure which parts of the original driver should be moved to the RPC server. In the original qemu_driver code, the QEMU process is started with the following command:
if err := qCmd.Start(); err != nil { return nil, err }
If I move this logic to the RPC server, then the QEMU process will run there, while some setup tasks—such as downloading the nerdctl archive and pulling the image—will still be handled on the client side.

I would appreciate any guidance on whether my understanding is correct and what parts of the original implementation should be moved to the server. Thank you!

[AkihiroSuda]

while some setup tasks—such as downloading the nerdctl archive and pulling the image—will still be handled on the client side.

Yes, correct, as these parts are not specific to the VM drivers.

[afbjorklund]

As mentioned, we might need a new API method to upload images - but that can be added to the framework afterwards...

The current drivers just assume that they can copy files from the cache to the instance directory, without any extra steps.

[AkihiroSuda]

Cross-posting an announcement here for visibility:

• [GSoC 2025] submit your proposal (VM plugins) by April 8 #3383

---

This year we participate in the Google Summer of Code(GSoC) as a CNCF project, so as to facilitate contributions from open source beginners and students.

Successful contributors will receive stipends.

Register to become a contributor by April 8 (Tue) 18:00 UTC, via the GSoC website.

(...)