DMA API Proposal for TinyGo

[soypat]

Background

TinyGo has long had non-standard DMA APIs for interacting with peripherals on MCUs. One example which has received attention recently is the implementation on the PIO library here:

• Implementation: piolib DMA
• Community asking for public DMA API: Suggestion about DMA interface pio#16
• Community designing amazing things with their own DMA API and completely flabbergasting me with what is possible with TinyGo: go-wspr

Due to these current events I have begun researching DMA APIs by asking my coworkers about DMA (many thanks to @ydnatag) and finding out interesting things about Linux's DMA API which is documented here:

• DMA engine Provider API documentation: https://github.com/torvalds/linux/blob/master/Documentation/driver-api/dmaengine/provider.rst
• DMA engine Client API documentation: https://github.com/torvalds/linux/blob/master/Documentation/driver-api/dmaengine/client.rst
• DMA engine implementation: https://github.com/torvalds/linux/blob/7f2ff7b6261742ed52aa973ccdf99151b7cc3a50/drivers/dma/dmaengine.c
• DMA driver implementation for STM32: https://github.com/torvalds/linux/blob/7f2ff7b6261742ed52aa973ccdf99151b7cc3a50/drivers/dma/stm32/stm32-dma.c

Proposal details

While having a DMA engine like Linux's would be useful I believe it is too early to work our way into something of the sort. I'd argue we can eventually get there (if we really want that) by building a common low level DMA channel API that is shared in part by Linux's DMA Engine. A couple things about the API:

• I believe DMAChannel start-transaction call should be asynchronous. After all we are abstracting the hardware, which is fundamentally and undeniably async. As much as I try to avoid async APIs I believe it is beneficial in this case to do go this way as that is the way the hardware works. Try to sidestep that and you'll eventually run into issues with whatever novel way you chose to represent something that is async as non-async.
• We should have a central structure which holds data on DMA channels in use to facilitate users not stepping on drivers, much like how linux does it with dma_request_channel. This is how piolib also does it and it seems to have worked for our users. Below is an API for a lightweight DMAArbiter implementation much like the one in piolib.
• Scatter-gather operations are very useful but might be too much to concentrate into the same API as a single buffer DMA transaction, which is why I've excluded it from this API for now. I'd imagine they probably might end up having their own DMAScatterGatherConfig type and corresponding function call
• I've chosen not to do DMA transaction in two steps: Configuration+Start. Not married to the current proposal below but I do like the simplicity of a single call which sort of implies consistent behaviour between calls since we are always configuring the transaction.

Proposed DMA Channel API

type DMATxConfig struct {
        // Allow DMA to be chained to another transaction in a different channel
	ChainTo          int
        // Size of data to be moved by transaction in bytes.
	TxSize           int
        // Pointer to start of source of data.
	Src              unsafe.Pointer
        // Pointer to start of the destination of data where data will be moved to.
	Dst              unsafe.Pointer
        // Width of data to be moved from source in bytes. If increment is enabled this will be the value added to the source address on every unit of data moved on the bus.
	SrcAddrWidth     uint8
        // Width of data at destination. If increment is enabled this value is added to the destination address on every unit of data moved on the bus.
	DstAddrWidth     uint8
       // Enable increment of source address. If false the data will be pulled from a single address which is useful for peripheral register interaction.
	SrcAddrIncrement bool
       // Enable increment of destination address. If both Src and Dst are incremented then DMA is effectively a memcopy.
	DstAddrIncrement bool
}

// StartTx begins a DMA transaction over the bus and returns immediately (non-blocking). 
// Use [DMAChannel.IsBusy] to know if the transaction terminated.
func (ch *DMAChannel) StartTx(cfg *DMATxConfig) error {
	return nil
}

// IsBusy returns true if the channel is currently performing a transaction over the bus.
func (ch *DMAChannel) IsBusy() bool {
	return false
}

// Abort terminates the DMA transaction early. 
// It should be called after a transaction finishes to disable the channel to save power.
func (ch *DMAChannel) Abort() {
}

// ChannelIndex returns the identifying index of this DMA channel.
func (ch *DMAChannel) ChannelIndex() int {
	return 0
}

Proposed DMA channel Arbiter API

type DMAArbiter struct {
	used   uint64 // bitmask of used DMA channels.
	nchans int
}

type DMAChannel struct {
	nchan int
}

var mdma = DMAArbiter{}

func (arb *DMAArbiter) RequestChannel() (*DMAChannel, error) {
	for i := 0; i < arb.nchans; i++ {
		bit := uint64(1 << i)
		if arb.used&bit == 0 {
			arb.used |= bit
			return &DMAChannel{}, nil
		}
	}
	return nil, errors.New("no free channels")
}

func (arb *DMAArbiter) ReleaseChannel(ch *DMAChannel) error {
	if ch.nchan >= arb.nchans {
		return errors.New("invalid channel number")
	}
	bit := uint64(1 << ch.nchan)
	if arb.used&bit == 0 {
		return errors.New("channel not claimed")
	}
	arb.used &^= bit
	return nil
}

@tdunning @aykevl @deadprogram

[eliasnaur]

Thanks for working on this. A central registry for reserving DMA channels is a good idea to allow disparate drivers to work together. However, I worry that a useful DMA abstraction isn't worth the complexity. For example, your proposed API doesn't cover

• Peripheral pacing (DREQ)
• Interrupts for DMA completion, halfway completed etc.
• Chaining by control blocks (perhaps a rp2-specific feature?)
• Quirks (read data must equal write data size on rp2, address decrement, increment 2 etc.)

In particular, your go-wspr example doesn't seem to fit your API because it uses control block chaining of DMA channels. Even if the API was sufficient, go-wspr is already platform-specific (because of pio) so won't gain much from a generic DMA API.

I propose adding just the central channel registry and add efficient bulk transfer API to each device (I2C, UART etc.) and driver, backed by the platform's DMA. That's a smaller API, and can serve as inspiration for a generic DMA abstraction later on.

I believe DMAChannel start-transaction call should be asynchronous. After all we are abstracting the hardware, which is fundamentally and undeniably async. As much as I try to avoid async APIs I believe it is beneficial in this case to do go this way as that is the way the hardware works. Try to sidestep that and you'll eventually run into issues with whatever novel way you chose to represent something that is async as non-async.

I'm not convinced by this argument. Goroutines are supposed to be lightweight enough that APIs can block and let the caller decide whether to introduce concurrency through extra goroutines and channel. With an async API you need a way to expose interrupt-based waiting, otherwise the IsBusy+Gosched loop probably (much of) eats the efficiency gains of using DMA in the first place.
Another key advantage to a synchronous bulk transfer API is that the ownership of the passed in buffers is clear: for the duration of the call, writing to the buffers is a race.
Finally, just as Big Go, I also don't like API that assumes a non-moving GC, and a synchronous bulk transfer API can use runtime.Pinner to tell a (hypthetical) moving GC not to move buffers during DMA. We're even more likely to eventually want a compacting/moving GC than Big Go, because fragmentation is a much bigger problem in memory constrained environments.

[soypat]

• Interrupts for DMA completion, halfway completed etc.

IsBusy+Gosched loop probably (much of) eats the efficiency gains of using DMA in the first place.

This could be solved by adding a BusyFor/WorkRemaining method which could also replace IsDone with a ==0 check:

// TimeRemaining returns the time remaining until DMA transfer is completed.
// It returns 1 if the time remaining is undefined and 0 if it is done with the transfer.
func (ch *DMAChannel) TimeRemaining() time.Duration

Regarding your other points I feel my vision is aligned with yours, at least somewhat. I mostly want a public DMA API so users can tinker with the DMA API to make peripheral transactions better (SPI/UART/I2C) before it is properly implemented in TinyGo. Things like go-wspr can be out of the scope off the initial DMA API and serve as a guide to where we want to eventually go.

Regarding your points against async DMA API I feel I do agree on some points- the drivers that use DMA should be synchronous, so SPI, UART and I2C that use DMA should not all of a sudden stop blocking, that would break our TinyGo I/O model and certainly many programs if not all. That said, DMA is async so part of the program is going to be async whether we like it or not. I favor having the async API since that is how DMA works. We can then implement a generic synchronous API which consumes the async API, might even be a package level function?

func DMATxAndWait(ch *DMAChannel, cfg *DMATxConfig) error {
}

So I feel we are agreeing on the DMAArbiter API and maybe exposing some X part of DMAChannel API. Heck maybe we can leave it all unexported for starters, though that would create code duplication between TinyGo and packages that do their own DMA so far

[eliasnaur]

I'm still not sure what you mean by "because that's how DMA works". How is waiting for a DMA completion different than, say, waiting for the response to http.Get? Also, TimeRemaining is not always well-defined, because you can't predict stalls (say, if an external device is pacing the DMA).

If the API is to be async, it should at least compose with the regular Go mechanism for concurrency such as select. For example, by having a Wait() <-chan struct{} method, similar to time.After.

[eliasnaur]

I thought a bit more about our difference in opinion, and realized that I do agree with you that the DMA abstraction (if any) should probably be async. I still think the user-facing API should be sync[0], but drivers that interact directly with DMA probably don't want to waste efficiency on additional goroutines, channels etc.

In other words, I had the perspective of user code, whereas you probably had the perspective of a driver.

[0] In the form of bulk transfer API on devices, and perhaps a DMA-aware memory-to-memory implementation of copy and other bulk memory transfers.