Skip to content

Debugging with dlv inside gvisor with systrap faults #11649

Open
@dany74q

Description

@dany74q

Description

Hey gvisor folks !

I have a use case where I need to debug a go binary with dlv, running w/ gvisor.
When I set a breakpoint, reach it, and then try to step - dlv is either stuck or faults.

This only happens when running with systrap, using --platform=ptrace, it works as expected;
as the ptrace platform usage is discouraged, I would really appreciate your help w/ making it work under systrap.

It happens with the latest runsc, go and dlv releases.

Steps to reproduce

mkdir /tmp/gvisor-dlv
cd /tmp/gvisor-dlv

# get runsc
ARCH=$(uname -m)
wget -q https://storage.googleapis.com/gvisor/releases/release/latest/$ARCH/runsc
chmod +x runsc
export PATH=$PATH:/tmp/gvisor-dlv

# get go binary
wget -qO- https://go.dev/dl/go1.24.2.linux-$(echo $ARCH | sed -E 's/x86_/amd/;s/aarch/arm/').tar.gz | tar xzf -
export PATH=$PATH:/tmp/gvisor-dlv/go/bin

# create oci bundle
mkdir bundle
mkdir --mode=0755 bundle/rootfs

# get dlv binary
GOBIN=/tmp/gvisor-dlv/bundle/rootfs go install github.com/go-delve/delve/cmd/dlv@latest

# compile go binary
go mod init main
cat <<EOF > main.go
package main

func main() {
        for i := range 1000 {
                print(i)
        }
}
EOF
go build -gcflags 'all=-N -l' -o bundle/rootfs/main .

# create mount dir for uds
mkdir mnt
chmod 777 mnt

# create oci config
cat <<EOF > bundle/config.json
{
    "ociVersion": "1.0.0",
    "process": {
        "args": [
            "/dlv",
            "--listen=unix:/mnt/dlv.sock",
            "--headless",
            "--api-version=2",
            "--accept-multiclient",
            "exec",
            "/main"
        ]
    },
    "root": {
        "path": "rootfs"
    },
    "mounts": [
        {
            "destination": "/mnt",
            "type": "bind",
            "source": "/tmp/gvisor-dlv/mnt"
        }
    ]
}
EOF

# run gvisor
sudo -E env PATH=$PATH runsc --host-uds=all run -bundle /tmp/gvisor-dlv/bundle test
# > API server listening at: /mnt/dlv.sock

# connect to dlv in another session
cd /tmp/gvisor-dlv
sudo chmod 666 mnt/dlv.sock
bundle/rootfs/dlv connect unix:mnt/dlv.sock
# > (dlv)

# set breakpoint 
(dlv) b main.main
# > Breakpoint 1 set at 0x470b0a for main.main() ./main.go:3

# reach breakpoint
(dlv) c
# > [Breakpoint 1] main.main() ./main.go:3 (hits goroutine(1):1 total:1) (PC: 0x470b0a)

# next
(dlv) n

# dlv should now hang or fault, it might take a few runs to hit the fault flow; 
# on rare occasions it works properly, if so rerunning will usually trigger the hang / fault
# > unexpected fault address 0xc0000547d0
#> [signal SIGSEGV: segmentation violation code=0x2 addr=0xc0000547d0 pc=0xc0000547d0]

# > [runtime-fatal-throw] runtime.throw() ./go/src/runtime/panic.go:1092 (hits goroutine(1):1 total:1) (PC: # 0x468084)

runsc version

runsc version release-20250414.0
spec: 1.1.0-rc.1

uname

Linux ip-10-0-1-16 6.8.0-1024-aws #26-Ubuntu SMP Tue Feb 18 17:22:37 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

runsc debug logs

runsc-debug-log.tar.gz

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions