Implementing debugger

[moretea]

I'm willing to take a stab at implementing debugger support as tracked in #119.

Are the maintainers interested in accepting a PR implementing a debugger? If so, I'd appreciate early feedback on the approach I plan to take, as my C is a bit rusty (pun intended, write most systems code in rust).

Overall plan

• Look at the existing implementation in https://github.com/koush/quickjs
• Get consensus on few questions up front-ish to ensure that the PR's will be accepted
• Submit a series of PR's

Existing implementation by koush

I have taken a look at the implementation at https://github.com/koush/quickjs, which seems reasonable; it generates two dispatch tables, and if the debugger is active, switches to the dispatch table that. The transport code (e.g. that allows an external debugger to open a TCP connection and control the quickjs debugger) seems to not be 100% compatible with the Debug Adapter Protocol. It also seems that no automated tests are included.

Upfront question

1. Determine if the debugger should always be included, or should be behind a feature toggle
2. Would the approach of using two dispatch tables be acceptable?
3. I would prefer to add a set of tests. I started out writing a C test program, but am considering using the quickjs javascript test harnass (tests/assert.js). Given that these tests would want to manipulate the runtime, this would entail adding a C module to expose those functions to quickjs itself. Would that be an acceptable route for writing debugger tests? If so, I would propose to create a new C module, qjs:qjs.

Questions to answer later
I'll do some prototyping and more investigation. Later I would like to get consensus on:
5. The API between QuickJS and a debugger stub
6. The debugging semantic model (options); per runtime, per context; should we do the same as koush w.r.t showing multiple threads for each context?

[bnoordhuis]

Determine if the debugger should always be included, or should be behind a feature toggle

I think it should be behind a compile-time feature flag, for people that want a minimal build (ex. quickjs-libc.c)

Would the approach of using two dispatch tables be acceptable?

Only if it doesn't regress the common case, which I suspect it will. If it does, what could work is breaking out JS_CallInternal to a separate source file and include it twice in quickjs.c, one with the regular dispatch table, and one with the debug table.

I would prefer to add a set of tests [..] I would propose to create a new C module, qjs:qjs

I'd rather not add new public modules without careful consideration. It's no problem to add a C module for use in your JS tests though.

The debugging semantic model (options); per runtime, per context

Per context, right? That's the unit of execution.

[sponge]

Was curious if anyone had messed with porting over koush changes to ng directly. I was able to port over the changes made for debugger support fairly easily, but I believe I'm running into issues with line number calculation preventing breakpoints from working, and some odd behavior in stepping.

[saghul]

I haven't tried it myself, but I think it would be cool to see how we could integrate it.

[sponge]

Here's what it looks like if you're curious!

sponge@73b5d0b

I am using a vendored version of the library, so I chopped out the changed files and added it to a branch. Didn't do any testing on this specific branch yet (I'm also using Premake and not CMake, but I don't think there are any build system changes needed besides the new files.)

I've also only tested this on MacOS so far, but when I was using this patch on a different QuickJS fork, it was working on Windows. This is all very much hacked together to just work for my tinkering level of quality :)

I extracted the changes by diffing the original debugger against that version of QuickJS, and then moved those changes over to quickjs-ng change by change, fixing up a handful of compiler issues along the way. The added files were brought over directly, minus a couple of other compiler changes.

Some thoughts:

• The core changes are pretty minimal overall. Most of it is just new functions at the bottom of quickjs.c, and the swapped out opcode table as mentioned.
• Hitting pause and stepping through functions seems to work. Variable and stackframe displays seem to work.
• Breakpoints don't work. As far as I can tell this statement never returns true. last_line_num always seems to be the line number the function starts on: https://github.com/sponge/quickjs-ng/blob/debugger/quickjs.c#L56066
  • This is where I start to hit the edges of my experience, not understanding terribly how language interpreters work, and QuickJS itself. I've tried to step through where I think the relevant points are, and I think this is where the breakage is happening.
• I don't love how the only interface to the debugger seems to be streams of JSON strings. I don't love the VSCode extension, it seems to be slow for breakpoints to start triggering, and I have problems getting it to connect when I launch the debugger and the host at the same time. I was hoping that I'd just have a bunch of plain functions and structs to let me hack up a little debugger UI in Dear ImGui or something.
  • I'm sure the extension could be improved, but considering Debugger Support #119 has been sitting around for a year, it's probably not a super in demand feature :)
• There were a couple of comments in the original version talking about how columns weren't tracked. They seem to be now, so I was able to fix those-- as long as I did it correctly!
• The debugger code seems to rely on reusing JSValue and such as generic containers for native debugger state. A bit weird, not sure if the runtime does this anywhere else, but it seems to work so vOv

[rosshadden]

@sponge Have you considered opening a PR? You're probably not satisfied with it but it would possibly get the ball moving. Maybe somebody here could even help out. I tried building from your branch and fixed one problem but ran into some others. Thanks for digging in to this in the first place!

[sponge]

I had some changes that I pushed from the version embedded in my dumb lil' game engine, but I'm not sure if it was anything that would've prevented building. I also only tested on Windows, so I have no idea if that'd help you on your side.

If there's any interest, I could open a PR, but didn't want to put the burden on anyone else since I'm probably not the best person to fix it up and get it over the finish line. This was very much a huge hack done for a personal project with absolutely no quality standards besides some semblance of Works On My Machine.

[bnoordhuis]

If nothing else, having a PR go through CI will tell you if it works on platforms besides Windows :-)