debug: use Delve's DAP implementation

[hyangah]

Delve supports debug adapter protocol natively (dlv dap).
https://github.com/go-delve/delve/tree/master/service/dap

Implement the debug feature using it and deprecate the dlv wrapper (src/debug/goDebug.ts).

@polinasok @quoctruong @ramya-rao-a

[polinasok]

DAP-in-delve work is tracked under go-delve/delve#1515
It relies on a new library repo for DAP support in Go: https://github.com/google/go-dap
You can reference delve's dap/server.go::handleRequest() for an account of which DAP requests are already supported in delve and which ones are still TODO.

[lggomez]

I would like to add that some of the current issues due to incompatibility between the current DAP and delve that should be solved (at least partially) are the following: #118 #119 #129 #130

Tacking these would help a lot to the goal of achieving a seamless debug experience

[polinasok]

Overview

This overview is being updated as new options and details surface in the comments below.

Before

VSCODE <=dap=> FULL ADAPTER <=json-rpc=> DELVE

• The current adaptor acts as an intermediary between VS code and Delve debugger.
• It is registered as an adapter in package.json debuggers under type "go".
• It is launched by the extension as a separate Node.js process at the beginning of each debug session of that debug type, using DebugAdapterExecutable.
• It communicates with the editor using DAP over stdin/stdout and with Delve over JSON-RPC.
• It handles initialize and shutdown sequences for the debug session and translates and routes everything in-between to delve.
• For remote debugging, it now provides local-to-remote path mapping using delve's RPCServer methods (debug: automatically Infer Path Mapping for Remote Debugging #45)

In response to `launch` or `attach` request, the DA launches `dlv --headless` command (that builds, executes or attaches to the debugee) or connects to an already running delve server.

• request=“launch” mode=“auto”
  • extension maps to mode=“debug” or “test”
• request=“launch” mode=“debug”
  • dlv debug <program> --headless=true --listen=<host>:<port> --api-version=<apiVersion>
• request=“launch” mode=“test"
  • dlv test <program> --headless=true --listen=<host>:<port> --api-version=<apiVersion>
• request=“launch” mode=“exec”
  • dlv exec <program> --headless=true --listen=<host>:<port> --api-version=<apiVersion>
• request=“launch” mode=“remote”
  • warning Request type of ‘launch’ with mode ‘remote’ is deprecated, please use request type ‘attach’ with mode ‘remote’ instead
  • connects to delve server already running at specified <host>:<port> (or default)
• request=“attach” mode=“local”
  • dlv attach <processId> --headless=true --listen=127.0.0.1:42309 --api-version=<apiVersion> --wd=<cwd>
• request=“attach” mode=“remote”
  • connects to delve server already running at specified <host>:<port>

After (Option 1)

VSCODE <=dap=> DELVE

• Delve with DAP will act as an adaptor and debugger in one without the middleman.
• It will be registered in package.json under a new debug type, so we can support both adaptors during the transition period.
• It will be launched by the extension as a separate server process with dlv dap command at the beginning of each debug session of the new debug type, using DebugAdapterServer. (To be revisited if dlv-dap starts supporting more than one debug session, so we can reuse a single server instance.) This can be done similarly to how the dlv command is spawned in the existing adaptor. The server will run at port specified by debugServer, defined dynamically within the extension as part of DebugConfigurationProvider implementation. The user-specified option in launch.json can still be used for debugging.
• It will communicate with the editor using DAP over TCP connection.
• It will handle the initialize and shutdown sequences for the debug session and everything in-between.
• Open Question: Are we guaranteed to always receive absolute file path from IDE via request args or does delve also need to keep track of the working directory?
• Open Question: For remote debugging, can local-to-remote path mapping be added on the delve or extension side while local source tree is passed via attach arguments?
• Open Question: Lack of adaptor could be an issue if we need to add any vscode-specific logic that does not belong in delve. Can it be shifted to the extension code instead?

In response to `launch` or `attach` request, delve-adapter will build, test, execute or attach to the debugee.

• request=“launch” mode=“debug”
  • run go debug, launch process
• request=“launch” mode=“test"
  • run go test, launch process
• request=“launch” mode=“exec”
  • launch process
• request=“launch” mode=“remote”
  • will not be supported
• request=“attach” mode=“local”
  • attach to process
• request=“attach” mode=“remote”
  • already attached to the debugger server as part of adaptor launch

After (Option 2)

VSCODE <=dap=> LEAN ADAPTER <=dap=> DELVE

• A lean TypeScript adaptor will exist between VS Code and Delve-on-DAP.
• It will be registered in package.json under a new debug type, so we can support both adaptors during the transition period.
• It will be launched by the extension as a separate Node.js process at the beginning of each debug session of that debug type, using DebugAdapterExecutable.
• It will communicate with the editor using DAP over stdin/stdout and with delve using DAP over TCP connection.
• TBD Will this adapter route the initialize and shutdown sequences for the debug session and everything in-between to delve, while intercepting the messages to adjust file paths and other settings?
  • Open Question: will delve or the thin adaptor handle initialize request? (delve supports it anyway, so it can be used by itself)
  • Open Question: should dlv dap be launched/connected to on initialize or launch/attach?
  • Open Question: which requests should be intercepted and why?
  • Open Question: will the rest of the communication happen directly between editor and delve?
• Open Question For remote debugging will the adapter support local-to-remote path mapping? What requests need to be added to DAP to support this?
• The lean adaptor can host vscode-specific logic that does not belong in delve.

Open Question: will this option make inline mode possible?
Open Question: will this option make RunInTerminal support easier?

[eliben]

I have an experimental branch where a new debug adapter is added alongside the existing one, and registered under a different type - godlvdap. A PR showing the diff is here.

As @polinasok's message above details, it registers a new debugger with a new entry in packages.json for the debuggers contribution point. For the purposes of this demo, the new adapter is just a copy of the existing adapter with some extra logging thrown in so we could distinguish between them. There are a couple more minor changes required, like registering a configuration provider for the new debugger in goMain.ts.

The prototype works. When debugging Go code, we can select between the two different debug adaptors by adjusting the "type" field in the project's launch.json.

[eliben]

After

VSCODE <=dap=> DELVE

[...]

Another alternative is to retain a thin debug adapter even when Delve's DAP functionality is complete. This debug adapter will take care of launching Delve and then will just pass-through all DAP commands back and forth.

The obvious cost of this alternative is having to maintain an additional debug adapter. That said, this adapter is expected to be very minimal.

The potential benefits of this approach:

• It will avoid pushing too much vscode-specific logic into Delve. As of now, the LaunchRequestArguments interface contains a lot of information that's outside the spec of the DAP protocol. It's specific to how vscode invokes debug adapters. If we talk to Delve directly, we'll have to encode all this logic in Delve itself, making it vscode-specific. If folks want to use Delve with other IDEs (which we already know is the case!) they will have to implement their own logic inside Delve or masquerade as vscode.
• It uses a familiar debug adapter invocation path. For example, the existing debug adapter has some logic for finding Delve and prints out customized error messages if Delve is not found. It's not clear how easy this is when dlv is specified directly in package.json. There's a difference between an adapter whose code ships with the extension, so the extension can safely assume it will find goDebug.ts, and between dlv which doesn't ship with the extension. It's likely that dlv won't be installed on new users' machines, and if it is, it's likely that the version will be wrong. We need a graceful way to handle this and it's not 100% clear how to do this without an adapter.

[hyangah]

The debug node client supports multiple ways of running a debug adapter.

https://github.com/microsoft/vscode-mock-debug/blob/master/src/extension.ts#L57

Can we utilize that and by default run the thin wrapper in inlined mode?

[eliben]

The debug node client supports multiple ways of running a debug adapter.

https://github.com/microsoft/vscode-mock-debug/blob/master/src/extension.ts#L57

Can we utilize that and by default run the thin wrapper in inlined mode?

Yes, though it's still not 100% clear to me what the tradeoff is. It seems like this allows us to debug the adapter and the extension in the same process, which is nice. This seems like a fairly new option in vscode, and I imagine the current Go debug adapter could also use this since it's written in TS/JS.

[adding more details]

Re-reading the documentation carefully again, when we use the DebugAdapterExecutable option (which is what happens when the path is specified in package.json), vscode expects to talk to the adapter via stdin/stdout, and not sockets. To talk to an adapter via sockets, we have to set debugServer. Therefore, if we want vscode to talk directly to Delve without an adapter in between, we'll have to either ask users to have debugServer in each
launch.json or we'll have to adapt Delve to talk DAP over stdin/stdout as well.

[quoctruong]

@eliben When you register the debug configuration provider, there is a method called resolveDebugConfiguration that you can use to intercept the configuration before sending it to the adapter. This is where you can set the debugServer on behalf of the user.

@polinasok I also agree with @eliben's suggestion that we should have a thin layer of DAP in VSCode unless we want to move some of the existing logic that we have like path inference into Delve.

[polinasok]

@quoctruong I was just going to ask you to chime in on the option of the thin adaptor, which we discussed as potentially necessary because of your path inference work. Could you please add more details/links as to what logic will end up in the thin adaptor for this?

For inspiration, python has a DAP adaptor to launch PTVSD, handling the rest of communication directly over DAP. The comment says

 * Primary purpose of this class is to perform the handshake with VS Code and launch PTVSD process.
 * I.e. it communicate with VS Code before PTVSD gets into the picture, once PTVSD is launched, PTVSD will talk directly to VS Code.
 * We're re-using DebugSession so we don't have to handle request/response ourselves.

[polinasok]

I updated the overview comment to include the 2nd option with the thin adapter, so we have both at a glance.

[polinasok]

@eliben

It will avoid pushing too much vscode-specific logic into Delve.

Agreed that we want to avoid pushing anything vscode-specific into Delve.
I believe so far we have been successful in avoid this, but please do let me know if you think otherwise.

As of now, the LaunchRequestArguments interface contains a lot of information that's outside the spec of the DAP protocol. It's specific to how vscode invokes debug adapters.

The current debug adaptor also gets all these extra arguments and just ignores them. That configuration passes through all the layers, so I think the idea is to just capture all attributes in one place and to let each layer handle those that of interest to it.

Since the arguments are a free-form map and not a struct, we do not have any of those vscode-specific fields coded anywhere. Are you concerned that they are polluting the request that delve receives even if it never reads those fields? Or do you think we should avoid having delve handle launch requests all-together and go back to specifying the program on the delve command line?

If folks want to use Delve with other IDEs (which we already know is the case!) they will have to implement their own logic inside Delve or masquerade as vscode.

Do you know of a specific example where the current approach might cause issues? I have not looked into other IDEs very closely yet.

My general understanding is that launch.json-like configuration is of part of the contract, even if DAP spec does not specify it in detail. Some fields are editor-specific. Others are adapter-specific. So the adapter (and in our case delve) has to make it clear, which argument fields it relies on and vscode or any other IDE have to supply them in launch/attach requests. And how they source them from the user or internally (via launch.json or something else), will be up to each client.

the existing debug adapter has some logic for finding Delve and prints out customized error messages if Delve is not found. It's not clear how easy this is when dlv is specified directly in package.json

Your concern about the dlv installation is very valid. I believe by default you get a "no debug adaptor" message, which is not very useful because the user would not know that delve is the adaptor and delve is missing and needs to be installed. However, if debugServer option is used, it takes precedence over the executable in package.json, which is the very last fallback. So whatever extension code launches delve as an adapter and sets debugServer, can in theory do the same kind of error-checking and messaging that the current adapter does.

Therefore, if we want vscode to talk directly to Delve without an adapter in between, we'll have to either ask users to have debugServer in each launch.json.

I tried to highlight in my overview comment that debugServer will be set in the extension code (goDebugConfiguration.ts). Looks like it was not clear that that we will not need any input from the user via launch.json (unless they want to debug an external adapter). Reworded a bit. Please let me know if it is still unclear.