Discussing Debugger Support For Xeus-cpp

[anutosh491]

Hey All,

I have been doing quite some back and forth with @kr-2003 for discussing some design details for the debugger. We think we've made quite some progress and would like to point out possible pain-points that came out of our discussion.

What has been achieved

We have already set up debugging in VSCode using LLDB-DAP. The following video demo showcases how JIT-compiled code can be debugged using LLDB, LLDB-DAP, and VSCode. For our project, we need to integrate LLDB-DAP with Jupyter Notebook.

debugg.1.mp4

Steps to replicate the above

PRE-REQUISITES : Install lldb and lldb-dap executable in your toolchain (should be pre-installed in macos). You can also install the lldb-dap extension for vs-code.

1. Create a program and name it as test.cxx for starters

// jitcode_with_cppinterop.cpp
#include "clang/Interpreter/CppInterOp.h"
#include <iostream>


void run_code(std::string code) {
 Cpp::Declare(code.c_str());
}


int main(int argc, char *argv[]) {
 Cpp::CreateInterpreter({"-g", "-O0"});
 std::vector<Cpp::TCppScope_t> Decls;
 std::string code = R"(
#include <iostream>
void f2() {
 int a = 4;
 int b = 10;
 std::cout << b - a << std::endl;
 std::cout << "kr-2003" << std::endl;
}
void f3() {
 int a = 1;
 int b = 10;
 std::cout << b - a << std::endl;
 std::cout << "in f3 function" << std::endl;
}
f2();
f3();
int a = 100;
int b = 1000;


 )";
 std::cout << code << std::endl;
 return 0;
}

2. Compile the above into an executable. We shall be attaching the debugger on this test executable

$LLVM_DIR/build/bin/clang++ -I$CPPINTEROP_DIR/include -g -O0 -lclangCppInterOp -Wl,-rpath,$CPPINTEROP_DIR/build/lib -L$CPPINTEROP_DIR/build/lib test.cxx -o test

3. Create a launch.json to launch the debugger through VS-code

{
 "version": "0.2.0",
 "configurations": [
     {
         "type": "lldb-dap",
         "request": "launch",
         "name": "Debug",
         "program": "/Users/abhinavkumar/Desktop/Coding/Testing/test", // change according to your test executable
         "sourcePath" : ["${workspaceFolder}"],
         "cwd": "${workspaceFolder}",
         "initCommands": [
             "settings set plugin.jit-loader.gdb.enable on", // This is crucial 
         ]
     },
 ]
}

By default, the VS Code extension will expect to find lldb-dap in your PATH. Alternatively, you can explictly specify the location of the lldb-dap binary using the lldb-dap.executable-path setting.

4. Create a input_line_1 file which acts as a source that the debugger uses for mapping addresses.

#include <iostream>
void f2() {
  int a = 4;
  int b = 10;
  std::cout << b - a << std::endl;
  std::cout << "kr-2003" << std::endl;
}
void f3() {
  int a = 1;
  int b = 10;
  std::cout << b - a << std::endl;
  std::cout << "in f3 function" << std::endl;
}
f2(); // applying breakpoiutn here
f3();
int a = 100;
int b = 1000;

5. Try running the debugger through vs-code. You should be able to achieve the above demo shown where we are able to set breakpoints, continue and step in for a single cell of JIT compiled code.

[anutosh491]

Pain Points until now

1. Debugging for Multiple cells of JIT compiled code.

While performing above checks and experiments, we found that if function definition and function call are in the same cell, then the step-in call from the function call behaves normally, i.e. it steps-in directly in one call. But when they are in different cells, then it takes multiple step-in calls to step into the function definition from the function call.

• Update test.cxx to the following.

#include "clang/Interpreter/CppInterOp.h"
#include <iostream>

void run_code(std::string code) {
  Cpp::Declare(code.c_str());
}

int main(int argc, char *argv[]) { 
  Cpp::CreateInterpreter({"-g", "-O0"});
  std::vector<Cpp::TCppScope_t> Decls;
  std::string code = R"(
#include <iostream>
void f2() {
  int a = 4;
  int b = 10;
  std::cout << b - a << std::endl;
  std::cout << "kr-2003" << std::endl;
}
void f3() {
  int a = 1;
  int b = 10;
  std::cout << b - a << std::endl;
  std::cout << "in f3 function" << std::endl;
}
f2();
f3();
int a = 100;
int b = 1000;

  )";
  std::cout << code << std::endl;
  run_code(code);
  code = R"(
std::cout << "a = " << a << std::endl;
std::cout << "b = " << b << std::endl;
f2();
  )";
  run_code(code);
  return 0;
}

As can be seen we call run_code multiple times to replicate Multiple cells of JIT compiled code !!!

2. Also remember to add a input_line_2 (we already have input_line_1)

std::cout << "a = " << a << std::endl;
std::cout << "b = " << b << std::endl;
f2();

fffffff.mp4

Probably @vgvassilev might know the root cause here and could educate us with some insights.

[kr-2003]

Continuing the discussion on the above pain point

Following are the lldb outputs when trying for both the cases:
a) When function definition and call are in the same cell
b) When function definition and call are in different cell

a) When function definition and call are in the same cell

1 location added to breakpoint 1
Process 93415 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
   frame #0: 0x00000001010841b0 JIT(0x101068000)`__stmts__9 at input_line_1:15:1
  12     std::cout << b - a << std::endl;
  13     std::cout << "in f3 function" << std::endl;
  14   }
-> 15   f2(); // applying breakpoiutn here
  16   f3();
  17   int a = 100;
  18   int b = 1000;
(lldb) s
Process 93415 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
   frame #0: 0x000000010108400c JIT(0x101068000)`f2() at input_line_1:4:7
  1  
  2    #include <iostream>
  3    void f2() {
-> 4      int a = 4;
  5      int b = 10;
  6      std::cout << b - a << std::endl;
  7      std::cout << "kr-2003" << std::endl;

b) When function definition and call are in different cell

Process 87843 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.8
   frame #0: 0x0000000100594068 JIT(0x100580000)`__stmts__0 at input_line_2:4:1
  1  
  2    std::cout << "a = " << a << std::endl;
  3    std::cout << "b = " << b << std::endl;
-> 4    f2();
(lldb) s
Process 87843 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
   frame #0: 0x00000001005940a8
->  0x1005940a8: adrp   x16, 0
   0x1005940ac: ldr    x16, [x16, #0xd8]
   0x1005940b0: br     x16
   0x1005940b4: adrp   x16, 0
(lldb) s
Process 87843 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step into
   frame #0: 0x00000001005940ac
->  0x1005940ac: ldr    x16, [x16, #0xd8]
   0x1005940b0: br     x16
   0x1005940b4: adrp   x16, 0
   0x1005940b8: ldr    x16, [x16, #0xf8]
(lldb) s
Process 87843 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step into
   frame #0: 0x00000001005940b0
->  0x1005940b0: br     x16
   0x1005940b4: adrp   x16, 0
   0x1005940b8: ldr    x16, [x16, #0xf8]
   0x1005940bc: br     x16
(lldb) s
Process 87843 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step into
   frame #0: 0x000000010057c000 JIT(0x100560000)`f2() at input_line_1:3
  1  
  2    #include <iostream>
-> 3    void f2() {
  4      int a = 4;
  5      int b = 10;
  6      std::cout << b - a << std::endl;
  7      std::cout << "kr-2003" << std::endl;

Here, I observed that the intermediate functions do not contain debug symbols, i.e. they don't have any function names. That's why the debugger in vs-code shows them as Unknown Sources.

[JohanMabille]

One possible workaround is to internally implement a loop of stepIn + stackTrace requests until the source field of the stackTraceResponse matches a known cell. Then the last stepIn response can be returned to the frontend. The only thing to be careful of is the request_seq of the response, that should be set to the sequence number of the initial stepIn request.

[kr-2003]

More Pain Points

Stepping-over(next) on a function call declared in previous block/cell does not work properly.

1. Let's say a user defines multiple functions(f1, f2 and f3) in a single cell. These functions are called from the same cell one after another. If the step-over(next) command is used on one function, then it flawlessly goes to next function call.
2. But let's say function call is made from another cell, then it takes multiple step-over(next) commands for the line to execute. Plus, it does not stop at the next function call. But it does execute it without any bugs/errors.

In the given video below,

1. input_line_1 shows the use-case explained in point-1. Breakpoint is at f1() call.
2. input_lint_2 shows the use-case explained in point-2. Breakpoint is at f3() call.

// input_line_1

#include <iostream>
void f1() {
  std::cout << "in f1 function" << std::endl;
}
void f2() {
  std::cout << "in f2 function" << std::endl;
}
void f3() {
  std::cout << "in f3 function" << std::endl;
}
std::cout << "In codeblock 1" << std::endl;
int a = 100;
int b = 1000;
f1();
f2();
f3();

// input_line_2 

std::cout << "In codeblock 2" << std::endl;
std::cout << "a = " << a << std::endl;
std::cout << "b = " << b << std::endl;
void f4() {
  std::cout << "in f4 function" << std::endl;
  f2();
}
f3();
f2();

Above is shown in this video.

mult-sing.mp4

But, let's say in input_line_2, if instead of using step-over(next), I continuosly use step-in on f3, then it goes to the next function call(i.e. f2).
Above is shown in this video.

mult2.mp4

The problem is how stepping-in inside the last call of f3 directs it to the call of f2 in input_line_2.

[anutosh491]

Check llvm/llvm-project#61152 (comment)

[kr-2003]

Progress

• Setup the base template(xdebugger class, xlldbdap_client etc.) for debugger in xeus-cpp.
• Able to start debugger when clicking on the debug-icon.
  • Running lldb-dap process as a forked process on specified port.
  • Binding the xlldbdap_client with the forked process of lldb-dap via threading.
  • Also, the client is now able to send request and receive responses from lldb-dap.
• All of the above progress can be find on this dummy PR.

Blockers

• Can't receive multiple responses from a single request in xeus-zmq.
• After the start of debugger, the Kernel remains busy and nothing else is called further.

[kr-2003]

Dummy PR - Link

Progress

• Now the kernel does not remain busy(Previous blocker solved most probably).
• Now, when clicked on debugger, the initialize_request is sent automatically.
• After that in attach_request instead of attaching a current running program, I sent hardcoded launch_request. Seems to work fine.
• Working on inspect_variables_request. Might require a lot of code to be written like xeus-python-shell for variables and inspection.
• UI for breakpoint(side padding on every cell) now opens up when clicked on debug button.

Blockers

• Can't receive multiple responses from a single request in xeus-zmq(might be working on this tomorrow).
• Cell and its code structure(For now I have hardcoded the executable. But to debug one might have to manage the JIT code in cells and send them as program which calls CppInterOp's Cpp::Process and debugs on that.
• Continuing point 2. How to structure the code? When to compile? Temp code files? For every cell need to have something like this. OR club all cells into one.

Video Demo

Note that the variables shown in the left panel are HARDCODED. We need to implement the inspect request feature. For now I just have like this in

    nl::json debugger::inspect_variables_request(const nl::json& message)
    {
        std::cout << "inspect_variables_request not implemented" << std::endl;
        std::cout << message.dump() << std::endl;
        nl::json reply = {
            {"type", "response"},
            {"request_seq", message["seq"]},
            {"success", true},
            {"command", message["command"]},
            {"body",
             {{"variables",
               {{{"name", "a"},
             {"value", "100"},
             {"type", "int"},
             {"evaluateName", "a"},
             {"variablesReference", 0}},
            {{"name", "b"},
             {"value", "1000"},
             {"type", "int"},
             {"evaluateName", "b"},
             {"variablesReference", 0}}}}}}
        };
        return reply;
    }

• Also regarding this is: You can see that inspect_variable_request is called infinitely many times. Why is this??
  • As far my understanding, I think its because the variables are wrong. Variables inside the test exec are something else. They have different properties(like reference, etc.).

debuguugug.mp4

[kr-2003]

Progress Regarding Attaching Kernel to lldb-dap process

EINTR.mp4

[kr-2003]

Progress Regarding Attaching LLDB/LLDB-DAP to jupyter-console

jup-console.mp4

[kr-2003]

Problem

1. Above video shows that lldb and lldb-dap can be connected to xcpp kernel(after some minor changes in xeus-zmq. And debugging can be carried via jupyter-console.
2. But the same is not true for jupyter-lab/jupyter-notebook. jupyter-lab/jupyter-notebook sends consecutive messages to the kernel. And if the kernel does not reply, then it assumes that the kernel dies.
3. So, whenever I was trying to attach debugger to xcpp kernel(spinned off via jupyterlab) using its pid, then the jupyter-lab notifies me with died kernel and starts another kernel process.

So kernel dying was the major issue while attaching debugger to xcpp kernel due to jupyter-lab continuous heartbeat checks.

Solution

I have tried the approach that Vipul suggested, i.e.

• Having another process running for CppInterOp's code processing.
• I have used shared memory to communicate between kernel and cppinterop_process(forked process).
• And then attached lldb to this cppinterop_process instead of attaching it to kernel(which resulted in its death).

Here are the results(tried for lldb only. i will update for lldb-dap too).

Result

• Debugger now working with jupyter-lab.

lab---working---bp.mp4

What I did?

• Extracted all the CppInterOp's processing from kernel to another process named cppInterop_process.
• Used shared buffer between parent(kernel) and child(cppinterop_process) for exchanging code and outputs.
• Attached lldb to cppinterop_process and applied breakpoints.

[kr-2003]

Since in the previous comment, I moved out the CppInterOp's process out of the main exection(kernel).
But I think it would be more feasible that if we move out the JIT execution of CppInterOp's process.(Not CppInterOp). Because, in clang-repl, out-of-process JIT execution has already been researched and implemented.

References

1. clang-repl's oop JIT execution PR
2. Remote JIT-ing docs