Technical discussion: Automated testing

[jkup]

I'm hoping this issue will serve as a good place for discussing automated testing of source maps.

• Some good discussions already happened on the RFC repository between @bomsy, @littledan and @jridgewell here Test Suite for Source Maps  ecma426#15
• This playground @kamilogorek made is a great start (https://github.com/kamilogorek/sourcemaps-playground); I'm curious about what approach we'd take for automated testing.

[jkup]

From the monthly call:

1. Maybe start with assertions for generators. Make a validator and test that each one created is a valid source map.
2. For consumers, we'll need a way to access the browser's. Jaro mentioned maybe we could expose something via puppeteer or Chrome DevTools where we could query them directly.
3. For testing post-hoc tools like Sentry maybe we could run tests against the symbolicator service example request
4. Maybe look at existing tools and bugs that have been filed against them.
5. Sentry had in the past taken every token mapping and gone back to the source to make sure they looked like right token.

[jkup]

From @brad4d (just didn't want it getting lost!)

Re: testing tools that generate source maps. Suppose the test cases were input JS code which includes string values that won't be stripped out by the compiler/processor. The output passes the test if the source maps correctly map the locations of those same strings in the output back to the original text?

[connor4312]

Re (2) and (3), could we publish a repository of test cases in some readable format and allow interested consumers to adopt the test suite? Running tests in Sentry versus Chrome Devtools versus js-debug would be very different, but it's likely people would happily adopt a nice easy-to-parse test suite. This is similar to other "bring your own tool" test suites like autobahn or pngsuite.

For dev tools at least, this could be a set of tests that comprise:

• A JS file and map file (or inline sourcemap). A source file would be nice to human parsibility but is not strictly required
• A JSON file containing one or more
  • Positions to pause at in the source file (tests the source -> compiled mapping)
  • Expected scopes at the position (tests the scopes proposal)
  • Expected variables and underlying expressions in each scope -- whether sourcemapped or just from the runtime
    • I would not test the evaluated value of variables in the scope, since that is engine-dependent; V8 tends to optimize away variables not used in a given function, for example
    • this also tests rename behavior
  • ...any other things to test?

One could also test generators by having them create the JS and map file from the source file and run it against similar assertions.