Skip to content

gathering_ideas.md

Latest commit

 

History

History
242 lines (184 loc) · 11.9 KB

gathering_ideas.md

File metadata and controls

242 lines (184 loc) · 11.9 KB

Project outline

The main design decisions we have not made and is stopping this project from progressing are:

  • do we want separate 'back-end' data collectors?
  • do we need a 'server' to collect data
  • does our data-visualization need to be CPU intensive ?
  • could it be browser based? ()
  • what platform and UI-library will we use

Collector -> Server -> Visualizer

One way to go would be to have python backends collecting data, send the data to a central server and have multiple clients. We could have browser-client and a native-client. The advantage of the server being that you can connect a client later after collection starts. And closing the visualization would not stop data-collection.

All-in-one

Another way could be to tail a log-file, much like debugview++, the visualization would be done locally on the system that has the log. Or it would run remote, and receive message by socket.

These setups are very different.

gathering requirements

  • gantt chart timeline with markers to visualize processes and bottlenecks
  • multi level / stacked gantt chart suggested by Peter Bindels to visualize chronological data that is spaced unevenly (decades vs. seconds)
  • time/frequency analysis (group events per period at plot counts/timebox over time)
  • pluggable input sources
  • pluggable visualizers
  • portability for MacOs, Linux and Windows
  • a 1 microsecond accurate storage will cover most usecases. On very time critical systems where time is measured in nanoseconds (1e-9), picoseconds (1e-12) or femtoseconds (1e-15) optimizations are usually measured/aggregated in loops. This seems to be generally available:
  • the description of the Trace Event Format contains both very good ideas for the storage format and visualization

ideas

limitations of existing tools to consider:

  • online tools (such as google's Perfetto) can (and do) change input format over time and become incompatible with older data (bugs or not, it is no longer ABI compatible), so consider a 'frozen' offline viewer to unsure that very data can be read anytime. Perfetto can be installed locally, so that can be a way to deal with that.
  • some trace formats (like VCD) require that you know up front what 'signals' are going to be present, this is problematic, because when 'streaming' logs all possible occurrences of names of function/events are not known up front. So this kind of format requires post-processing.
  • some trace format (like chrome's trace.json) require special begin and end markers, [ {"first": "v1"} , {"second": "v2"} ] this may pose a problem when a program crashes and it does not write the end marker.

more...

  • online filters/highlighting storage to sync settings over difference workstations (login with google account?)
  • what do we do with internationalization? Chinese, Japanese characters, etc...

https://simplicable.com/new/information-analysis

  • gather
  • aggregation / correlation
  • map (different sources may provide the same information in slightly different ways)
  • categorization
  • summaries (frequency, totals occurrences)
  • analysis
  • visualization

Concepts:

  • multi-in/multi-out
    • in: for example, take multiple logfiles, sockets, or other input
    • out: stream to disk, stream to memory-ring-buffer
  • client/server
    • for example, multiple server instances on linux and a client/viewer on windows (or vice versa)
    • server side buffer (late connecting client can get messages from the past x time)
    • Debugview++ could be one of the 'servers' for windows?
    • consider .dotnet core or java for a front-end visualization, to make the UI multi-platform

Visualization:

communication:

Filter:

  • separation for input and view filter, meaning filtered messages from input filters is not kept at all, and filtered messages from view filters and only not shown. (debugview++ only has the latter, which can make it difficult to setup very long lasting tests.)

Search:

  • search
  • filter !
  • (trigger action)

Possible inputs:

  • compact binary format to collect high-resolution data (telemetry) for games and financial trading applications.
  • VCD (Value Change Dump) an ASCII-based format described by IEEE standard 1364-2001
  • logs from a text formatted file
  • logs from OutputDebugString API (Win32 specfic)
  • logs from http(s)
  • logs from stdin
  • logs from structured CSV (TICS results)
  • logs from binary formatted file (use customizable plugin?)
  • remote log sources, such as ssh/sftp
  • remote instance of the same tool, attach/monitor it remotely
  • [windows specific] Event log !
  • [windows specific] Event Tracing for Windows (ETW) is designed for C and C++ developers who write user-mode applications but can also be used in kernel drivers
  • network messages, listening to a TCP/UDP socket, specifically from multiple systems that work together.
  • network traffic, as captured by tcp-dump

Formatting:

  • allow to edit columns, on tabs that read form files, maybe apply restrictions on OutputDebugString/Event log tabs.
  • unformating/parsing by allowing specification of column separator or fixed width (per column)
  • allow/be agnostic to (log)files without timestamps (create timestamps for such input as the input arrives)

Visualize:

  • open event source in new tab, allow to merge tabs
  • scope-view
  • listview, coloring / highlighting
  • timelines (gantt charts) with annotations
  • summaries / reports ?
  • scaling from scattered datapoints milliseconds <=> years ?

Other considerations:

  • Theming (ie. dark mode etc)
  • Internationalization / multi-language support

Outputs:

  • choose filtered / unfiltered
  • to file
  • to socket? (for transport / remote monitoring)

commercial logo?

google keywords

Telemetry is an automated communications process by which measurements and other data are collected at remote or inaccessible points and transmitted to receiving equipment for monitoring.

process mining

We almost certainly have overlap between our goals and the goals from process mining

references

print debugging

site references

TimeScale+Grafana -> grouping by seconds an lowest timeunit may be a problem?

  • Flamegraph?

free similar applications:

non-free similar applications: