Boyuc/observatory draft demo

backends/qualcomm/debugger/README.md

@@ -90,6 +90,38 @@ Note: Files ending with `.bin ` do not support graph visualization in qairt_visu
 For more details, visit the [QAIRT Visualizer](https://pypi.org/project/qairt-visualizer/).
 
 
+# Observatory
+
+A new, review-focused Observatory implementation is available under:
+
+`backends/qualcomm/debugger/observatory`
+
+Use the Observatory CLI to wrap any Qualcomm AOT export script. Use `--lens_recipe=accuracy`
+to enable accuracy lenses.
+
+```bash
+python -m executorch.backends.qualcomm.debugger.observatory \
+    --output-html obs_report.html \
+    --lens_recipe=accuracy \
+    {original script and args}
+```
+
+For example:
+
+```bash
+python -m executorch.backends.qualcomm.debugger.observatory \
+    --output-html obs_report.html \
+    --lens_recipe=accuracy \
+    examples/qualcomm/oss_scripts/mobilevit_v2.py \
+    --backend htp --model SM8650 -d ./imagenet-mini-val/ -b build-android/ --compile_only
+```
+
+> **Note**: Qualcomm example scripts (e.g. `oss_scripts/roberta.py`) use only absolute imports
+> and are run as plain scripts. The Observatory CLI auto-selects `runpy.run_path` for these since
+> their directories do not contain `__init__.py`.
+
+See `backends/qualcomm/debugger/observatory/README.md` for full documentation.
+
 # ExecuTorch QNN Intermediate Output Debugger
 
 ExecuTorch QNN Intermediate Output Debugger is a tool that helps users debug intermediate output accuracy by comparing CPU outputs with QNN outputs. This tool offers a variety of output formats and flexibility for users to define their own metrics when debugging.

backends/qualcomm/debugger/observatory/README.md

@@ -0,0 +1,146 @@
+# Qualcomm Observatory CLI
+
+Qualcomm-specific Observatory CLI that wraps `devtools/observatory` with QNN backend patches and
+accuracy lenses. Requires a QNN SDK environment (source `$QNN_SDK_ROOT/bin/envsetup.sh` before
+running on-device jobs).
+
+## Usage
+
+### Collection mode (default)
+
+```bash
+python -m executorch.backends.qualcomm.debugger.observatory \
+    [--output-html PATH] [--output-json PATH] SCRIPT [SCRIPT_ARGS...]
+```
+
+### With accuracy debugging
+
+```bash
+python -m executorch.backends.qualcomm.debugger.observatory \
+    --lens_recipe=accuracy \
+    [--output-html PATH] [--output-json PATH] \
+    SCRIPT [SCRIPT_ARGS...]
+```
+
+### Visualize mode (JSON → HTML, no re-execution)
+
+```bash
+python -m executorch.backends.qualcomm.debugger.observatory visualize \
+    --input-json report.json --output-html report.html
+```
+
+## Qualcomm examples
+
+Qualcomm example scripts use only absolute imports and live in directories without `__init__.py`,
+so the Observatory CLI runs them as plain scripts via `runpy.run_path` (no special invocation
+needed).
+
+### Vision model (ImageNet)
+
+```bash
+source $QNN_SDK_ROOT/bin/envsetup.sh
+
+python -m executorch.backends.qualcomm.debugger.observatory \
+    --output-html /tmp/obs_vit/report.html \
+    --output-json /tmp/obs_vit/report.json \
+    --lens_recipe=accuracy \
+    examples/qualcomm/scripts/torchvision_vit.py \
+    -m SM8650 -b ./build-android \
+    --dataset imagenet-mini-val/ \
+    -H mlgtw-linux -s <device_serial> \
+    -a /tmp/obs_vit --seed 1126 --compile_only
+```
+
+### NLP model (Wikipedia sentences)
+
+```bash
+python -m executorch.backends.qualcomm.debugger.observatory \
+    --output-html /tmp/obs_roberta/report.html \
+    --lens_recipe=accuracy \
+    examples/qualcomm/oss_scripts/roberta.py \
+    -m SM8650 -b ./build-android \
+    -H mlgtw-linux -s <device_serial> \
+    -a /tmp/obs_roberta --compile_only
+```
+
+### Compile-only (no device required)
+
+Add `--compile_only` to any Qualcomm script to export and lower without pushing to device.
+This is useful for inspecting the compilation pipeline in CI or on a dev machine.
+
+## Available example scripts
+
+### `examples/qualcomm/scripts/` — vision models
+
+| Script | Model |
+|---|---|
+| `torchvision_vit.py` | Vision Transformer |
+| `mobilenet_v2.py` | MobileNetV2 |
+| `mobilenet_v3.py` | MobileNetV3 |
+| `inception_v3.py` | InceptionV3 |
+| `inception_v4.py` | InceptionV4 |
+
+Dataset: ImageNet (pass with `--dataset <path>` or `-d <path>`).
+
+### `examples/qualcomm/oss_scripts/` — NLP/open-source models
+
+| Script | Model |
+|---|---|
+| `roberta.py` | RoBERTa |
+| `bert.py` | BERT |
+| `albert.py` | ALBERT |
+| `distilbert.py` | DistilBERT |
+| `eurobert.py` | EuroBERT |
+
+Dataset: Wikipedia sentences (`wikisent2.txt`). Pass with `-d <path>`.
+
+Common flags: `-m <SOC_MODEL>` (e.g. `SM8650`), `-b <build_folder>`, `-H <host>`,
+`-s <device_serial>`, `-a <artifact_dir>`, `--compile_only`.
+
+## Accuracy lenses (`--lens_recipe=accuracy`)
+
+Registers `AccuracyLens` and `PerLayerAccuracyLens` (with QNN dataset patches) on top of the
+default `PipelineGraphCollectorLens`. These produce:
+
+- Per-stage accuracy metrics (PSNR, cosine similarity, MSE, top-k)
+- Per-layer accuracy heat-map overlaid on the graph
+- Cross-stage diff labels in the left panel of the HTML report
+
+QNN dataset patches (`lenses/qnn_dataset_patches.py`) wire the on-device inference output back
+into the accuracy lens so metrics reflect true QNN outputs, not emulated CPU results.
+
+## Two-step workflow
+
+Collect on-device in CI, visualize locally without re-running:
+
+```bash
+# Step 1 — collect (e.g., in CI with device attached)
+python -m executorch.backends.qualcomm.debugger.observatory \
+    --output-html /tmp/obs/report.html \
+    --output-json /tmp/obs/report.json \
+    examples/qualcomm/scripts/torchvision_vit.py \
+    -m SM8650 -b ./build-android -d imagenet-mini-val/ \
+    -H mlgtw-linux -s <device_serial> -a /tmp/obs
+
+# Step 2 — re-generate HTML from JSON (e.g., locally after lens update)
+python -m executorch.backends.qualcomm.debugger.observatory visualize \
+    --input-json /tmp/obs/report.json \
+    --output-html /tmp/obs/report_v2.html
+```
+
+## Backend patches
+
+`lenses/qnn_patches.py` installs a monkey-patch on `ptq_calibrate` so the
+`PipelineGraphCollectorLens` can intercept the QNN quantization calibration stage and capture
+the graph at that point. The patch is active only while the Observatory context is open.
+
+`lenses/qnn_dataset_patches.py` wires on-device inference results into `AccuracyLens` so that
+accuracy metrics use real QNN outputs.
+
+## See also
+
+- `backends/qualcomm/debugger/README.md` — broader Qualcomm debugger overview (QAIRT visualizer,
+  intermediate output debugger)
+- `devtools/observatory/README.md` — framework overview, Python API, custom lens guide
+- `devtools/observatory/USAGE.md` — full CLI reference
+- `devtools/observatory/lenses/LENSES.md` — built-in lens details

backends/qualcomm/debugger/observatory/__init__.py

@@ -0,0 +1,5 @@
+# Copyright (c) Qualcomm Innovation Center, Inc.
+# All rights reserved
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.

backends/qualcomm/debugger/observatory/__main__.py

@@ -0,0 +1,9 @@
+# Copyright (c) Qualcomm Innovation Center, Inc.
+# All rights reserved
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+from .cli import main
+
+main()

backends/qualcomm/debugger/observatory/cli.py

@@ -0,0 +1,81 @@
+# Copyright (c) Qualcomm Innovation Center, Inc.
+# All rights reserved
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Qualcomm Observatory CLI -- QNN-specific lens configuration.
+
+Collection mode (default):
+    python -m executorch.backends.qualcomm.debugger.observatory \\
+        [--output-html PATH] [--output-json PATH] SCRIPT [SCRIPT_ARGS...]
+
+With accuracy debugging:
+    python -m executorch.backends.qualcomm.debugger.observatory \\
+        --lens_recipe=accuracy SCRIPT [SCRIPT_ARGS...]
+
+Visualize mode (JSON -> HTML):
+    python -m executorch.backends.qualcomm.debugger.observatory visualize \\
+        --input-json report.json --output-html report.html
+"""
+
+from __future__ import annotations
+
+import sys
+
+
+def main():
+    from executorch.devtools.observatory.cli import (
+        make_collect_parser,
+        make_visualize_parser,
+        run_observatory,
+        run_visualize,
+    )
+
+    if len(sys.argv) > 1 and sys.argv[1] == "visualize":
+        parser = make_visualize_parser()
+        args = parser.parse_args(sys.argv[2:])
+        run_visualize(args.input_json, args.output_html)
+        return
+
+    parser = make_collect_parser(
+        prog="python -m executorch.backends.qualcomm.debugger.observatory"
+    )
+    parser.add_argument(
+        "--lens_recipe",
+        choices=["accuracy"],
+        default=None,
+        help="Lens recipe to enable (e.g. accuracy)",
+    )
+    args = parser.parse_args(sys.argv[1:])
+
+    from executorch.devtools.observatory.observatory import Observatory
+    from executorch.devtools.observatory.lenses.pipeline_graph_collector import (
+        PipelineGraphCollectorLens,
+    )
+    from .lenses.qnn_patches import install_qnn_patches
+
+    Observatory.clear()
+    PipelineGraphCollectorLens.register_backend_patches(install_qnn_patches)
+    Observatory.register_lens(PipelineGraphCollectorLens)
+
+    if args.lens_recipe == "accuracy":
+        from executorch.devtools.observatory.lenses.accuracy import AccuracyLens
+        from .lenses.qnn_dataset_patches import install_qnn_dataset_patches
+
+        AccuracyLens.register_dataset_patches(install_qnn_dataset_patches)
+        Observatory.register_lens(AccuracyLens)
+
+        from executorch.devtools.observatory.lenses.per_layer_accuracy import (
+            PerLayerAccuracyLens,
+        )
+
+        Observatory.register_lens(PerLayerAccuracyLens)
+
+    run_observatory(
+        args.script, args.script_args, Observatory, args.output_html, args.output_json
+    )
+
+
+if __name__ == "__main__":
+    main()

backends/qualcomm/debugger/observatory/lenses/__init__.py

@@ -0,0 +1,5 @@
+# Copyright (c) Qualcomm Innovation Center, Inc.
+# All rights reserved
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.

backends/qualcomm/debugger/observatory/lenses/qnn_dataset_patches.py

@@ -0,0 +1,75 @@
+# Copyright (c) Qualcomm Innovation Center, Inc.
+# All rights reserved
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""QNN dataset patches for AccuracyLens.
+
+Installs monkey-patches on executorch.examples.qualcomm.utils dataset functions
+to capture targets and task type for accuracy evaluation.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from executorch.devtools.observatory.lenses.accuracy import AccuracyLens
+
+
+def install_qnn_dataset_patches(cls: type[AccuracyLens]) -> None:
+    """Install QNN dataset capture patches on AccuracyLens."""
+    try:
+        import executorch.examples.qualcomm.utils as utils_module
+
+        if hasattr(utils_module, "get_imagenet_dataset"):
+            original = utils_module.get_imagenet_dataset
+            cls._originals["get_imagenet_dataset"] = original
+
+            def patched_imagenet(*args, **kwargs):
+                inputs, targets = original(*args, **kwargs)
+                cls._captured_targets = targets
+                cls._task_type = "classification"
+                logging.info(
+                    "[AccuracyLens] Captured ImageNet targets (%d samples)",
+                    len(targets),
+                )
+                return inputs, targets
+
+            utils_module.get_imagenet_dataset = patched_imagenet
+            logging.info("[AccuracyLens] Installed patch: get_imagenet_dataset")
+
+        if hasattr(utils_module, "get_masked_language_model_dataset"):
+            original_mlm = utils_module.get_masked_language_model_dataset
+            cls._originals["get_masked_language_model_dataset"] = original_mlm
+
+            def patched_mlm(*args, **kwargs):
+                inputs, targets = original_mlm(*args, **kwargs)
+                cls._captured_targets = targets
+                cls._task_type = "mlm"
+                logging.info(
+                    "[AccuracyLens] Captured MLM targets (%d samples)",
+                    len(targets),
+                )
+                return inputs, targets
+
+            utils_module.get_masked_language_model_dataset = patched_mlm
+            logging.info(
+                "[AccuracyLens] Installed patch: get_masked_language_model_dataset"
+            )
+
+        def _uninstall():
+            try:
+                for key, orig in cls._originals.items():
+                    if hasattr(utils_module, key):
+                        setattr(utils_module, key, orig)
+            except Exception:
+                pass
+
+        cls._dataset_uninstallers.append(_uninstall)
+    except ImportError:
+        logging.debug(
+            "[AccuracyLens] qualcomm utils not available, skipping dataset patches"
+        )