[15] Usability enhancements to CONTROL-CORE

[GaneshPatil7517]

Hello @pradeeban Sir, @Rahuljagwani Sir, and @mayureshkothare Sir,
While drafting my proposal for [15] Usability Enhancements to CONTROL-CORE, I came across a few questions and would really appreciate your guidance.

Q1.The current configuration is scattered across four separate files (concore.tools, concore.octave, concore.mcr, concore.sudo). Would you be open to consolidating these into a single TOML/YAML configuration file (e.g., concore.toml) while keeping backward compatibility with the old files? Or is the separation intentional for modularity?

Q2.I noticed that mkconcore.py generates separate shell scripts for Windows (start /B, taskkill), POSIX (xterm -e, kill), and Docker (docker run/stop). Currently, users must choose one target at generation time with -t. Would it be valuable to generate a unified cross-platform runner (e.g., using Python's subprocess module) so that a single generated study can run on any OS without regeneration?

Q3.The CLI's validate command already performs 12 checks including shell injection prevention and ZMQ port conflict detection. For the usability enhancement, should I focus on making validation part of a pre-flight check before concore run, or should validation and running remain decoupled?

Q4. I see that Phase 1 protocol conformance tests exist in tests/test_protocol_conformance.py covering Python's parse_params, initval, and write_zmq. Phase 2 is planned for C++, Java, and MATLAB cross-language validation. Is completing Phase 2 conformance testing in scope for this usability project, or should I focus purely on installation/deployment UX?

Q5.The wire format uses Python ast.literal_eval() syntax. The C++ parser in concore.hpp and Java parser in TestLiteralEval.java re-implement this independently. Have there been interoperability bugs between language implementations? I'm thinking about whether a formal wire format specification document would help prevent future drift.

Q6.The Docker workflow currently generates individual build/run/stop scripts but doesn't produce a docker-compose.yml. Since concore studies often have 3-5+ nodes running concurrently with shared volumes, would generating a Docker Compose file be a high-value usability improvement? This would also naturally pave the way for Kubernetes deployment via kompose.

Q7.I noticed Dockerfile.py installs g++ and libgl1-mesa-glx even though Python-only nodes don't need them. Dockerfile.m uses mtmiller/octave as the base image. Should each study node get a minimal per-language Dockerfile, or is the current 'fat image' approach preferred for simplicity? This has implications for image size and CI/CD build times.

Q8.The DOCKEREXE variable defaults to docker but previous versions defaulted to sudo docker. I see the three-level override (config file > env var > default). For platforms like macOS/Windows with Docker Desktop, Podman, or Rancher Desktop, should the installer auto-detect the container runtime?

Q9. For the installer, I'm considering three approaches: (a) a Python-based concore setup wizard that auto-detects compilers/runtimes and writes config files, (b) platform-native installers (MSI for Windows, .deb/.rpm for Linux, Homebrew formula for macOS), or (c) a container-first approach where docker run concore/studio gives you everything. Which direction aligns best with your users' needs — researchers running locally, or teams deploying on clusters?

Q10.The concore.tools config requires users to manually find paths for g++, iverilog, python3, octave, etc. Tools like shutil.which() in Python can auto-detect these on PATH. Should the concore init command include a --detect-tools phase that scans the system and writes concore.tools automatically?

Q11.Given the 350-hour budget, which is roughly 12 weeks full-time, I'd like to prioritize. From your perspective, which of these areas would deliver the most impact: (a) installation automation, (b) Docker Compose + container-first workflow, (c) cross-platform unified runner, or (d) documentation and guided tutorials? I want to make sure Week 1-4 tackles the highest-value items."

Q12.The concore-editor (DHGWorkflow fork) is a ReactJS frontend. Should usability enhancements include improvements to the editor's setup process (e.g., a single concore editor CLI command to launch it), or should I keep focus strictly on the backend protocol and CLI?

[GaneshPatil7517]

What I Understand So Far
CONTROL-CORE is a decentralized, broker-less simulation framework for closed-loop neuromodulation control systems. Unlike traditional DAG-based workflow tools (CWL, WDL), concore uniquely supports dicycles (feedback loops), which are essential for modeling neural control systems.

The core protocol uses a file-based data-sharing mechanism with a wire format based on Python's ast.literal_eval() syntax, and it supports development in Python, C++, MATLAB/Octave, Verilog, and Java. An optional ZMQ transport layer enables distributed execution. The workflow generator (mkconcore.py) parses GraphML files from the concore Editor (DHGWorkflow) and produces platform-specific scripts for Windows, POSIX, and Docker environments.

The CLI (concore_cli) already offers 7 commands - init, run, validate, inspect, status, stop, and watch - with comprehensive validation (12 checks including shell injection detection and ZMQ port conflict analysis). The Docker workflow generates per-node containers with shared-volume communication.

However, the multi-platform and multi-language nature of the project introduces friction: configuration is scattered across four separate files (concore.tools, concore.octave, concore.mcr, concore.sudo), tool paths must be set manually, platform-specific scripts must be regenerated when switching between Windows/POSIX/Docker, and the Docker workflow lacks modern orchestration (e.g., Docker Compose).

Proposed Approach: "Zero-to-Study in Minutes"
Goal: A new user should go from pip install concore to running their first study in under 5 minutes, on any platform.

Phase 1 - Installation & Configuration Automation (Weeks 1–4)
1. Unified Configuration: Consolidate the four separate config files into a single concore.toml, with a backward-compatibility layer that still reads the old files.
2. Auto-detection Wizard: A new concore setup command that scans the system for installed compilers/runtimes (g++, python3, iverilog, octave, Docker) using shutil.which() and writes the configuration automatically.
3. Platform Installers: A Homebrew formula for macOS, an install script for Linux, and a PowerShell installer for Windows - each installs concore and triggers auto-detection.
4. Actionable Error Messages: When a tool is missing (e.g., iverilog not found), display platform-specific installation instructions instead of a generic error.

Phase 2 - Container-First Workflow (Weeks 5–8)
1. Docker Compose Generation: Extend concore run --target docker to produce a docker-compose.yml with proper service definitions, shared volumes, and dependency ordering - replacing the current separate shell scripts.
2. Minimal Per-Language Dockerfiles: Slim images for each language (Python ~150 MB, C++ ~200 MB) vs. the current larger images that bundle unnecessary packages.
3. Container Runtime Auto-detection: Detect Docker Desktop, Podman, or Rancher Desktop and configure DOCKEREXE automatically.
4. One-Command Studio: docker run -p 8080:8080 concore/studio - a pre-built image containing the full development environment.
5. CI/CD Templates: GitHub Actions workflow for running concore studies in CI.

Phase 3 - Cross-Platform Runner & UX Polish (Weeks 9–12)
1. Unified Python Runner: A Python-based runner (using subprocess) that replaces platform-specific shell scripts, so a single generated study runs on Windows, macOS, and Linux without regeneration.
2. Pre-flight Validation: Automatic concore validate before every concore run, with actionable hints.
3. Interactive Init Wizard: concore init asks the user for language, number of nodes, and generates a working starter study.
4. Watch Improvements: Configurable pagination, CSV export, and an optional web-based dashboard.
5. Cross-Language Smoke Tests: Phase 2 protocol conformance - C++ and MATLAB nodes communicating with Python nodes.
6. Documentation: "Your First Study" tutorials for each platform (Windows, macOS, Ubuntu, Docker), plus a troubleshooting FAQ.

Why This Approach
1. Root-cause focused: Systematically removes every friction point in the new-user journey.
2. Incrementally valuable: Each phase delivers standalone improvements. Even if Phase 3 isn't 100% complete, Phases 1–2 alone transform the onboarding experience.
3. Fully backward-compatible: The unified config reads old files, Docker Compose is an addition (not a replacement), and the Python runner is opt-in.
4. Measurable: Success = time from pip install concore to first running study, benchmarked on Windows/macOS/Ubuntu. Target: < 5 minutes.

Given my existing deep involvement in the project, I'm confident I can hit the ground running from Day 1 and deliver meaningful usability improvements across all target platforms.
I look forward to your feedback on this approach.

[pradeeban]

Sounds good, @GaneshPatil7517.

[pradeeban]

@GaneshPatil7517 While making the usability enhancements, we should refrain from excessive changes to the backend logic or "old" files as that is a mere distraction. Bug fixes are great. But refactoring maybe not. Just to be careful.

We can figure out the priorities eventually. You can list your approach in your proposal and we will comment if something is listed out of order. Perhaps some of these enhancements should be an "if time permits."

[GaneshPatil7517]

@GaneshPatil7517 While making the usability enhancements, we should refrain from excessive changes to the backend logic or "old" files as that is a mere distraction. Bug fixes are great. But refactoring maybe not. Just to be careful.

We can figure out the priorities eventually. You can list your approach in your proposal and we will comment if something is listed out of order. Perhaps some of these enhancements should be an "if time permits."

Thank you for the guidance, @pradeeban Sir
I've restructured my proposal based on your feedback. The key changes:

1. All core deliverables (D1–D7) are purely additive - new files and new CLI commands only. No modifications to mkconcore.py, concore.py, concore_base.py, or any existing backend logic.

2. Bug fixes only for old files - for example, I just fixed a Windows UnicodeEncodeError in the concore doctor output (commit 888f143), but the fix was in the new doctor.py file itself, not in any old code.

3. Anything touching backend logic is now a stretch goal (S1–S3), explicitly gated behind "if time permits" and mentor approval:

• S1: Unified concore.toml config (would need to modify mkconcore.py)
• S2: Cross-platform Python runner (would replace shell/batch generation)
• S3: Pre-flight validation

4. The concore setup wizard writes the existing config files (concore.tools, concore.octave, etc.) that mkconcore.py already reads - so zero backend change is needed.

I'll list the full priority order in the proposal so you can flag anything that's out of order. The timeline is: Weeks 1–4 doctor+setup, Weeks 5–8 Docker Compose+slim images, Weeks 9–10 init wizard+docs, Weeks 11–12 stretch goals only if core is merged.

The concore doctor command is already implemented and pushed as PR #495 - 22 tests passing, CI green. Happy to iterate on it based on your review.

[pradeeban]

Sounds good. Also, if a new feature is implemented, we could create new test cases. No need to change all the previous examples to demonstrate the new feature. That will be a waste of effort. At some point, we could always document - which of these samples are for the new changes. Something to consider for the main GSoC coding period. Not now.

[GaneshPatil7517]

hello @pradeeban Sir i have one question:
When concore compose generates docker-compose.yml, should the containers communicate through shared Docker volumes (mimicking the current file-based edge protocol), or should we use Docker networking with the ZMQ layer that already exists in 0mq/? Shared volumes are simpler and keep backward compatibility, but ZMQ over the network would make the Docker workflow truly distributed -which seems like where CONTROL-CORE is heading

[GaneshPatil7517]

That's a fair point My current thinking is to support both modes start with shared Docker volumes by default (since that maps directly to the existing file-based edge protocol and requires zero changes to how nodes communicate), and then offer a --network flag that uses the ZMQ layer from 0mq/ for truly distributed setups. That way concore compose works out of the box for existing studies, and users who need multi-host execution can opt into ZMQ networking. Would love to hear @pradeeban's & @Rahuljagwani take on this approach.

[pradeeban]

I don't think we should overthink the backward compatibility of the past use cases. We just need to create some improved/ported use cases and document the new ones vs old ones (during the GSoC coding period, that is. No break changes at this point). We do not have to limit ourselves too much for backward compatibility and we should not spend too much time porting old use cases/examples either.

[GaneshPatil7517]

Thank you for clarifying, @pradeeban That makes sense I was being overly conservative with the "don't touch anything" approach. I'll focus on building the best possible new workflows (shared volumes by default, ZMQ --network flag for distributed) and document how they compare to the existing Docker scripts. I won't break existing behavior, but I also won't constrain myself to only wrapping around the old logic. Will update my proposal to reflect this.

[pradeeban]

No, you were right about not touching old stuff, such as ratc, testsou, etc. Not because we do not want to break them (even if you break, we can always go back to previous tags and releases). But because we cannot spend too much time fixing research products that are already published.

[GaneshPatil7517]

That makes perfect sense, @pradeeban. The rationale is clear time is better spent building improved tooling than fixing published research outputs that already serve their purpose. My proposal focuses entirely on new CLI commands (setup, compose, init) and documentation, leaving ratc/, testsou/, and other published studies untouched. Thanks for the clarification.......