AI-Based-Tip-and-Cue

Under development.
Expected release: March 2026

Find out more at: ESA Φ-lab Collaborative Innovation Network - AI-Based Tip and Cue

Dependencies

• Mitsuba-3: physics-based rendering engine to model radiance and geometric effects at off-nadir viewing angles.
• pySMARTS (Simple Model of the Atmospheric Radiative Transfer of Sunshine): to simulate incident solar spectral irradiance based on solar geometry, atmosphere, and surface characteristics.
• PASEOS: to simulate spacecraft subsystem states, including field of view windows, attitude pointing, and stabilization (through custom modules).
• Orekit: to propagate satellite trajectories and state vectors.

Dataset

Whales from Space Dataset : 633 patches of Very High Resolution (VHR) whale imagery from WorldView-2, -3, QuickBird-2, and GeoEye-1 satellites.

Installation

Requirements

Hardware:

• Nvidia GPU for Mitsuba rendering.

Software:

• Anaconda / Miniconda: https://www.anaconda.com/download/success
• PyCharm (or another Python IDE): https://www.jetbrains.com/pycharm/download/?section=windows
• Visual Studio 2022, including optional feature 'Desktop Development with C++' (not 2026, due to incompatibility with CUDA): https://www.junian.net/dev/visual-studio-community-download-links/
• CUDA Toolkit: https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64
• Git: https://git-scm.com/install/windows

Create Environment

From Git Bash, clone the AI-Based Tip and Cue repository.

git clone https://github.com/ESA-PhiLab/AI-Based-Tip-and-Cue.git

From Anaconda prompt, create your Conda environment.

cd AI-Based-Tip-and-Cue
conda env create -f environment.yml
conda activate tipandcue

Fix rasterio installation manually.

conda install -c conda-forge --yes gdal rasterio pyproj proj proj-data 

Launch PyCharm.

pycharm64

Add your new environment:

Add new interpreter -> Add local interpreter -> Select existing -> Conda -> Path to Conda: C:\Users\*username*\miniconda3\Scripts\conda.exe , Environment: C:\Users\*username*\miniconda3\envs\tipandcue

Install PASEOS

From Git Bash, inside the AI-Based-Tip-and-Cue directory, clone the PASEOS repository and merge with the custom PASEOS packages.

cd AI-Based-Tip-and-Cue
git clone https://github.com/aidotse/PASEOS.git PASEOS_tmp
cp -r PASEOS_tmp/* PASEOS/
cp -r PASEOS_tmp/.* PASEOS/ 2>/dev/null
rm -rf PASEOS_tmp

From Anaconda prompt, install additional dependencies.

cd PASEOS
pip install -e . --no-deps

Install Orekit

1. From https://jdk.java.net/25/ (or a newer version), download the Windows/x64 .zip file.
2. Extract the zip file to: C:\Program Files\Java\jdk-25.
3. Edit Windows Environment variables: (search -> Environment variables -> Edit the system environment variables)

System variables:
Name: JAVA_HOME
Value: C:\Program Files\Java\jdk-25*

User variables: Edit Path -> New -> %JAVA_HOME%\bin

Click Ok to complete.

From Command prompt, verify the the installation.

java --version

From Anaconda prompt, install Orekit and download the Orekit data.

conda install -c conda-forge orekit

pip install git+https://gitlab.orekit.org/orekit/orekit-data.git

Complete Mitsuba installation

Install LLVM 18.1 (no newer version, otherwise it crashes with DrJit) from: https://github.com/llvm/llvm-project/releases?page=4. Expand Assets, then download + execute LLVM-18.1.6-win64.exe file.

Add Environment variable:
System variables:
Name: DRJIT_LIBLLVM_PATH
Value: C:\Program Files\LLVM\bin\LLVM-C.dll

Install pySMARTS

Download SMARTS software from: https://www.nrel.gov/grid/solar-resource/smarts

Extract the .zip and place it in: C:\Program Files\SMARTS_295_PC

Add Environment variable:
System variables:
Name: SMARTSPATH
Value: C:\Program Files\SMARTS_295_PC

Installation Troubleshooting

Mitsuba

On some PCs, 'import mitsuba' crashes with a silent error after run, error code (0xC0000005). If this happens, try the following.

• !! First check your LLVM version. It should be lower than LLVM 18.1, otherwise it is imcompatible with DrJit. To install, obtain the LLVM-18.1.6-win64.exe file from https://github.com/llvm/llvm-project/releases?page=4 and execute.
• Re-install Microsoft VS Code redistributables (both x64 and x86): https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist
• (Re-)install Visual Studio Code: https://visualstudio.microsoft.com/downloads/
• (Re-)install Visual Studio 2022: https://www.junian.net/dev/visual-studio-community-download-links/ . Enable 'Desktop Development with C++' and thick all optional feature boxes.
• (Re-)install CUDA Toolkit: https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64

Dataset

Request the Whales from Space dataset at: https://data.bas.ac.uk/full-record.php?id=GB/NERC/BAS/PDC/01592

Place the obtained data in dataset/whales_from_space with the following structure:

dataset/
└── whales_from_space/
    ├── Auckland2006/
    ├── Auckland2011/
    ├── Ignacio2017/
    ├── Maui2015/
    ├── Pelagos2016/
    ├── Valdes2012/
    ├── Valdes2014/
    ├── Valdes2016/
    ├── Witsand2009/
    └── WhaleFromSpaceDB_Whales.csv

Verification

Run the file: run_simulation.py

Please note that the simulation framework is still under development. The code and the results have not yet been verified.
Full release is expected around March 2026.

Usage

to be completed.

Contact

Nadine Duursma, N.A.Duursma at outlook.com

Reference

@software{AI-Based-Tip-and-Cue,
  title        = {AI-Based-Tip-and-Cue Simulation Framework},
  author       = {Duursma, Nadine Anje},
  organization = {{European Space Agency (ESA) Phi-Lab}},
  version      = {0.1.1},
  year         = {2025},
  url          = {https://github.com/ESA-PhiLab/AI-Based-Tip-and-Cue}
}