Skip to content

RemoteMedia-SDK/lfm2-audio-mlx-python-source

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

lfm2-audio-mlx-python-source-plugin — LFM2.5-Audio (MLX) as a Python source plugin

Apple Silicon only. Sibling of examples/lfm2-audio-python-source-plugin/ (torch + CUDA path) — same source-load mechanism, different backend.

lfm2-audio-mlx-python-source-plugin/
├── plugin.toml                  ← metadata: language=python, mlx-audio + numpy
└── lfm2_audio_mlx_source.py     ← LFM2AudioMlxNode, vendored from in-tree

Plus the usual manifest.json / consume.py / run.sh for the smoke demo.

Why a separate plugin?

mlx-audio (Apple's MLX port of LFM2-Audio) only publishes wheels for darwin-arm64. Bundling it into the torch plugin would break uv resolve on every other platform. Keeping it in its own plugin means:

  • Linux / Windows hosts use lfm2-audio-python-source-plugin (torch + CUDA, ~5–7 GB venv).
  • macOS Apple Silicon hosts use this one (mlx-audio, ~2 GB venv).
  • A multi-platform manifest can switch via env-substitution on the plugins entry; both register the same conceptual node (just with different node_type strings).

How it loads

Same Phase 1C source-load resolver dispatch as the torch sibling. When a manifest references this directory (or a tagged GitHub release):

  1. Parses plugin.toml.
  2. Sees language = "python".
  3. Provisions a uv-managed venv from [python].requires (mlx-audio + numpy, ~2 GB).
  4. Spawns the runner with --module-root <this dir> --register-module lfm2_audio_mlx_source.
  5. Registers LFM2AudioMlxNode into the executor's registry.

Model weights download lazily on first turn (~1.95 GB for the 4-bit quant at mlx-community/LFM2.5-Audio-1.5B-4bit).

Try it

cd examples/lfm2-audio-mlx-python-source-plugin
./run.sh

The smoke consume.py generates a 1-second 220 Hz sine wave and sends it as audio (no WAV file needed). LFM2AudioMlxNode only accepts audio input — there's no text-input TTS branch like the torch sibling has, so a synthetic tone is the easiest input to manufacture. The model will produce some response (probably not real speech given the synthetic input), but the goal is verifying the plugin path works end-to-end.

For real audio I/O against this plugin, point one of the heavier examples at this manifest — for example examples/lfm2_audio_typed_rpc_chat.py with --node-id audio.

Aux ports beyond the base set

LFM2AudioMlxNode adds three control-bus aux ports that the torch sibling doesn't expose, intended for the Phase 2/4 affect-steering work:

Aux port Payload What it does
audio.in.set_steering {"target_vad": [v, a, d], "alpha": float} Channel D — bias the forward pass at model.lfm.layers[steering_layer] along the calibrated VAD axes
audio.in.clear_steering (empty) disable steering for next turn (history preserved)
audio.in.set_sampling {"temperature": float, "max_new_tokens": int} Channel A — per-turn sampling override
audio.in.set_system_augmentation {"text": str} Channel B — affect-state summary appended to the persona prompt

These only fire when steering_enabled=true in the node params AND a valid steering_directions_path NPZ exists from the Phase 5 calibration tool. See clients/python/remotemedia/nodes/ml/lfm2_audio_mlx.py for the steering wiring.

Vendoring & upstream drift

lfm2_audio_mlx_source.py is a byte-for-byte vendor of clients/python/remotemedia/nodes/ml/lfm2_audio_mlx.py with a header note and a _NODE_REGISTRY full-path alias appended. When the upstream changes:

cp clients/python/remotemedia/nodes/ml/lfm2_audio_mlx.py \
   examples/lfm2-audio-mlx-python-source-plugin/lfm2_audio_mlx_source.py
# then re-add the vendor header + registry-alias tail

External plugin authors maintaining a fork can drop the vendor note and treat lfm2_audio_mlx_source.py as their own.

Cross-references

About

LFM2.5-Audio speech-to-speech (MLX backend, Apple Silicon) — RemoteMedia SDK Path-5 Python-source plugin

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Contributors