Can not Integrate LocalAI with ultraworkers/claw-code

[1511563371]

LocalAI version: Version: v4.3.6 (aee4611)

Environment, CPU architecture, OS, and Version:
zc@zc-MS-7E13:~/LocalAI$ uname -a
Linux zc-MS-7E13 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug
I wanna integrate LocalAI with ultraworkers/claw-code.
However, there was an issue of being unable to find the model when using it.

My operation
1 model location

zc@zc-MS-7E13:~/LocalAI/models$ tree .
.
├── DeepSeek-R1-Distill-Qwen-1.5B-BF16.yaml
└── local
    └── DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf

2 yaml configuration

zc@zc-MS-7E13:~/LocalAI/models$ cat DeepSeek-R1-Distill-Qwen-1.5B-BF16.yaml 
name: DeepSeek-R1-Distill-Qwen-1.5B-BF16
parameters: 
  model: local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf # 匹配模型文件名
context_size: 8192
threads: 8
backend: llama-cpp
preload: true

3 run LocalAI

zc@zc-MS-7E13:~/LocalAI$ ./localai run --models-path="./models" --address localhost:8080

4 run ultraworkers/claw-code

export ANTHROPIC_BASE_URL=http://127.0.0.1:8080/
export ANTHROPIC_API_KEY=anthropic

or
export OPENAI_BASE_URL="http://localhost:8080/v1"
export OPENAI_API_KEY="localai"

./claw --model local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf

5 problems
LocalAI try to Downloading url="https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf" rather than directly Using local/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf

Jun 03 08:03:23 INFO  HTTP request method="POST" path="/v1/chat/completions" status=200 
Jun 03 08:04:08 INFO  Downloading url="https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf" 
Jun 03 08:04:38 ERROR failed to install model from gallery error=failed to download file "/home/zc/LocalAI/models/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf": Get "https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf": dial tcp [2a03:2880:f117:83:face:b00c:0:25de]:443: i/o timeout model="local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf" 
Jun 03 08:04:38 INFO  Trying to load the model modelID="Deepseek-R1-Distill-Qwen-1.5B" backends=[cpu-llama-cpp llama-cpp] 
Jun 03 08:04:38 INFO  Attempting to load backend="cpu-llama-cpp" 
Jun 03 08:04:38 INFO  BackendLoader starting modelID="Deepseek-R1-Distill-Qwen-1.5B" backend="cpu-llama-cpp" model="local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf" 
Jun 03 08:04:42 INFO  Loads OK backend="cpu-llama-cpp" 
Jun 03 08:05:09 INFO  HTTP request method="POST" path="/v1/chat/completions" status=200 

[localai-bot]

Thanks for the detailed report. This is a configuration mismatch, but LocalAI's reaction to it was confusing, and that part is now being fixed.

Root cause

The model you pass in an API request must match the top-level name in your YAML, not the parameters.model weight-file path. Your config is:

name: DeepSeek-R1-Distill-Qwen-1.5B-BF16          # <- the API model id
parameters:
  model: local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf   # <- just the weights path

But you launched claw-code with the weights path:

./claw --model local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf

So LocalAI never matched your configured model. Use the name instead:

./claw --model DeepSeek-R1-Distill-Qwen-1.5B-BF16

(You can confirm the exact id any time with GET /v1/models.)

Why it tried to download

Two behaviors combined to turn "wrong name" into a HuggingFace download:

1. AUTOLOAD_GALLERIES defaults to true, so an unknown model triggers a gallery install attempt. You can disable it with --autoload-galleries=false (env LOCALAI_AUTOLOAD_GALLERIES=false).
2. The "model not found" guard was skipped for any name containing / (intended for org/repo IDs that backends like diffusers download on the fly). Because local/...gguf also contains a /, it bypassed the guard and silently fell through to the autoloader instead of returning a clear error.

Fix

We've tightened that guard so it only treats a /-containing name as a remote org/repo ID when it does not end in a recognized weight-file extension. A name pointing at a concrete file (e.g. local/model.gguf) is now verified like any other: if it doesn't exist you get a clean 404 model "..." not found instead of a surprise download. Legitimate HuggingFace IDs (and loose weight files that genuinely exist by relative path) are unaffected.

For your immediate setup, just switching --model to DeepSeek-R1-Distill-Qwen-1.5B-BF16 should resolve it.

[1511563371]

zc@zc-MS-7E13:~/agent/claw-code-main/rust/target/debug$ ./claw --model DeepSeek-R1-Distill-Qwen-1.5B-BF16 
[error-kind: invalid_model_syntax]
error: invalid model syntax: 'DeepSeek-R1-Distill-Qwen-1.5B-BF16'.
Expected provider/model (e.g., anthropic/claude-opus-4-7)

Run `claw --help` for usage.

ultraworkers/claw-code except provider/model (e.g., anthropic/claude-opus-4-7)，

Note:./localai run --autoload-galleries="false" --models-path="./models" --address="127.0.0.1:8080"

Result1

zc@zc-MS-7E13:~/agent/claw-code-main/rust/target/debug$ ./claw --model local/DeepSeek-R1-Distill-Qwen-1.5B-BF16
██████╗██╗      █████╗ ██╗    ██╗
██╔════╝██║     ██╔══██╗██║    ██║
██║     ██║     ███████║██║ █╗ ██║
██║     ██║     ██╔══██║██║███╗██║
╚██████╗███████╗██║  ██║╚███╔███╔╝
╚═════╝╚══════╝╚═╝  ╚═╝ ╚══╝╚══╝ Code 🦞

Model            local/DeepSeek-R1-Distill-Qwen-1.5B-BF16
Permissions      danger-full-access
Branch           unknown
Workspace        clean
Directory        /home/zc/agent/claw-code-main/rust/target/debug
Session          session-1780574602978-0
Auto-save        .claw/sessions/7bcdecb4f9f43786/session-1780574602978-0.jsonl

Type /help for commands · /status for live context · /resume latest jumps back to the newest session · /diff then /commit to ship · Tab for workflow completions · Shift+Enter for newline
Connected: local/DeepSeek-R1-Distill-Qwen-1.5B-BF16 via anthropic
> hello ,what is your name?
✘ ❌ Request failed
[error-kind: unknown]
error: failed to parse Anthropic response for model local/DeepSeek-R1-Distill-Qwen-1.5B-BF16: unknown variant `error`, expected one of `message_start`, `message_delta`, `content_block_start`, `content_block_delta`, `content_block_stop`, `message_stop` at line 1 column 15; first 200 chars of body: {"type":"error","error":{"type":"api_error","message":"model inference failed: could not load model - all backends returned error: [cpu-llama-cpp]: failed to load model with internal loader: could not…

Run `claw --help` for usage.
zc@zc-MS-7E13:~/agent/claw-code-main/rust/target/debug$ 

AND LocalAI Log is:

Jun 04 20:03:42 ERROR Failed to load model modelID="local/DeepSeek-R1-Distill-Qwen-1.5B-BF16" error=failed to load model with internal loader: could not load model: rpc error: code = Internal desc = Failed to load model: /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16. Error: gguf_init_from_file: failed to open GGUF file '/home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16' (No such file or directory); llama_model_load: error loading model: llama_model_loader: failed to load model from /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16; llama_model_load_from_file_impl: failed to load model; gguf_init_from_file: failed to open GGUF file '/home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16' (No such file or directory); llama_model_load: error loading model: llama_model_loader: failed to load model from /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16; llama_model_load_from_file_impl: failed to load model backend="llama-cpp" 
Jun 04 20:03:42 INFO  Fails backend="llama-cpp" error="failed to load model with internal loader: could not load model: rpc error: code = Internal desc = Failed to load model: /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16. Error: gguf_init_from_file: failed to open GGUF file '/home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16' (No such file or directory); llama_model_load: error loading model: llama_model_loader: failed to load model from /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16; llama_model_load_from_file_impl: failed to load model; gguf_init_from_file: failed to open GGUF file '/home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16' (No such file or directory); llama_model_load: error loading model: llama_model_loader: failed to load model from /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16; llama_model_load_from_file_impl: failed to load model" 
Jun 04 20:03:42 ERROR Anthropic stream model inference failed error=could not load model - all backends returned error: [cpu-llama-cpp]: failed to load model with internal loader: could not load model: rpc error: code = Internal desc = Failed to load model: /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16. Error: gguf_init_from_file: failed to open GGUF file '/home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16' (No such file or directory); llama_model_load: error loading model: llama_model_loader: failed to load model from /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16; llama_model_load_from_file_impl: failed to load model; gguf_init_from_file: failed to open GGUF file '/home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16' (No such file or directory); llama_model_load: error loading model: llama_model_loader: failed to load model from /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16; llama_model_load_from_file_impl: failed to load model
[llama-cpp]: failed to load model with internal loader: could not load model: rpc error: code = Internal desc = Failed to load model: /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16. Error: gguf_init_from_file: failed to open GGUF file '/home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16' (No such file or directory); llama_model_load: error loading model: llama_model_loader: failed to load model from /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16; llama_model_load_from_file_impl: failed to load model; gguf_init_from_file: failed to open GGUF file '/home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16' (No such file or directory); llama_model_load: error loading model: llama_model_loader: failed to load model from /home/zc/LocalAI/models/local/DeepSeek-R1-Distill-Qwen-1.5B-BF16; llama_model_load_from_file_impl: failed to load model 
Jun 04 20:03:42 INFO  HTTP request method="POST" path="/v1/messages" status=200 

Result2

zc@zc-MS-7E13:~/agent/claw-code-main/rust/target/debug$ ./claw --model local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf
██████╗██╗      █████╗ ██╗    ██╗
██╔════╝██║     ██╔══██╗██║    ██║
██║     ██║     ███████║██║ █╗ ██║
██║     ██║     ██╔══██║██║███╗██║
╚██████╗███████╗██║  ██║╚███╔███╔╝
╚═════╝╚══════╝╚═╝  ╚═╝ ╚══╝╚══╝ Code 🦞

Model            local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf
Permissions      danger-full-access
Branch           unknown
Workspace        clean
Directory        /home/zc/agent/claw-code-main/rust/target/debug
Session          session-1780574842374-0
Auto-save        .claw/sessions/7bcdecb4f9f43786/session-1780574842374-0.jsonl

Type /help for commands · /status for live context · /resume latest jumps back to the newest session · /diff then /commit to ship · Tab for workflow completions · Shift+Enter for newline
Connected: local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf via anthropic
> hello,what is your name?
✔ ✨ Done


Hello! How can I assist you today?

> can you tell me ,what is your name ?
The user’s question is unclear and contains a typo. They asked “what is your name?” with a period at the end, which might be a mistake. Without context, I can’t determine their name or provide a helpful respo✔ ✨ Done


The user's question is unclear and contains a typo. They asked "what is your name?" with a period at the end, which might be a mistake. Without context, I can't determine their name or provide a helpful response. Please clarify or provide additional information so I can assist you better.

> who are you?
I’m an AI assistant created by DeepSeek. I’m here to help with tasks and provide information, but I don’t have personal identities or capabilities to assist with human inquiries. Could you clarify your questi✔ ✨ Done


I'm an AI assistant created by DeepSeek. I'm here to help with tasks and provide information, but I don't have personal identities or capabilities to assist with human inquiries. Could you clarify your question or provide more context so I can assist you better? I'm here to help!

AND LocalAI Log is:

Jun 04 20:03:42 INFO  HTTP request method="POST" path="/v1/messages" status=200 
Jun 04 20:07:33 INFO  Trying to load the model modelID="Deepseek-R1-Distill-Qwen-1.5B" backends=[cpu-llama-cpp llama-cpp] 
Jun 04 20:07:33 INFO  Attempting to load backend="cpu-llama-cpp" 
Jun 04 20:07:33 INFO  BackendLoader starting modelID="Deepseek-R1-Distill-Qwen-1.5B" backend="cpu-llama-cpp" model="local/DeepSeek-R1-Distill-Qwen-1.5B-BF16.gguf" 
Jun 04 20:07:37 INFO  Loads OK backend="cpu-llama-cpp" 
Jun 04 20:08:08 INFO  HTTP request method="POST" path="/v1/messages" status=200