Skip to content

[WIP] OpenWebUI update#4102

Open
przepeck wants to merge 2 commits intomainfrom
przepeck/openwebui_update
Open

[WIP] OpenWebUI update#4102
przepeck wants to merge 2 commits intomainfrom
przepeck/openwebui_update

Conversation

@przepeck
Copy link
Copy Markdown
Collaborator

🛠 Summary

CVS-183785
Changing models used in OpenWebUI, adding sections about new agentic features.

TODO: Update screenshots to use ovms-model model name instead of Godreign/llama-3.2-3b-instruct-openvino-int4-model

🧪 Checklist

  • Unit tests added.
  • The documentation updated.
  • Change follows security best practices.
    ``

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the OpenWebUI integration demo documentation to use newer default models (including a VLM) and adds guidance for newer “agentic” OpenWebUI features like Web Search, Memory, and Code Interpreter.

Changes:

  • Switched the primary chat model example to OpenVINO/gpt-oss-20b-int4-ov and standardized the OpenWebUI Model ID to ovms-model.
  • Replaced the VLM example model with Junrui2021/Qwen3-VL-8B-Instruct-int4 and added a new screenshot for image upload.
  • Added new documentation sections for Web Search, Memory/context, and Code Interpreter configuration in OpenWebUI.

Reviewed changes

Copilot reviewed 1 out of 13 changed files in this pull request and generated 11 comments.

File Description
demos/integration_with_OpenWebUI/README.md Updates model pull/config instructions and adds new OpenWebUI feature sections (Web Search, Memory, Code Interpreter).
demos/integration_with_OpenWebUI/upload_images.png Adds/updates a screenshot used by the VLM “upload images” step.

```bash
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model OpenVINO/InternVL2-2B-int4-ov --model_repository_path models --model_name OpenVINO/InternVL2-2B-int4-ov --task text_generation
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --add_to_config --config_path /models/config.json --model_path OpenVINO/InternVL2-2B-int4-ov --model_name OpenVINO/InternVL2-2B-int4-ov
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_repository_path models --model_name ovms-model-vl --task text_generation
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the Docker VLM pull command, --model_repository_path is set to models (relative path inside the container), while the volume is mounted to /models. This likely writes to a non-persisted path and breaks the subsequent --add_to_config --config_path /models/config.json step. Use /models here for consistency with other Docker examples above.

Suggested change
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_repository_path models --model_name ovms-model-vl --task text_generation
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_repository_path /models --model_name ovms-model-vl --task text_generation

Copilot uses AI. Check for mistakes.
```bash
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model OpenVINO/InternVL2-2B-int4-ov --model_repository_path models --model_name OpenVINO/InternVL2-2B-int4-ov --task text_generation
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --add_to_config --config_path /models/config.json --model_path OpenVINO/InternVL2-2B-int4-ov --model_name OpenVINO/InternVL2-2B-int4-ov
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_repository_path models --model_name ovms-model-vl --task text_generation
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows VLM section uses --pipeline_type VLM_CB, but the Linux/Docker command omits it. If AUTO doesn’t reliably detect VLM, the Docker path may start the wrong pipeline. Align the Linux command with Windows by specifying the same --pipeline_type (and keep both examples consistent).

Suggested change
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_repository_path models --model_name ovms-model-vl --task text_generation
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_repository_path models --model_name ovms-model-vl --task text_generation --pipeline_type VLM_CB

Copilot uses AI. Check for mistakes.
Comment on lines 306 to 309
### Step 2: Chat with VLM

1. Start a **New Chat** with model set to `OpenVINO/InternVL2-2B-int4-ov`
1. Start a **New Chat**
2. Click **+More** to upload images, by capturing the screen or uploading files. The image used in this demo is [http://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/releases/2025/3/demos/common/static/images/zebra.jpeg](http://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/releases/2025/3/demos/common/static/images/zebra.jpeg).
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 2 for VLM no longer tells the reader to select the ovms-model-vl model in OpenWebUI (it just says “Start a New Chat”). Without selecting the correct model, the subsequent image-upload chat steps may run against the default text model instead of the VLM endpoint.

Copilot uses AI. Check for mistakes.

### Step 3: Use Web Search in the chat

1. Open new Chat (it should have blue Web Search icon blow)
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: “blow” should be “below” in “blue Web Search icon blow”.

Suggested change
1. Open new Chat (it should have blue Web Search icon blow)
1. Open new Chat (it should have blue Web Search icon below)

Copilot uses AI. Check for mistakes.

### Reference

![https://docs.openwebui.com/features/chat-conversations/web-search/agentic-search/](https://docs.openwebui.com/features/chat-conversations/web-search/agentic-search/)
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The “Reference” is currently written as an image link (![...](...)), which renders as a broken image instead of a hyperlink. Use a normal markdown link here.

Suggested change
![https://docs.openwebui.com/features/chat-conversations/web-search/agentic-search/](https://docs.openwebui.com/features/chat-conversations/web-search/agentic-search/)
[Open WebUI Web Search documentation](https://docs.openwebui.com/features/chat-conversations/web-search/agentic-search/)

Copilot uses AI. Check for mistakes.

![add memory](./add_memory.png)

It's possible to have multiple manageble memory records.
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typos: “manageble” should be “manageable”.

Suggested change
It's possible to have multiple manageble memory records.
It's possible to have multiple manageable memory records.

Copilot uses AI. Check for mistakes.

1. Go to **Workspace** → **Models**
2. Choose model or create it.
3. In **Buildin Tools** section enable **Memory**
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: “Buildin Tools” should be “Built-in Tools” (or whatever wording matches the OpenWebUI UI label).

Suggested change
3. In **Buildin Tools** section enable **Memory**
3. In **Built-in Tools** section enable **Memory**

Copilot uses AI. Check for mistakes.

![memory usage](./memory_usage.png)

> **Note**: There is no way to make searching memory default on the beggingng of the conversation in Open Web UI. User should tell model to use it to make it work.
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: “beggingng” should be “beginning” in the note about searching memory.

Suggested change
> **Note**: There is no way to make searching memory default on the beggingng of the conversation in Open Web UI. User should tell model to use it to make it work.
> **Note**: There is no way to make searching memory default on the beginning of the conversation in Open Web UI. User should tell model to use it to make it work.

Copilot uses AI. Check for mistakes.
Comment on lines +287 to 289
ovms.exe --pull --source_model Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_repository_path models --model_name ovms-model-vl --task text_generation --pipeline_type VLM_CB
ovms.exe --add_to_config --config_path models\config.json --model_path Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_name ovms-model-vl
```
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the VLM Windows example, --pull uses --model_name ovms-model-vl, but --add_to_config points --model_path to Junrui2021/Qwen3-VL-8B-Instruct-int4. If pull mode follows the documented behavior (downloads into <model_repository_path>/<model_name>), the model path should match ovms-model-vl (or omit --model_name in the pull step). As written, --add_to_config will likely fail to find the pulled model directory.

Copilot uses AI. Check for mistakes.
Comment on lines +294 to +295
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_repository_path models --model_name ovms-model-vl --task text_generation
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --add_to_config --config_path /models/config.json --model_path Junrui2021/Qwen3-VL-8B-Instruct-int4 --model_name ovms-model-vl
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly in the Docker VLM example, --pull sets --model_name ovms-model-vl, but --add_to_config uses --model_path Junrui2021/Qwen3-VL-8B-Instruct-int4. If pull stores the model under /models/ovms-model-vl, --model_path should point to ovms-model-vl (or the pull step should not override --model_name).

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants