Skip to content

add image search support in agents sdk #3474

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yuning-oai wants to merge 1 commit into
base: main
Choose a base branch
from
+47 −2
Open

add image search support in agents sdk #3474

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion src/agents/models/openai_responses.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2021,9 +2021,16 @@ def _convert_tool(
}
if tool.external_web_access is not None:
web_search_tool["external_web_access"] = tool.external_web_access
if tool.search_content_types is not None:
web_search_tool["search_content_types"] = list(tool.search_content_types)
Comment on lines +2024 to +2025
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the preview web-search tool for content types

When WebSearchTool(search_content_types=...) is used, this still serializes the tool as type: "web_search" and adds search_content_types to that GA payload. I checked the generated OpenAI Responses types: search_content_types is defined on WebSearchPreviewToolParam (web_search_preview...), not on the GA WebSearchToolParam, so callers exercising the new option send a tool shape the API rejects instead of enabling image search.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member

@seratch seratch May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yuning-oai Can you resolve all the Codex review comments? This repo runs reviews per push.

Copy link
Copy Markdown
Author

@yuning-oai yuning-oai May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. The backend accepts search_content_types on web_search, but the field is intentionally still hidden from public OpenAPI for now, so openai-python does not expose it on WebSearchToolParam yet.

I’m going to keep the Agents SDK payload as web_search rather than switch to preview; this is relying on the Responses API hidden field until we’re ready to expose it publicly.

web_search_include: ResponseIncludable | None = (
"web_search_call.results"
if tool.search_content_types is not None and "image" in tool.search_content_types
else None
)
return (
_require_responses_tool_param(web_search_tool),
None,
web_search_include,
)
elif isinstance(tool, FileSearchTool):
file_search_tool_param: FileSearchToolParam = {
Expand Down
5 changes: 4 additions & 1 deletion src/agents/tool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import json
import math
import weakref
from collections.abc import Awaitable, Callable, Mapping
from collections.abc import Awaitable, Callable, Mapping, Sequence
from dataclasses import dataclass, field
from enum import Enum
from types import UnionType
Expand Down Expand Up @@ -600,6 +600,9 @@ class WebSearchTool:
indexed-only behavior where supported.
"""

search_content_types: Sequence[Literal["text", "image"]] | None = None
"""The content types to include in the search results. Omitting this uses the API default."""

@property
def name(self):
return "web_search"
Expand Down
35 changes: 35 additions & 0 deletions tests/models/test_openai_responses_converter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,7 @@ def test_convert_tools_basic_types_and_includes():
assert web_params.get("user_location") == web_tool.user_location
assert web_params.get("search_context_size") == web_tool.search_context_size
assert "external_web_access" not in web_params
assert "search_content_types" not in web_params
# Verify computer tool uses the GA built-in tool payload.
comp_params = next(ct for ct in converted.tools if ct["type"] == "computer")
assert comp_params == {"type": "computer"}
Expand Down Expand Up @@ -468,6 +469,40 @@ def test_convert_tools_includes_explicit_false_external_web_access() -> None:
]


def test_convert_tools_forwards_web_search_content_types() -> None:
web_tool = WebSearchTool(search_content_types=["text", "image"])

converted = Converter.convert_tools([web_tool], handoffs=[], model="gpt-5.4")

assert converted.includes == ["web_search_call.results"]
assert converted.tools == [
{
"type": "web_search",
"filters": None,
"user_location": None,
"search_context_size": "medium",
"search_content_types": ["text", "image"],
}
]


def test_convert_tools_includes_results_for_image_only_web_search() -> None:
web_tool = WebSearchTool(search_content_types=["image"])

converted = Converter.convert_tools([web_tool], handoffs=[], model="gpt-5.4")

assert converted.includes == ["web_search_call.results"]
assert converted.tools == [
{
"type": "web_search",
"filters": None,
"user_location": None,
"search_context_size": "medium",
"search_content_types": ["image"],
}
]


def test_convert_tools_uses_preview_computer_payload_for_preview_model() -> None:
comp_tool = ComputerTool(computer=DummyComputer())

Expand Down
Loading