External models (Gemini Nano Banana & OpenAI GPT Image) (#8633)

[CypherNaught-0x]

Summary

This PR adds support for external model provider APIs with Google and OpenAI added for now.
It supports txt2img, img2img and image references.
I tried to make it fit well within the application and be easily extensible for future models.

Related Issues / Discussions

#8633 includes functionality requested here

QA Instructions

Select an external provider in the model setup dialog and add an API key.
Select a new model from the dropdown list.
...
Profit

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[Pfannkuchensack]

I did some testing. Works fine (only did gemini)

A few comments

• Reidentify button: The "Reidentify" button in the Model Manager should not be shown for external models.
• Auto-install starter models: Auto-install should always be enabled for external starter models. When an API key is removed, the associated external models should also be removed.
• Install queue status: The install queue shows "Unknown" when installing external models. This needs to display the correct model name/status.
• Starter model description: The text for external starter models needs to clearly indicate that an API key is required and that usage may incur costs. (And the Starter Models are not needed if the Autoinstall is always on)
• Canvas settings for external models: It is currently unclear which canvas settings are actually passed to external models. Right now all standard settings (Scheduler, Steps, CFG Scale, and everything under Advanced) are displayed, but most of these are not used by external models. We need a solution where external models can define their required settings as JSON, and the frontend renders only the relevant controls based on that definition.
• External Image Generation node: The "External Image Generation" node also contains these values. The core issue is that we cannot have dynamic nodes. Instead, we should have a dedicated settings node for each external model node.

[Pfannkuchensack]

Some Changes that should be done:

In invokeai/app/services/external_generation/external_generation_default.py, the method _refresh_model_capabilities does:

from invokeai.app.api.dependencies import ApiDependencies
record = ApiDependencies.invoker.services.model_manager.store.get_model(request.model.key)

No other service in the codebase imports from invokeai.app.api.dependencies. All other services receive their dependencies via constructor injection through InvocationServices. This is an architectural violation that makes the service harder to test in isolation and creates a hidden coupling between the service and API layers.

---

In invokeai/app/services/model_install/model_install_common.py:

MODEL_SOURCE_TO_TYPE_MAP = {
    ...
    ExternalModelSource: ModelSourceType.Url,
}

ExternalModelSource is not a URL source. There is no ModelSourceType.External enum value in taxonomy.py. This means external models get recorded as Url-type sources in the database, which is semantically incorrect and could cause issues in any code that branches on source_type.

---

In invokeai/app/api/routers/app_info.py:

for config in (runtime_config, file_config):
    config.update_config(updates)
    for field_name, value in updates.items():
        if value is None:
            config.model_fields_set.discard(field_name)

This directly mutates the model_fields_set of the global singleton InvokeAIAppConfig, bypassing Pydantic's field-tracking internals. Concurrent requests to set_external_provider_config or reset_external_provider_config could race on this shared mutable set.

---

In invokeai/app/services/model_install/model_install_default.py, _register_external_model generates a deterministic key via slugify(f"{provider_id}-{provider_model_id}"). Installing the same external model twice produces the same key. While the DB layer catches this with DuplicateModelException, there is no proactive check or update-if-exists logic, resulting in an unhelpful error for the user.

---

In invokeai/app/api/routers/model_manager.py, list_model_records uses setattr(model, "capabilities", ...) and setattr(model, "default_settings", ...) on Pydantic model instances. Pydantic v2 models may not support direct attribute mutation without validate_assignment = True. The PR itself uses model_copy(update=...) correctly in other places (e.g., _apply_starter_overrides in external_generation_default.py), so this is inconsistent.

---

I think to save the API keys in the invoke.yaml is not the best choice here. This is something that the development team in https://discord.com/channels/1020123559063990373/1049495067846524939 should discuss.

[lstein]

@CypherNaught-0x Please see the changes requested from @Pfannkuchensack above.

[CypherNaught-0x]

Thanks @Pfannkuchensack for the valuable feedback!

I didn't feel comfortable disabling all the inputs as I didn't see this done elsewhere but you are right of course in that it makes no sense to show for example CFG when that property is not used.
For the other details I went with the interface of model_supports_x. We could do this for the advanced properties as well.
From a UX standpoint I am wondering if it's better to disable these properties or hide them completely.
Do you have any preference or insight on how this is handled elsewhere?

This was more of a first draft since I wasn't sure how such a large addition would be received so I haven't yet spent much time polishing things like the install queue. I was positively surprised with the feedback so I'll try and get things to a more polished state for the next review round.
I also haven't really done extensive testing on the OpenAI implementation so I will get that done also. Glad things worked for Gemini on your end though. I tested on different systems with fresh installs but it's nice to have external confirmation.

How are the discussions on the API Key storage coming along? I saw that the model marketplace can store API keys there as well so figured with a decently restricted API key this might be ok though I'd obviously also prefer at least non-plain-text storage.

[Pfannkuchensack]

Pfannkuchensack@3c83692 i did some work on the hiding of unneeded things in the ui. Maybe take a Look Or copy the whole Thing from there.

[Pfannkuchensack]

The API keys require a separate YAML file. This is better because it allows the API key to be kept separate.

[lstein]

The API keys require a separate YAML file. This is better because it allows the API key to be kept separate.

We need a unified place to stash user's security tokens and API keys. I just now proposed a "Token Manager" in Issue #8904 . Temporarily, you could add nano_banana_key and gpt_image_key to the InvokeAIAppConfig class in invokeai.app.services.config.config_default and stash the keys in invokeai.yaml .

@Pfannkuchensack Does this seem like a reasonable interim solution to API key storage or would it be better to have a completely separate API keys file, like ~/invokeai/api_keys.yaml?

[Pfannkuchensack]

I would prefer the separate file, especially since there will be another solution later, thus avoiding major changes to the invoke.yaml file.

[lstein]

@CypherNaught-0x I'm wondering what you see as the timetable for this? I'm thinking we'll be ready for a 6.12 release in the second week of March. Would that be targetable, or later? The release after that will likely be mid April.

[CypherNaught-0x]

Pfannkuchensack@3c83692 i did some work on the hiding of unneeded things in the ui. Maybe take a Look Or copy the whole Thing from there.

I had already started work on this and your implementation looks very similar so I'll try and integrate them. Also very much looking forward to Seedream support 👍

[CypherNaught-0x]

@lstein I've had some time to work on it. I'll try and get things into a polished state and push the changes. I believe mid march should be very much reasonable for a release target.