Skip to content

version 4 of replace_all_objects() no longer supports iterators #584

@iloveagent57

Description

@iloveagent57

def replace_all_objects(
self,
index_name: str,
objects: List[Dict[str, Any]],
batch_size: int = 1000,
scopes=["settings", "rules", "synonyms"],
request_options: Optional[Union[dict, RequestOptions]] = None,
) -> ReplaceAllObjectsResponse:

I'm in the process of updating from v3 to v4 and noticed that this function no longer seems to support iterators, which will cause problems for our systems that use it to bulk replace large datasets. It's possible that making some fix inside of chunked_batch() is all thats needed to support iterators.

The legacy/v1 API used to support iterators: https://www.algolia.com/doc/libraries/sdk/v1/methods/replace-all-objects#param-objects

Python: Use an iterator instead of a list to prevent memory issues, especially if you want to replace many records.

  1. Is this an intentional design decision?
  2. Short of re-architecting our syncrhonization strategy, do you have any recommended best practice for those of us using replace_all_objects on sets of data that are too large to fit in a list/memory?

Thanks!

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions