feat: distributed index building for ivf index#218
feat: distributed index building for ivf index#218goodqiang wants to merge 3 commits intolance-format:mainfrom
Conversation
github-actions
bot
added
enhancement
| Dataset.open() | ||
| .allocator(LanceRuntime.allocator()) | ||
| .namespace(readOptions.getNamespace) | ||
| .readOptions(readOptions.toReadOptions) |
There was a problem hiding this comment.
this should not be removed
| import java.util.*; | ||
| import java.util.stream.Collectors; | ||
|
|
||
| public class BaseVectorIndexTest extends BaseAddIndexTest { |
There was a problem hiding this comment.
this test is not really inheriting anything from BaseAddIndexTest?
| <properties> | ||
| <lance-spark.version>0.2.0</lance-spark.version> | ||
| <lance.version>2.0.0</lance.version> | ||
| <lance.version>3.0.0-beta.2</lance.version> |
There was a problem hiding this comment.
do we depend on any feature not in 2.0?
jackye1995
left a comment
jackye1995
left a comment
There was a problem hiding this comment.
thanks for the work! Please also add related documentations, and make sure CI is passing 🙏
hamersaw
left a comment
hamersaw
left a comment
There was a problem hiding this comment.
Can we add a small integration test in docker/tests/test_lance_spark.py?
|
Looks like there are a few integration test failures, can you fix those? The logs are extremely verbose making it difficult to discover the root cause, I'm wondering if we can reduce the log-level so they're more easily digested? |
|
hi @goodqiang ，Are you still working on this PR? If there's no time, I can take over and continue. |
4 participants
feat: distributed index building for ivf index support #176
BREAKING CHANGE: AddIndexExec adds "alter table create index index_name using ivf_flat/ivf_pq/ivf_sq"