In #738 we added and tested support for decoding an IPC schema that contains dictionary-encoded fields; however, the decoder does not yet provide a way to export the dictionary IDs with the schema. We probably need something like:
struct ArrowIpcSchemaDictionaries;
ArrowIpcDecoderDecodeSchemaWithDictionaries(struct ArrowIpcDecoder* decoder, struct ArrowSchema* schema, struct ArrowIpcSchemaDictionaries*);
ArrowIpcDecoderSetSchemaWithDictionaries(struct ArrowIpcDecoder* decoder, struct ArrowSchema* schema, struct ArrowIpcSchemaDictionaries*);
I was thinking that ArrowIpcSchemaDictionaries could be essentially a Vec<(const struct ArrowSchema*, int64_t)> where the pointers correspond to the dictionary pointers in the decoded schema. This allows the setter to locate the dictionary IDs when looping over the schema with the downside the pointers are invalidated if anybody decides to ArrowSchemaDeepCopy(). I'm not also not sure there's a better solution in the presence of potentially nested dictionaries.
Part of #622.
In #738 we added and tested support for decoding an IPC schema that contains dictionary-encoded fields; however, the decoder does not yet provide a way to export the dictionary IDs with the schema. We probably need something like:
I was thinking that
ArrowIpcSchemaDictionariescould be essentially aVec<(const struct ArrowSchema*, int64_t)>where the pointers correspond to the dictionary pointers in the decoded schema. This allows the setter to locate the dictionary IDs when looping over the schema with the downside the pointers are invalidated if anybody decides toArrowSchemaDeepCopy(). I'm not also not sure there's a better solution in the presence of potentially nested dictionaries.Part of #622.