Skip to content

[Question] Creating lazy matrix from a function #16

@PeterMitrano

Description

@PeterMitrano

first off thanks for the great library. I'm wondering if there's a better way to create a matrix from a function that only knows how to load a single example. I thought this would work:

from lazyarray import larray
import numpy as np

pretend_this_is_a_file = [
        [1,2,3],
        [1,2,3],
        [1,2,3],
]

def load(i, j):
    print(type(i), type(j))
    return pretend_this_is_a_file[i][j]

So load only works if i and j are integers. Not, for example, lists or arrays of integers. That means this usage fails:

z = larray(load, shape=[3,3])
print(z[0,0])
print(z[0])
print(z[:, 0])

Is there a better way? I have a workaround, which implements the case for when i or j and arrays of integers, but that seems like it should be done by lazyarray

EX:

    def _load_examples(i, j):
        if isinstance(i, np.ndarray):
            results = []
            for _i in i:
                _name = f'{_i}-{j}.pkl.gz'
                _results_filename = results_dir / _name
                if _results_filename.exists():
                    _result = load_gzipped_pickle(_results_filename)[k]
                else:
                    _result = None
                results.append(_result)
            return np.array(results)
        elif isinstance(j, np.ndarray):
            results = []
            for _j in j:
                _name = f'{i}-{_j}.pkl.gz'
                _results_filename = results_dir / _name
                if _results_filename.exists():
                    _result = load_gzipped_pickle(_results_filename)[k]
                else:
                    _result = None
                results.append(_result)
            return np.array(results)
        else:
            name = f'{i}-{j}.pkl.gz'
            results_filename = results_dir / name
            result = load_gzipped_pickle(results_filename)[k]
            return result

If this is a good idea and you're interested in contributions, I could probably help put a PR together

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions