Is your feature request related to a problem? Please describe.
There is interest in using the engine as a pull-based iterator (#243). While that is hard to achieve, we can have a partial solution of supporting a channel between threads as a Sink. A user can then spin the engine in a separate thread and then use the other end of the channel as a source of results from which to pull.
It might be beneficial to check if one of those impls should be used in rq itself. If the overhead is small, we might default to it on large files where writing the results to output may be a heavy operation. This might also be an option for online output.
Describe the solution you'd like
We should have two implementations.
std::sync::mpsc, the stdlib implementation of a channel.crossbeam_channel, the most popular channel crate.
The other should be behind a feature flag to enable the optional dependency – we won't force users to pay for it if they don't need it.
Describe alternatives you've considered
We could implement Sink on other primitives, like crossbeam's ArrayQueue, but that seems too specific. People trying to get a lower-level primitive to work with the engine probably would be better suited by a custom Sink impl anyway.
Is your feature request related to a problem? Please describe.
There is interest in using the engine as a pull-based iterator (#243). While that is hard to achieve, we can have a partial solution of supporting a channel between threads as a
Sink. A user can then spin the engine in a separate thread and then use the other end of the channel as a source of results from which to pull.It might be beneficial to check if one of those impls should be used in
rqitself. If the overhead is small, we might default to it on large files where writing the results to output may be a heavy operation. This might also be an option for online output.Describe the solution you'd like
We should have two implementations.
std::sync::mpsc, the stdlib implementation of a channel.crossbeam_channel, the most popular channel crate.The other should be behind a feature flag to enable the optional dependency – we won't force users to pay for it if they don't need it.
Describe alternatives you've considered
We could implement
Sinkon other primitives, like crossbeam's ArrayQueue, but that seems too specific. People trying to get a lower-level primitive to work with the engine probably would be better suited by a customSinkimpl anyway.