Skip to content

Faster filtering mapping#53

Closed
jeswr wants to merge 7 commits into
RubenVerborgh:mainfrom
jeswr:jeswr/faster-filtering-mapping
Closed

Faster filtering mapping#53
jeswr wants to merge 7 commits into
RubenVerborgh:mainfrom
jeswr:jeswr/faster-filtering-mapping

Conversation

@jeswr

@jeswr jeswr commented Mar 27, 2022

Copy link
Copy Markdown
Collaborator

Supercedes #48
Supercedes #50

Makes use of this suggestion by @RubenVerborgh to provide the ~2x speedup to chained maps and transforms; without having the problems associated with immutability (to give you a sense, the test suite is now running at ~600ms on my machine vs ~1000ms before).

@jeswr

jeswr commented Mar 27, 2022

Copy link
Copy Markdown
Collaborator Author

The performance improvements greatly depend on the number of filters. In the example @jacoscaz had with only maps - there was no gain (if anything a slight degradation of 5-10%), but if there are filters involved the speedup can be much more significant. For instance running

range(0, 2_000_000)
.map((item: number) => item)
.filter((item: number) => item % 2 === 0)
.map((item: number) => item)
.filter((item: number) => item % 3 === 0)
.map((item: number) => item)
.filter((item: number) => item % 4 === 0)
.map((item: number) => item)
.filter((item: number) => item % 5 === 0)
.map((item: number) => item)
.filter((item: number) => item % 6 === 0)

Goes from ~80ms to ~20ms; so a 4x speedup

As a result - it may be worth only triggering this optimized iterator when we reach the first FilterIterator which we can do by simply removing the following lines https://github.com/jeswr/AsyncIterator/blob/1ec39fbb12497ad46ea5409ba683deb4cbb2079c/asynciterator.ts#L1331-L1351

@jeswr

jeswr commented Mar 27, 2022

Copy link
Copy Markdown
Collaborator Author

I'm not sure I understand the cause of the failing test on node v10

@jeswr

jeswr commented Mar 27, 2022

Copy link
Copy Markdown
Collaborator Author

As discussed on gitter @jacoscaz will merge this into #48 - I will close this once that has been done.

@jacoscaz

Copy link
Copy Markdown
Collaborator

Done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants