-
Notifications
You must be signed in to change notification settings - Fork 448
Reduce memory consumption in FilesCollection #1219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ckuran
wants to merge
1
commit into
phpro:v2.x
Choose a base branch
from
ckuran:reduce-memory-consumption
base: v2.x
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The signature however is
ArrayCollection<array-key, \SplFileInfo>.Now it also contains strings making it result in broken public API contracts (see e.g; the getFirst() vs first() and optionally similar methods like last, current, ...)
I'm also wondering: now every time an iterator goes over it, it will be hydrated.
So if you have more tasks (or file filters), it will always increase memory.
This makes me wonder: what is the ACTUAL memory issue that we are trying to solve?
Can we find a better solution than keeping it as strings intermediate?
It also has been a while but: what benifit does SymfonySplFileInfo give over regular SplFileInfo here. Which one does add the overhead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your comment. It allowed me to rethink the solution and actually understand where the problem is. As you already indicated,
FilesCollectionwas holdingSplFileInfoobjects, and I broke the contract by adding pure paths there. I understood that the main problem is thatFilesCollectionhas two responsibilities: holding the array of files and filtering the collection.Now I separated those responsibilities into
FilesCollection, which holds the array of strings, andFilesCollectionFilter, which filters theFilesCollection. The filter is converting the paths from strings toSymfonySplFileInfo, doing the filtering and returning theFilesCollectionwith strings. TheSymfonySplFileInfoobjects are garbaged immediately after filtering. Also we must use it for filtering instead of\SplFileInfo.This solution is fully backwards-compatible, but I marked using the filtering methods in
FilesCollectionas deprecated, and it should be removed in v3. The result? Previously, my app was taking 126 MBs in peak memory usage. Now it's only 50 MBs for the same suite. I had to migrate a few places which were directly working onSplFileInfoobjects to work with file paths. I haven't fixed phpspec yet, and the tasks themselves, but that can be covered later. Waiting for your opinion on that.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I consider this a massive breaking change in the contract of the FilesCollection, especially since its not final. + You added a new 'filter' collection class that isnt really a filter, but rather another type of files collection.
I don't think this is the way forward to be fair.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree it's not a small change in FilesCollection, but the whole solution is in the final state. The contract for public methods in FilesCollection was not changed so that's why I consider this solution backward-compatible. The new 'filter' class can be renamed but basically it takes the FilesCollection as an argument and returns filtered FilesCollection. That was the easiest way to make it backward compatible. Running
bin/grumphp runreturns only failing phpspec, which is todo after we agree what next. If you have a better idea how to organize this change, please let me know.