Fix fuse_dfs flush skipping hdfsHFlush when fi->flags is 0#8545
Open
shanrui wants to merge 3 commits into
Open
Fix fuse_dfs flush skipping hdfsHFlush when fi->flags is 0#8545shanrui wants to merge 3 commits into
shanrui wants to merge 3 commits into
Conversation
Author
|
@ajfabbri May i ask a review on this? |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Contributor
|
Thanks for the PR. This needs a Jira which describes the bug and how it manifests. Please use our PR template for the main description here and follow its instructions. This should also include a test case. Once you've done this and Yetus is happy I'll take another look. I'm not familiar with why newer Fuse impls. would clear those flags, so any reference to that information would be helpful as well. |
Author
|
@ajfabbri Thanks for the feedback. There is a related historical Jira, HDFS-2551. While the root cause is different, the observed behavior is conceptually similar: if data is not properly flushed (especially when hdfs flush() not hdfs hflush() is invoked), HDFS cannot guarantee that newly written data will be immediately visible to readers. In this case, skipping hdfsHFlush() in dfs_flush() can lead to the same symptom: a reader opened after the write may observe an empty file or stale data until the writer closes the file. Regarding fi->flags, I found that as early as FUSE 2.4 the documentation did not guarantee that the original open flags would be reliably preserved across all callback invocations(https://github.com/libfuse/libfuse/blob/fuse_2_4_0/include/fuse_common.h#L36). Because of this, using fi->flags inside flush() to determine whether the file was opened for writing may not be reliable. In my environment, fi->flags was observed to be zero in dfs_flush(), which caused hdfsHFlush() to be skipped entirely. I'm working on creating a Jira and adding a test case as suggested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
In modern libfuse implementations, fi->flags in flush() callback
may be reset to 0. The original fuse_dfs code relies on:
if (fi->flags & O_WRONLY)
As a result, hdfsHFlush/hdfsHSync is skipped, causing newly written
data to be invisible to readers until file close.
Root cause:
fi->flags is not guaranteed to be preserved in flush().
Fix:
Store open flags in dfs_fh during dfs_open(), and use it in flush()
instead of fi->flags.
Testing:
Verified write -> flush -> read consistency
Verified hdfsHFlush is triggered correctly