[Bug]: Filesystem partition saving run with overwrite removes all files not just the related ones

[stloyd]

What happened?

Right now, when having two data frames working on the same partitioning, and using the native any filesystem on the run of the second data frame, it will remove files from the previous one.

How to reproduce?

df()
    ->extract(/* */)
    ->partitionBy('order-year', 'order-month')
    ->mode(overwrite())
    ->load(to_csv(__DIR__.'/output/orders.csv'))
    ->run()
;

df()
    ->extract(/* */)
    ->partitionBy('order-year', 'order-month')
    ->mode(overwrite())
    ->load(to_csv(__DIR__.'/output/sales.csv'))
    ->run()
;

Playground snippet

No response

Data required to reproduce bug locally

If the bug requires specific data, provide it here:

• PHP array
• CSV
• JSON
• Other format if bug is format-specific

Version

0.30+

Relevant error output

Copy and paste any relevant error output (no backticks needed).

[norberttech]

Could you prep a sandbox example of this behavior? (you can transform purchase date from orders dataset to year / month / day entries and use them for partitioning)

[stloyd]

I create fix and test case: #2240. When you revert the changes it fails like:

1) Flow\ETL\Tests\Integration\Filesystem\FilesystemStreams\Partitioned\OverwriteModeTest::test_open_stream_for_existing_partition_with_existing_file
Failed asserting that actual size 1 matches expected size 2

[norberttech]

yea but I'm not sure if what you are reporting is an actual bug or expected behavior of how overwrite should works, thats why I would like to see it in playground first

[stloyd]

If it's expected, then I would vote for a rename or adding an explicit comment that says any existing file in the given partition will be deleted.

For simple repro, I think this is "good enough": https://flow-php.com/playground/?snippet=6ffd177f02c5a2c8e4facc9b6b1573b19f7983103dcf9edbe1d3e2a0bc307b39

[norberttech]

yea thats not a bug, look here:

https://flow-php.com/playground/?snippet=2e54526d7ab44e01ac071af09d38e980aa9dabf8124d954150c8ad358661fd8f

day=99 partition stays regardless how many time you execute it, byt you are right, all partitions that are present in the dataset will be overwritten. I thought it's removing that day=99 which would be a bug. Because partitions that are not present in the dataset should stay untouched, only those that are present in the dataset should be overwritten.

On the other hand, I think that your example is logically incorrect, you are trying to save 2 different datasets in the same path. This is how it should be: https://flow-php.com/playground/?snippet=8af85e667b4aabb7b848f302fb1c905fad0f1925b10d778c216859930bdb189a

What you are doing here is the same as saving 2 datasets into the same path with overwrite mode (the second run will overwrite the first one).

[norberttech]

I'm closing this one for now

[stloyd]

TBH. I'm not sure why it's not logical (not perfect, but still valid), cause i.e., if I use append, it will allow everything I want. It's not clear why the same partitioning used in two different data frames is not logical, cause i.e. your example with:

    ->write(\Flow\ETL\Adapter\CSV\to_csv(__DIR__ . '/output/orders/orders.csv'))
// ...
    ->write(\Flow\ETL\Adapter\CSV\to_csv(__DIR__ . '/output/plan/plan.csv'))

Is already pointing to different partitioning IMO. Also, there is nothing in the documentation that would point to you must not have, i.e., raw, clean, and "perfect" data in the same partition.

Anyway, a small change to have a file with a different name than the one from the data frame shows it's removed: https://flow-php.com/playground/?snippet=2f4acf908bfde32055db86b6f43b4ae965b3af3a6969fe6e721d8773adf6e955

[norberttech]

Because it's not about DataFrames it's about the same destination path.

/output/file.csv
/output/sales.csv

this is would work without partitions, but
with partitions it can't, at least not right now. Right now overwrite is doing exactly what its supposed to, remove everything from a given partition.

I don't think any other DataFrames supports something like this, at least I never had a use case for it

But partitioning should be better documented, no doubts there.

More examples in playground are very welcome.

In general I think there should be a section in the documentation for file based datasets explaining in details how different save mode, partitioning and other configs impacts the output

[norberttech]

it will get even more complicated with parallelism - as we can't make writing to a single file a bottleneck, multiple processes will generate multiple output files in the same partition - that's why before parallelism is in place I would not like to make any changes to the partitioning/fs as they all might impact it. But again, we should 100% work on docs and document all those behaviors better, at least for each save mode