[Proposal]: Allow path pattern names with partitioning

[stloyd]

Describe the Proposal

Right now, code like this:

df()
    ->extract(/* */)
    ->partitionBy('order-year', 'order-month', 'order-name')
    ->load(
        to_csv(__DIR__.'/output/{order-name}.csv')
    );

Would fail because Path doesn't allow patterns.

API Adjustments

# UnixPath.php

    public function addPartitions(Partition $partition, Partition ...$partitions) : self
    {
        /** @var Partition[] $partitions */
        $partitions = [$partition, ...$partitions];

        // Here are first changes
        if ($this->isPathPattern($this->path) && [] === $partitions) {
            throw new InvalidArgumentException("Can't use partitions with non-pattern path.");
        }

        $pathInfo = \pathinfo($this->path);
        $dirname = $pathInfo['dirname'] ?? '';
        $filename = $pathInfo['filename'] ?? '';
        $extension = $pathInfo['extension'] ?? '';

        // Here lies most of the "magic"
        if (str_starts_with($filename, '{') && str_ends_with($filename, '}')) {
            foreach ($partitions as $index => $partition) {
                if (!str_contains($filename, $partition->name)) {
                    continue;
                }

                $filename = $partition->value;
                unset($partitions[$index]);
            }
        }

        $basename = $filename . '.' . $extension;
        if ($this->isPathPattern($basename)) {
            throw new InvalidArgumentException("Can't resolve path pattern.");
        }

        $partitionsString = \implode('/', \array_map(static fn (Partition $p) => $p->name . '=' . $p->value, $partitions));

        return match ($dirname) {
            '', '.' => new self($this->protocol->scheme() . '/' . $partitionsString . '/' . $basename, $this->options),
            '/', '\\' => new self($this->protocol->scheme() . '/' . $partitionsString . '/' . $basename, $this->options),
            default => new self($this->protocol->scheme() . $dirname . '/' . $partitionsString . '/' . $basename, $this->options),
        };
    }

Are you intending to also work on proposed change?

Yes

Are you interested in sponsoring this change?

None

Integration & Dependencies

No response

[norberttech]

Let me first make sure I understand the idea here.

So right now how it works is that when you do:

->partitionBy('order-year', 'order-month', 'order-name')

This line tells DataFrame to partition dataset like this (on fs) /order-year=xxx/order-month=xxx/order-name=xxx

So essentially you pass the path as to a single file _DIR__.'/output/file.csv' and data frame will save it as:

/output
  /order-year=xxx
    /order-month=xxx
      /order-name=xxx
        /file.csv

You would like to make it possible to make the last partition to be a output file name instead of a folder?

Something like:

/output
  /order-year=xxx
    /order-month=xxx
      /{order-name}.csv

Do I understand it correctly?

[stloyd]

Yes, and the last part /{order-name}.csv will become the value of data, from the data frame, similarly to how it does for directories.

Like:

/output
  /order-year=xxx
    /order-month=xxx
      /123456-PL.csv

[norberttech]

There are few edge cases to consider here but it could be really nice addition - it's one of the problems with spark that it does not support something like that.

One of the challenges is how to support reading this, because now partitions detection happens on the path directly xxx=xxx if path has something like that, dataframe consider this a partition
and rebuild Rows with that recognized partition.

Another one, is that should be only limited to the last partition, or any partition can become a part of the path like for example: /output/{order_year}_{order_month}/{order_name}.csv

And probably the biggest one is append save mode:

Try to run this one few times: https://flow-php.com/playground/?snippet=5d67c20d874483a17a114dc86f5fc2b354e9325dfc74b78886b7291a62f05df2

So this change would need to be aligned with those existing mechanisms