Using Pipeline Driver in workload

[fschlimb]

This PR makes workloads (examples based on workload class) use the smae pipepline definition and Driver infrastructure as lh-opt

• move type dispatch from TransformStage to Transform
• Accept ready modules as payload and TransformStages
• rename schedule_modules -> pipeline

[rengolin]

Do we really need _ for all field names?

[fschlimb]

We do not need to, but that's python's way to mark them private.

[rengolin]

I think it should be the other way around: Driver imports the rest of the infra.

From an earlier conversation with @tkarna, the Workload object is not a driver nor it is a pipeline.

We need to reach those agreements before starting the cross dependencies between our modules.

[fschlimb]

We can split the Driver, but the workload and Driver will use something which handles the pipeline. I didn't want to create yet another abstraction for no good reason, but if you insist.
I don't see the point of running examples through the Driver. Unless we get rid of workload.

[fschlimb]

IN my mind Workload is a different insfrastructure than pipeline etc. It is a user of most of the other infra.

[tkarna]

One possible way to separate the Workload and Driver is to move the workload.lower_payload method to runner as a helper function.

[rengolin]

We should separate the runner anyway at some point

[rengolin]

But there's also the discussion is the workload is part of the pipeline, or if it contains the pipeline, or if they're orthogonal to each other...

[fschlimb]

In my mind, the workload is a helper/infrastructure class, which helps writing end-to-end examples, including benchmarking and correctness checking. Hence it uses a pipeline etc. and hides away boilerplate code. It is orthogonal to things like lh-opt or lh-run. A different design is of course possible. If we want to change the current approach, this should go into separate PR.

[tkarna]

I'm not particularly fond of this filename or ir.Module dualism.

The way that I see this is Transform is a utility tool to help importing transform schedules from a python or mlir file. With your changes, Transform is needed in TransformStage only to check whether the schedule module contains the expected named sequence and if not, it raises a descriptive error message. So the __str__ method is the only added value of the Transform object.

Suggested changes:

• Rename Transform to something like TransformFileImporter. Move named sequence check there. This could actually also be a helper function, we don't really need the object.
• TransformStage takes a schedule module. (Optionally it can also accept a TransformFileImporter). It could still do another named sequence check if needed.

[fschlimb]

Yes, I had similar thoughts. I am not sure, it that was a motivation in the orig design, but one can create a Transform when there is no context available yet. If we want to keep that possibility (of course only for files/strings) we need a way to defer the Stage assembly. In your approach this would mean we'll need some dualism when creating the TransformStage. So it would only move it from Transform to TransformStage.

[tkarna]

I am not sure, it that was a motivation in the orig design, but one can create a TransformPass when there is no context available yet

I guess you meant Transform here. Hmm, TransformStage does need a context though. So we can only have the how-to-read-from-file metadata without a context.

In your approach this would mean we'll need some dualism when creating the TransformStage.

Yes, if we allow TransformStage to take either TransformFileImporter object or ir.module. We could just accept the latter. The functional implementation of create-transform-stage-from-file would then be something like
TransformStage(import_schedule_module(filename))

[tkarna]

Or we could have TransformStage with the api: TransformStage(schedule_module) and TransformStage.from_file(filename). The latter could set _source_file etc metadata which could be used in error messages if needed. In this case the Transform class would not be needed at all.

[fschlimb]

Would be fine with me.
Still, I don't think it makes the whole thing better. The dualism is basically simply polymorphism, which I don't see as problematic in any way but actually like a lot. With your latest suggestion the user/creator of TransformStage is required to do the dispatch and not the Transform "wrapper". I could argue that the Transform is the right place to do this, not the stage. But as I said, both are fine with me.
What do others think? @rengolin