Dynamic Workflow Support

[markovejnovic]

Issue Description

Currently, hm expects you to define static pipelines, which define the build graph completely at bake time.

hm takes the given Python/TS and converts it to an IR. The IR is a JSON definition of the build graph which fully encodes the necessary steps to run a CI/CD pipeline. Converting the py/ts definition into the IR is what we call baking. Pipelines run in a bi-phasic manner — first we bake the definition into the IR and then we execute the IR.

Unfortunately, this adds some friction — we cannot run conditional (or even complex) logic within each step. If you were to write

if hm.env("FOO") == "bar":
  return hm.sh("xyz")
else:
  return hm.sh("abc")

we'd have no mechanism to actually execute the conditional logic during the bake step, as the IR cannot express that.

Proposed Solutions

Dependency Injection at bake time

There is a relatively naive way to solve the aforementioned problem — during bake, we can inject environment variables. For example, doing FOO=bar hm run, would cause the bake step to observe the FOO environment variable, which would then bake a static IR which we then blindly execute.

Not bad for a first pass, but has some problems — runtime information is unavailable. For example, the following pipeline would not be expressible:

@hm.target()
def some_step():
  if datetime.now().minute == 30:
    return hm.sh("curl -X POST https://foo.bar/logic-for-half-hour")
  else:
    return hm.sh("curl -X POST https://foo.bar/logic-for-else")

This example leads me to believe that we should further granulize our bake-execute logic to be per-step.

Granulized Baking

Rather than bake the whole pipeline, we can delay baking for dynamic targets. I propose that we add a new construct to the IR dynamic, which causes the given step, and all subsequent steps to be baked during the execution of the pipeline.

I am imagining something like

@hm.target(dynamic=True)
def some_step():
  if datetime.now().minute == 30:
    return hm.sh("curl -X POST https://foo.bar/logic-for-half-hour")
  else:
    return hm.sh("curl -X POST https://foo.bar/logic-for-else")

results in an IR step:

{
  "key": "...",
  "label": "...",
  "eval": {
    "type": "dynamic",
    "target_name": "some_step"
  }
}

and which would migrate all our current steps to use the following IR:

 {
   "key": "...",
   "label": "...",
-  "cmd": "<...>",
+  "eval": {
+    "type": "cmd",
+    "cmd": "<...>"
+  }
 }

I suspect this will mostly work as expected, however, there is a caveat to be mindful of. We support a concept of groups, which allow users to bundle multiple steps in one target. One immediate problem I see is that if a dynamic pipeline returns a group of steps, it's not immediately clear on which of the steps in a group subsequent steps need to run in.