Bad handling of CPU scalars passed to `actx.compile`d functions

[inducer]

(Currently: based on guesswork and second-hand information from @matthiasdiener. Away at a conference, will investigate more when I have time.)

Right now:

• Sees CPU scalar passed, generates pytato Placeholder with shape=().
• That becomes an ArrayArg from Loopy's perspective
• We then spend tons of time transferring these itty bitty things to the GPU.

Proposed remedy:

• Introduce a tag in pytato that says "this placeholder should become a ValueArg.
• Apply that in arraycontext when creating the Placeholders if appropriate.
• Respect the tag in pytato codegen.

This should remove the cost of these transfers.

[matthiasdiener]

Here is a profile of a smoke_test_shock1d run on 1 rank Tuolumne (AMD APU):

before:

With the two PRs, #304 and inducer/pytato#587, as well as disabling all host-side computation (status, health, viz, restart) in the driver, time step time for that case is reduced from 0.5s to 0.025s. 🎉

after:
(note that this is higher than 0.025s due to profiling overhead)

[inducer]

👀 ❗

Well, it seems that that was worth doing. 🙂

I'm curious what's inside that my_rhs call. (Probably a lot of argument upnacking.) At any rate, it's likely that we can shrink this quite a bit still.