Code to run after final job failure

[mitar]

I am looking and cannot find a good way to run a piece of code after the job finally fails (after multiple attempts, etc).

The use case here is that I have a web app and I want to offload work to a background job while the user waits for it.

• So when user requests the work (e.g., uploads a file) I make a row into a database where the status is "working" and submit a new job to river, all inside same transaction.
• Then web app polls that row to see when will the status change.
• Now job runs, and if it succeeds it updates the status to "success". I update status and call JobCompleteTx inside a transaction to assure that job's state and row's status are always in sync.
• But what if the job fails? It is repeated. Great. But what if it fails for the last time. How do I get the row's status to be set to "failed" so that the user sees the error?

I was thinking about the following options:

1. I could wrap up my whole work code with logic which checks if this is the last run and in that case it does not really return the err to River, but it sets status to "failed" inside a transaction and call JobCompleteTx. But that means that if I look at River's stats, such job will be counted as succeeded.
2. I could subscribe and then on final failure I update row's status. But what if my subscription fails. Or the client doing the subscription. That is not ideal. Job failures could be missed and user would be stuck with "working" status.
3. I could change my API to fetch status of the row from both row and if that one says "working", it would go and fetch job's status. And then if it notices that job has completed, it would update the row's status. To me it feels like bad separation of concerns.
4. Maybe a workflow job could be done to run after the first job. But that requires River Pro which I do not have. So I didn't look too much into this.

So to me it looks like there is a need for JobFailTx function which would be similar to JobCompleteTx but it would mark job as failed?

I also use river.JobCancel for errors when I know job has failed in a way that it makes no sense to retry. But there is no way to also inside same transaction update my row's status. So I think that approach with river.JobCancel is not the best one. Probably we need JobCancelTx and JobFailTx.

Or is there some other way?

[brandur]

So to me it looks like there is a need for JobFailTx function which would be similar to JobCompleteTx but it would mark job as failed?

Hmm, I think this might turn out to be quite complex for the user code. You'd end up having to internalize all the job failure code into your job's body wouldn't you? Like you wouldn't want to call it unless we're on attempt == max_attempts. It seems like it'd leave a lot of room for error.

I was going to suggest something like a new hook that you could use to plug into the last failure of a job, but that wouldn't provide transactional guarantees. Given this is a very specific use case, it might be more practical just to have a poll loop that looks for discarded jobs and which can be triggered preemptively through the use of a subscription (so it's low latency).

[mitar]

Like you wouldn't want to call it unless we're on attempt == max_attempts.

Yes, is there any other condition like this? Maybe this could be exposed on job argument. Like FinalAttemp() bool method or something.

Given this is a very specific use case

It is? To me it looks like a very general use case, I mean, the whole idea of "transactional enqueueing" is exactly about this. How to keep two separates states in sync when submitting a job. And making it possible to keep states in sync when job is failing is just the other side of the same story. If we are "transactional enqueueing" we should also be doing "transactional cleanup". :-)

Now I am thinking, maybe the easiest way to do this is that I call JobFailTx to mark job as failed and JobFailTx returns to me some flag telling me if job will be retried. I check that flag, and based on that I update my status in my row. And then whole transaction commits or fails. I think this would work well.

So we would have:

• JobCompleteTx - I know it will not repeat - status success
• JobCancelTx - I know it will not repeat - status failure
• JobFailTx - returns the flag, based on that I maybe set status failure

[brandur]

I suppose that makes a certain amount of sense. I'm just trying to think this through though, and it kind of feels like even if such a feature existed, it'd be very impractical to use. Like you'd need a special case around every potential error path in your job body right? And then what does the error path do if JobFailTx fails?

Do you think you could write a sample job implementation with transaction that shows how this would be used, and make sure to include a few different sample return err to show how those would interact with the new feature? This'd help me get a clearer understanding of how you're looking to use this sort of thing.

[mitar]

Like you'd need a special case around every potential error path in your job body right?

I think this can be one defer (+ with named return value) at the beginning and this is it? Or just a simple wrapper around the whole "real" body which on any error calls JobFailTx and this is it. Check the flag, does something about it, and then return the error out.

And then what does the error path do if JobFailTx fails?

Yea, I think JobFailTx will have to handle few cases and details here. You will probably have to store in memory of the worker running the job that JobFailTx was called. And whether it returned an error. Once job finishes (with or without returning an error - JobFailTx should work even if I do not return an error - maybe we should even say that one should not return an error if one does call JobFailTx and maybe returning an error after JobFailTx should mean "disregard me calling JobFailTx, something else happened afterwards, use that") you check if in River database state is really "job failed" and if not, you know that transaction with it was reverted.

So we really have to go through multiple combinations here and define what is a proper behavior:

• JobFailTx is not called, job does not return an error - River marks it as successful outside of a transaction
• JobFailTx is not called, job does return an error - River marks it as failed outside of a transaction
• JobFailTx was called, it did not return an error, job does not return an error, job is marked as failed in the database - happy path, River does not do anything special, repeats the job if retries are still possible
• JobFailTx was called, it did not return an error, job does return an error, job is marked as failed in the database - happy path, River does not do anything special, repeats the job if retries are still possible, maybe stores the error additionally in the database just to be sure it is stored if it is different for easier debugging, maybe AttemptError can be extended with Followup error field or something
• JobFailTx was called, it did not return an error, job does not return an error, job is not marked as failed in the database - ...
• JobFailTx was called, it did not return an error, job does return an error, job is not marked as failed in the database - ...
• JobFailTx was called, it did return an error, job does not return an error, job is marked as failed in the database - ...
• JobFailTx was called, it did return an error, job does return an error, job is marked as failed in the database - ...
• JobFailTx was called, it did return an error, job does not return an error, job is not marked as failed in the database - ...
• JobFailTx was called, it did return an error, job does return an error, job is not marked as failed in the database - ...

I will think about this a bit more in few days, I have one deadline coming up first. I can prepare an example then, too.