Add `workflow_job.{queued,waiting,in_progress}` spans to github receiver to allow measuring queueing latency

[arianvp]

Component(s)

receiver/github

Is your feature request related to a problem? Please describe.

Currently the webhook receiver only handles workflow_job.completed event
however it would be very useful to also handle the workflow_job.queued, workflow_job.waiting and workflow_job.in_progress transitions

as with this we can measure the latency of how long it took for a job to be picked up by a runner. This is a useful signal for self-hosted runners as you can detect under-provisioning.

Similarly there is also workflow_run.{requested,in_progress,completed} that would be useful.

Describe the solution you'd like

the code is extended to also add spans for the periods where the workflow job is in queued, waiting and in_progress state.

Describe alternatives you've considered

None

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/github: @adrielp @andrzej-stencel @crobert-1 @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[adrielp]

@arianvp - thanks for opening this enhancement request. This is definitely an area where improvement can be made. Right now you can derive the queue time but in the visualization it's basically "white space" -- so certainly not ideal. Having queue spans would be significantly better.

I'm not sure if it should necessarily come from those events though. Do you have an idea on if the same information is natively available within jobs run events? (I don't remember off hand)

In the code, we actually only create the spans on workflow run/job completed events so as to be able to actually set the span start / end times w/ minimal overhead.

[arianvp]

I'm not sure if it should necessarily come from those events though. Do you have an idea on if the same information is natively available within jobs run events? (I don't remember off hand)

Ah! I just checked and workflow_job has a created_at and a started_at field. I assume we can deduce the queue time from that. and use that to create a span for it

[adrielp]

@arianvp Thank you for checking! That sounds right. We could add a function that runs this calculation and creates the corresponding span.

I think in general, this is something that could really help inform semantics. Would you be able to come to the CICD SIG call this week and discuss this issue with us in more detail? My thinking is that as a group, this could help inform whether or not we should add a queue span to the general semantics.

[arianvp]

Sure! Can you send me an invite to arian [at] mercury [dot] com?

[adrielp]

Thanks! @arianvp it's on the otel calendar under the CICD SIG on Thursdays.

[arianvp]

Just thought of: A benefit of handling the queued,waiting,in_progress events is that we can open spans live as opposed to only create them after a job completed. Imagine a job that is hanging for 6 hours+. You'd only see the span for this appear after it completes as opposed to seeing it get opened immediately.

[adrielp]

After discussion in the CICD SIG today, will be implementing this ASAP using the completed event. The first implementation will be considered a prototype informing the semantics Likely will result in a dedicated span for the scheduled/queue.

[adrielp]

Pull request is open for Job queue spans. This should also help inform the semantic conventions, and aligns with some of the discussions/WIP.

One thing to note, I did not implement workflow RUN queues in this pull request. This is largely because queues cannot be calculated like jobs can. It actually requires the queue event to be handled which is a larger change.

Also, we modeled out last week how queue spans should look and this reflects that modeling in the CICD SIG call.