Skipt test suite for doc-only changes

[pedro-psb]

Skip tests when there are no code changes.

Test PRs:

• [CI TEST] skip-tests: has code changes pulp_rpm#4112
• [CI TEST] skip-tests: no code changes pulp_rpm#4113

---

Important

This requires plugins to disable "required checks" (on the github repository settings) on anything but the ready-to-ship job. Otherwise, the checks will forever wait those required checks (which are checked in ready-to-ship).
Example (Settings > Branches > Branch Protection Rules):

[pedro-psb]

Results:

With Code Changes

The CI doesn't skip tests, as expected.

Doesnt skip tests

No Code Changes

The CI skips the test, but github is waiting for the skipped ones.
Probably its required to actually read the "ready-to-ship"

Skip tests

But waiting pending

[pedro-psb]

Conclusion: that the validation logic should be handled in "ready-to-ship", not in another check

[pedro-psb]

Moving the logic to ready-to-ship did not resolve the no-code changes situation.
Maybe we can run the test jobs but bypass the script run?

PR looks like waiting checks, even if its effectively done

[pedro-psb]

New conclusion

The check hangs when skipped because they are "required checks" by branch protection rules. This rule applies to main and all release branches. There are ways of bypassing this, but if we can bypass them, why have them in the first place? We could use only the "ready-to-ship" check and put the required logic there.

Proposal

As this is an admin per-repository setting, I'm proposing of making this skip-doc-only-changes feature opt-in.
If a repository wants to use it, it should disable protection for those checks [0], toggle the skip_tests_on_doc_only_changes flag in template_config.yml and apply plugin-template again.

[0] Protection rules are located in: {repo} > Settings > Branches > Branch protection rules

Examples ways of bypassing but which are very messy

• creating a twin ci.yml, one with ignore-path: docs and the other with the opposite. One of them can use the job names required by the checks but do nothing.
• using conditionals, because skipping is usually considered success for those required checks, but they must be skipped very explicitly (e.g, if test doesnt run because lint didnt run, it'll hang. Also that's tricky with test.yml matrix, because skipping the whole test job is not enough, and github doesn't allow skipping individual matrix runs (which is what would work in this case).

About the test PRs

On the pulp/pulp_rpm#4113 (doc only changes) test PR, I've created a branch that doesn't have those checks to assert that it will correctly skip and not hang waiting for checks.

On the pulp/pulp_rpm#4112 (has code changes) test PR, the tests run as expected.

On the default case where skip_tests_on_doc_only_changes is left as false, the skip is trivially ignored.

[mdellweg]

Suggested change

	needs: check-changes
	if: needs.check-changes.outputs.run_tests == '1'
	{%- if pre_job_template %}
	needs: {{ pre_job_template.name }}
	needs: [check-changes, {{ pre_job_template.name }}]
	needs:
	- "check-changes"
	{%- if pre_job_template %}
	- "{{ pre_job_template.name }}"
	{%- endif %}
	if: needs.check-changes.outputs.run_tests == '1'

The endif below needs to be removed...