job reconfigurations


Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco)
 

I spent some time investigating the current behavior.

I do not claim to understand it fully,

but I have formed an opinion.

I am sharing it here, so similarly curious minds

(e.g. me in a year, after forgetting half of this)

do not have to investigate again.

 

Some years ago, the job [0] that edits job configurations

upon merge into ci-management repository was acting smart.

It detected which jobs should have a different configuration,

and updated only those, skipping over the rest.

Nowadays, it "reconfigures" all the jobs, no matter what.

 

Furthermore, job configuration history shows many changes (example: [1]),

although most of them look like this: [2].

The Older Change shows the state after the jjb merge job,

the Newer Change shows the state after the reconfigured job was run

(that is why less active jobs show less frequent configuration changes).

On Sandbox, the Newer Change edits are also applied

when configuring the job manually (via webUI).

 

My opinion: Newer Jenkins version are postponing their edits

(such as adding plugin versions) to avoid lagging

when many jobs are configured (or when plugins are upgraded).

Jenkins Job Builder either does not have enough visibility

into the intended edits, or it does not care (to save CPU cycles).

Also, Jenkins has troubles keeping XML formatting consistent for some time [4].

Config history plugin already hides some edits,

so maybe later versions can hide some of the noise.

In [2] there is already button Hide Version Changes,

although it does not do anything for me.

 

Overall, it seems the lazy behaviors (of both Jenkins and JJB)

seem to work fine together, the typical merge run spends [3]

only 30 seconds (out of its almost 3 minutes of runtime)

reconfiguring 434 jobs and 8 views.

Noisy history and unability to tell which jobs were "really"

reconfigured are the only downsides.

 

Vratko.

[0] https://jenkins.fd.io/view/ci-management/job/ci-management-jjb-merge/

[1] https://jenkins.fd.io/view/ci-management/job/ci-management-jjb-verify/jobConfigHistory/
[2] https://jenkins.fd.io/view/ci-management/job/ci-management-jjb-verify/jobConfigHistory/showDiffFiles?timestamp1=2021-08-26_08-32-22&timestamp2=2021-08-26_08-48-07
[3] https://logs.fd.io/production/vex-yul-rot-jenkins-1/ci-management-jjb-merge/795/console-timestamp.log.gz
[4] https://issues.jenkins.io/browse/JENKINS-41177