1. tsc@lists.fd.io

Re: FD.io Production Jenkins Restart Required

Vanessa Valderrama
 

Status Update

Issue: Gateway Timeout Errors

  • Summary: Intermittent Gateway Timeout Errors on the ci-management-jjb-merge jobs are causing stability issues with Jenkins causing unplanned downtime
    • We have put in a change to take Nginx out of the picture and allow the build node to talk directly to Jenkins
    • We'll be monitoring closely to ensure this resolves the issue

Issue: Gerrit cloning timeouts

  • Summary: Intermittent job failures caused by a timeout when closing a Gerrit repo
    • We have opened a Vexxhost ticket for Vexxhost and Ed Kern to troubleshoot the latency within the network the Nomad cluster is on
    • We are also setting up a local Gerrit mirror which should help resolve/improve cloning - this should be complete by the end of the week

Issue: CSIT: s3-t21-sut1 (10.30.51.44) failure

  • Summary: The device s3-t21-sut1 device is having an SSH disk read only issue and is unreachable over NW
    • We've opened a Vexxhost ticket to check the machine

Issue: Hung jobs

  • Summary: Intermittent jobs stuck/hung requiring the job to be aborted
    • We believe this issue was resolved with the latest Jenkins upgrade

Please let me know if you need additional information. If you experience any hung jobs or gateway timeout errors, please open a ticket at support.linuxfoundation.org.

Thank you,
Vanessa

On 11/6/19 9:56 AM, Maciek Konstantynowicz (mkonstan) wrote:
Hi Vanessa, Thanks for the note. CSIT project keeps experiencing issues
due to Jenkins outages. Do you have ETA for the fix that will stop these
outages?

-Maciek


On 5 Nov 2019, at 23:18, Vanessa Valderrama <vvalderrama@...> wrote:

Jenkins has been restarted, job views restored, jobs are running.

We will continue to investigate the Gateway Timeout and JNLP errors
we've been seeing the last couple of days.

If you experience any issues, please open a ticket at
support.linuxfoundation.org

Thank you,
Vanessa


On 11/5/19 4:39 PM, Vanessa Valderrama wrote:

We continue having issues with Gateway Timeouts on the CI merge job which has
corrupted the Jenkins job views.

Jenkins will need to be restarted to resolve this issue.

Thank you,
Vanessa


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#1152): https://lists.fd.io/g/tsc/message/1152
Mute This Topic: https://lists.fd.io/mt/42686762/675185
Group Owner: tsc+owner@...
Unsubscribe: https://lists.fd.io/g/tsc/unsub  [mkonstan@...]
-=-=-=-=-=-=-=-=-=-=-=-

Join tsc@lists.fd.io to automatically receive all group messages.