On 5 Nov 2019, at 23:18, Vanessa Valderrama <vvalderrama@linuxfoundation.org> wrote:

Jenkins has been restarted, job views restored, jobs are running.

We will continue to investigate the Gateway Timeout and JNLP errors
we've been seeing the last couple of days.

If you experience any issues, please open a ticket at
support.linuxfoundation.org

Thank you,
Vanessa


On 11/5/19 4:39 PM, Vanessa Valderrama wrote:

We continue having issues with Gateway Timeouts on the CI merge job which has
corrupted the Jenkins job views.

Jenkins will need to be restarted to resolve this issue.

Thank you,
Vanessa

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#1152): https://lists.fd.io/g/tsc/message/1152
Mute This Topic: https://lists.fd.io/mt/42686762/675185
Group Owner: tsc+owner@lists.fd.io
Unsubscribe: https://lists.fd.io/g/tsc/unsub [mkonstan@cisco.com]
-=-=-=-=-=-=-=-=-=-=-=-

We continue having issues with Gateway Timeouts on the CI merge job which has
corrupted the Jenkins job views.

Jenkins will need to be restarted to resolve this issue.

Thank you,
Vanessa

Jenkins has been restarted.

We're pushing the jobs now.


On 11/4/19 2:22 PM, Vanessa Valderrama wrote:

We are having issues with Gateway Timeouts on the CI merge job which has
corrupted the Jenkins job views. We are aware aware of the issue and
will have it resolved shortly.

Jenkins will be placed in shutdown mode while we generate the jobs.

Thank you,
Vanessa

We are having issues with Gateway Timeouts on the CI merge job which has
corrupted the Jenkins job views. We are aware aware of the issue and
will have it resolved shortly.

Jenkins will be placed in shutdown mode while we generate the jobs.

Thank you,
Vanessa

We need to extend the maintenance window until 2300 to allow the volume migration to complete.

On 10/23/2019 11:10 AM, Vanessa Valderrama wrote:

Jenkins has been placed in shtudown mode in preparation for maintenance.

On 10/22/2019 09:35 AM, Vanessa Valderrama wrote:

Maintenance reminder

On Oct 17, 2019, at 1:04 PM, Vanessa Valderrama <vvalderrama@...> wrote:

What:

LF will be performing standard system maintenance and a Jenkins migration

• Jenkins
  
• Migrate to two new SSD volumes
• OS updates
  
• Jenkins upgrade to 2.190.1
• Plugin upgrades
• Nexus
  
• OS updates
• Nexus upgrade to 2.14.15-01
• Jira
  
• OS updates
• Jira upgrade to 7.13.9
• Gerrit
  
• OS updates
• Gerrit upgrade to 2.16.12
• Sonar
• OS updates
• OpenGrok
• OS updates

When:

2019-10-23 1700 UTC to 2100 UTC

Impact:

Maintenance will require a reboot of each FD.io system. Jenkins will be placed in shutdown mode at 1600 UTC. Please let us know if specific jobs cannot be aborted.

The following systems will be unavailable during the maintenance window:

• Jenkins sandbox
• Jenkins production
• Nexus
• Jira
• Gerrit
• Sonar
• OpenGrok
  

Jenkins has been placed in shtudown mode in preparation for maintenance.

On 10/22/2019 09:35 AM, Vanessa Valderrama wrote:

Maintenance reminder

On Oct 17, 2019, at 1:04 PM, Vanessa Valderrama <vvalderrama@...> wrote:

What:

LF will be performing standard system maintenance and a Jenkins migration

• Jenkins
  
• Migrate to two new SSD volumes
• OS updates
  
• Jenkins upgrade to 2.190.1
• Plugin upgrades
• Nexus
  
• OS updates
• Nexus upgrade to 2.14.15-01
• Jira
  
• OS updates
• Jira upgrade to 7.13.9
• Gerrit
  
• OS updates
• Gerrit upgrade to 2.16.12
• Sonar
• OS updates
• OpenGrok
• OS updates

When:

2019-10-23 1700 UTC to 2100 UTC

Impact:

Maintenance will require a reboot of each FD.io system. Jenkins will be placed in shutdown mode at 1600 UTC. Please let us know if specific jobs cannot be aborted.

The following systems will be unavailable during the maintenance window:

• Jenkins sandbox
• Jenkins production
• Nexus
• Jira
• Gerrit
• Sonar
• OpenGrok
  

Maintenance reminder

On Oct 17, 2019, at 1:04 PM, Vanessa Valderrama <vvalderrama@...> wrote:

What:

LF will be performing standard system maintenance and a Jenkins migration

• Jenkins
  
• Migrate to two new SSD volumes
• OS updates
  
• Jenkins upgrade to 2.190.1
• Plugin upgrades
• Nexus
  
• OS updates
• Nexus upgrade to 2.14.15-01
• Jira
  
• OS updates
• Jira upgrade to 7.13.9
• Gerrit
  
• OS updates
• Gerrit upgrade to 2.16.12
• Sonar
• OS updates
• OpenGrok
• OS updates

When:

2019-10-23 1700 UTC to 2100 UTC

Impact:

Maintenance will require a reboot of each FD.io system. Jenkins will be placed in shutdown mode at 1600 UTC. Please let us know if specific jobs cannot be aborted.

The following systems will be unavailable during the maintenance window:

• Jenkins sandbox
• Jenkins production
• Nexus
• Jira
• Gerrit
• Sonar
• OpenGrok
  

Starting maintenance

On 09/25/2019 11:02 AM, Vanessa Valderrama wrote:

Jenkins has been placed in shut down mode to prepare for maintenance.

On 09/23/2019 04:08 PM, Vanessa Valderrama wrote:

What:

LF will performing maintenance on the Nexus server to migrate data to two new SSD volumes in attempt to resolve the intermittent issue with hung jobs we are experiencing

• Migrate Nexus data to new volumes

When:
2019-09-25 1700 UTC to 2100 UTC

Impact: We will place Jenkins in shutdown mode at 1600 UTC. Jobs will be aborted at 1700 UTC
Jenkins and Nexus will be unavailable during this time

You can subscribe to status updates here:
https://fdio.statuspage.io/incidents/f5tfpfdyd8l0

Jenkins has been placed in shut down mode to prepare for maintenance.

On 09/23/2019 04:08 PM, Vanessa Valderrama wrote:

What:

LF will performing maintenance on the Nexus server to migrate data to two new SSD volumes in attempt to resolve the intermittent issue with hung jobs we are experiencing

• Migrate Nexus data to new volumes

When:
2019-09-25 1700 UTC to 2100 UTC

Impact: We will place Jenkins in shutdown mode at 1600 UTC. Jobs will be aborted at 1700 UTC
Jenkins and Nexus will be unavailable during this time

You can subscribe to status updates here:
https://fdio.statuspage.io/incidents/f5tfpfdyd8l0

What:

LF will performing maintenance on the Nexus server to migrate data to two new SSD volumes in attempt to resolve the intermittent issue with hung jobs we are experiencing

• Migrate Nexus data to new volumes

When:
2019-09-25 1700 UTC to 2100 UTC

Impact: We will place Jenkins in shutdown mode at 1600 UTC. Jobs will be aborted at 1700 UTC
Jenkins and Nexus will be unavailable during this time

You can subscribe to status updates here:
https://fdio.statuspage.io/incidents/f5tfpfdyd8l0

What:

LF will performing maintenance on the Nexus server to migrate data to two new SSD volumes in attempt to resolve the intermittent issue with hung jobs we are experiencing

• Migrate Nexus data to new volumes

When:
2019-09-25 1700 UTC to 2100 UTC

Impact: We will place Jenkins in shutdown mode at 1600 UTC. Jobs will be aborted at 1700 UTC
Jenkins and Nexus will be unavailable during this time

You can subscribe to status updates here:
https://fdio.statuspage.io/incidents/f5tfpfdyd8l0

We're going to terminate the two CSIT jobs. I spoke with Dave Wallace
and we felt it'd be better to terminate the jobs and restart Jenkins.

Thank you,
Vanessa

On 09/11/2019 02:17 PM, Vanessa Valderrama wrote:

Jenkins is still in shutdown mode. We'll do the restart when these jobs
are complete.

https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/124/
https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/125/

On 09/11/2019 01:09 PM, Vanessa Valderrama wrote:

Jenkins has been placed in shutdown mode in preparation for a restart.

We were seeing Gateway Time-out errors on ci-management-jjb-merge jobs
which has caused problems with the job configurations. I have tried to
push the jobs manually but I'm getting the same errors.

We are going to restart Jenkins and try to do another manual push of the
job configs.

Thank you,
Vanessa

Jenkins is still in shutdown mode. We'll do the restart when these jobs
are complete.

https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/124/
https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/125/

On 09/11/2019 01:09 PM, Vanessa Valderrama wrote:

Jenkins has been placed in shutdown mode in preparation for a restart.

We were seeing Gateway Time-out errors on ci-management-jjb-merge jobs
which has caused problems with the job configurations. I have tried to
push the jobs manually but I'm getting the same errors.

We are going to restart Jenkins and try to do another manual push of the
job configs.

Thank you,
Vanessa

Jenkins has been placed in shutdown mode in preparation for a restart.

We were seeing Gateway Time-out errors on ci-management-jjb-merge jobs
which has caused problems with the job configurations. I have tried to
push the jobs manually but I'm getting the same errors.

We are going to restart Jenkins and try to do another manual push of the
job configs.

Thank you,
Vanessa