Date   

FD.io - JIRA Maintenance 2020-11-23 at 0300 UTC to 0600 UTC

Vanessa Valderrama
 

What:  JIRA maintenance to migrate users from LDAP to Internal directory

When:   2020-11-23 at 0300 UTC to 0600 UTC

Impact:  JIRA will be unavailable during this time

Why:  This change is related to the JIRA Auth0 migration

Thank you,
Vanessa


CANCELED - Re: FDIO Gerrit Maintenance - 2020-11-04 at 1700 UTC to 1900 UTC

Vanessa Valderrama
 

This maintenance is canceled. The Gerrit resize was complete today during the Jenkins restart.

Thank you,

Vanessa

On 10/29/20 11:51 AM, Vanessa Valderrama wrote:

Correction

When:   2020-11-04 at 1700 UTC to 1900 UTC

Thank you,
Vanessa


On 10/28/20 12:12 PM, Vanessa Valderrama wrote:

What:  Gerrit maintenance to resize the instance

When:   2020-10-04 at 1700 UTC to 1900 UTC

Impact:  Gerrit and Jenkins will be unavailable at this time. Jenkins will be placed in shutdown mode at 1700 UTC. At 1800 UTC jobs will be aborted

Why:  This increase is being done at the request of the FD.io community

Thank you,
Vanessa


Re: FD.io Nomad Issue

Vanessa Valderrama
 

Jenkins has been restarted and jobs are running again.

We got approval to the Gerrit resize at the time so next week's
maintenance will be cancelled.

Thank you,

Vanessa

On 10/29/20 1:33 PM, Vanessa Valderrama wrote:
The community has requested a restart of Jenkins. We're placing Jenkins
in shutdown mode to prepare for the restart.

Thank you,

Vanessa

On 10/29/20 11:26 AM, Vanessa Valderrama wrote:
Nomad executors are not starting in Jenkins. This was due to the DNS for
the Nomad URL in Jenkins which is configured to use
nomad.fdiopoc.net:4646 pointing to the wrong IP address.

; <<>> DiG 9.11.14-RedHat-9.11.14-2.fc30 <<>> nomad.fdiopoc.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22624
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;nomad.fdiopoc.net.        IN    A
;; ANSWER SECTION:
nomad.fdiopoc.net.    180    IN    A    157.230.67.179
;; Query time: 38 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Oct 29 09:55:49 CDT 2020
;; MSG SIZE  rcvd: 62

We tried hard-coding the Nomad URL to the IP address 10.30.51.32:4646
and 10.39.51.33:4646. Unfortunately that is not resolving the issue.

We will continue to work with the community to resolve this issue as
quickly as possible.

Thank you,
Vanessa


Re: FD.io Nomad Issue

Vanessa Valderrama
 

The community has requested a restart of Jenkins. We're placing Jenkins
in shutdown mode to prepare for the restart.

Thank you,

Vanessa

On 10/29/20 11:26 AM, Vanessa Valderrama wrote:
Nomad executors are not starting in Jenkins. This was due to the DNS for
the Nomad URL in Jenkins which is configured to use
nomad.fdiopoc.net:4646 pointing to the wrong IP address.

; <<>> DiG 9.11.14-RedHat-9.11.14-2.fc30 <<>> nomad.fdiopoc.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22624
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;nomad.fdiopoc.net.        IN    A
;; ANSWER SECTION:
nomad.fdiopoc.net.    180    IN    A    157.230.67.179
;; Query time: 38 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Oct 29 09:55:49 CDT 2020
;; MSG SIZE  rcvd: 62

We tried hard-coding the Nomad URL to the IP address 10.30.51.32:4646
and 10.39.51.33:4646. Unfortunately that is not resolving the issue.

We will continue to work with the community to resolve this issue as
quickly as possible.

Thank you,
Vanessa


Re: FDIO Gerrit Maintenance - 2020-11-04 at 1700 UTC to 1900 UTC

Vanessa Valderrama
 

Correction

When:   2020-11-04 at 1700 UTC to 1900 UTC

Thank you,
Vanessa


On 10/28/20 12:12 PM, Vanessa Valderrama wrote:

What:  Gerrit maintenance to resize the instance

When:   2020-10-04 at 1700 UTC to 1900 UTC

Impact:  Gerrit and Jenkins will be unavailable at this time. Jenkins will be placed in shutdown mode at 1700 UTC. At 1800 UTC jobs will be aborted

Why:  This increase is being done at the request of the FD.io community

Thank you,
Vanessa


FD.io Nomad Issue

Vanessa Valderrama
 

Nomad executors are not starting in Jenkins. This was due to the DNS for
the Nomad URL in Jenkins which is configured to use
nomad.fdiopoc.net:4646 pointing to the wrong IP address.

; <<>> DiG 9.11.14-RedHat-9.11.14-2.fc30 <<>> nomad.fdiopoc.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22624
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;nomad.fdiopoc.net.        IN    A
;; ANSWER SECTION:
nomad.fdiopoc.net.    180    IN    A    157.230.67.179
;; Query time: 38 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Oct 29 09:55:49 CDT 2020
;; MSG SIZE  rcvd: 62

We tried hard-coding the Nomad URL to the IP address 10.30.51.32:4646
and 10.39.51.33:4646. Unfortunately that is not resolving the issue.

We will continue to work with the community to resolve this issue as
quickly as possible.

Thank you,
Vanessa


FDIO Gerrit Maintenance - 2020-10-04 at 1700 UTC to 1900 UTC

Vanessa Valderrama
 

What:  Gerrit maintenance to resize the instance

When:   2020-10-04 at 1700 UTC to 1900 UTC

Impact:  Gerrit and Jenkins will be unavailable at this time. Jenkins will be placed in shutdown mode at 1700 UTC. At 1800 UTC jobs will be aborted

Why:  This increase is being done at the request of the FD.io community

Thank you,
Vanessa


Re: FD.io Jenkins

Vanessa Valderrama
 

Jenkins service has been restored. The UI instability was a result of a
Jenkins OOM issue. We increased the heap size, increased the
KeepAliveTimeout, upgraded the Jenkins version and restarted the service.

We'll continue to monitor Jenkins for performance issues. If you
experience any issue, please open a ticket at support.linuxfoundation.org.

Thank you,
Vanessa

On 10/19/20 7:52 AM, Vanessa Valderrama wrote:
We are currently seeing intermittent slowness on jenkins.fd.io. We're
placing Jenkins in shutdown mode while we troubleshoot the issue and in
preparation for a possible restart.

Thank you,

Vanessa


FD.io Jenkins

Vanessa Valderrama
 

We are currently seeing intermittent slowness on jenkins.fd.io. We're
placing Jenkins in shutdown mode while we troubleshoot the issue and in
preparation for a possible restart.

Thank you,

Vanessa


Re: [tsc] Emergency upgrade of Nexus system

Andrew Grimberg
 

This work has been completed.

-Andy-

On 2020-10-15 12:04, Andrew Grimberg via lists.fd.io wrote:
We just received word that both Nexus 2 and Nexus 3 are vulnerable to a
security issue that was just announced with possible attacks already in
the wild.

Given the nature of the security issue we will be performing emergency
upgrades on the single Nexus system that FD.io has in play. I have
already placed both Jenkins systems into shutdown mode to stop new jobs
from starting.

Given the length of the jobs already running and their ETA for
completion. We will be running the upgrade in while the jobs are in
progress. If you see a failure in your job due to this forced upgrade we
apologize but due to the nature of this we must get this update rolled out.

-Andy-




--
Andrew J Grimberg
Manager Release Engineering
The Linux Foundation


Emergency upgrade of Nexus system

Andrew Grimberg
 

We just received word that both Nexus 2 and Nexus 3 are vulnerable to a
security issue that was just announced with possible attacks already in
the wild.

Given the nature of the security issue we will be performing emergency
upgrades on the single Nexus system that FD.io has in play. I have
already placed both Jenkins systems into shutdown mode to stop new jobs
from starting.

Given the length of the jobs already running and their ETA for
completion. We will be running the upgrade in while the jobs are in
progress. If you see a failure in your job due to this forced upgrade we
apologize but due to the nature of this we must get this update rolled out.

-Andy-
--
Andrew J Grimberg
Manager Release Engineering
The Linux Foundation


Re: FD.io Jenkins Maintenance: 2020-10-19 1700 UTC to 2200 UTC

Vanessa Valderrama
 

Maintenance has been completed successfully. All service are available.

As part of maintenance we downgraded the Gerrit Trigger plugin in Jenkins to avoid a re-occurrence of the incident we had today. It appears there is a defect in the latest version of the plugin that causes Gerrit triggers to stop triggering builds in Jenkins causing the queue to grow out of control and affect the stability of the Jenkins system. The root cause is unknown at this time. The workaround is to restart Jenkins.

If you experience any issues, please open a ticket at support.linuxfoundation.org.

Thank you,
Anton & Vanessa


On 10/7/20 9:36 AM, Vanessa Valderrama wrote:

Due to the unexpected Jenkins outage, we are going to perform this maintenance now. This maintenance has been approved by the VPP and CSIT teams.

Thank you,
Vanessa

On 10/5/20 12:11 PM, Vanessa Valderrama wrote:

What:
  • Ingress
    • Increase the size of the instance
    • OS and security updates
  • Jenkins
    • OS and security updates
    • Upgrade to 2.249.1
    • Plugin updates
  • Nexus
    • OS updates
    • Upgrade to 2.14.19-01
  • Jira
    • OS updates
    • Upgrade to 8.12.2
  • Gerrit
    • OS updates
When:  2020-10-19 1700 UTC to 2200 UTC

Impact:

All systems will be unavailable during the maintenance window. Jenkins will be placed in shutdown mode at 1600 UTC. We will abort all jobs at 1700 UTC.


Re: FD.io Jenkins Maintenance: 2020-10-19 1700 UTC to 2200 UTC

Vanessa Valderrama
 

Due to the unexpected Jenkins outage, we are going to perform this maintenance now. This maintenance has been approved by the VPP and CSIT teams.

Thank you,
Vanessa

On 10/5/20 12:11 PM, Vanessa Valderrama wrote:

What:
  • Ingress
    • Increase the size of the instance
    • OS and security updates
  • Jenkins
    • OS and security updates
    • Upgrade to 2.249.1
    • Plugin updates
  • Nexus
    • OS updates
    • Upgrade to 2.14.19-01
  • Jira
    • OS updates
    • Upgrade to 8.12.2
  • Gerrit
    • OS updates
When:  2020-10-19 1700 UTC to 2200 UTC

Impact:

All systems will be unavailable during the maintenance window. Jenkins will be placed in shutdown mode at 1600 UTC. We will abort all jobs at 1700 UTC.


FD.io Jenkins Unavailable

Vanessa Valderrama
 

We are currently experiencing issues with Jenkins production. We are
investigating this issue and working on resolving it as quickly as possible.

Thank you,
Vanessa


FD.io Jenkins Maintenance: 2020-10-19 1700 UTC to 2200 UTC

Vanessa Valderrama
 

What:
  • Ingress
    • Increase the size of the instance
    • OS and security updates
  • Jenkins
    • OS and security updates
    • Upgrade to 2.249.1
    • Plugin updates
  • Nexus
    • OS updates
    • Upgrade to 2.14.19-01
  • Jira
    • OS updates
    • Upgrade to 8.12.2
  • Gerrit
    • OS updates
When:  2020-10-19 1700 UTC to 2200 UTC

Impact:

All systems will be unavailable during the maintenance window. Jenkins will be placed in shutdown mode at 1600 UTC. We will abort all jobs at 1700 UTC.


Re: FDIO Gerrit Maintenance - 2019-09-10 @ 1700 UTC

Vanessa Valderrama
 

If you are using an HTTP protocol for Gerrit instead of using your LFID password you will need to request a new token from Gerrit. If you have any issues authenticating, please open a ticket at support.linuxfoundation.org.

Thanks,
Vanessa

On 9/10/20 3:29 PM, Vanessa Valderrama wrote:

Maintenance is complete. Thank you for your patience. Please open a ticket at support.linuxfoundation.org if you have any issues.

Thank you,
Vanessa

On 9/10/20 3:17 PM, Vanessa Valderrama wrote:

Again maintenance is taking longer than expected. We should be finished at the top of the hours.

Thank you,

Vanessa

On 9/10/20 1:55 PM, Vanessa Valderrama wrote:

We'll need to extend this maintenance window by approximately 30 minutes. I apologize for the inconvenience, we got a later start than planned to allow some existing jobs to finish.

Thank you for your patience.

On 9/10/20 12:11 PM, Vanessa Valderrama wrote:

Starting maintenance.


On 9/4/20 2:53 PM, Vanessa Valderrama wrote:

What:

Linux Foundation will be performing maintenance to migrate to our new SSO service in effort to roll out a more consistent login experience across all Linux Foundation project services.

The login process will use sso.linuxfoundation.org. It will be similar to the JSD login.

There is a FAQ on the transition available at https://identity.linuxfoundation.org/migration-faq
  • Gerrit upgrade to 3.2
  • Gerrit migration to Auth0
When:
Jenkins sandbox - 2019-09-10 @ 1700 UTC

Impact:
Jenkins will be placed in shutdown mode during  Gerrit upgrade and migration. Gerrit will be unavailble during the upgrade and restart.


Re: FDIO Gerrit Maintenance - 2019-09-10 @ 1700 UTC

Vanessa Valderrama
 

Maintenance is complete. Thank you for your patience. Please open a ticket at support.linuxfoundation.org if you have any issues.

Thank you,
Vanessa

On 9/10/20 3:17 PM, Vanessa Valderrama wrote:

Again maintenance is taking longer than expected. We should be finished at the top of the hours.

Thank you,

Vanessa

On 9/10/20 1:55 PM, Vanessa Valderrama wrote:

We'll need to extend this maintenance window by approximately 30 minutes. I apologize for the inconvenience, we got a later start than planned to allow some existing jobs to finish.

Thank you for your patience.

On 9/10/20 12:11 PM, Vanessa Valderrama wrote:

Starting maintenance.


On 9/4/20 2:53 PM, Vanessa Valderrama wrote:

What:

Linux Foundation will be performing maintenance to migrate to our new SSO service in effort to roll out a more consistent login experience across all Linux Foundation project services.

The login process will use sso.linuxfoundation.org. It will be similar to the JSD login.

There is a FAQ on the transition available at https://identity.linuxfoundation.org/migration-faq
  • Gerrit upgrade to 3.2
  • Gerrit migration to Auth0
When:
Jenkins sandbox - 2019-09-10 @ 1700 UTC

Impact:
Jenkins will be placed in shutdown mode during  Gerrit upgrade and migration. Gerrit will be unavailble during the upgrade and restart.


Re: FDIO Gerrit Maintenance - 2019-09-10 @ 1700 UTC

Vanessa Valderrama
 

Again maintenance is taking longer than expected. We should be finished at the top of the hours.

Thank you,

Vanessa

On 9/10/20 1:55 PM, Vanessa Valderrama wrote:

We'll need to extend this maintenance window by approximately 30 minutes. I apologize for the inconvenience, we got a later start than planned to allow some existing jobs to finish.

Thank you for your patience.

On 9/10/20 12:11 PM, Vanessa Valderrama wrote:

Starting maintenance.


On 9/4/20 2:53 PM, Vanessa Valderrama wrote:

What:

Linux Foundation will be performing maintenance to migrate to our new SSO service in effort to roll out a more consistent login experience across all Linux Foundation project services.

The login process will use sso.linuxfoundation.org. It will be similar to the JSD login.

There is a FAQ on the transition available at https://identity.linuxfoundation.org/migration-faq
  • Gerrit upgrade to 3.2
  • Gerrit migration to Auth0
When:
Jenkins sandbox - 2019-09-10 @ 1700 UTC

Impact:
Jenkins will be placed in shutdown mode during  Gerrit upgrade and migration. Gerrit will be unavailble during the upgrade and restart.


Re: FDIO Gerrit Maintenance - 2019-09-10 @ 1700 UTC

Vanessa Valderrama
 

We'll need to extend this maintenance window by approximately 30 minutes. I apologize for the inconvenience, we got a later start than planned to allow some existing jobs to finish.

Thank you for your patience.

On 9/10/20 12:11 PM, Vanessa Valderrama wrote:

Starting maintenance.


On 9/4/20 2:53 PM, Vanessa Valderrama wrote:

What:

Linux Foundation will be performing maintenance to migrate to our new SSO service in effort to roll out a more consistent login experience across all Linux Foundation project services.

The login process will use sso.linuxfoundation.org. It will be similar to the JSD login.

There is a FAQ on the transition available at https://identity.linuxfoundation.org/migration-faq
  • Gerrit upgrade to 3.2
  • Gerrit migration to Auth0
When:
Jenkins sandbox - 2019-09-10 @ 1700 UTC

Impact:
Jenkins will be placed in shutdown mode during  Gerrit upgrade and migration. Gerrit will be unavailble during the upgrade and restart.


Re: FDIO Gerrit Maintenance - 2019-09-10 @ 1700 UTC

Vanessa Valderrama
 

Starting maintenance.


On 9/4/20 2:53 PM, Vanessa Valderrama wrote:

What:

Linux Foundation will be performing maintenance to migrate to our new SSO service in effort to roll out a more consistent login experience across all Linux Foundation project services.

The login process will use sso.linuxfoundation.org. It will be similar to the JSD login.

There is a FAQ on the transition available at https://identity.linuxfoundation.org/migration-faq
  • Gerrit upgrade to 3.2
  • Gerrit migration to Auth0
When:
Jenkins sandbox - 2019-09-10 @ 1700 UTC

Impact:
Jenkins will be placed in shutdown mode during  Gerrit upgrade and migration. Gerrit will be unavailble during the upgrade and restart.