Date   

FD.io Nexus Maintenance: 2019-09-25 1700 UTC to 2100 UTC

Vanessa Valderrama
 

What:

LF will performing maintenance on the Nexus server to migrate data to two new SSD volumes in attempt to resolve the intermittent issue with hung jobs we are experiencing

  • Migrate Nexus data to new volumes
When:
2019-09-25 1700 UTC to 2100 UTC

Impact:
We will place Jenkins in shutdown mode at 1600 UTC. Jobs will be aborted at 1700 UTC
Jenkins and Nexus will be unavailable during this time

You can subscribe to status updates here:
https://fdio.statuspage.io/incidents/f5tfpfdyd8l0



Re: FD.io Jenkins Restart

Vanessa Valderrama
 

Jenkins has been restarted and jobs have been restored.

On 09/11/2019 03:11 PM, Vanessa Valderrama wrote:
We're going to terminate the two CSIT jobs. I spoke with Dave Wallace
and we felt it'd be better to terminate the jobs and restart Jenkins.

Thank you,
Vanessa

On 09/11/2019 02:17 PM, Vanessa Valderrama wrote:
Jenkins is still in shutdown mode. We'll do the restart when these jobs
are complete.

https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/124/
https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/125/

On 09/11/2019 01:09 PM, Vanessa Valderrama wrote:
Jenkins has been placed in shutdown mode in preparation for a restart.

We were seeing Gateway Time-out errors on ci-management-jjb-merge jobs
which has caused problems with the job configurations. I have tried to
push the jobs manually but I'm getting the same errors.

We are going to restart Jenkins and try to do another manual push of the
job configs.

Thank you,
Vanessa


Re: FD.io Jenkins Restart

Vanessa Valderrama
 

We're going to terminate the two CSIT jobs. I spoke with Dave Wallace
and we felt it'd be better to terminate the jobs and restart Jenkins.

Thank you,
Vanessa

On 09/11/2019 02:17 PM, Vanessa Valderrama wrote:
Jenkins is still in shutdown mode. We'll do the restart when these jobs
are complete.

https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/124/
https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/125/

On 09/11/2019 01:09 PM, Vanessa Valderrama wrote:
Jenkins has been placed in shutdown mode in preparation for a restart.

We were seeing Gateway Time-out errors on ci-management-jjb-merge jobs
which has caused problems with the job configurations. I have tried to
push the jobs manually but I'm getting the same errors.

We are going to restart Jenkins and try to do another manual push of the
job configs.

Thank you,
Vanessa


Re: FD.io Jenkins Restart

Vanessa Valderrama
 

Jenkins is still in shutdown mode. We'll do the restart when these jobs
are complete.

https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/124/
https://jenkins.fd.io/job/csit-vpp-perf-verify-1908-2n-skx/125/

On 09/11/2019 01:09 PM, Vanessa Valderrama wrote:
Jenkins has been placed in shutdown mode in preparation for a restart.

We were seeing Gateway Time-out errors on ci-management-jjb-merge jobs
which has caused problems with the job configurations. I have tried to
push the jobs manually but I'm getting the same errors.

We are going to restart Jenkins and try to do another manual push of the
job configs.

Thank you,
Vanessa


FD.io Jenkins Restart

Vanessa Valderrama
 

Jenkins has been placed in shutdown mode in preparation for a restart.

We were seeing Gateway Time-out errors on ci-management-jjb-merge jobs
which has caused problems with the job configurations. I have tried to
push the jobs manually but I'm getting the same errors.

We are going to restart Jenkins and try to do another manual push of the
job configs.

Thank you,
Vanessa


Re: FD.io Nexus Outage

Vanessa Valderrama
 

Nexus is back up. We disabled the plugin that caused the outage.

We also had to restart Jenkins which is back up now.

Thank you,
Vanessa

On 09/06/2019 03:18 PM, Vanessa Valderrama wrote:
Nexus is currently down. To prevent necessary job failures we've placed
Jenkins into shutdown mode while we resolve this issue.


FD.io Nexus Outage

Vanessa Valderrama
 

Nexus is currently down. To prevent necessary job failures we've placed
Jenkins into shutdown mode while we resolve this issue.


VPP 19.08 artifacts deferral notice and 19.08.1 schedule

Dave Wallace
 

Dear FD.io Project Members,

[ Note: FD.io <project>-dev email lists BCC'd to reduce noise ]

As discussed in this week's VPP Bi-Weekly Community Meeting, there
is the necessity to introduce a critical fix that breaks VPP API
compatibility. Therefore we need to defer ALL 19.08 release artifacts
and publish 19.08.1 as soon as practicable.

We apologize in advance for this inconvenience. The details are
in the VPP 19.08.1 release note change set [0].

We plan to merge the API-breaking fix into stable/1908 on Monday
9 September 2019 at 9am CET, and would like to cut the VPP 19.08.1
release on Thursday 12 September. The timing is rather tight to
avoid accumulating too many additional commits to stable/1908 and
to minimize the changes between 19.08 and 19.08.1.

Please review the release note change set [0] and add yourself as
a reviewer to be informed of any changes to the plan.

PTLs, please indicate that your project is okay with these dates or
add your desired dates on-gerrit and save the changes to [0].

We would also kindly ask that even if you are okay with these dates,
that you add 1 or 2 persons from your project as reviewers to [0],
to be able to efficiently communicate any changes to the plan to all
interested project teams.

Thanks,
Andrew & Dave
"Your Friendly VPP Release Managers"

[0] https://gerrit.fd.io/r/#/c/vpp/+/21829/


Re: [csit-dev] FD.io Jenkins Maintenance: 2019-08-29 @ 1700 UTC and 2019-08-30 @ 1700 UTC

Maciek Konstantynowicz (mkonstan)
 

Thank you Vanessa for this good news!
We are aiming to complete the critical part of the CSIT-1908 testing
jobs by 04-Sep, will keep you and the team posted.

Cheers,
-Maciek

On 29 Aug 2019, at 18:18, Vanessa Valderrama <vvalderrama@...> wrote:

Sandbox upgrade is complete


On 08/29/2019 10:10 AM, Vanessa Valderrama wrote:

Maintenance update

Jenkins production - The upgrade is not considered critical. We'll be rescheduling to allow CSIT to finish the 19.08 release.

Jenkins sandbox - Today at 1700 UTC

Thank you,
Vanessa


On 08/26/2019 11:36 AM, Vanessa Valderrama wrote:

Maintenance Reminder


On 08/22/2019 12:00 PM, Vanessa Valderrama wrote:


What:
  • Jenkins upgrade to version 2.176.3 LTS
  • Jenkins OS updates
  • Jenkins plug-in updates

When:
Jenkins sandbox - 2019-08-29 @ 1700 UTC
Jenkins production - 2019-08-30 @ 1700 UTC

Impact:
This maintenance requires a reboot of Jenkins.  Jenkins will be placed in shutdown mode at 1600 UTC. We will terminate jobs at 1700 UTC

Why:
Jenkins version 2.176.3 LTS will contain fixes for high severity security issues present in the current version of Jenkins.





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#3700): https://lists.fd.io/g/csit-dev/message/3700
Mute This Topic: https://lists.fd.io/mt/32991954/675185
Group Owner: csit-dev+owner@...
Unsubscribe: https://lists.fd.io/g/csit-dev/unsub  [mkonstan@...]
-=-=-=-=-=-=-=-=-=-=-=-


Re: FD.io Jenkins Maintenance: 2019-08-29 @ 1700 UTC and 2019-08-30 @ 1700 UTC

Vanessa Valderrama
 

Sandbox upgrade is complete


On 08/29/2019 10:10 AM, Vanessa Valderrama wrote:

Maintenance update

Jenkins production - The upgrade is not considered critical. We'll be rescheduling to allow CSIT to finish the 19.08 release.

Jenkins sandbox - Today at 1700 UTC

Thank you,
Vanessa


On 08/26/2019 11:36 AM, Vanessa Valderrama wrote:

Maintenance Reminder


On 08/22/2019 12:00 PM, Vanessa Valderrama wrote:

What:
  • Jenkins upgrade to version 2.176.3 LTS
  • Jenkins OS updates
  • Jenkins plug-in updates

When:
Jenkins sandbox - 2019-08-29 @ 1700 UTC
Jenkins production - 2019-08-30 @ 1700 UTC

Impact:
This maintenance requires a reboot of Jenkins.  Jenkins will be placed in shutdown mode at 1600 UTC. We will terminate jobs at 1700 UTC

Why:
Jenkins version 2.176.3 LTS will contain fixes for high severity security issues present in the current version of Jenkins.






Re: FD.io Jenkins Maintenance: 2019-08-29 @ 1700 UTC and 2019-08-30 @ 1700 UTC

Vanessa Valderrama
 

Maintenance update

Jenkins production - The upgrade is not considered critical. We'll be rescheduling to allow CSIT to finish the 19.08 release.

Jenkins sandbox - Today at 1700 UTC

Thank you,
Vanessa


On 08/26/2019 11:36 AM, Vanessa Valderrama wrote:

Maintenance Reminder


On 08/22/2019 12:00 PM, Vanessa Valderrama wrote:

What:
  • Jenkins upgrade to version 2.176.3 LTS
  • Jenkins OS updates
  • Jenkins plug-in updates

When:
Jenkins sandbox - 2019-08-29 @ 1700 UTC
Jenkins production - 2019-08-30 @ 1700 UTC

Impact:
This maintenance requires a reboot of Jenkins.  Jenkins will be placed in shutdown mode at 1600 UTC. We will terminate jobs at 1700 UTC

Why:
Jenkins version 2.176.3 LTS will contain fixes for high severity security issues present in the current version of Jenkins.





Re: FD.io Nexus Outage

Vanessa Valderrama
 

The issue appears to be hardware related at our cloud provider, we are
working with them to come up with a solution. In the mean time, service
appears to be tentatively restored and we are monitoring the situation

On 08/28/2019 02:16 PM, Vanessa Valderrama wrote:
We're investigating an issue causing instability with cloud disks. I'll
send updates as they are available or you can check the status at
https://status.linuxfoundation.org/incidents/3f31vqs63095?utm_source=slack

Thank you,
Vanessa

On 08/28/2019 02:14 PM, Vanessa Valderrama wrote:
We are currently experiencing issues with Nexus. We're investigating the
issue and will have it resolved as quickly as possible.

Thank you,
Vanessa


Re: FD.io Nexus Outage

Vanessa Valderrama
 

We're investigating an issue causing instability with cloud disks. I'll
send updates as they are available or you can check the status at
https://status.linuxfoundation.org/incidents/3f31vqs63095?utm_source=slack

Thank you,
Vanessa

On 08/28/2019 02:14 PM, Vanessa Valderrama wrote:
We are currently experiencing issues with Nexus. We're investigating the
issue and will have it resolved as quickly as possible.

Thank you,
Vanessa


FD.io Nexus Outage

Vanessa Valderrama
 

We are currently experiencing issues with Nexus. We're investigating the
issue and will have it resolved as quickly as possible.

Thank you,
Vanessa


Re: FD.io Jenkins Maintenance: 2019-08-29 @ 1700 UTC and 2019-08-30 @ 1700 UTC

Vanessa Valderrama
 

Maintenance Reminder


On 08/22/2019 12:00 PM, Vanessa Valderrama wrote:

What:
  • Jenkins upgrade to version 2.176.3 LTS
  • Jenkins OS updates
  • Jenkins plug-in updates

When:
Jenkins sandbox - 2019-08-29 @ 1700 UTC
Jenkins production - 2019-08-30 @ 1700 UTC

Impact:
This maintenance requires a reboot of Jenkins.  Jenkins will be placed in shutdown mode at 1600 UTC. We will terminate jobs at 1700 UTC

Why:
Jenkins version 2.176.3 LTS will contain fixes for high severity security issues present in the current version of Jenkins.




Re: FD.io Jenkins Maintenance: 2019-08-29 @ 1700 UTC and 2019-08-30 @ 1700 UTC

Vanessa Valderrama
 

Correction

When:
Jenkins sandbox - 2019-08-29 @ 1700 UTC
Jenkins production - 2019-08-30 @ 1700 UTC


On 08/22/2019 12:00 PM, Vanessa Valderrama wrote:

What:
  • Jenkins upgrade to version 2.176.3 LTS
  • Jenkins OS updates
  • Jenkins plug-in updates

When:
Jenkins sandbox - 2019-08-29 @ 1700 UTC
Jenkins production - 2019-08-29 @ 1700 UTC

Impact:
This maintenance requires a reboot of Jenkins.  Jenkins will be placed in shutdown mode at 1600 UTC. We will terminate jobs at 1700 UTC

Why:
Jenkins version 2.176.3 LTS will contain fixes for high severity security issues present in the current version of Jenkins.




FD.io Jenkins Maintenance: 2019-08-29 @ 1700 UTC and 2019-08-30 @ 1700 UTC

Vanessa Valderrama
 

What:
  • Jenkins upgrade to version 2.176.3 LTS
  • Jenkins OS updates
  • Jenkins plug-in updates

When:
Jenkins sandbox - 2019-08-29 @ 1700 UTC
Jenkins production - 2019-08-29 @ 1700 UTC

Impact:
This maintenance requires a reboot of Jenkins.  Jenkins will be placed in shutdown mode at 1600 UTC. We will terminate jobs at 1700 UTC

Why:
Jenkins version 2.176.3 LTS will contain fixes for high severity security issues present in the current version of Jenkins.



Re: [vpp-dev] FD.io - Gerrit 2.16 Changes

Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco)
 

Hi Klement.

Using
%private avoids triggering verify jobs,

and you can use UI to set the desired state after that.

 

Based on my testing [0], "Unmark private" action triggers

verify jobs if and only if "fd.io JJB" is in reviewers list

(same as with "recheck" comment).

 

From what I can see, Jenkins allows some tweaking

using "Exclude WIP Changes" flag in trigger configuration,

but I do not see any support for that flag in Jenkins Job Builder.

 

Vratko.

 

[0] https://gerrit.fd.io/r/c/csit/+/20983

 

From: vpp-dev@... <vpp-dev@...> On Behalf Of Klement Sekera via Lists.Fd.Io
Sent: Wednesday, July 31, 2019 4:29 PM
To: Vanessa Valderrama <vvalderrama@...>; ci-management-dev@...; cicn-dev@...; csit-dev@...; deb-dpdk@...; discuss@...; dmm-dev@...; govpp-dev@...; hc2vpp@...; honeycomb-dev@...; infra-container@...; infra-steering@...; nsh_sfc-dev@...; odp4vpp-dev@...; one-dev@...; p4vpp-dev@...; pma-tools-dev@...; puppet-fdio@...; rpm_dpdk@...; sweetcomb-dev@...; tldk-dev@...; trex-dev@...; tsc@...; vpp-dev@...; jvpp-dev@...; hicn-dev@...
Cc: vpp-dev@...
Subject: Re: [vpp-dev] FD.io - Gerrit 2.16 Changes

 

Hi Vanessa,

 

I pushed a couple of changes via your suggested

 

git push origin HEAD:refs/for/master%wip

 

and I noticed that on all of these the verify jobs were (attempted to) run. One of the advantages of draft changes is that it doesn’t eat resources for pointless verify jobs.

 

Thanks,

Klement

 

 

Changes that will happen with Gerrit:

1) The 'New UI' for Gerrit will become the default UI

2) The Draft work flow is removed and replaced with 'Work in Progress'
aka 'WIP' and 'Private' workflows. Unfortunately git-review does not
support either of these workflows directly. Utilizing them means either
pushing your changes the manual way for either system or pushing them up
with git-review and then marking the change via the UI into either of
the workflows.

To push a private change you may do so as follows:
git push origin HEAD:refs/for/master%private

To pull it out of private you may do so as follows:
git push origin HEAD:refs/for/master%remove-private

To push a WIP you may do so as follows:
git push origin HEAD:refs/for/master%wip

To mark it ready for review you may do so as follows:
git push origin HEAD:refs/for/master%ready

Once a change is in either private or WIP state it does not switch the
change to a ready state until the current state has been removed.

In both cases, the state can be set via the UI by selecting the tripple
dot menu option on the change and selecting the appropriate option.

To remove WIP state press the 'START REVIEW' button. To remove the
private state you must do so via the menu.


NOTE: We are not moving to Gerrit 3 at this time. That is on the road
map but we need to come to the latest 2.x as we have to do various
migrations that are only available at the 2.16 level before we can move
to Gerrit 3.

Thank you,
Vanessa


Re: FD.io Maintenance: 2019-07-30 @ 1700 UTC (10:00am PDT) - 2100 UTC (2:00pm PDT)

Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco)
 

> If you experience any issues please open a ticket at support.linuxfoundation.org.

 

Executors for ci-management jjb jobs are not available, opened [3].

 

[3] https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-17012

 

From: discuss@... <discuss@...> On Behalf Of Vanessa Valderrama
Sent: Tuesday, July 30, 2019 8:58 PM
To: tsc@...; discuss@...; infra-steering@...; vpp-dev@...; csit-dev@...; Ed Kern <ejkern@...>; Anton Baranov <abaranov@...>; Trishan de Lanerolle <tdelanerolle@...>
Subject: Re: [discuss] FD.io Maintenance: 2019-07-30 @ 1700 UTC (10:00am PDT) - 2100 UTC (2:00pm PDT)

 

Maintenance is complete. All systems are now available. We'll be doing some additional testing and monitoring throughout the day. If you experience any issues please open a ticket at support.linuxfoundation.org.

Thank you,
Vanessa

On 07/30/2019 12:01 PM, Vanessa Valderrama wrote:

Starting maintenance

 

On 07/30/2019 11:02 AM, Vanessa Valderrama wrote:

Jenkins has been placed in shutdown mode for maintenance

 

 

On 07/29/2019 10:29 AM, Vanessa Valderrama wrote:

Maintenance reminder

 

On 07/25/2019 04:09 PM, Vanessa Valderrama wrote:

What:

FD.io system maintenance

  • Jenkins
    • OS updates
    • Upgrade from 2.164.3 to 2.176.2
    • Update plug-ins
    • Memory upgrade
  • Nexus
    • OS updates
  • Gerrit
    • OS updates
    • Upgrade from 2.14.6 to 2.16
  • JIRA
    • OS updates
  • Sonar
    • OS updates
  • OpenGrok OS updates

When:

2019-07-30 @ 1700 UTC (10:00am PDT) - 2100 UTC (2:00pm PDT)

Impact:

The maintenance will require a reboot of each FD.io system. Jenkins will be placed in shutdown mode at 1600 UTC.  Please let us know if there are any jobs that cannot be aborted.  The following systems will be impacted:

  • Jenkins production
  • Jenkins sandbox
  • Nexus
  • Gerrit
  • JIRA
  • Sonar
  • OpenGrok

 

 

 

 

 


FD.io - Gerrit 2.16 Changes

Vanessa Valderrama
 

Changes that will happen with Gerrit:

1) The 'New UI' for Gerrit will become the default UI

2) The Draft work flow is removed and replaced with 'Work in Progress'
aka 'WIP' and 'Private' workflows. Unfortunately git-review does not
support either of these workflows directly. Utilizing them means either
pushing your changes the manual way for either system or pushing them up
with git-review and then marking the change via the UI into either of
the workflows.

To push a private change you may do so as follows:
git push origin HEAD:refs/for/master%private

To pull it out of private you may do so as follows:
git push origin HEAD:refs/for/master%remove-private

To push a WIP you may do so as follows:
git push origin HEAD:refs/for/master%wip

To mark it ready for review you may do so as follows:
git push origin HEAD:refs/for/master%ready

Once a change is in either private or WIP state it does not switch the
change to a ready state until the current state has been removed.

In both cases, the state can be set via the UI by selecting the tripple
dot menu option on the change and selecting the appropriate option.

To remove WIP state press the 'START REVIEW' button. To remove the
private state you must do so via the menu.

NOTE: We are not moving to Gerrit 3 at this time. That is on the road
map but we need to come to the latest 2.x as we have to do various
migrations that are only available at the 2.16 level before we can move
to Gerrit 3.

Thank you,
Vanessa