Date   

CSIT failing perf tests for week 40 (09/19 – 09/25)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

===OUTSTANDING UNFIXED===

 

1) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/402/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/98/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/691/log.html.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1857

 

2) error: all testbeds DPDK l3fwd tests have wrong configuration

   rca:

   test: DPDK l3fwd

   frequency: all

   testbed: all

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/25/log.html.gz#s1-s1-s1-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-verify-master-3n-icx/3/log.html.gz#s1-s1-s1-s1-t1-k2-k5-k14

 

TICKET: https://jira.fd.io/browse/CSIT-1858

 

NOTE: tests are passing but the traffic returns back to TG instead of being forwarded to DUT2.

 

3) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1865

 

NOTE: This week 3n-icx didn't run due to low cadency of runs because of iterative tests.

 

4) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s4-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/159/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: week37: issue appeared

NOTE: week39: happened again but only on ip6base e810Cq - build #158.

NOTE: week40: This week issue happened only on ip4base e810Cq, but 99,9% of packets are lost now. Build #159

 

TICKET: https://jira.fd.io/browse/CSIT-1864

 

5) error: 2n-icx: all tests failed with parent suite setup time-out

   rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup

   test: all

   frequency: all

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/39/console.log.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1863

 

 

6) error: 3n-alt: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/25/log.html.gz#s1-s1-s1-s1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

7) error: 3n-icx: IP4 tunnels GTPU and WIREGUARD tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/39/log.html.gz#s1-s1-s1-s1-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

 

TICKET: https://jira.fd.io/browse/CSIT-1862

 

8) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on VM VHOST tests

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR daily : 2n-clx: VM VHOST vppl2xc

                             MRR weekly: 2n-clx, 2n-icx: VM VHOST chain vppip4

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1192/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/141/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/620/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/160/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/39/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/159/log.html.gz#s1-s1-s1-s6-s2

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/40/log.html.gz#s1-s1-s1-s6-s2-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

TICKET: https://jira.fd.io/browse/CSIT-1859

 

9) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1192/log.html.gz#s1-s1-s1-s2-s18

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/159/log.html.gz#s1-s1-s1-s2-s18

 

TICKET: https://jira.fd.io/browse/CSIT-1861

 

10) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

TICKET: https://jira.fd.io/browse/CSIT-1860

 

11) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/133/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/39/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

12) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1245/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

13) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/133/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

14) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

15) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

16) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1245/log.html.gz#s1-s1-s1-s1-s3-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1255/log.html.gz#s1-s1-s1-s1-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

17) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/141/log.html.gz#s1-s1-s1-s1-s2-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/620/log.html.gz#s1-s1-s1-s2-s15-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

18) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

  frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/620/log.html.gz#s1-s1-s1-s2-s21-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

#) error: 2n-aws all tests failing with InvalidPlacementGroup

   rca:

   test: all

   frequency: all

   testbed: 2n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-aws/64/console.log.gz

  

   message:

   Error: error creating EC2 Placement Group (csit-2n-aws-c5n-csit-vpp-perf-mrr-weekly-master-2n-aws-pg): InvalidPlacementGroup.Duplicate: The placement group 'csit-2n-aws-c5n-csit-vpp-perf-mrr-weekly-master-2n-aws-pg' already exists.

              status code: 400, request id: e9120991-2048-4f8b-89d2-4cd0868e5e0e

 

TICKET: https://jira.fd.io/browse/CSIT-1866

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


Re: CSIT failing perf tests for week 39 (09/12 – 09/18)

Juraj Linkeš
 

I also have an update regarding the Arm failures. Tianyu from Arm submitted a patch that address the issue: https://gerrit.fd.io/r/c/vpp/+/37199

 

Juraj Linkeš

 

From: csit-report@... <csit-report@...> On Behalf Of Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco) via lists.fd.io
Sent: Thursday, September 22, 2022 1:50 PM
To: csit-report@...
Subject: Re: [csit-report] CSIT failing perf tests for week 39 (09/12 – 09/18)

 

Update:

 

We’ve got issue with all ARM testbeds failing to start VPP. Assigned the highest priority. ARM is aware and are working on fix.

 

1) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/396/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/94/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/687/log.html.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1857

 

Best regards,

Viliam Luc

 

From: Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
Sent: 19 September 2022 11:29
To: csit-report@...
Subject: CSIT failing perf tests for week 39 (09/12 – 09/18)

 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

2) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s4-s4-t1

 

NOTE: happened again but only on ip6base e810Cq - build #158.

 

3) error: 2n-icx: all tests failed with parent suite setup time-out

   rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup

   test: all

   frequency: all

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/39/console.log.gz

 

===OUTSTANDING UNFIXED===

 

4) error: 3n-icx: NDR tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

 

5) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR: VM VHOST vppl2xc

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/136/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/615/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/157/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s6-s2-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

6) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s2-s18

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s2-s18

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1239/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1240/log.html.gz#s1-s1-s1-s2-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1250/log.html.gz#s1-s1-s1-s1-s3-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s2-s1-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


Re: CSIT failing perf tests for week 39 (09/12 – 09/18)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

Update:

 

We’ve got issue with all ARM testbeds failing to start VPP. Assigned the highest priority. ARM is aware and are working on fix.

 

1) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/396/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/94/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/687/log.html.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1857

 

Best regards,

Viliam Luc

 

From: Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
Sent: 19 September 2022 11:29
To: csit-report@...
Subject: CSIT failing perf tests for week 39 (09/12 – 09/18)

 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

2) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s4-s4-t1

 

NOTE: happened again but only on ip6base e810Cq - build #158.

 

3) error: 2n-icx: all tests failed with parent suite setup time-out

   rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup

   test: all

   frequency: all

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/39/console.log.gz

 

===OUTSTANDING UNFIXED===

 

4) error: 3n-icx: NDR tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

 

5) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR: VM VHOST vppl2xc

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/136/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/615/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/157/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s6-s2-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

6) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s2-s18

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s2-s18

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1239/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1240/log.html.gz#s1-s1-s1-s2-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1250/log.html.gz#s1-s1-s1-s1-s3-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s2-s1-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 39 (09/12 – 09/18)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

2) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s4-s4-t1

 

NOTE: happened again but only on ip6base e810Cq - build #158.

 

3) error: 2n-icx: all tests failed with parent suite setup time-out

   rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup

   test: all

   frequency: all

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/39/console.log.gz

 

===OUTSTANDING UNFIXED===

 

4) error: 3n-icx: NDR tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

 

5) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR: VM VHOST vppl2xc

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/136/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/615/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/157/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s6-s2-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

6) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s2-s18

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s2-s18

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1239/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1240/log.html.gz#s1-s1-s1-s2-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1250/log.html.gz#s1-s1-s1-s1-s3-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s2-s1-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 38 (09/04 – 09/11)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 3n-icx: NDR tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s3-s1-t1

 

===OUTSTANDING UNFIXED===

 

2) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR: VM VHOST vppl2xc

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1182/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/131/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/610/log.html.gz#s1-s1-s1-s6-s2-t1

                                                          https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/157/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                          https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/37/log.html.gz#s1-s1-s1-s1-s2-s1-s1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

3) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1182/log.html.gz#s1-s1-s1-s2-s18

 

4) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

5) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/123/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

6) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1234/log.html.gz#s1-s1-s1-s1-s7-t6

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

7) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

8) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

9) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

  tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

10) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1234/log.html.gz#s1-s1-s1-s1-s1-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1244/log.html.gz#s1-s1-s1-s1-s2-t4

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

11) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/131/log.html.gz#s1-s1-s1-s3-s4-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

12) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

13) error: 2n-clx, 2n-zn2: Failed to create AVF interfaces on 710 NICs

   rca:

   test: AVF 710

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s10-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s2-s10-t1

 

NOTE: 2n-clx X710-AVF tests failing with issue #3

NOTED: FIXED on 09th of September 2n-clx build #1182

 

14) error: 2n-clx: half of the packets lost on PDR tests

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: intermitent.

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


Re: CSIT failing perf tests for week 36 (08/28 – 09/04)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

Update

  • #3 not only AVF but all x710 tests affected
  • Added one new issue with #5

 

CSIT failing perf tests for week 36 (08/28 – 09/04)

 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 2n-clx, 2n-zn2: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: VM VHOST vppl2xc

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s6-s10-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

2) error: 2n-clx, 2n-zn2: Failed to create AVF interfaces on 710 NICs

   rca:

   test: AVF 710

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s10-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s2-s10-t1

 

NOTE: 2n-clx X710-AVF tests failing with issue #3

 

3) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s2-s18

 

4) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/385/log.html.gz#

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/676/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

5) error: 2n-clx: half of the packets lost on PDR tests

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

 

===OUTSTANDING UNFIXED===

 

6) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/117/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

7) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s1-s7-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

8) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/208/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

9) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

10) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

11) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1240/log.html.gz#s1-s1-s1-s1-s3-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s2-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

12) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/116/log.html.gz#s1-s1-s1-s3-s5-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

13) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1190/log.html.gz#s1-s1-s1-s1-s9-t4

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: During week28 1st time happened on 2n-icx

NOTE: During week29 1st time observed on 2n-dnv

NOTE: wasn't observed this week

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc

 

From: csit-report@... <csit-report@...> On Behalf Of Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco) via lists.fd.io
Sent: 05 September 2022 11:39
To: csit-report@...
Subject: [csit-report] CSIT failing perf tests for week 36 (08/28 – 09/04)

 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 2n-clx, 2n-zn2: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: VM VHOST vppl2xc

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s6-s10-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

2) error: 2n-clx, 2n-zn2: Failed to create AVF interfaces on 710 NICs

   rca:

   test: AVF 710 (both X710 and XXV710)

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s10-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s2-s10-t1

 

NOTE: 2n-clx X710-AVF tests failing with issue #3

 

3) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s2-s18

 

4) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/385/log.html.gz#

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/676/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

===OUTSTANDING UNFIXED===

 

5) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/117/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

6) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

  rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s1-s7-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

7) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/208/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

8) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

9) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

10) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1240/log.html.gz#s1-s1-s1-s1-s3-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s2-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

11) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/116/log.html.gz#s1-s1-s1-s3-s5-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

12) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1190/log.html.gz#s1-s1-s1-s1-s9-t4

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: During week28 1st time happened on 2n-icx

NOTE: During week29 1st time observed on 2n-dnv

NOTE: wasn't observed this week

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 36 (08/28 – 09/04)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 2n-clx, 2n-zn2: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: VM VHOST vppl2xc

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s6-s10-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

2) error: 2n-clx, 2n-zn2: Failed to create AVF interfaces on 710 NICs

   rca:

   test: AVF 710 (both X710 and XXV710)

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s10-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s2-s10-t1

 

NOTE: 2n-clx X710-AVF tests failing with issue #3

 

3) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: AVF X710

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s2-s18

 

4) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/385/log.html.gz#

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/676/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

===OUTSTANDING UNFIXED===

 

5) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/117/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

6) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

  rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s1-s7-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

7) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/208/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

8) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

9) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

10) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1240/log.html.gz#s1-s1-s1-s1-s3-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s2-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

11) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/116/log.html.gz#s1-s1-s1-s3-s5-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

12) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1190/log.html.gz#s1-s1-s1-s1-s9-t4

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: During week28 1st time happened on 2n-icx

NOTE: During week29 1st time observed on 2n-dnv

NOTE: wasn't observed this week

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


master: The build number 483 of the job csit-trending-merge-daily-master-ubuntu2004-x86_64 failed #email

noreply@...
 


master: The build number 482 of the job csit-trending-merge-daily-master-ubuntu2004-x86_64 failed #email

noreply@...
 


CSIT failing perf tests for week 34 (08/16 – 08/21)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

===OUTSTANDING UNFIXED===

 

1) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/110/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

2) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1220/log.html.gz#s1-s1-s1-s1-s7-t6

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

3) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/208/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

NOTE: After a while failed on MRR daily tests. Only 1518B tests failed, IMIX passed.

 

4) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

5) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

6) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1230/log.html.gz#s1-s1-s1-s1-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1220/log.html.gz#s1-s1-s1-s1-s2-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

7) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/594/log.html.gz#s1-s1-s1-s5-s13-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

8) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1190/log.html.gz#s1-s1-s1-s1-s9-t4

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: During week28 1st time happened on 2n-icx

NOTE: During week29 1st time observed on 2n-dnv

 

===OUTSTANDING FIXED===

 

9) error: 3n-alt, 2n-zn2, 2n-tx2: FOR loop has no closing END.

   rca: possible cause upgrade of RobotFramework

   test: 3n-alt, 2n-tx2: cache-ip4base-policy

                             2n-zn2: RDMA

   frequency: all

   testbed: 3n-alt, 2n-zn2, 2n-tx2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/71/log.html.gz#s1-s1-s1-s1-s4-t1

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/591/log.html.gz#s1-s1-s1-s2-s1-t1

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/371/log.html.gz#s1-s1-s1-s2-s4-t1

 

NOTE: VL to look into the issue.

NOTE: partial fix: 36888: fix(hoststack): add a missing end | https://gerrit.fd.io/r/c/csit/+/36888

FIX: 36906: fix: add missing closing END | https://gerrit.fd.io/r/c/csit/+/36906

 

 

10) error: 2n-clx, 3n-skx: First port is down when running traffic profile

   rca:

   test: 10Ge2P1X710

   frequency: all

   testbed: 2n-clx, 3n-skx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-skx/1616/log.html.gz#s1-s1-s1-s2-s1-t1

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-skx/209/log.html.gz#s1-s1-s1-s1-t1

 

NOTE: Issue had been observed on 2n-clx and fixed with workaround. Now (7/3) it appeared on 3n-skx. The same x710 NIC.

TICKET: https://jira.fd.io/browse/CSIT-1831

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


master: The build number 481 of the job csit-trending-merge-daily-master-ubuntu2004-x86_64 failed #email

noreply@...
 


master: The build number 480 of the job csit-trending-merge-daily-master-ubuntu2004-x86_64 failed #email

noreply@...
 


master: The build number 479 of the job csit-trending-merge-daily-master-ubuntu2004-x86_64 failed #email

noreply@...
 


master: Progressions as of 2022-08-18 14:04:47 UTC #email

noreply@...
 

Following progressions occured in the last trending job runs, listed per testbed type.

Legend: Test-name NIC Frame-size Trend[Mpps] Runs[#] Long-Term change[%]

vpp-3n-alt-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/74, vpp version: 22.10-rc0~119-g3ebe1efce~b1479

1t1c-eth-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 3.46 1 -1.36
1t1c-ethip4-ip4base-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 3.5 1 -0.8
2t2c-ethip4-ip4base-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 6.24 1 -0.27
4t4c-eth-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 11.17 1 -1.83
4t4c-ethip4-ip4base-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 10.93 1 -1.43


vpp-2n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-icx/116, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

No progressions

vpp-3n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-icx/109, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

2t1c-ethip4gtpusw-ip4base-mrr 25ge2p1e810xxv 64B 5.32 1 3.89


vpp-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-skx/1780, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

No progressions

vpp-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-skx/1616, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

No progressions

vpp-2n-zn2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-zn2/594, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

No progressions

vpp-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/665, vpp version: 22.10-rc0~119-g3ebe1efce~b1479

No progressions

vpp-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-tx2/374, vpp version: 22.10-rc0~119-g3ebe1efce~b1479

No progressions

vpp-2n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-dnv/1219, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

No progressions

vpp-3n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-dnv/1229, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

2t2c-ethip4-ip4scale20k-mrr 10ge2p1x553 64B 5.82 1 -5.54
4t4c-ethip4-ip4scale20k-rnd-mrr 10ge2p1x553 64B 10.17 1 -1.07
1t1c-ethip6-ip6base-mrr 10ge2p1x553 78B 4.96 1 -0.71
2t2c-ethip4ipsec4tnlsw-ip4base-int-aes256gcm-mrr 10ge2p1x553 1518B 0.63 1 -0.53
4t4c-ethip4ipsec1000tnlsw-ip4base-int-aes256gcm-mrr 10ge2p1x553 1518B 1.03 1 -1.58
2t2c-eth-l2patch-mrr 10ge2p1x553 64B 13.66 1 -0.97
4t4c-eth-l2xcbase-mrr 10ge2p1x553 64B 17.66 1 1.8
4t4c-eth-l2bdbasemaclrn-mrr 10ge2p1x553 64B 14.63 1 1.57


dpdk-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-skx/210, dpdk version: 22.07.0

8t4c-eth-l2xcbase-testpmd-mrr 25ge2p1xxv710 64B 31.35 1 4.98


dpdk-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-skx/209, dpdk version: 22.07.0

No progressions

dpdk-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/102, dpdk version: 22.07.0

No progressions

dpdk-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-tx2/85, dpdk version: 22.07.0

No progressions

vpp-2n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/208, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No progressions

vpp-3n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/209, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No progressions


master: Regressions as of 2022-08-18 14:04:47 UTC #email

noreply@...
 

Following regressions occured in the last trending job runs, listed per testbed type.

Legend: Test-name NIC Frame-size Trend[Mpps] Runs[#] Long-Term change[%]

vpp-3n-alt-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/74, vpp version: 22.10-rc0~119-g3ebe1efce~b1479

No regressions

vpp-2n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-icx/116, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

No regressions

vpp-3n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-icx/109, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

4t2c-ethip4gtpusw-ip4base-mrr 25ge2p1e810xxv 64B 5.12 1 0.19


vpp-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-skx/1780, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

4t2c-avf-eth-l2bdbasemaclrn-mrr 10ge2p1x710 64B 29.53 1 -0.44


vpp-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-skx/1616, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

No regressions

vpp-2n-zn2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-zn2/594, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

No regressions

vpp-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/665, vpp version: 22.10-rc0~119-g3ebe1efce~b1479

1t1c-ethip4vxlan-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 10ge2p1x520 64B 0.61 1 -58.13
4t4c-ethip4vxlan-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 10ge2p1x520 64B 1.82 1 -58.11


vpp-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-tx2/374, vpp version: 22.10-rc0~119-g3ebe1efce~b1479

No regressions

vpp-2n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-dnv/1219, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

No regressions

vpp-3n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-dnv/1229, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

2t2c-ethip4vxlan-l2xcbase-mrr 10ge2p1x553 64B 1.86 1 -68.17
2t2c-ethip6-ip6base-mrr 10ge2p1x553 78B 2.55 1 -64.72
4t4c-ethip4ipsec4tnlsw-ip4base-int-aes256gcm-mrr 10ge2p1x553 IMIX 0.58 1 -72.53
4t4c-ethip4ipsec4tnlsw-ip4base-int-aes256gcm-mrr 10ge2p1x553 1518B 0.88 1 -4.19
4t4c-eth-l2patch-mrr 10ge2p1x553 64B 2.95 1 -88.19


dpdk-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-skx/210, dpdk version: 22.07.0

2t1c-ethip4-ip4base-l3fwd-mrr 25ge2p1xxv710 64B 35.97 1 -0.98


dpdk-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-skx/209, dpdk version: 22.07.0

No regressions

dpdk-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/102, dpdk version: 22.07.0

No regressions

dpdk-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-tx2/85, dpdk version: 22.07.0

No regressions

vpp-2n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/208, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions

vpp-3n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/209, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions


master: Failed tests as of 2022-08-18 14:04:47 UTC #email

noreply@...
 

Following tests failed in the last trending job runs, listed per testbed type.



vpp-3n-alt-mrr, 23 tests failed, 145 tests passed, duration: 03:44, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/74, vpp version: 22.10-rc0~119-g3ebe1efce~b1479

ethip4ipsec10000tnlsw-ip4base-int-aes128cbc-hmac512sha 40ge2p1xl710 1518b imix 1t1c[0] 2t2c[0] 4t4c[0]
ethip4ipsec10000tnlsw-ip4base-int-aes128gcm 40ge2p1xl710 1518b imix 1t1c[0] 2t2c[0] 4t4c[0]
ethip4ipsec10000tnlsw-ip4base-int-aes256gcm 40ge2p1xl710 1518b imix 1t1c[0] 2t2c[0] 4t4c[0]
ethip4ipsec1000spe-cache-ip4base-policy-outbound-nocrypto 40ge2p1xl710 64b 1t1c[0] 2t2c[0] 4t4c[0]
ethip4ipsec1000spe-ip4base-policy-outbound-nocrypto 40ge2p1xl710 64b 1t1c[0] 2t2c[0]


vpp-2n-icx-mrr, 0 tests failed, 240 tests passed, duration: 06:43, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-icx/116, vpp version: 22.10-rc0~119-g3ebe1efce~b2795



vpp-3n-icx-mrr, 9 tests failed, 189 tests passed, duration: 04:22, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-icx/109, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

ethip4ipsec1000tnlsw-fixtnlip-ip4base-policy-aes256gcm 25ge2p1e810xxv 1518b 2t1c[1] 4t2c[1] 8t4c[1]
ethip4ipsec1000tnlsw-fixtnlip-ip4base-policy-flow-dir-aes256gcm 25ge2p1e810xxv 1518b 2t1c[1] 4t2c[1] 8t4c[1]
ethip4ipsec1000tnlsw-fixtnlip-ip4base-policy-flow-rss-aes256gcm 25ge2p1e810xxv 1518b 2t1c[1] 4t2c[1] 8t4c[1]


vpp-2n-skx-mrr, 2 tests failed, 280 tests passed, duration: 06:28, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-skx/1780, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

avf-eth-l2bdbasemaclrn-eth-2vhostvr1024-1vm-vppl2xc 25ge2p1xxv710 64b 8t4c[2]
avf-eth-l2patch 25ge2p1xxv710 64b 4t2c[2]


vpp-3n-skx-mrr, 15 tests failed, 171 tests passed, duration: 03:58, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-skx/1616, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

avf-eth-l2bdbasemaclrn 10ge2p1x710 64b 2t1c[3] 4t2c[3] 8t4c[3]
avf-eth-l2patch 10ge2p1x710 64b 2t1c[3] 4t2c[3] 8t4c[3]
avf-eth-l2xcbase 10ge2p1x710 64b 2t1c[3] 4t2c[3] 8t4c[3]
avf-ethip4-ip4base 10ge2p1x710 64b 2t1c[3] 4t2c[3] 8t4c[3]
avf-ethip6-ip6base 10ge2p1x710 78b 2t1c[3] 4t2c[3] 8t4c[3]


No input data available for vpp-2n-clx-mrr. See CSIT job https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-clx for more information.


vpp-2n-zn2-mrr, 1 tests failed, 242 tests passed, duration: 03:51, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-zn2/594, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

avf-eth-l2patch 25ge2p1xxv710 64b 4t2c[2]


vpp-3n-tsh-mrr, 0 tests failed, 156 tests passed, duration: 04:16, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/665, vpp version: 22.10-rc0~119-g3ebe1efce~b1479



vpp-2n-tx2-mrr, 0 tests failed, 192 tests passed, duration: 03:14, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-tx2/374, vpp version: 22.10-rc0~119-g3ebe1efce~b1479



vpp-2n-dnv-mrr, 6 tests failed, 48 tests passed, duration: 01:51, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-dnv/1219, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

eth-l2bdbasemaclrn 10ge2p1x553 64b 1t1c[3] 2t2c[3]
eth-l2bdscale10kmaclrn 10ge2p1x553 64b 1t1c[3] 4t4c[3]
ethip4-ip4scale20k 10ge2p1x553 64b 1t1c[3]
ethip6-ip6scale20k 10ge2p1x553 78b 4t4c[3]


vpp-3n-dnv-mrr, 17 tests failed, 55 tests passed, duration: 01:50, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-dnv/1229, vpp version: 22.10-rc0~119-g3ebe1efce~b2795

eth-l2bdbasemaclrn 10ge2p1x553 64b 1t1c[3]
eth-l2bdscale10kmaclrn 10ge2p1x553 64b 1t1c[3] 2t2c[3]
eth-l2patch 10ge2p1x553 64b 1t1c[3] 2t2c[3]
eth-l2xcbase 10ge2p1x553 64b 1t1c[3] 4t4c[3]
ethip4-ip4scale20k 10ge2p1x553 64b 1t1c[3]
ethip4-ip4scale20k-rnd 10ge2p1x553 64b 1t1c[3]
ethip4ipsec1000tnlsw-ip4base-int-aes128cbc-hmac512sha 10ge2p1x553 1518b 1t1c[3]
ethip4ipsec1tnlswasync-scheduler-ip4base-int-aes256gcm 10ge2p1x553 1518b 1t2c[3]
ethip4ipsec4tnlsw-ip4base-int-aes128cbc-hmac512sha 10ge2p1x553 1518b 2t2c[3]
ethip4ipsec4tnlsw-ip4base-int-aes256gcm 10ge2p1x553 1518b 1t1c[3] 2t2c[3]
ethip4vxlan-l2bdbasemaclrn 10ge2p1x553 64b 1t1c[3]
ethip6-ip6base 10ge2p1x553 78b 1t1c[3]
ethip6-ip6scale20k-rnd 10ge2p1x553 78b 2t2c[3]


No input data available for . See CSIT job https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-weekly-master-2n-aws for more information.


dpdk-2n-skx-mrr, 0 tests failed, 12 tests passed, duration: 00:25, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-skx/210, dpdk version: 22.07.0



dpdk-3n-skx-mrr, 6 tests failed, 6 tests passed, duration: 00:42, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-skx/209, dpdk version: 22.07.0

eth-l2xcbase-testpmd 10ge2p1x710 64b 1t1c[3] 2t2c[3] 4t4c[3]
ethip4-ip4base-l3fwd 10ge2p1x710 64b 1t1c[3] 2t2c[3] 4t4c[3]


No input data available for dpdk-2n-clx-mrr. See CSIT job https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-clx for more information.


dpdk-3n-tsh-mrr, 0 tests failed, 6 tests passed, duration: 00:16, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/102, dpdk version: 22.07.0



dpdk-2n-tx2-mrr, 0 tests failed, 6 tests passed, duration: 00:09, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-tx2/85, dpdk version: 22.07.0



vpp-2n-skx-ndrpdr, 1 tests failed, 281 tests passed, duration: 22:43, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/208, vpp version: 22.10-rc0~110-g57cf5087e~b2786

avf-ethip6-ip6scale20k-rnd 25ge2p1xxv710 78b 4t2c[2]


vpp-3n-skx-ndrpdr, 0 tests failed, 186 tests passed, duration: 17:08, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/209, vpp version: 22.10-rc0~110-g57cf5087e~b2786



No input data available for vpp-2n-clx-ndrpdr. See CSIT job https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-clx for more information.


No input data available for . See CSIT job https://jenkins.fd.io/view/csit/job/csit-trex-perf-ndrpdr-weekly-master-2n-skx for more information.

For detailed information visit: https://s3-docs.fd.io/csit/master/trending/introduction/failures.html


Legend:

[0] - Timeout, interfaces not up:['FortyGigabitEthernet4/0/0']
[1] - No traffic forwarded
[2] - Failed to create AVF interface on host xxx.xxx.xxx.xxx
[3] - T-Rex STL runtime error


master: Failed tests as of 2022-08-17 14:04:51 UTC #email

noreply@...
 

Following tests failed in the last trending job runs, listed per testbed type.



vpp-3n-alt-mrr, 0 tests failed, 168 tests passed, duration: 03:48, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/73, vpp version: 22.10-rc0~113-gc7f93b321~b1473



vpp-2n-icx-mrr, 0 tests failed, 240 tests passed, duration: 06:37, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-icx/115, vpp version: 22.10-rc0~113-gc7f93b321~b2789



vpp-3n-icx-mrr, 12 tests failed, 186 tests passed, duration: 04:23, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-icx/108, vpp version: 22.10-rc0~113-gc7f93b321~b2789

avf-ethip4ipsec10000tnlsw-ip4base-int-aes256gcm 25ge2p1xxv710 1518b 4t2c[1]
avf-ethip4ipsec40tnlsw-ip4base-int-aes256gcm 25ge2p1xxv710 1518b 8t4c[1]
avf-ethip4vxlan-l2xcbase 25ge2p1xxv710 64b 4t2c[1]
ethip4ipsec1000tnlsw-fixtnlip-ip4base-policy-aes256gcm 25ge2p1e810xxv 1518b 2t1c[0] 4t2c[0] 8t4c[0]
ethip4ipsec1000tnlsw-fixtnlip-ip4base-policy-flow-dir-aes256gcm 25ge2p1e810xxv 1518b 2t1c[0] 4t2c[0] 8t4c[0]
ethip4ipsec1000tnlsw-fixtnlip-ip4base-policy-flow-rss-aes256gcm 25ge2p1e810xxv 1518b 2t1c[0] 4t2c[0] 8t4c[0]


vpp-2n-skx-mrr, 0 tests failed, 282 tests passed, duration: 06:33, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-skx/1779, vpp version: 22.10-rc0~113-gc7f93b321~b2789



vpp-3n-skx-mrr, 15 tests failed, 171 tests passed, duration: 03:59, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-skx/1615, vpp version: 22.10-rc0~113-gc7f93b321~b2789

avf-eth-l2bdbasemaclrn 10ge2p1x710 64b 2t1c[2] 4t2c[2] 8t4c[2]
avf-eth-l2patch 10ge2p1x710 64b 2t1c[2] 4t2c[2] 8t4c[2]
avf-eth-l2xcbase 10ge2p1x710 64b 2t1c[2] 4t2c[2] 8t4c[2]
avf-ethip4-ip4base 10ge2p1x710 64b 2t1c[2] 4t2c[2] 8t4c[2]
avf-ethip6-ip6base 10ge2p1x710 78b 2t1c[2] 4t2c[2] 8t4c[2]


No input data available for vpp-2n-clx-mrr. See CSIT job https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-clx for more information.


vpp-2n-zn2-mrr, 0 tests failed, 243 tests passed, duration: 03:54, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-zn2/593, vpp version: 22.10-rc0~113-gc7f93b321~b2789



vpp-3n-tsh-mrr, 0 tests failed, 156 tests passed, duration: 04:17, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/664, vpp version: 22.10-rc0~113-gc7f93b321~b1473



vpp-2n-tx2-mrr, 0 tests failed, 192 tests passed, duration: 03:14, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-tx2/373, vpp version: 22.10-rc0~113-gc7f93b321~b1473



vpp-2n-dnv-mrr, 11 tests failed, 43 tests passed, duration: 01:39, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-dnv/1218, vpp version: 22.10-rc0~113-gc7f93b321~b2789

eth-l2bdbasemaclrn 10ge2p1x553 64b 1t1c[2]
eth-l2bdscale10kmaclrn 10ge2p1x553 64b 2t2c[2]
eth-l2xcbase 10ge2p1x553 64b 2t2c[2] 4t4c[2]
ethip4-ip4scale20k 10ge2p1x553 64b 1t1c[2]
ethip4-ip4scale20k-rnd 10ge2p1x553 64b 4t4c[2]
ethip4tcp-nat44ed-h1024-p63-s64512-tput 10ge2p1x553 1518b 4t4c[3]
ethip4tcp-nat44ed-h16384-p63-s1032192-tput 10ge2p1x553 1518b 1t1c[3] 2t2c[3]
ethip4tcp-nat44ed-h65536-p63-s4128768-tput 10ge2p1x553 1518b 1t1c[3]
ethip6-ip6scale20k 10ge2p1x553 78b 4t4c[2]


vpp-3n-dnv-mrr, 19 tests failed, 53 tests passed, duration: 01:50, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-dnv/1228, vpp version: 22.10-rc0~113-gc7f93b321~b2789

eth-l2bdscale10kmaclrn 10ge2p1x553 64b 1t1c[2]
ethip4-ip4base 10ge2p1x553 64b 4t4c[2]
ethip4-ip4scale20k 10ge2p1x553 64b 4t4c[2]
ethip4-ip4scale20k-rnd 10ge2p1x553 64b 2t2c[2]
ethip4ipsec1000tnlsw-ip4base-int-aes128cbc-hmac512sha 10ge2p1x553 1518b 1t1c[2] 2t2c[2]
ethip4ipsec1000tnlsw-ip4base-int-aes256gcm 10ge2p1x553 1518b imix 1t1c[2] 2t2c[2] 4t4c[2]
ethip4ipsec1tnlswasync-scheduler-ip4base-int-aes128cbc-hmac512sha 10ge2p1x553 1518b 1t2c[2]
ethip4ipsec1tnlswasync-scheduler-ip4base-int-aes256gcm 10ge2p1x553 imix 1t2c[2] 1t4c[2]
ethip4ipsec4tnlsw-ip4base-int-aes256gcm 10ge2p1x553 1518b 2t2c[2]
ethip4vxlan-l2bdbasemaclrn 10ge2p1x553 64b 1t1c[2] 2t2c[2]
ethip4vxlan-l2xcbase 10ge2p1x553 64b 4t4c[2]
ethip6-ip6base 10ge2p1x553 78b 1t1c[2] 4t4c[2]
ethip6-ip6scale20k 10ge2p1x553 78b 4t4c[2]


No input data available for . See CSIT job https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-weekly-master-2n-aws for more information.


dpdk-2n-skx-mrr, 0 tests failed, 12 tests passed, duration: 00:25, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-skx/210, dpdk version: 22.07.0



dpdk-3n-skx-mrr, 6 tests failed, 6 tests passed, duration: 00:42, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-skx/209, dpdk version: 22.07.0

eth-l2xcbase-testpmd 10ge2p1x710 64b 1t1c[2] 2t2c[2] 4t4c[2]
ethip4-ip4base-l3fwd 10ge2p1x710 64b 1t1c[2] 2t2c[2] 4t4c[2]


No input data available for dpdk-2n-clx-mrr. See CSIT job https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-clx for more information.


dpdk-3n-tsh-mrr, 0 tests failed, 6 tests passed, duration: 00:16, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/102, dpdk version: 22.07.0



dpdk-2n-tx2-mrr, 0 tests failed, 6 tests passed, duration: 00:09, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-tx2/85, dpdk version: 22.07.0



vpp-2n-skx-ndrpdr, 1 tests failed, 281 tests passed, duration: 22:43, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/208, vpp version: 22.10-rc0~110-g57cf5087e~b2786

avf-ethip6-ip6scale20k-rnd 25ge2p1xxv710 78b 4t2c[1]


vpp-3n-skx-ndrpdr, 0 tests failed, 186 tests passed, duration: 17:08, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/209, vpp version: 22.10-rc0~110-g57cf5087e~b2786



No input data available for vpp-2n-clx-ndrpdr. See CSIT job https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-clx for more information.


No input data available for . See CSIT job https://jenkins.fd.io/view/csit/job/csit-trex-perf-ndrpdr-weekly-master-2n-skx for more information.

For detailed information visit: https://s3-docs.fd.io/csit/master/trending/introduction/failures.html


Legend:

[0] - No traffic forwarded
[1] - Failed to create AVF interface on host xxx.xxx.xxx.xxx
[2] - T-Rex STL runtime error
[3] - Not all NAT44 TCP sessions have been established: 51397 != 64512


master: Progressions as of 2022-08-17 14:04:51 UTC #email

noreply@...
 

Following progressions occured in the last trending job runs, listed per testbed type.

Legend: Test-name NIC Frame-size Trend[Mpps] Runs[#] Long-Term change[%]

vpp-3n-alt-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/73, vpp version: 22.10-rc0~113-gc7f93b321~b1473

No progressions

vpp-2n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-icx/115, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No progressions

vpp-3n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-icx/108, vpp version: 22.10-rc0~113-gc7f93b321~b2789

4t2c-ethip4gtpusw-ip4base-mrr 25ge2p1e810xxv 64B 7.47 1 46.31


vpp-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-skx/1779, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No progressions

vpp-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-skx/1615, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No progressions

vpp-2n-zn2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-zn2/593, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No progressions

vpp-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/664, vpp version: 22.10-rc0~113-gc7f93b321~b1473

1t1c-ethip4-ip4base-eth-2vhostvr1024-1vm-mrr 10ge2p1x520 64B 1.34 1 -22.26
2t2c-ethip4vxlan-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 10ge2p1x520 64B 2.51 1 -0.8


vpp-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-tx2/373, vpp version: 22.10-rc0~113-gc7f93b321~b1473

No progressions

vpp-2n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-dnv/1218, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No progressions

vpp-3n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-dnv/1228, vpp version: 22.10-rc0~113-gc7f93b321~b2789

4t4c-ethip4vxlan-l2xcbase-mrr 10ge2p1x553 64B 10.01 1 5.61
1t1c-ethip6-ip6base-mrr 10ge2p1x553 78B 4.96 1 -0.71
1t1c-ethip4ipsec1000tnlsw-ip4base-int-aes256gcm-mrr 10ge2p1x553 1518B 0.28 1 -1.68
2t2c-ethip4ipsec4tnlsw-ip4base-int-aes256gcm-mrr 10ge2p1x553 1518B 0.63 1 -0.53
2t2c-eth-l2patch-mrr 10ge2p1x553 64B 13.66 1 -0.97
2t2c-eth-l2xcbase-mrr 10ge2p1x553 64B 10.59 1 -0.5
4t4c-eth-l2xcbase-mrr 10ge2p1x553 64B 17.66 1 1.8


dpdk-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-skx/210, dpdk version: 22.07.0

8t4c-eth-l2xcbase-testpmd-mrr 25ge2p1xxv710 64B 31.35 1 5.0


dpdk-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-skx/209, dpdk version: 22.07.0

No progressions

dpdk-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/102, dpdk version: 22.07.0

No progressions

dpdk-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-tx2/85, dpdk version: 22.07.0

No progressions

vpp-2n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/208, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No progressions

vpp-3n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/209, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No progressions


master: Regressions as of 2022-08-17 14:04:51 UTC #email

noreply@...
 

Following regressions occured in the last trending job runs, listed per testbed type.

Legend: Test-name NIC Frame-size Trend[Mpps] Runs[#] Long-Term change[%]

vpp-3n-alt-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/73, vpp version: 22.10-rc0~113-gc7f93b321~b1473

1t1c-eth-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 0.76 1 -78.24
2t2c-ethip4-ip4base-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 1.35 1 -78.37
4t4c-eth-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 2.76 1 -75.71
4t4c-ethip4-ip4base-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 2.64 1 -76.21


vpp-2n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-icx/115, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No regressions

vpp-3n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-icx/108, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No regressions

vpp-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-skx/1779, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No regressions

vpp-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-skx/1615, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No regressions

vpp-2n-zn2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-zn2/593, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No regressions

vpp-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/664, vpp version: 22.10-rc0~113-gc7f93b321~b1473

No regressions

vpp-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-tx2/373, vpp version: 22.10-rc0~113-gc7f93b321~b1473

No regressions

vpp-2n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-dnv/1218, vpp version: 22.10-rc0~113-gc7f93b321~b2789

No regressions

vpp-3n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-dnv/1228, vpp version: 22.10-rc0~113-gc7f93b321~b2789

2t2c-ethip4-ip4scale20k-mrr 10ge2p1x553 64B 2.98 1 -51.65
4t4c-ethip4-ip4scale20k-mrr 10ge2p1x553 64B 2.98 1 -70.88
4t4c-ethip4-ip4scale20k-rnd-mrr 10ge2p1x553 64B 2.97 1 -71.07
4t4c-ethip4ipsec4tnlsw-ip4base-int-aes128cbc-hmac512sha-mrr 10ge2p1x553 IMIX 0.47 1 -9.13
4t4c-ethip4ipsec1000tnlsw-ip4base-int-aes256gcm-mrr 10ge2p1x553 1518B 0.16 1 -85.06
4t4c-eth-l2bdbasemaclrn-mrr 10ge2p1x553 64B 2.96 1 -79.43


dpdk-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-skx/210, dpdk version: 22.07.0

2t1c-ethip4-ip4base-l3fwd-mrr 25ge2p1xxv710 64B 35.97 1 -0.98


dpdk-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-skx/209, dpdk version: 22.07.0

No regressions

dpdk-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/102, dpdk version: 22.07.0

No regressions

dpdk-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-tx2/85, dpdk version: 22.07.0

No regressions

vpp-2n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/208, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions

vpp-3n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/209, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions


master: Regressions as of 2022-08-16 14:03:32 UTC #email

noreply@...
 

Following regressions occured in the last trending job runs, listed per testbed type.

Legend: Test-name NIC Frame-size Trend[Mpps] Runs[#] Long-Term change[%]

vpp-3n-alt-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/72, vpp version: 22.10-rc0~110-g57cf5087e~b1470

1t1c-ethip4-ip4base-eth-2vhostvr1024-1vm-mrr 40ge2p1xl710 64B 0.73 1 -79.26


vpp-2n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-icx/114, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions

vpp-3n-icx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-icx/107, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions

vpp-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-skx/1778, vpp version: 22.10-rc0~110-g57cf5087e~b2786

2t1c-af-xdp-ethip4-ip4scale20k-rnd-mrr 25ge2p1xxv710 64B 0.73 1 -38.68


vpp-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-skx/1614, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions

vpp-2n-zn2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-zn2/592, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions

vpp-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/663, vpp version: 22.10-rc0~110-g57cf5087e~b1470

1t1c-eth-l2xcbase-eth-2vhostvr1024-1vm-mrr 10ge2p1x520 64B 0.61 1 -57.87
1t1c-ethip4-ip4base-eth-2vhostvr1024-1vm-mrr 10ge2p1x520 64B 0.56 1 -67.66
2t2c-eth-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 10ge2p1x520 64B 1.04 1 -67.03
2t2c-ethip4vxlan-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 10ge2p1x520 64B 1.04 1 -58.85


vpp-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-tx2/372, vpp version: 22.10-rc0~110-g57cf5087e~b1470

No regressions

vpp-2n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-2n-dnv/1217, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions

vpp-3n-dnv-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-dnv/1227, vpp version: 22.10-rc0~110-g57cf5087e~b2786

4t4c-ethip4-ip4scale20k-mrr 10ge2p1x553 64B 2.98 1 -70.88
1t1c-ethip4ipsec1000tnlsw-ip4base-int-aes256gcm-mrr 10ge2p1x553 1518B 0.16 1 -45.62
2t2c-eth-l2patch-mrr 10ge2p1x553 64B 2.98 1 -78.41
2t2c-eth-l2xcbase-mrr 10ge2p1x553 64B 2.98 1 -72.01
4t4c-eth-l2xcbase-mrr 10ge2p1x553 64B 2.96 1 -82.94


dpdk-2n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-skx/210, dpdk version: 22.07.0

2t1c-ethip4-ip4base-l3fwd-mrr 25ge2p1xxv710 64B 35.97 1 -0.98


dpdk-3n-skx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-skx/209, dpdk version: 22.07.0

No regressions

dpdk-2n-clx-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-clx/153, dpdk version: 22.07.0

No regressions

dpdk-3n-tsh-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/102, dpdk version: 22.07.0

No regressions

dpdk-2n-tx2-mrr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-2n-tx2/85, dpdk version: 22.07.0

No regressions

vpp-2n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/208, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions

vpp-3n-skx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/209, vpp version: 22.10-rc0~110-g57cf5087e~b2786

No regressions

vpp-2n-clx-ndrpdr, CSIT build: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152, vpp version: 22.10-rc0~88-g389aae573~b2764

No regressions

1 - 20 of 5984