Date   

CSIT failing perf tests for week 47 (11/14 – 11/20)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

New issues - 4

Unfixed issues - 14

Fixed issues - 0

 

===NEW ISSUES===

1) [H] 2n-aws: All tests fail to start VPP

   rca: Module uio_pci_generic not found in directory /lib/modules/5.4.0-1009-aws

   test: all

   frequency: allways

   testbed: 2n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-aws/73/log.html.gz#s1-s1-s1-s1-s1-t1

 

2) [H] 2n-aws: All tests fail to initialize interface

   rca: '/sys/bus/pci/devices/0000:3b:00.0/virtfn0/driver/unbind': No such file or directory

   test: all

   frequency: allways

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/168/log.html.gz#s1-s1-s1-s1-s1-s1-s1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/48/log.html.gz#s1-s1-s1-s1-s1

 

3) [H] 2n-icx: NFV density tests breaks VPP which fails to start (re-opened)

   rca:

   test: all

   frequency: all

   testbed: 2n-icx, 3n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/47/log.html.gz#s1-s1-s1-s1-s1-s1-s1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/48/log.html.gz#s1-s1-s1-s5-s8-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1881

 

4) [H] 2n-clx: dpdk tests failing to unbind PCI device

   rca: DRV_VFIO_PCI

   test: all DPDK

   frequency: allways

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/167/log.html.gz#s1-s1-s1-s2-s16-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-aws/73/log.html.gz#s1-s1-s1-s1-s1-t4

 

===OUTSTANDING UNFIXED===

5) [M] 3n-snr: All hwasync wireguard tests failing when trying to verify device

   rca: Failed to bind PCI device 0000:f4:00.0 to c4xxx on host 10.30.51.93

   test: hwasync wireguard

   frequency: allways

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/45/log.html.gz#s1-s1-s1-s3-s1

 

TICKET: https://jira.fd.io/browse/CSIT-1883

 

6) [M] 1n-aws: TRex NDR PDR ALL IP4 scale and L2 scale tests failing with 50% packet loss

   rca:

   test: ip4scale2m

   frequency: all

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/8/log.html.gz#s1-s1-s1-s1-s2-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1876

NOTE: The root cause can be shared environment in aws cloud.

 

7) [H] 2n-clx, 2n-zn2: all RDMA tests failing with cli_inband clear runtime command

   rca:

   test: RDMA with CX556A NIC

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1212/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/639/log.html.gz#s1-s1-s1-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/167/log.html.gz#s1-s1-s1-s2-s5-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1882

 

8) [M] 3n-tsh: VM tests failing to boot VM

   rca:

   test: 3n-tsh: sporadic VM vhost

   frequency: all

   testbed: 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/710/log.html.gz#s1-s1-s1-s7-s2-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1877

NOTE: 3n-alt testbed was fixed. 3n-tsh still failing

 

9) [M] 3n-snr: 25Ge Interface goes down randomly

   rca:

   test: all

   frequency: sporadic

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/45/log.html.gz#s1-s1-s1-s3-s12-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1871

NOTE: Sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.

 

10) [H] 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/167/log.html.gz#s1-s1-s1-s2-s8-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1864

 

11) [M] 3n-alt, 3n-snr: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/33/log.html.gz#s1-s1-s1-s1-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-iterative-2210-3n-snr/6/log.html.gz#s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

NOTE: 3n-alt was fixed. Only 3n-tsh failing

 

12) [L] 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1264/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

13)[H] 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/148/log.html.gz#s1-s1-s1-s1-s4-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/32/log.html.gz#s1-s1-s1-s1-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/43/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

14)[L] all testbeds: AF-XDP - NDR tests failing from time to time

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

NOTE: This is mainly observed in iterative and coverage. It's very low frequency ~ 1 out of 100

 

15)[M] 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/47/log.html.gz#s1-s1-s1-s2-s37-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

16)[L] T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1264/log.html.gz#s1-s1-s1-s1-s3-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1274/log.html.gz#s1-s1-s1-s2-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

17)[L] failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/639/log.html.gz#s1-s1-s1-s2-s18-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/152/log.html.gz#s1-s1-s1-s5-s1-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

18)[L] Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/160/log.html.gz#s1-s1-s1-s2-s35-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/164/log.html.gz#s1-s1-s1-s2-s54-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 46 (11/07 – 11/13)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

New issues - 0

Unfixed issues - 13

Fixed issues - 2

 

===NEW ISSUES===

 

===OUTSTANDING UNFIXED===

1) [M] 1n-aws: TRex NDR PDR ALL IP4 scale and L2 scale tests failing with 50% packet loss

   rca:

   test: ip4scale2m

   frequency: all

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/7/log.html.gz#s1-s1-s1-s1-s2-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1876

NOTE: The root cause can be shared environment in aws cloud.

 

2) [H] 2n-clx, 2n-zn2: all RDMA tests failing with cli_inband clear runtime command

   rca:

   test: RDMA with CX556A NIC

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1207/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/631/log.html.gz#s1-s1-s1-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/164/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1882

 

3) [M] 3n-alt, 3n-tsh: VM tests failing to boot VM

   rca:

   test: 3n-alt all VM vhost

         3n-tsh: sporadic VM vhost

   frequency: all

   testbed: 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/112/log.html.gz#s1-s1-s1-s6-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2210-3n-tsh/5/log.html.gz#s1-s1-s1-s7-s1-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1877

 

4) [M] 3n-snr: 25Ge Interface goes down randomly

   rca:

   test: all

   frequency: sporadic

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/40/log.html.gz#s1-s1-s1-s1-s3-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1871

NOTE: Sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.

 

5) [H] 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/166/log.html.gz#s1-s1-s1-s2-s8-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1864

 

6) [M] 3n-alt, 3n-snr: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/33/log.html.gz#s1-s1-s1-s1-t2

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-iterative-2210-3n-snr/6/log.html.gz#s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

7) [L] 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1259/log.html.gz#s1-s1-s1-s1-s9-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

8)[H] 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/148/log.html.gz#s1-s1-s1-s1-s4-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/32/log.html.gz#s1-s1-s1-s1-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/43/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

9)[L] all testbeds: AF-XDP - NDR tests failing from time to time

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

NOTE: This is mainly observed in iterative and coverage. It's very low frequency ~ 1 out of 100

 

10)[M] 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/47/log.html.gz#s1-s1-s1-s2-s37-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

11)[L] T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1259/log.html.gz#s1-s1-s1-s1-s1-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1269/log.html.gz#s1-s1-s1-s1-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

12)[L] failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1207/log.html.gz#s1-s1-s1-s2-s37-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/634/log.html.gz#s1-s1-s1-s5-s7-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

13)[L] Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/634/log.html.gz#s1-s1-s1-s2-s21-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/164/log.html.gz#s1-s1-s1-s2-s54-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

#) [H] 3n-icx: VPP failed to start!

   rca:

   test: all

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/143/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/44/log.html.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1881

NOTE: We noticed that nf_density tests are breaking the VPP

 

===FIXED ISSUES===

 

#) [M] 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/139/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

NOTE: Hasn't failed since TRex upgrade. I guess this was fixed by

FIX: https://gerrit.fd.io/r/c/csit/+/37359

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 45 (10/31 – 11/06)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

New issues - 1

Unfixed issues - 12

Fixed issues - 1

 

===NEW ISSUES===

1) [M] 1n-aws: ~260 000 packets lost on ip4scale2m

   rca:

   test: ip4scale2m

   frequency: all

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/6/log.html.gz#s1-s1-s1-s1-s2-t1

 

NOTE: The root cause can be shared environment in aws cloud.

 

===OUTSTANDING UNFIXED===

2) [H] 2n-clx, 2n-zn2: all RDMA tests failing with cli_inband clear runtime command

   rca:

   test: RDMA with CX556A NIC

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1205/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/631/log.html.gz#s1-s1-s1-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/164/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1882

 

3) [M] 3n-alt, 3n-tsh: VM tests failing to boot VM

   rca:

   test: 3n-alt all VM vhost

         3n-tsh: sporadic VM vhost

   frequency: all

   testbed: 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/110/log.html.gz#s1-s1-s1-s6-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2210-3n-tsh/5/log.html.gz#s1-s1-s1-s7-s1-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1877

 

4) [M] 3n-snr: 25Ge Interface goes down randomly

   rca:

   test: all

   frequency: sporadic

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/38/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1871

NOTE: sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.

NOTE: builds #33 and #34 all 72/72 tests failed with interface down

 

5) [H] 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2210-2n-clx/46/log.html.gz#s1-s1-s1-s4-s4-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1864

NOTE: not observed in trending, but in iterative (release)

 

6) [M] 3n-alt, 3n-snr: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/32/log.html.gz#s1-s1-s1-s1-t2

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-iterative-2210-3n-snr/6/log.html.gz#s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

7) [M] 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/139/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

#) [L] 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1257/log.html.gz#s1-s1-s1-s2-s1-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

9)[H] 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s4-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/32/log.html.gz#s1-s1-s1-s1-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/43/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

10)[L] all testbeds: AF-XDP - NDR tests failing from time to time

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

11)[M] 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

12)[L] T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1257/log.html.gz#s1-s1-s1-s1-s7-t4

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1267/log.html.gz#s1-s1-s1-s1-s5-t4

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

13)[L] failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/153/log.html.gz#s1-s1-s1-s2-s17-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/145/log.html.gz#s1-s1-s1-s6-s4-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

14)[L] Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/630/log.html.gz#s1-s1-s1-s2-s22-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/164/log.html.gz#s1-s1-s1-s2-s54-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

#) [H] 3n-icx: VPP failed to start!

   rca:

   test: all

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/143/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/44/log.html.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1881

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 44 (10/24 – 10/30)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

New issues - 2

Unfixed issues - 12

Fixed issues - 2

 

===NEW ISSUES===

1) [H] 3n-icx: VPP failed to start!

   rca:

   test: all

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/143/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/44/log.html.gz

 

2) [H] 2n-clx, 2n-zn2: all RDMA tests failing with cli_inband clear runtime command

   rca:

   test: RDMA with CX556A NIC

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1204/log.html.gz#s1-s1-s1-s2-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/630/log.html.gz#s1-s1-s1-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/164/log.html.gz#s1-s1-s1-s1-s1-t1

 

===OUTSTANDING UNFIXED===

3) [M] 3n-alt, 3n-tsh: VM tests failing to boot VM

   rca:

   test: 3n-alt all VM vhost

         3n-tsh: sporadic VM vhost

   frequency: all

   testbed: 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/109/log.html.gz#s1-s1-s1-s6-s1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2210-3n-tsh/5/log.html.gz#s1-s1-s1-s7-s1-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1877

 

4) [M] 3n-snr: 25Ge Interface goes down randomly

   rca:

   test: all

   frequency: sporadic

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/37/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1871

NOTE: sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.

NOTE: builds #33 and #34 all 72/72 tests failed with interface down

 

5) [H] 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: sporadic

   testbed: 2n-clx

   example: hhttps://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2210-2n-clx/46/log.html.gz#s1-s1-s1-s4-s4-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1864

NOTE: not observed in trending, but in iterative (release)

 

6) [M] 3n-alt, 3n-snr: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/30/log.html.gz#s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-iterative-2210-3n-snr/6/log.html.gz#s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

7) [M] 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/139/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

8)[L] 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1256/log.html.gz#s1-s1-s1-s1-s9-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

9)[H] 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s4-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/32/log.html.gz#s1-s1-s1-s1-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/43/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

10)[L] all testbeds: AF-XDP - NDR tests failing from time to time

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

11)[M] 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

12)[L] T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1256/log.html.gz#s1-s1-s1-s1-s2-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1266/log.html.gz#s1-s1-s1-s1-s1-t5

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

13)[L] failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1204/log.html.gz#s1-s1-s1-s2-s29-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/151/log.html.gz#s1-s1-s1-s4-s10-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

14)[L] Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/630/log.html.gz#s1-s1-s1-s2-s22-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/164/log.html.gz#s1-s1-s1-s2-s54-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

#) [M] 1n-aws: TRex NDRPDR ALL TPUT and CPS tests failing to start TRex

   rca:

   test: all NAT

   frequency: all

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/3/log.html.gz#s1-s1-s1-s1-s3

 

TICKET: https://jira.fd.io/browse/CSIT-1869

 

#) [M] 1n-aws: TRex NDRPDR ALL IP4 scale and L2 scale tests failing with 50% packet loss

   rca:

   test: IP4 scale and L2 scale

   frequency: all

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/3/log.html.gz#s1-s1-s1-s1-s3

 

TICKET: https://jira.fd.io/browse/CSIT-1876

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 43 (10/17 – 10/23)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

New issues - 1

Unfixed issues - 13

Fixed issues - 4

 

===NEW ISSUES===

1) [M] 3n-alt, 3n-tsh: VM tests failing to boot VM

   rca:

   test: 3n-alt all VM vhost

         3n-tsh: sporadic VM vhost

   frequency: all

   testbed: 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/106/log.html.gz#s1-s1-s1-s6-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2210-3n-tsh/5/log.html.gz#s1-s1-s1-s7-s1-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1877

 

===OUTSTANDING UNFIXED===

 

2) [M] 1n-aws: TRex NDRPDR ALL TPUT and CPS tests failing to start TRex

   rca:

   test: all NAT

   frequency: all

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/3/log.html.gz#s1-s1-s1-s1-s3

 

TICKET: https://jira.fd.io/browse/CSIT-1869

 

3) [M] 1n-aws: TRex NDRPDR ALL IP4 scale and L2 scale tests failing with 50% packet loss

   rca:

   test: IP4 scale and L2 scale

   frequency: all

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/3/log.html.gz#s1-s1-s1-s1-s3

 

TICKET: https://jira.fd.io/browse/CSIT-1876

 

4) [M] 3n-snr: 25Ge Interface goes down randomly

   rca:

   test: all

   frequency: sporadic

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/34/log.html.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1871

NOTE: sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.

NOTE: builds #33 and #34 all 72/72 tests failed with interface down

 

5) [M] 3n-alt, 3n-snr: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/30/log.html.gz#s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-iterative-2210-3n-snr/6/log.html.gz#s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

6) [M] 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/139/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

7)[L] 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1253/log.html.gz#s1-s1-s1-s1-s8-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

8)[H] 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s4-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/32/log.html.gz#s1-s1-s1-s1-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/43/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

9)[L] all testbeds: AF-XDP - NDR tests failing from time to time

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

10)[M] 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

11)[L] T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1253/log.html.gz#s1-s1-s1-s1-s1-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1263/log.html.gz#s1-s1-s1-s1-s3-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

12)[L] failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1201/log.html.gz#s1-s1-s1-s2-s36-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/149/log.html.gz#s1-s1-s1-s2-s41-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

13)[L] Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/44/log.html.gz#s1-s1-s1-s2-s35-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

#) [H] ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/408/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/104/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/697/log.html.gz

                                                           

TICKET: https://jira.fd.io/browse/CSIT-1857

 

#) [M] 3n-icx: IP4 tunnels GTPU and WIREGUARD tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

TICKET: https://jira.fd.io/browse/CSIT-1862

NOTE: #43 passed. Fixed by TRex upgrade.

FIX: https://gerrit.fd.io/r/c/csit/+/37359

 

#) [M] 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

NOTE: ARM testbeds still failing so this wasn't observed.

TICKET: https://jira.fd.io/browse/CSIT-1860

FIX: https://gerrit.fd.io/r/c/vpp/+/37442

 

#) [H] 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1197/log.html.gz#s1-s1-s1-s4-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/161/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: week37: issue appeared

NOTE: week39: happened again but only on ip6base e810Cq - build #158.

NOTE: week40: happened only on ip4base e810Cq, but 99,9% of packets are lost now. Build #159

NOTE: week41: This week issue happened only on ip4base e810Cq

NOTE: week42: Issue happened on daily MRR tests. Only Ip6Base (E810Cq) with no traffic.

 

TICKET: https://jira.fd.io/browse/CSIT-1864

NOTE: Fixed by TRex upgrade.

FIX: https://gerrit.fd.io/r/c/csit/+/37359

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 42 (10/10 – 10/16)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

New issues - 2

Unfixed issues - 14

Fixed issues – 0

 

===NEW ISSUES===

 

1) error: 1n-aws: TRex NDRPDR ALL NAT tests failing to start TRex

   rca: Requested RX offload TCP_LRO is not supported

   priority: medium

   test: all NAT TPUT and CPS

   frequency: all

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/3/log.html.gz#s1-s1-s1-s1-s3

 

TICKET: https://jira.fd.io/browse/CSIT-1869

 

2) error: 1n-aws: TRex NDRPDR ALL IP4 scale and L2 scale tests failing with 50% packet loss

   rca:

   priority: medium

   test: IP4 scale and L2 scale

   frequency: all

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/3/log.html.gz#s1-s1-s1-s1-s3

 

TICKET: https://jira.fd.io/browse/CSIT-1876

 

===OUTSTANDING UNFIXED===

 

3) error: 3n-snr: 25Ge Interface goes down randomly

   rca:

   priority: medium

   test: all

   frequency: sporadic

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/32/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1871

NOTE: sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.

 

4) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   priority: high

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/408/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/104/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/697/log.html.gz

                                                           

TICKET: https://jira.fd.io/browse/CSIT-1857

 

5) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   priority: high

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1197/log.html.gz#s1-s1-s1-s4-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/161/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: week37: issue appeared

NOTE: week39: happened again but only on ip6base e810Cq - build #158.

NOTE: week40: happened only on ip4base e810Cq, but 99,9% of packets are lost now. Build #159

NOTE: week41: This week issue happened only on ip4base e810Cq

NOTE: week42: Issue happened on daily MRR tests. Only Ip6Base (E810Cq) with no traffic.

NOTE: due RC2 runs it hasn't finished yet

 

TICKET: https://jira.fd.io/browse/CSIT-1864

 

6) error: 3n-alt: testpmd tests fail with no traffic

   rca:

   test: testpmd

   priority: medium

   frequency: all

   testbed: 3n-alt

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/29/log.html.gz#s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

7) error: 3n-icx: IP4 tunnels GTPU and WIREGUARD tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   priority: medium

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

TICKET: https://jira.fd.io/browse/CSIT-1862

NOTE: due RC2 runs it hasn't finished yet

 

8) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   priority: medium - low

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

NOTE: ARM testbeds still failing so this wasn't observed.

TICKET: https://jira.fd.io/browse/CSIT-1860

 

9) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   priority: medium

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/139/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

10) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   priority: low

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1251/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

11) error: 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   priority: high

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s4-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/32/log.html.gz#s1-s1-s1-s1-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

12) error: all testbeds: AF-XDP - NDR tests failing from time to time

   rca:

   test: af-xdp multicore tests

   priority: low

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

13) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   priority: medium

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

14) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   priority: low

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1251/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1261/log.html.gz#s1-s1-s1-s1-s1-t3

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

15) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   priority: low

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1199/log.html.gz#s1-s1-s1-s5-s25-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

16) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   priority: low

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1199/log.html.gz#s1-s1-s1-s2-s54-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


Re: CSIT failing perf tests for week 42 (10/03 – 10/09)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

Update. I’ve added new row with priority. We can discuss  and make some changes to priority during public call today.

 

CSIT failing perf tests for week 42 (10/03 – 10/09)

 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 3n-snr: 25Ge Interface goes down randomly

   rca:

   priority: medium

   test: all

   frequency: sporadic

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/30/log.html.gz#s1-s1-s1-s1-s5-t6

 

TICKET: https://jira.fd.io/browse/CSIT-1871

NOTE: sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.

 

===OUTSTANDING UNFIXED===

 

2) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   priority: high

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/406/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/102/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/695/log.html.gz

                                                           

TICKET: https://jira.fd.io/browse/CSIT-1857

 

3) error: all testbeds DPDK l3fwd tests have wrong configuration

   rca:

   test: DPDK l3fwd

   priority: medium

   frequency: all

   testbed: all

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/28/log.html.gz#s1-s1-s1-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-verify-master-3n-icx/3/log.html.gz#s1-s1-s1-s1-t1-k2-k5-k14

 

TICKET: https://jira.fd.io/browse/CSIT-1858

NOTE: tests are passing but the traffic returns back to TG instead of being forwarded to DUT2.

FIX: https://gerrit.fd.io/r/c/csit/+/37240

 

4) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   priority: high

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1197/log.html.gz#s1-s1-s1-s4-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/161/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: week37: issue appeared

NOTE: week39: happened again but only on ip6base e810Cq - build #158.

NOTE: week40: happened only on ip4base e810Cq, but 99,9% of packets are lost now. Build #159

NOTE: week41: This week issue happened only on ip4base e810Cq

NOTE: week42: Issue happened on daily MRR tests. Only Ip6Base (E810Cq) with no traffic.

 

TICKET: https://jira.fd.io/browse/CSIT-1864

 

5) error: 3n-alt: testpmd tests fail with no traffic

   rca:

   test: testpmd

   priority: medium

   frequency: all

   testbed: 3n-alt

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/28/log.html.gz#s1-s1-s1-s1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

6) error: 3n-icx: IP4 tunnels GTPU and WIREGUARD tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   priority: medium

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

TICKET: https://jira.fd.io/browse/CSIT-1862

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   priority: medium - low

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

NOTE: ARM testbeds still failing so this wasn't observed.

TICKET: https://jira.fd.io/browse/CSIT-1860

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   priority: medium

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   priority: low

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1249/log.html.gz#s1-s1-s1-s1-s9-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   priority: high

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s4-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/30/log.html.gz#s1-s1-s1-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: all testbeds: AF-XDP - NDR tests failing from time to time

   rca:

   test: af-xdp multicore tests

   priority: low

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/30/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   priority: medium

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   priority: low

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1249/log.html.gz#s1-s1-s1-s1-s1-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1259/log.html.gz#s1-s1-s1-s1-s2-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   priority: low

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1197/log.html.gz#s1-s1-s1-s5-s26-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/145/log.html.gz#s1-s1-s1-s2-s39-t3

                 

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   priority: low

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/624/log.html.gz#s1-s1-s1-s2-s21-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

#) error: 3n-tsh: failed to bind/unbind interface

   rca:

   test: all

   frequency: all

   testbed: 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-tsh/110/log.html.gz#s1-s1-s1-s1-k1-k5-k3-k2-k1-k1-k3-k2-k1

 

TICKET: https://jira.fd.io/browse/CSIT-1874

TODO: JL to fix.

FIX: echo 1 | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

 

#) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1865

 

Best regards,

Viliam Luc

 

From: Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
Sent: 10 October 2022 12:09
To: csit-report@...
Subject: CSIT failing perf tests for week 42 (10/03 – 10/09)

 

=====SUMMARY=====

New issues - 1

Unfixed issues - 15

Fixed issues - 2

 

===NEW ISSUES===

 

1) error: 3n-snr: 25Ge Interface goes down randomly

   rca:

   priority: medium

   test: all

   frequency: sporadic

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/30/log.html.gz#s1-s1-s1-s1-s5-t6

 

TICKET: https://jira.fd.io/browse/CSIT-1871

NOTE: sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.

 

===OUTSTANDING UNFIXED===

 

2) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/406/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/102/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/695/log.html.gz

                                                           

TICKET: https://jira.fd.io/browse/CSIT-1857

 

3) error: all testbeds DPDK l3fwd tests have wrong configuration

   rca:

   test: DPDK l3fwd

   frequency: all

   testbed: all

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/28/log.html.gz#s1-s1-s1-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-verify-master-3n-icx/3/log.html.gz#s1-s1-s1-s1-t1-k2-k5-k14

 

TICKET: https://jira.fd.io/browse/CSIT-1858

 

NOTE: tests are passing but the traffic returns back to TG instead of being forwarded to DUT2.

 

4) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1197/log.html.gz#s1-s1-s1-s4-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/161/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: week37: issue appeared

NOTE: week39: happened again but only on ip6base e810Cq - build #158.

NOTE: week40: happened only on ip4base e810Cq, but 99,9% of packets are lost now. Build #159

NOTE: week41: This week issue happened only on ip4base e810Cq

NOTE: week42: Issue happened on daily MRR tests. Only Ip6Base (E810Cq) with no traffic.

 

TICKET: https://jira.fd.io/browse/CSIT-1864

 

5) error: 3n-alt: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/28/log.html.gz#s1-s1-s1-s1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

6) error: 3n-icx: IP4 tunnels GTPU and WIREGUARD tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

TICKET: https://jira.fd.io/browse/CSIT-1862

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

NOTE: ARM testbeds still failing so this wasn't observed.

TICKET: https://jira.fd.io/browse/CSIT-1860

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1249/log.html.gz#s1-s1-s1-s1-s9-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s4-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/30/log.html.gz#s1-s1-s1-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: all testbeds: AF-XDP - NDR tests failing from time to time

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/30/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1249/log.html.gz#s1-s1-s1-s1-s1-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1259/log.html.gz#s1-s1-s1-s1-s2-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1197/log.html.gz#s1-s1-s1-s5-s26-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/145/log.html.gz#s1-s1-s1-s2-s39-t3

                 

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/624/log.html.gz#s1-s1-s1-s2-s21-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

#) error: 3n-tsh: failed to bind/unbind interface

   rca:

   priority: low

   test: all

   frequency: all

   testbed: 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-tsh/110/log.html.gz#s1-s1-s1-s1-k1-k5-k3-k2-k1-k1-k3-k2-k1

 

TICKET: https://jira.fd.io/browse/CSIT-1874

TODO: JL to fix.

FIX: echo 1 | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

 

#) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

NOTE: This week 3n-icx didn't run due to low cadency of runs because of iterative tests for RC1.

TICKET: https://jira.fd.io/browse/CSIT-1865

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 42 (10/03 – 10/09)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

New issues - 1

Unfixed issues - 15

Fixed issues - 2

 

===NEW ISSUES===

 

1) error: 3n-snr: 25Ge Interface goes down randomly

   rca:

   priority: medium

   test: all

   frequency: sporadic

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/30/log.html.gz#s1-s1-s1-s1-s5-t6

 

TICKET: https://jira.fd.io/browse/CSIT-1871

NOTE: sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.

 

===OUTSTANDING UNFIXED===

 

2) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/406/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/102/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/695/log.html.gz

                                                           

TICKET: https://jira.fd.io/browse/CSIT-1857

 

3) error: all testbeds DPDK l3fwd tests have wrong configuration

   rca:

   test: DPDK l3fwd

   frequency: all

   testbed: all

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/28/log.html.gz#s1-s1-s1-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-verify-master-3n-icx/3/log.html.gz#s1-s1-s1-s1-t1-k2-k5-k14

 

TICKET: https://jira.fd.io/browse/CSIT-1858

 

NOTE: tests are passing but the traffic returns back to TG instead of being forwarded to DUT2.

 

4) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1197/log.html.gz#s1-s1-s1-s4-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/161/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: week37: issue appeared

NOTE: week39: happened again but only on ip6base e810Cq - build #158.

NOTE: week40: happened only on ip4base e810Cq, but 99,9% of packets are lost now. Build #159

NOTE: week41: This week issue happened only on ip4base e810Cq

NOTE: week42: Issue happened on daily MRR tests. Only Ip6Base (E810Cq) with no traffic.

 

TICKET: https://jira.fd.io/browse/CSIT-1864

 

5) error: 3n-alt: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/28/log.html.gz#s1-s1-s1-s1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

6) error: 3n-icx: IP4 tunnels GTPU and WIREGUARD tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

TICKET: https://jira.fd.io/browse/CSIT-1862

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

NOTE: ARM testbeds still failing so this wasn't observed.

TICKET: https://jira.fd.io/browse/CSIT-1860

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1249/log.html.gz#s1-s1-s1-s1-s9-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/137/log.html.gz#s1-s1-s1-s1-s4-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/30/log.html.gz#s1-s1-s1-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/41/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: all testbeds: AF-XDP - NDR tests failing from time to time

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/30/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1249/log.html.gz#s1-s1-s1-s1-s1-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1259/log.html.gz#s1-s1-s1-s1-s2-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1197/log.html.gz#s1-s1-s1-s5-s26-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/145/log.html.gz#s1-s1-s1-s2-s39-t3

                 

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/624/log.html.gz#s1-s1-s1-s2-s21-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

#) error: 3n-tsh: failed to bind/unbind interface

   rca:

   priority: low

   test: all

   frequency: all

   testbed: 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-tsh/110/log.html.gz#s1-s1-s1-s1-k1-k5-k3-k2-k1-k1-k3-k2-k1

 

TICKET: https://jira.fd.io/browse/CSIT-1874

TODO: JL to fix.

FIX: echo 1 | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

 

#) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

NOTE: This week 3n-icx didn't run due to low cadency of runs because of iterative tests for RC1.

TICKET: https://jira.fd.io/browse/CSIT-1865

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 41 (09/26 – 10/02)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

New issues - 0

Unfixed issues - 15

Fixed issues - 3

 

===NEW ISSUES===

 

===OUTSTANDING UNFIXED===

 

1) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/404/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/99/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/693/log.html.gz

                                                           

TICKET: https://jira.fd.io/browse/CSIT-1857

 

2) error: all testbeds DPDK l3fwd tests have wrong configuration

   rca:

   test: DPDK l3fwd

   frequency: all

   testbed: all

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/27/log.html.gz#s1-s1-s1-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-verify-master-3n-icx/3/log.html.gz#s1-s1-s1-s1-t1-k2-k5-k14

 

TICKET: https://jira.fd.io/browse/CSIT-1858

 

NOTE: tests are passing but the traffic returns back to TG instead of being forwarded to DUT2.

 

3) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

NOTE: This week 3n-icx didn't run due to low cadency of runs because of iterative tests for RC1.

TICKET: https://jira.fd.io/browse/CSIT-1865

 

4) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s4-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/159/log.html.gz#s1-s1-s1-s2-s8-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/160/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: week37: issue appeared

NOTE: week39: happened again but only on ip6base e810Cq - build #158.

NOTE: week40: happened only on ip4base e810Cq, but 99,9% of packets are lost now. Build #159

NOTE: week41: This week issue happened only on ip4base e810Cq

 

TICKET: https://jira.fd.io/browse/CSIT-1864

 

5) error: 3n-alt: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/26/log.html.gz#s1-s1-s1-s1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

6) error: 3n-icx: IP4 tunnels GTPU and WIREGUARD tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/39/log.html.gz#s1-s1-s1-s1-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

NOTE: Job failed with another issue so this wasn't observed this week.

TICKET: https://jira.fd.io/browse/CSIT-1862

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

NOTE: ARM testbeds still failing so this wasn't observed.

TICKET: https://jira.fd.io/browse/CSIT-1860

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/135/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/39/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1247/log.html.gz#s1-s1-s1-s1-s9-t6

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/133/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

NOTE: This is wasn't observed this week.

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1247/log.html.gz#s1-s1-s1-s3-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1257/log.html.gz#s1-s1-s1-s1-s3-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1194/log.html.gz#s1-s1-s1-s4-s17-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/622/log.html.gz#s1-s1-s1-s2-s12-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/620/log.html.gz#s1-s1-s1-s2-s21-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

#) error: 2n-aws all tests failing with InvalidPlacementGroup

   rca:

   test: all

   frequency: all

   testbed: 2n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-aws/64/console.log.gz

  

   message:

   Error: error creating EC2 Placement Group (csit-2n-aws-c5n-csit-vpp-perf-mrr-weekly-master-2n-aws-pg): InvalidPlacementGroup.Duplicate: The placement group 'csit-2n-aws-c5n-csit-vpp-perf-mrr-weekly-master-2n-aws-pg' already exists.

              status code: 400, request id: e9120991-2048-4f8b-89d2-4cd0868e5e0e

 

TICKET: https://jira.fd.io/browse/CSIT-1866

 

===FIXED ISSUES===

#) error: 2n-icx: all tests failed with parent suite setup time-out

   rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup

   test: all

   frequency: all

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/39/console.log.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1863

 

#) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on VM VHOST tests

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR daily : 2n-clx: VM VHOST vppl2xc

                             MRR weekly: 2n-clx, 2n-icx: VM VHOST chain vppip4

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1194/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/143/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/622/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/161/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/40/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/160/log.html.gz#s1-s1-s1-s6-s2

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/41/log.html.gz#s1-s1-s1-s6-s2-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

NOTE: fix: 37254: fix(qemu): VM tests | https://gerrit.fd.io/r/c/csit/+/37254 (applied to oper branch on 10/03)

 

TICKET: https://jira.fd.io/browse/CSIT-1859

 

#) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1192/log.html.gz#s1-s1-s1-s2-s18

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/159/log.html.gz#s1-s1-s1-s2-s18

 

TICKET: https://jira.fd.io/browse/CSIT-1861

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 40 (09/19 – 09/25)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

===OUTSTANDING UNFIXED===

 

1) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/402/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/98/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/691/log.html.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1857

 

2) error: all testbeds DPDK l3fwd tests have wrong configuration

   rca:

   test: DPDK l3fwd

   frequency: all

   testbed: all

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/25/log.html.gz#s1-s1-s1-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-verify-master-3n-icx/3/log.html.gz#s1-s1-s1-s1-t1-k2-k5-k14

 

TICKET: https://jira.fd.io/browse/CSIT-1858

 

NOTE: tests are passing but the traffic returns back to TG instead of being forwarded to DUT2.

 

3) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1865

 

NOTE: This week 3n-icx didn't run due to low cadency of runs because of iterative tests.

 

4) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s4-s4-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/159/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: week37: issue appeared

NOTE: week39: happened again but only on ip6base e810Cq - build #158.

NOTE: week40: This week issue happened only on ip4base e810Cq, but 99,9% of packets are lost now. Build #159

 

TICKET: https://jira.fd.io/browse/CSIT-1864

 

5) error: 2n-icx: all tests failed with parent suite setup time-out

   rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup

   test: all

   frequency: all

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/39/console.log.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1863

 

 

6) error: 3n-alt: testpmd tests fail with no traffic

   rca:

   test: testpmd

   frequency: all

   testbed: 3n-alt

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/25/log.html.gz#s1-s1-s1-s1

 

TICKET: https://jira.fd.io/browse/CSIT-1848

 

7) error: 3n-icx: IP4 tunnels GTPU and WIREGUARD tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/39/log.html.gz#s1-s1-s1-s1-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

 

TICKET: https://jira.fd.io/browse/CSIT-1862

 

8) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on VM VHOST tests

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR daily : 2n-clx: VM VHOST vppl2xc

                             MRR weekly: 2n-clx, 2n-icx: VM VHOST chain vppip4

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1192/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/141/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/620/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/160/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/39/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/159/log.html.gz#s1-s1-s1-s6-s2

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/40/log.html.gz#s1-s1-s1-s6-s2-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

TICKET: https://jira.fd.io/browse/CSIT-1859

 

9) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1192/log.html.gz#s1-s1-s1-s2-s18

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/159/log.html.gz#s1-s1-s1-s2-s18

 

TICKET: https://jira.fd.io/browse/CSIT-1861

 

10) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

TICKET: https://jira.fd.io/browse/CSIT-1860

 

11) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/133/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/39/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

12) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1245/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

13) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/133/log.html.gz#s1-s1-s1-s1-s4-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

14) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

15) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

16) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1245/log.html.gz#s1-s1-s1-s1-s3-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1255/log.html.gz#s1-s1-s1-s1-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

17) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/141/log.html.gz#s1-s1-s1-s1-s2-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/620/log.html.gz#s1-s1-s1-s2-s15-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

18) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

  frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/620/log.html.gz#s1-s1-s1-s2-s21-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

#) error: 2n-aws all tests failing with InvalidPlacementGroup

   rca:

   test: all

   frequency: all

   testbed: 2n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-aws/64/console.log.gz

  

   message:

   Error: error creating EC2 Placement Group (csit-2n-aws-c5n-csit-vpp-perf-mrr-weekly-master-2n-aws-pg): InvalidPlacementGroup.Duplicate: The placement group 'csit-2n-aws-c5n-csit-vpp-perf-mrr-weekly-master-2n-aws-pg' already exists.

              status code: 400, request id: e9120991-2048-4f8b-89d2-4cd0868e5e0e

 

TICKET: https://jira.fd.io/browse/CSIT-1866

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


Re: CSIT failing perf tests for week 39 (09/12 – 09/18)

Juraj Linkeš
 

I also have an update regarding the Arm failures. Tianyu from Arm submitted a patch that address the issue: https://gerrit.fd.io/r/c/vpp/+/37199

 

Juraj Linkeš

 

From: csit-report@... <csit-report@...> On Behalf Of Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco) via lists.fd.io
Sent: Thursday, September 22, 2022 1:50 PM
To: csit-report@...
Subject: Re: [csit-report] CSIT failing perf tests for week 39 (09/12 – 09/18)

 

Update:

 

We’ve got issue with all ARM testbeds failing to start VPP. Assigned the highest priority. ARM is aware and are working on fix.

 

1) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/396/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/94/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/687/log.html.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1857

 

Best regards,

Viliam Luc

 

From: Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
Sent: 19 September 2022 11:29
To: csit-report@...
Subject: CSIT failing perf tests for week 39 (09/12 – 09/18)

 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

2) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s4-s4-t1

 

NOTE: happened again but only on ip6base e810Cq - build #158.

 

3) error: 2n-icx: all tests failed with parent suite setup time-out

   rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup

   test: all

   frequency: all

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/39/console.log.gz

 

===OUTSTANDING UNFIXED===

 

4) error: 3n-icx: NDR tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

 

5) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR: VM VHOST vppl2xc

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/136/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/615/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/157/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s6-s2-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

6) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s2-s18

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s2-s18

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1239/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1240/log.html.gz#s1-s1-s1-s2-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1250/log.html.gz#s1-s1-s1-s1-s3-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s2-s1-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


Re: CSIT failing perf tests for week 39 (09/12 – 09/18)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

Update:

 

We’ve got issue with all ARM testbeds failing to start VPP. Assigned the highest priority. ARM is aware and are working on fix.

 

1) error: ALL ARM testbeds are failing with VPP failed to start

   rca:

   test: all

   frequency: all

   testbed: 2n-tx2, 3n-alt, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/396/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/94/log.html.gz

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/687/log.html.gz

 

TICKET: https://jira.fd.io/browse/CSIT-1857

 

Best regards,

Viliam Luc

 

From: Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
Sent: 19 September 2022 11:29
To: csit-report@...
Subject: CSIT failing perf tests for week 39 (09/12 – 09/18)

 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

2) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s4-s4-t1

 

NOTE: happened again but only on ip6base e810Cq - build #158.

 

3) error: 2n-icx: all tests failed with parent suite setup time-out

   rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup

   test: all

   frequency: all

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/39/console.log.gz

 

===OUTSTANDING UNFIXED===

 

4) error: 3n-icx: NDR tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

 

5) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR: VM VHOST vppl2xc

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/136/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/615/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/157/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s6-s2-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

6) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s2-s18

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s2-s18

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1239/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1240/log.html.gz#s1-s1-s1-s2-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1250/log.html.gz#s1-s1-s1-s1-s3-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s2-s1-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 39 (09/12 – 09/18)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded

   rca:

   test: all chain ipsec

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-3n-icx/35/log.html.gz#s1-s1-s1-s1-s1-s1-s1-t1

 

2) error: 2n-clx: half of the packets lost on PDR tests (re-opened)

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s4-s4-t1

 

NOTE: happened again but only on ip6base e810Cq - build #158.

 

3) error: 2n-icx: all tests failed with parent suite setup time-out

   rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup

   test: all

   frequency: all

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/39/console.log.gz

 

===OUTSTANDING UNFIXED===

 

4) error: 3n-icx: NDR tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s3-s1-t1

 

NOTE: TRex doesn't support E810Xxv (Columbiaville).

 

5) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR: VM VHOST vppl2xc

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/136/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/615/log.html.gz#s1-s1-s1-s6-s2-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/157/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s6-s2-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

6) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1187/log.html.gz#s1-s1-s1-s2-s18

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/158/log.html.gz#s1-s1-s1-s2-s18

 

7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: sporadic

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/38/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: sporadic

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1239/log.html.gz#s1-s1-s1-s1-s7-t4

 

TICKET: https://jira.fd.io/browse/CSIT-1850

NOTE: #1240 all tput test passed

 

10) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

11) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

13) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1240/log.html.gz#s1-s1-s1-s2-s1-t1

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1250/log.html.gz#s1-s1-s1-s1-s3-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

14) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/38/log.html.gz#s1-s1-s1-s1-s2-s2-s1-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

15) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                  https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 38 (09/04 – 09/11)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 3n-icx: NDR tests failing with ~1700 packets lost

   rca:

   test: IP4 tunnels with E810Xxv nic

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s3-s1-t1

 

===OUTSTANDING UNFIXED===

 

2) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: MRR: VM VHOST vppl2xc

         NDRPDR: VM VHOST vppip4

   frequency: all

   testbed: 2n-clx, 2n-zn2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1182/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/131/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/610/log.html.gz#s1-s1-s1-s6-s2-t1

                                                          https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-clx/157/log.html.gz#s1-s1-s1-s1-s2-s1-s1-t1

                                                          https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/37/log.html.gz#s1-s1-s1-s1-s2-s1-s1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

3) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710 (ip4base, ip6base, l2bd)

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1182/log.html.gz#s1-s1-s1-s2-s18

 

4) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: all Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/386/log.html.gz#s1-s1-s1-s1-s1-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/679/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

5) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/123/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

6) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1234/log.html.gz#s1-s1-s1-s1-s7-t6

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

7) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/37/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

8) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

9) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

  tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

10) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1234/log.html.gz#s1-s1-s1-s1-s1-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1244/log.html.gz#s1-s1-s1-s1-s2-t4

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

11) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/131/log.html.gz#s1-s1-s1-s3-s4-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

12) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

 

===OUTSTANDING FIXED===

13) error: 2n-clx, 2n-zn2: Failed to create AVF interfaces on 710 NICs

   rca:

   test: AVF 710

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s10-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s2-s10-t1

 

NOTE: 2n-clx X710-AVF tests failing with issue #3

NOTED: FIXED on 09th of September 2n-clx build #1182

 

14) error: 2n-clx: half of the packets lost on PDR tests

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

 

NOTE: intermitent.

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


Re: CSIT failing perf tests for week 36 (08/28 – 09/04)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

Update

  • #3 not only AVF but all x710 tests affected
  • Added one new issue with #5

 

CSIT failing perf tests for week 36 (08/28 – 09/04)

 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 2n-clx, 2n-zn2: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: VM VHOST vppl2xc

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s6-s10-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

2) error: 2n-clx, 2n-zn2: Failed to create AVF interfaces on 710 NICs

   rca:

   test: AVF 710

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s10-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s2-s10-t1

 

NOTE: 2n-clx X710-AVF tests failing with issue #3

 

3) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s2-s18

 

4) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/385/log.html.gz#

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/676/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

5) error: 2n-clx: half of the packets lost on PDR tests

   rca:

   test: e810Cq ip4base, ip6base

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/156/log.html.gz#s1-s1-s1-s2-s8-t1

 

===OUTSTANDING UNFIXED===

 

6) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/117/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

7) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s1-s7-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

8) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/208/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

9) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

10) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

11) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1240/log.html.gz#s1-s1-s1-s1-s3-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s2-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

12) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/116/log.html.gz#s1-s1-s1-s3-s5-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

13) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1190/log.html.gz#s1-s1-s1-s1-s9-t4

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: During week28 1st time happened on 2n-icx

NOTE: During week29 1st time observed on 2n-dnv

NOTE: wasn't observed this week

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc

 

From: csit-report@... <csit-report@...> On Behalf Of Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco) via lists.fd.io
Sent: 05 September 2022 11:39
To: csit-report@...
Subject: [csit-report] CSIT failing perf tests for week 36 (08/28 – 09/04)

 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 2n-clx, 2n-zn2: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: VM VHOST vppl2xc

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s6-s10-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

2) error: 2n-clx, 2n-zn2: Failed to create AVF interfaces on 710 NICs

   rca:

   test: AVF 710 (both X710 and XXV710)

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s10-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s2-s10-t1

 

NOTE: 2n-clx X710-AVF tests failing with issue #3

 

3) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: X710

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s2-s18

 

4) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/385/log.html.gz#

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/676/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

===OUTSTANDING UNFIXED===

 

5) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/117/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

6) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

  rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s1-s7-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

7) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/208/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

8) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

9) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

10) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1240/log.html.gz#s1-s1-s1-s1-s3-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s2-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

11) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/116/log.html.gz#s1-s1-s1-s3-s5-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

12) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1190/log.html.gz#s1-s1-s1-s1-s9-t4

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: During week28 1st time happened on 2n-icx

NOTE: During week29 1st time observed on 2n-dnv

NOTE: wasn't observed this week

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


CSIT failing perf tests for week 36 (08/28 – 09/04)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: 2n-clx, 2n-zn2: QEMU NF failed to run on vppl2xc

   rca: svm_region_map(mmap open): No such file or directory

   test: VM VHOST vppl2xc

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s2-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s6-s10-t1

 

NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.

 

2) error: 2n-clx, 2n-zn2: Failed to create AVF interfaces on 710 NICs

   rca:

   test: AVF 710 (both X710 and XXV710)

   frequency: all

   testbed: 2n-clx, 2n-zn2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s6-s10-t1

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/605/log.html.gz#s1-s1-s1-s2-s10-t1

 

NOTE: 2n-clx X710-AVF tests failing with issue #3

 

3) error: 2n-clx: X710 NICs interfere with TRex

   rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2

   test: AVF X710

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1175/log.html.gz#s1-s1-s1-s2-s18

 

4) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1

   rca:

   test: 2n-tx2: Container Memif

         3n-tsh: SRv6_PROXY

   frequency: all

   testbed: 2n-tx2, 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/385/log.html.gz#

                 https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/676/log.html.gz#s1-s1-s1-s6-s4-t1

  

NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.

NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.

 

===OUTSTANDING UNFIXED===

 

5) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/117/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

6) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

  rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s1-s7-t5

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

7) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/208/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

 

8) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

9) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

10) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1240/log.html.gz#s1-s1-s1-s1-s3-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1230/log.html.gz#s1-s1-s1-s2-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

11) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/116/log.html.gz#s1-s1-s1-s3-s5-t3

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

12) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1190/log.html.gz#s1-s1-s1-s1-s9-t4

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: During week28 1st time happened on 2n-icx

NOTE: During week29 1st time observed on 2n-dnv

NOTE: wasn't observed this week

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


master: The build number 483 of the job csit-trending-merge-daily-master-ubuntu2004-x86_64 failed #email

noreply@...
 


master: The build number 482 of the job csit-trending-merge-daily-master-ubuntu2004-x86_64 failed #email

noreply@...
 


CSIT failing perf tests for week 34 (08/16 – 08/21)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

===OUTSTANDING UNFIXED===

 

1) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss

   rca:

   test: 1518B crypto

   frequency: all

   testbed: 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/110/log.html.gz#s1-s1-s1-s1-s1-t1

 

TICKET: https://jira.fd.io/browse/CSIT-1844

 

2) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions

   rca:

   test: 1518B tput

   frequency: all

   testbeds: 2n-dnv

   examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1220/log.html.gz#s1-s1-s1-s1-s7-t6

 

TICKET: https://jira.fd.io/browse/CSIT-1850

 

3) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: 3n-skx, 3n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/208/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

NOTE: After a while failed on MRR daily tests. Only 1518B tests failed, IMIX passed.

 

4) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3

 

TICKET: https://jira.fd.io/browse/CSIT-1802

 

5) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.

   rca:

   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/151/log.html.gz#s1-s1-s1-s2-s56-t3

                                          

TICKET: https://jira.fd.io/browse/CSIT-1804

 

6) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1230/log.html.gz#s1-s1-s1-s1-s4-t3

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1220/log.html.gz#s1-s1-s1-s1-s2-t2

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

7) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/594/log.html.gz#s1-s1-s1-s5-s13-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

8) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/112/log.html.gz#s1-s1-s1-s2-s35-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1190/log.html.gz#s1-s1-s1-s1-s9-t4

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1145/log.html.gz#s1-s1-s1-s2-s48-t2

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1776/log.html.gz#s1-s1-s1-s2-s37-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: During week28 1st time happened on 2n-icx

NOTE: During week29 1st time observed on 2n-dnv

 

===OUTSTANDING FIXED===

 

9) error: 3n-alt, 2n-zn2, 2n-tx2: FOR loop has no closing END.

   rca: possible cause upgrade of RobotFramework

   test: 3n-alt, 2n-tx2: cache-ip4base-policy

                             2n-zn2: RDMA

   frequency: all

   testbed: 3n-alt, 2n-zn2, 2n-tx2

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/71/log.html.gz#s1-s1-s1-s1-s4-t1

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/591/log.html.gz#s1-s1-s1-s2-s1-t1

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-tx2/371/log.html.gz#s1-s1-s1-s2-s4-t1

 

NOTE: VL to look into the issue.

NOTE: partial fix: 36888: fix(hoststack): add a missing end | https://gerrit.fd.io/r/c/csit/+/36888

FIX: 36906: fix: add missing closing END | https://gerrit.fd.io/r/c/csit/+/36906

 

 

10) error: 2n-clx, 3n-skx: First port is down when running traffic profile

   rca:

   test: 10Ge2P1X710

   frequency: all

   testbed: 2n-clx, 3n-skx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-skx/1616/log.html.gz#s1-s1-s1-s2-s1-t1

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-skx/209/log.html.gz#s1-s1-s1-s1-t1

 

NOTE: Issue had been observed on 2n-clx and fixed with workaround. Now (7/3) it appeared on 3n-skx. The same x710 NIC.

TICKET: https://jira.fd.io/browse/CSIT-1831

 

===FIXED ISSUES===

 

Best regards,

Viliam Luc


master: The build number 481 of the job csit-trending-merge-daily-master-ubuntu2004-x86_64 failed #email

noreply@...
 

1 - 20 of 5993