CSIT failing perf tests for week 25 (06/13 – 06/19)


Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
 

=====SUMMARY=====

 

===NEW ISSUES===

 

1) error: QEMU: NF failed to run on 10.32.8.22!

   rca:

   test: VM vhost non Vppl2Xc

   frequency: all

   testbed: 2n-clx, 2n-zn2 (daily)

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1131/log.html.gz#s1-s1-s1-s6-s1-t1

            https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/562/log.html.gz#s1-s1-s1-s6-s1

 

TICKET: https://jira.fd.io/browse/CSIT-1839

 

===OUTSTANDING UNFIXED===

 

2) error: all tcp tput tests failing

   rca:

   test: 100b tcp tput

   frequency: all

   testbed: 2n-clx, 2n-icx, 2n-skx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1743/log.html.gz#s1-s1-s1-s2-s17-t1

                                           https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/84/log.html.gz#s1-s1-s1-s2-s15-t1

 

NOTE: 'reset' TRex is not valid while disconnected.

TICKET: https://jira.fd.io/browse/CSIT-1830

 

3) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss

   rca:

   test: all AVF crypto

   frequency: sporadic

   testbed: NDRPDR: 3n-skx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/200/log.html.gz#s1-s1-s1-s1-s1-t1

                                          

TICKET: https://jira.fd.io/browse/CSIT-1827

NOTE: Wasn't observed for 1 week

 

4) error: NDR sporadic packet lost

   rca:

   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/140/log.html.gz#s1-s1-s1-s2-s10-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1802

NOTE: wasn't observed for 5 weeks

 

5) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1188/log.html.gz#s1-s1-s1-s1-s1-t1

                                           https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-dnv/1197/log.html.gz#s1-s1-s1-s2-s1-t1

 

TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010

 

6) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-skx/1585/log.html.gz#s1-s1-s1-s4-s1-t2

 

NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782

 

7) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/84/log.html.gz#s1-s1-s1-s2-s29-t2

 

TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: 1st time happaned on 2n-icx

 

===OUTSTANDING FIXED===

 

===FIXED ISSUES===

 

8) error: port 0 is down when running traffic profile

   rca:

   test: 10Ge2P1X710

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1130/log.html.gz#s1-s1-s1-s2-s22

 

NOTE: Probably disconnected 10Ge cable. PM opened ticket which was resolved, but that didn't fix the issue.

NOTE: cable connection was verified and it's ok. It might be a problem with driver.

TICKET: https://jira.fd.io/browse/CSIT-1831

NOTE: All testbeds affected.

 

Best regards,

Viliam Luc