CSIT failing perf tests for week 25 (06/13 – 06/19)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)





1) error: QEMU: NF failed to run on!


   test: VM vhost non Vppl2Xc

   frequency: all

   testbed: 2n-clx, 2n-zn2 (daily)

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1131/log.html.gz#s1-s1-s1-s6-s1-t1



TICKET: https://jira.fd.io/browse/CSIT-1839




2) error: all tcp tput tests failing


   test: 100b tcp tput

   frequency: all

   testbed: 2n-clx, 2n-icx, 2n-skx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-skx/1743/log.html.gz#s1-s1-s1-s2-s17-t1



NOTE: 'reset' TRex is not valid while disconnected.

TICKET: https://jira.fd.io/browse/CSIT-1830


3) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss


   test: all AVF crypto

   frequency: sporadic

   testbed: NDRPDR: 3n-skx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-skx/200/log.html.gz#s1-s1-s1-s1-s1-t1


TICKET: https://jira.fd.io/browse/CSIT-1827

NOTE: Wasn't observed for 1 week


4) error: NDR sporadic packet lost


   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-skx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/140/log.html.gz#s1-s1-s1-s2-s10-t2


TICKET: https://jira.fd.io/browse/CSIT-1802

NOTE: wasn't observed for 5 weeks


5) error: T-Rex STL runtime error

   rca: VPP code - X557 speed_capability set 1GE instead of 10GE

   test: sporadic

   frequency: all

   testbed: 2n-dnv and 3n-dnv

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-dnv/1188/log.html.gz#s1-s1-s1-s1-s1-t1



TODO: VPP to fix speed_capability.

TICKET: https://jira.fd.io/browse/VPP-2010


6) error: failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-skx/1585/log.html.gz#s1-s1-s1-s4-s1-t2


NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782


7) error: Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/84/log.html.gz#s1-s1-s1-s2-s29-t2


TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: 1st time happaned on 2n-icx






8) error: port 0 is down when running traffic profile


   test: 10Ge2P1X710

   frequency: all

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1130/log.html.gz#s1-s1-s1-s2-s22


NOTE: Probably disconnected 10Ge cable. PM opened ticket which was resolved, but that didn't fix the issue.

NOTE: cable connection was verified and it's ok. It might be a problem with driver.

TICKET: https://jira.fd.io/browse/CSIT-1831

NOTE: All testbeds affected.


Best regards,

Viliam Luc