CSIT failing perf tests for week 01 (01/02 – 01/08)

Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco) <vluc@...>

All updated information also on wiki page:




New issues - 0

Unfixed issues - 13

Fixed issues - 0






01) (H) all testbeds: all DPDK tests not run because DPDK failed to install meson

  - rca: meson.build:4:0: ERROR:  Meson version is 0.49.2 but project requires >= 0.53.2.

  - test: all

  - frequency: always

  - testbed: all

  - examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-2n-clx/175/log.html.gz


02) (M) 3n-icx: Wireguard tests with 100 and more tunnels are failing PDR criteria

  - rca:

  - test: wireguard 100 tunnels and more

  - frequency: high

  - testbed: 3n-icx

  - examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/54/log.html.gz#s1-s1-s1-s3-s9-t2


TICKET: https://jira.fd.io/browse/CSIT-1886

NOTE: issue already had been identified in release coverage testing. Now it's being observed in trending too.


03) [H] 2n-icx: NFV density tests breaks VPP which fails to start (re-opened)


   test: all subsequent

   frequency: sporadic

   testbed: 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/53/console.log.gz


TICKET: https://jira.fd.io/browse/CSIT-1881

NOTE: issue wasn't observed this week


04) [M] 3n-snr: All hwasync wireguard tests failing when trying to verify device

   rca: Failed to bind PCI device 0000:f4:00.0 to c4xxx on host

   test: hwasync wireguard

   frequency: always

   testbed: 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/80/log.html.gz#s1-s1-s1-s3-s1


TICKET: https://jira.fd.io/browse/CSIT-1883


05) [M] 1n-aws: TRex NDR PDR ALL IP4 scale and L2 scale tests failing with 50% packet loss


   test: ip4scale2m

   frequency: always

   testbed: 1n-aws

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-1n-aws/15/log.html.gz#s1-s1-s1-s1-s2-t1


TICKET: https://jira.fd.io/browse/CSIT-1876

NOTE: The root cause can be shared environment in aws cloud.


06) [M] 3n-tsh: all VM tests failing to boot VM


   test: all VM

   frequency: sporadic

   testbed: 3n-tsh

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-tsh/733/log.html.gz#s1-s1-s1-s7-s2-t1


TICKET: https://jira.fd.io/browse/CSIT-1877

NOTE: 3n-alt testbed was fixed. 3n-tsh still failing


07) [H] 2n-clx: half of the packets lost on PDR tests (re-opened)


   test: e810Cq ip4base, ip6base

   frequency: sporadic

   testbed: 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/174/log.html.gz#s1-s1-s1-s2-s8-t1


TICKET: https://jira.fd.io/browse/CSIT-1864


08) [M] 3n-alt, 3n-snr: testpmd tests fail with no traffic


   test: testpmd

   frequency: always

   testbed: 3n-alt, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-3n-alt/34/log.html.gz#s1-s1-s1-s1-t1



TICKET: https://jira.fd.io/browse/CSIT-1848


09)[H] 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss


   test: all AVF crypto with Xxv710

   frequency: sporadic

   testbed: 3n-skx, 3n-icx, 3n-snr

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/148/log.html.gz#s1-s1-s1-s1-s4-t1




TICKET: https://jira.fd.io/browse/CSIT-1827

NOTE: issue wasn't observed this week


10)[L] all testbeds: AF-XDP - NDR tests failing from time to time


   test: af-xdp multicore tests

   frequency: low

   testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3



TICKET: https://jira.fd.io/browse/CSIT-1802

NOTE: This is mainly observed in iterative and coverage. It's very low frequency ~ 1 out of 100


11)[M] 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.


   tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)

   frequency: medium

   testbed: 3n-tsh, 3n-alt, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/170/log.html.gz#s1-s1-s1-s2-s55-t2


TICKET: https://jira.fd.io/browse/CSIT-1804


12)[L] failed creating AVF interface

   rca: issue in Intel FVL driver

   test: multicore AVF

   frequency: sporadic

   testbed: all testbeds

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/195/log.html.gz#s1-s1-s1-s2-s9-t2





NOTE: A long standing issue without a final permanent fix.

TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782


13)[L] Not all DET44 sessions have been established: 4128767 != 4128768

   rca: unknown

   test: nat44det udp 4m and 16m (64k and 1m are ok)

   frequency: very sporadic. It failed in 1 out of 8 runs.

   testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx

   example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/657/log.html.gz#s1-s1-s1-s2-s21-t1


TICKET: https://jira.fd.io/browse/CSIT-1795

NOTE: issue wasn't observed this week






Best regards,

Viliam Luc

Join csit-report@lists.fd.io to automatically receive all group messages.