CSIT failing perf tests for week 46 (11/07 – 11/13)
=====SUMMARY=====
New issues - 0
Unfixed issues - 13
Fixed issues - 2
===NEW ISSUES===
===OUTSTANDING UNFIXED===
1) [M] 1n-aws: TRex NDR PDR ALL IP4 scale and L2 scale tests failing with 50% packet loss
rca:
test: ip4scale2m
frequency: all
testbed: 1n-aws
TICKET: https://jira.fd.io/browse/CSIT-1876
NOTE: The root cause can be shared environment in aws cloud.
2) [H] 2n-clx, 2n-zn2: all RDMA tests failing with cli_inband clear runtime command
rca:
test: RDMA with CX556A NIC
frequency: all
testbed: 2n-clx, 2n-zn2
TICKET: https://jira.fd.io/browse/CSIT-1882
3) [M] 3n-alt, 3n-tsh: VM tests failing to boot VM
rca:
test: 3n-alt all VM vhost
3n-tsh: sporadic VM vhost
frequency: all
testbed: 3n-alt, 3n-tsh
TICKET: https://jira.fd.io/browse/CSIT-1877
4) [M] 3n-snr: 25Ge Interface goes down randomly
rca:
test: all
frequency: sporadic
testbed: 3n-snr
TICKET: https://jira.fd.io/browse/CSIT-1871
NOTE: Sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail.
5) [H] 2n-clx: half of the packets lost on PDR tests (re-opened)
rca:
test: e810Cq ip4base, ip6base
frequency: sporadic
testbed: 2n-clx
TICKET: https://jira.fd.io/browse/CSIT-1864
6) [M] 3n-alt, 3n-snr: testpmd tests fail with no traffic
rca:
test: testpmd
frequency: all
testbed: 3n-alt, 3n-snr
TICKET: https://jira.fd.io/browse/CSIT-1848
7) [L] 2n-dnv: sporadic 1518B tput tests failing to establish required sessions
rca:
test: 1518B tput
frequency: sporadic
testbeds: 2n-dnv
TICKET: https://jira.fd.io/browse/CSIT-1850
8)[H] 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss
rca:
test: all AVF crypto
frequency: sporadic
testbed: 3n-skx, 3n-icx, 3n-snr
TICKET: https://jira.fd.io/browse/CSIT-1827
9)[L] all testbeds: AF-XDP - NDR tests failing from time to time
rca:
test: af-xdp multicore tests
frequency: low
testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx
TICKET: https://jira.fd.io/browse/CSIT-1802
NOTE: This is mainly observed in iterative and coverage. It's very low frequency ~ 1 out of 100
10)[M] 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.
rca:
tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)
frequency: medium
testbed: 3n-tsh, 3n-alt, 2n-clx
TICKET: https://jira.fd.io/browse/CSIT-1804
11)[L] T-Rex STL runtime error
rca: VPP code - X557 speed_capability set 1GE instead of 10GE
test: sporadic
frequency: all
testbed: 2n-dnv and 3n-dnv
TODO: VPP to fix speed_capability.
TICKET: https://jira.fd.io/browse/VPP-2010
12)[L] failed creating AVF interface
rca: issue in Intel FVL driver
test: multicore AVF
frequency: sporadic
testbed: all testbeds
NOTE: A long standing issue without a final permanent fix.
TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782
13)[L] Not all DET44 sessions have been established: 4128767 != 4128768
rca: unknown
test: nat44det udp 4m and 16m (64k and 1m are ok)
frequency: very sporadic. It failed in 1 out of 8 runs.
testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx
TICKET: https://jira.fd.io/browse/CSIT-1795
===OUTSTANDING FIXED===
#) [H] 3n-icx: VPP failed to start!
rca:
test: all
frequency: all
testbed: 3n-icx
example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/143/log.html.gz
https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/44/log.html.gz
TICKET: https://jira.fd.io/browse/CSIT-1881
NOTE: We noticed that nf_density tests are breaking the VPP
===FIXED ISSUES===
#) [M] 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss
rca:
test: 1518B crypto
frequency: sporadic
testbed: 3n-icx
TICKET: https://jira.fd.io/browse/CSIT-1844
NOTE: Hasn't failed since TRex upgrade. I guess this was fixed by
FIX: https://gerrit.fd.io/r/c/csit/+/37359
Best regards,
Viliam Luc