CSIT-2001 update: Xeon Skylake Performance and Progressions/Regressions RCAs
Maciek Konstantynowicz (mkonstan)
Slides used on today’s VPP call: https://wiki.fd.io/view/File:200512-csit-vpp-readout.pptxtoggle quoted messageShow quoted text
On 12 May 2020, at 15:18, Maciek Konstantynowicz (mkonstan) <firstname.lastname@example.org> wrote:
Maciek Konstantynowicz (mkonstan)
We have finally pushed out an update to CSIT-2001 report with VPP
performance data for testbeds with Intel Xeon Skylake processors (2n-skx
and 3n-skx testbeds), with SUT and TG servers impacted by firmware and
OS upgrades (BIOS, ucode, kernel updates with mitigations against the
newly discovered Spectre-Meltdown security vulnerabilities).
The updated CSIT-2001 report should be available for browsing just
before 15:00 UTC today, subject to Jenkins job execution (will have
updated version timestamp):
In addition to 2n-skx and 3n-skx performance data available at the usual
locations in the report (see links [r1] to [r4] referenced below), we
have expanded the way we do VPP release-to-release comparisons and root
cause analysis (RCA) for any identified performance progressions and
- CSIT test environment is now versioned, with ver. 1 associated
with CSIT rls1908 git branch as of 2019-08-21, and ver. 2
associated with CSIT master and rls2001 git branches as of
- To identify SUT performance change(s) due to CSIT test environment
change(s) from ver. 1 to ver. 2, VPP v19.08.1 has been re-tested
in ver. 2 and results compared against the past data obtained with
ver. 1. RCA1 analysis has been applied to this part. See [r5].
- To identify SUT performance change(s) due to VPP code change(s)
from v19.08.1 to v20.01.0, both VPP versions have been tested in
CSIT environment ver. 2 and results compared. Separate RCA2
analysis has been applied to this part. See [r5].
- At this stage RCA1 and RCA2 analyses are focusing on progressions > +5%
and regressions < -5%.
Attached pasted complete list of RCAs identified as part of this
exercise  to .
Hope it makes sense. For any questions and comments please contact
(on behalf of FD.io CSIT team)
Specific links within the report:
[r1] VPP throughput graphs,
[r2] VPP throughput speedup multi-core,
[r3] VPP packet latency,
[r4] VPP soak tests,
[r5] 2n-skx PDR comparison with RCA,
[r6] 3n-skx PDR comparison with RCA,
 DONE, Impact of upgrades: i) Skx ucode from 0x2000043 to 0x2000065,
[ii) Linux kernel from 4.15.0-60 to 4.15.0-72 and iii) SuperMicro
[motherboard BIOS from 3.0c to 3.2.
 DONE, Applied fix of FVL NIC firmware 6.0.1 for increasing TRex pps
rate from 27 Mpps to 37 Mpps, [CSIT-1503], [TRex-519].
 DONE, Applied VPP PAPI fix to enable memif zero-copy, [CSIT-1592],
 OPEN, Higher than before StDev of PDR throughput for VPP vhost-user
with VPP-inside-VM, under investigation, [CSIT-1699], [CSIT-1704].
 OPEN, dot1q-l2xcbase progression, retro-inspection of weekly ndrpdr
tests points to ge-22805, automated bisect script does not work
due to frequent API changes, [CSIT-1699], [CSIT-1705].
 DONE, ip4base-nat44 regression, ge-23963
 WIP, avf-ip4scale regression, CANDIDATE(S) before ge-22699, [
 OPEN, VPP vhost-user with VPP-inside-VM higher than before stdev
of PDR throughput, under investigation, [CSIT-1699], [CSIT-1704].
 WIP, vhost-user with testpmd-in-VM progression, CANDIDATE(S)
before 22277, [CSIT-1699], [CSIT-1707].
 WIP, avf-ip4base regression, CANDIDATE(S) range
ge-18361..ge-24505, [CSIT-1699], [CSIT-1708].
 DONE, memif regression, CANDIDATE(S) confirmed ge-23801.
 WIP, ipsec tnl sw scale regression, CANDIDATE(S) before ge-23557,