Published: FD.io CSIT-1908 Release Report

Maciek Konstantynowicz (mkonstan)
 

Hi All,

FD.io CSIT-1908 report has been published on FD.io docs site:

html: https://docs.fd.io/csit/rls1908/report/
pdf: https://docs.fd.io/csit/rls1908/report/_static/archive/csit_rls1908.pdf

Many thanks to All in CSIT, VPP and wider FD.io community who
contributed and worked hard to make CSIT-1908 happen!

Below two summaries:
- CSIT-1908 Release Summary, a high-level summary.
- Points of Note in CSIT-1908 Report, with specific links to report.

Welcome all comments, best by email to csit-dev@....

Cheers,
-Maciek

------------------------------------------------------------------------
CSIT-1908 Release Summary
------------------------------------------------------------------------
1. CSIT-1908 Report

- html link: https://docs.fd.io/csit/rls1908/report/
- pdf link: https://docs.fd.io/csit/rls1908/report/_static/archive/csit_rls1908.pdf

2. New Tests

- VM/VNF service chains with DPDK Testpmd and VPP L2/IPv4 workloads
and external VXLAN encapsulation.
- IPsec with new VPP native cipher algorithms, baseline and large
scale (up to 60k tunnels).
- VM/VNF service chains with VPP IPsec workloads, baseline and
horizontal scaling (experimental, in-testing).
- GBP (Group Based Policy) with external dot1q encapsulation.
- Extended test coverage with VPP native AVF driver: IPv4 scale tests.
- Refreshed VPP TCP/HTTP tests.
- Number of VPP functional device tests running in container based
environment.
- Good VPP PAPI (Python API) test coverage, PAPI used for all VPP
tests.

3. Benchmarking

- Added new processor micro-architectures: ARM/AArch64 (TaiShan) and
Atom (Denverton).

- New testbeds onboarded into FD.io CSIT CI/CD functional and
performance test pipelines.
- Daily trending with throughput changes monitoring, analytics and
anomaly auto-detection.
- Release reports with benchmarking data including throughput,
latency, test repeatibility.

- Updated CSIT benchmarking report specification

- Consistent selection of tests across all testbeds and processor
microarchitectures present in FD.io labs (Xeon, Atom, ARM) for
iterative benchmarking tests to verify results repeatibility. Data
presented in graphs conveying NDR (non-drop rate, zero packet
loss) and PDR (partial drop rate) throughput statistics.
Multi-core speedup and latency are also presented.
- Consistent comparison of NDR and PDR throughput results across the
releases.
- Updated graph naming and test grouping to improve browsability and
access to test data.

- Increased test coverage in 2-node testbed environment (2n-skx).

- Updated soak testing methodology and new results, aligned with
latest IETF draft specification draft-vpolak-bmwg-plrsearch-02.

4. Infrastructure

- API
- PAPI (Python API) used for all VPP tests, migrated away from VAT
(VPP API Test).
- VPP API change detection and gating in VPP and CSIT CI/CD.

- Test Environments
- VPP functional device tests: migrated away from VIRL (VM based) to
container environment (with Nomad).
- Added new physical testbeds: ARM/AArch64 (TaiShan) and Atom
(Denverton).

- CSIT Framework
- Conifguration keyword alignment across 2-node and 3-node testbeds to
ease test portability across environments.

- Installer
- Updated bare-metal CSIT performance testbed installer (ansible).

------------------------------------------------------------------------
Points of Note in CSIT-1908 Report
------------------------------------------------------------------------
Indexed specific links listed at the bottom.

1. VPP release notes
a. Changes in CSIT-1908: [1]
b. Known issues: [2]

2. VPP performance - 64B/IMIX throughput graphs (selected NIC models):
a. Graphs explained: [3]
b. L2 Ethernet Switching: [4]
c. IPv4 Routing: [5]
d. IPv6 Routing: [6]
e. SRv6 Routing: [7]
f. IPv4 Tunnels: [8]
g. KVM VMs vhost-user: [9]
h. LXC/DRC Container Memif: [10]
e. IPsec IPv4 Routing: [11]
f. Virtual Topology System: [12]

3. VPP performance - multi-core and latency graphs:
a. Speedup Multi-Core: [13]
b. Latency: [14]

4. VPP system performance - NFV service density and TCP/IP:
a. VNF (VM) Service Chains: [15]
b. CNF (Container) Service Chains: [16]
c. CNF (Container) Service Pipelines: [17]
d. HTTP and TCP/IP: [18]

4. VPP performance comparisons
a. VPP-19.08 vs. VPP-19.04: [19]

5. VPP performance test details - all NICs:
a. Detailed results 64B IMIX 1518B 9kB: [20]
b. Configuration: [21]

DPDK Testpmd and L3fwd performance sections follow similar structure.

6. DPDK applications:
a. Release notes: [22]
b. DPDK performance - 64B throughput graphs: [23]
c. DPDK performance - latency graphs: [24]
d. DPDK performance - DPDK-1905 vs. DPDK-19.01: [25]

Functional tests, including VPP_Device (functional device tests),
VPP_VIRL and HoneyComb are all included in the report.

Specific links within the report:

[1] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/csit_release_notes.html#changes-in-csit-release
[2] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/csit_release_notes.html#known-issues
[3] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/index.html
[4] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/l2.html
[5] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/ip4.html
[6] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/ip6.html
[7] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/srv6.html
[8] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/ip4_tunnels.html
[9] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/vm_vhost.html
[10] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/container_memif.html
[11] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/ipsec.html
[12] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/vts.html
[13] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/throughput_speedup_multi_core/index.html
[14] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_latency_graphs/index.html
[15] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/nf_service_density/vnf_service_chains.html
[16] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/nf_service_density/cnf_service_chains.html
[17] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/nf_service_density/cnf_service_pipelines.html
[18] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/http_server_performance/index.html
[19] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/comparisons/current_vs_previous_release.html
[20] https://docs.fd.io/csit/rls1908/report/detailed_test_results/vpp_performance_results/index.html
[21] https://docs.fd.io/csit/rls1908/report/test_configuration/vpp_performance_configuration/index.html
[22] https://docs.fd.io/csit/rls1908/report/dpdk_performance_tests/csit_release_notes.html
[23] https://docs.fd.io/csit/rls1908/report/dpdk_performance_tests/packet_throughput_graphs/index.html
[24] https://docs.fd.io/csit/rls1908/report/dpdk_performance_tests/packet_latency_graphs/index.html
[25] https://docs.fd.io/csit/rls1908/report/dpdk_performance_tests/comparisons/current_vs_previous_release.html
------------------------------------------------------------------------

Join honeycomb-dev@lists.fd.io to automatically receive all group messages.