Date   

AMD and NAT44ed stateful benchmarks in updated CSIT-2009 Report

Maciek Konstantynowicz (mkonstan)
 

Hi All,

Quick update re recent major additions to CSIT-2009 report available at https://docs.fd.io/csit/rls2009/report/ :

1. Benchmark results for AMD 2n-zn2 testbed (EPYC Zen2 7532) with VPP and DPDK performance data:

- Testbed description in [12]
- Sample throughput graphs in [13]
- Sample latency graphs in [14]
- Sample benchmark tables in [15]
- Sample VPP operational data (“show runtime”) in [16]

2. New NAT44 Endpoint Dependent stateful benchmarks

- Methodology for Connections-Per-Second (CPS) and Packets-Per-Second (PPS) for UDP and TCP/IP [17]
- Sample benchmark graphs UDP CPS [18], TCP CPS [19], UDP PPS [20], TCP PPS [21]

Happy browsing! :)

Cheers,
Maciek

[12] https://docs.fd.io/csit/rls2009/report/introduction/physical_testbeds.html#node-amd-epyc-zen2-2n-zn2
[13] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/packet_throughput_graphs/ip4-2n-zn2-xxv710.html
[14] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/packet_latency/ip4-2n-zn2-xxv710.html
[15] https://docs.fd.io/csit/rls2009/report/detailed_test_results/vpp_performance_results_2n_zn2/ip4_xxv710.html
[16] https://docs.fd.io/csit/rls2009/report/test_operational_data/vpp_performance_operational_data_2n_zn2/ip4_xxv710.html

[17] https://docs.fd.io/csit/rls2009/report/introduction/methodology_nat44.html#nat44-endpoint-dependent
[18] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/packet_throughput_graphs/nat44-ed-udp-cps.html
[19] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/packet_throughput_graphs/nat44-ed-tcp-cps.html
[20] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/packet_throughput_graphs/nat44-ed-udp-pps.html
[21] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/packet_throughput_graphs/nat44-ed-tcp-pps.html

On 14 Oct 2020, at 21:58, Maciek Konstantynowicz (mkonstan) <mkonstan@...> wrote:

Hi All,

FD.io CSIT-2009 report is available on FD.io docs site:

https://docs.fd.io/csit/rls2009/report/

Great thanks to all contributors in CSIT and VPP communities!

Below summary and pointers to specific sections in the report.
Welcome all comments, best by email to csit-dev@....

Cheers,
-Maciek


CSIT-2009 Release Summary
-------------------------

NEW TESTS

- A new category of tests using TRex ASTF stateful APIs and traffic
profiles with up to 16M UDP and TCP/IP sessions. Initial stateful
tests include VPP NAT44 Endpoint Dependent (NAT44ed)
connections-per-second and packets-per-second throughput (with
controlled packet size). (Note: report test runs are still to be
executed in their fullness, expect them to appear in maintenance
report versions next week and week after next. Maintenance reports are
published on a weekly basis if there are changes.)

- Refactored existing NAT44 Deterministic (NAT44det) throughput tests
and added higher session scale, up to 16M UDP sessions. Continue to
use TRex STL stateless APIs and traffic profiles.

- Added NAT44ed uni-directional UDP throughput tests using TRex STL
stateless APIs and traffic profiles, as a way to verify stateful tests
performance.

- IPsec async mode VPP performance tests, with HW crypto only for now,
meaning Xeon Haswell testbeds only.

- Full suite of tests now running on Mellanox ConnectX5-2p100GE NICs in
2n-clx (Intel Xeon Cascadelake) testbeds using VPP native rdma driver.
For the first time one case see linear multi-core speedup into 72 Mpps
region (L2 on 2 cores, IPv4 on 4 cores), in some cases NIC is the
limit again (like it was to date in CSIT labs with FVL 2p25GE NICs).

BENCHMARKING

- AMD 2n-zn2 testbed onboarded with EPYC 7532 32-Core Processor. Full
set of CSIT-2009 results to be included in one of the upcoming
maintenance reports, following completion of calibrating dry runs that
are currently ongoing.

- Optimization and calibration of TRex STL and ASTF multi-core
configurations, a small impact on test results as captured in current
vs. previous release performance comparisons.


Pointers to CSIT-2009 Report sections
-------------------------------------

1. FD.io CSIT test methodology [1]
2. VPP release notes [2]
3. VPP 64B/IMIX throughput graphs [3]
4. VPP throughput speedup multi-core [4]
5. VPP latency under load [5]
6. VPP comparisons v20.09 vs. v20.05 [6]
7. VPP performance all pkt sizes & NICs [7]
8. DPDK 20.08 apps release notes [8]
9. DPDK 64B throughput graphs [9]
10. DPDK latency under load [10]
11. DPDK comparisons 20.08 vs. 20.02 [11]

Functional device tests (VPP_Device) are also included in the report.

[1] https://docs.fd.io/csit/rls2009/report/introduction/methodology.html
[2] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/csit_release_notes.html
[3] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/packet_throughput_graphs/index.html
[4] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/throughput_speedup_multi_core/index.html
[5] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/packet_latency/index.html
[6] https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/comparisons/current_vs_previous_release.html
[7] https://docs.fd.io/csit/rls2009/report/detailed_test_results/vpp_performance_results/index.html
[8] https://docs.fd.io/csit/rls2009/report/dpdk_performance_tests/csit_release_notes.html
[9] https://docs.fd.io/csit/rls2009/report/dpdk_performance_tests/packet_throughput_graphs/index.html
[10] https://docs.fd.io/csit/rls2009/report/dpdk_performance_tests/packet_latency/index.html
[11] https://docs.fd.io/csit/rls2009/report/dpdk_performance_tests/comparisons/current_vs_previous_release.html


REMINDER: Please Take the Intelligent Network and AI Survey by Jan 5 (LFN)

Brandon Wick
 

LFN Community Members:

The LFN End User Advisory Group (EUAG) recently created a new survey titled the "Intelligent Network and AI for Survey for Telecom Operators, Vendors and AI communities". The survey is designed to determine industry interest, readiness, priorities, scenarios, and methods for adopting intelligent networks using Artificial Intelligence and Machine Learning (AI/ML).

In order to gain a larger sample size and achieve a broader perspective across the industry, we are opening up this particular survey to the broader community. We'll forward this email to the LF Networking lists and feel free to share it with anyone in the ecosystem you feel would be appropriate. Respondent data will be parsed by the company type (network operator, vendor, systems integrator, etc.) Survey results will be made public and shared in Q1 2021. We hope you decide to take the survey and help enrich the known landscape of this exciting new phase in networking evolution. Please take the survey before Jan 5, 2021


Please let us know if you have any questions or comments and thank you!

Best,

Brandon Wick
Senior Integrated Marketing Manager
The Linux Foundation
+1.917.282.0960


REMINDER: Register for the LFN Developer & Testing Forum (Feb 1-4)

Brandon Wick
 

LFN Technical Communities, 

The LFN Developer & Testing Forum will be held virtually over four days, February 1-4, 2021. Once again we will be gathering the LFN projects to progress our releases; discuss project architecture, direction, and integration points; and further innovate through the open source networking stack. We will use a combination of online tools to create a productive and interactive virtual meeting environment. Projects planning participation include CNTT/OPNFV, ONAP, OpenDaylight, and Tungsten Fabric. The event is free of charge and registration is required.
Session proposals are due January 15 so start thinking now about what you'd like to submit. The schedule will be published January 22. Please let us know directly if you have any questions or feedback. Thank you!

Best,

Brandon Wick
Senior Integrated Marketing Manager
The Linux Foundation
+1.917.282.0960


Re: Regrets

George Zhao
 

No worries, we had a quick one today. Happy holidays.

 

George

 

From: tsc@... <tsc@...> On Behalf Of Joel Halpern via lists.fd.io
Sent: Thursday, December 17, 2020 6:59 AM
To: tsc@...
Subject: [tsc] Regrets

 

Last minute meeting conflict; sorry.

Joel


Regrets

Joel Halpern
 

Last minute meeting conflict; sorry.

Joel


Re: [csit-dev] Summary of Jenkins issues

Vanessa Valderrama
 

Andrew,

I've created a new ticket to track the merge failures. I don't believe they are related to the ongoing issues.

https://jira.linuxfoundation.org/browse/RELENG-3343

Thank you,
Vanessa

On 12/16/20 3:19 PM, Andrew 👽 Yourtchenko wrote:

Vanessa,

Today is the VPP 21.01 RC1 milestone, and two of the 21.01-RC1 merge jobs failed during the remerge, namely:



Immediate “remerge” on https://gerrit.fd.io/r/c/vpp/+/30441 didn’t take the effect until the subsequent Jenkins restart.

The RC1 milestone thus got delayed due to need to do another remerge.

Which of the three buckets that you mentioned does this occurrence fall into ?

Thanks !

--a

On 16 Dec 2020, at 21:25, Vanessa Valderrama <vvalderrama@...> wrote:



There has been some confusion regarding the current state of Jenkins and issues we're tracking. Below is a summary with the tickets where you can find and provide updates.

As per our standard support process all communication regarding these issues will be handled via the tickets.

Jenkins slowness

ci-management merge job intermittent timeouts

  • https://jira.linuxfoundation.org/browse/RELENG-3324
    • This issue is not resolved. The last steps taken were to upgrade the JJB version and increase the HTTP connectors from 100/20 to 300/50.
    • Next steps:
      • We would like to focus on the Jenkins slowness issue for now and then continue to troubleshoot the merge job timeout issue once Jenkins is stable

Jenkins build history

  • https://jira.linuxfoundation.org/browse/RELENG-3335
    • The current build discarder macro is not working properly.  We need to clean up build history and migrate to the global-jjb build discarder macro.
    • Cleaning up the existing build history will cause slowness on Jenkins. Our options are to take down time to clean up the build history and migrate jobs to the global-jjb build discarder or clean up the build history when we migrate Jenkins to the new Vexxhost V3 instance. The V3 instances are considerably cheaper and faster than the current V2 instance.
    • We would prefer to clean up the build history during the migration. We do not believe the build history is contributing to the Jenkins slowness or merge job timeout issue.







Summary of Jenkins issues

Vanessa Valderrama
 

There has been some confusion regarding the current state of Jenkins and issues we're tracking. Below is a summary with the tickets where you can find and provide updates.

As per our standard support process all communication regarding these issues will be handled via the tickets.

Jenkins slowness

ci-management merge job intermittent timeouts

  • https://jira.linuxfoundation.org/browse/RELENG-3324
    • This issue is not resolved. The last steps taken were to upgrade the JJB version and increase the HTTP connectors from 100/20 to 300/50.
    • Next steps:
      • We would like to focus on the Jenkins slowness issue for now and then continue to troubleshoot the merge job timeout issue once Jenkins is stable

Jenkins build history

  • https://jira.linuxfoundation.org/browse/RELENG-3335
    • The current build discarder macro is not working properly.  We need to clean up build history and migrate to the global-jjb build discarder macro.
    • Cleaning up the existing build history will cause slowness on Jenkins. Our options are to take down time to clean up the build history and migrate jobs to the global-jjb build discarder or clean up the build history when we migrate Jenkins to the new Vexxhost V3 instance. The V3 instances are considerably cheaper and faster than the current V2 instance.
    • We would prefer to clean up the build history during the migration. We do not believe the build history is contributing to the Jenkins slowness or merge job timeout issue.




Re: Proxies for Dec 10 FD.io TSC

Ole Troan (otroan) <otroan@...>
 

Dear Ed,


Apologies, I am unwell.

Ole, would you be willing to be my proxy at the FD.io TSC tomorrow (Dec 10)?
Yes, that would be fine.
Hope you recover quickly!

Ray, would you be willing to chair the meeting?
Best regards,
Ole



Re: Proxies for Dec 10 FD.io TSC

Ray Kinsella
 

NP - I will chair, get well soon!

 

Ray K

 

From: tsc@... <tsc@...> On Behalf Of Edward Warnicke
Sent: Wednesday 9 December 2020 23:55
To: tsc@...; Ole Troan (otroan) <otroan@...>; Kinsella, Ray <ray.kinsella@...>
Subject: [tsc] Proxies for Dec 10 FD.io TSC

 

Apologies, I am unwell.

 

Ole, would you be willing to be my proxy at the FD.io TSC tomorrow (Dec 10)?

 

Ray, would you be willing to chair the meeting?

 

Ed


Proxies for Dec 10 FD.io TSC

Edward Warnicke
 

Apologies, I am unwell.

Ole, would you be willing to be my proxy at the FD.io TSC tomorrow (Dec 10)?

Ray, would you be willing to chair the meeting?

Ed


Please Take the Intelligent Network and AI Survey by Jan 5 (LFN)

Brandon Wick
 

LFN Community Members:

The LFN End User Advisory Group (EUAG) recently created a new survey titled the "Intelligent Network and AI for Survey for Telecom Operators, Vendors and AI communities". The survey is designed to determine industry interest, readiness, priorities, scenarios, and methods for adopting intelligent networks using Artificial Intelligence and Machine Learning (AI/ML).

In order to gain a larger sample size and achieve a broader perspective across the industry, we are opening up this particular survey to the broader community. We'll forward this email to the LF Networking lists and feel free to share it with anyone in the ecosystem you feel would be appropriate. Respondent data will be parsed by the company type (network operator, vendor, systems integrator, etc.) Survey results will be made public and shared in Q1 2021. We hope you decide to take the survey and help enrich the known landscape of this exciting new phase in networking evolution. Please take the survey before Jan 5, 2021


Please let us know if you have any questions or comments and thank you!

Best,

Brandon Wick
Senior Integrated Marketing Manager
The Linux Foundation
+1.917.282.0960


Ole Troan is my proxy for Dec 3 FD.io TSC meeting

Damjan Marion
 

Ole Troan is my proxy for Dec 3 FD.io TSC meeting

Thanks,

Damjan


Register for the LFN Developer & Testing Forum (Feb 1-4)

Brandon Wick
 

LF Technical Communities, 

The LFN Developer & Testing Forum will be held virtually over four days, February 1 - 4, 2021. Once again we will be gathering the LFN project technical communities to progress our releases; discuss project architecture, direction, and integration points; and further innovate through the open source networking stack. We will use a combination of online tools to create a productive and interactive virtual meeting environment. Projects planning participation include CNTT / OPNFV, ONAP, OpenDaylight, and Tungsten Fabric. The event is free of cost and registration is required.
Session proposals are due January 15 so start thinking now about what you'd like to submit. The schedule will be published January 22. Please let us know directly if you have any questions or feedback. Thank you!

Best, 

Brandon Wick
Senior Integrated Marketing Manager
The Linux Foundation
+1.917.282.0960


Docker Hub Rate Limit issue RESOLVED (was: [vpp-dev] Jenkins jobs UNSTABLE due to failure to upload logs to nexus.fd.io)

Dave Wallace
 

Folks,

The Docker Hub Rate Limit issue has been resolved (details below) and the FD.io CI jobs are operating normally.  Please let me know if you encounter any errant failure signatures.

Thanks to Vanessa & Trishan for their help resolving the outage.

Cheers,
-daw-

---- %< ----
Docker Hub Rate Limit Resolution:

It turns out I misunderstood their rate limiting scheme -- the limit is imposed on anonymous & unauthorized docker id based pull requests, not on the repository accounts. Therefore we needed to create an authenticated account, add it to the 'fdiotools' 'users' team and then configure Nomad to login with the docker id for pull requests from the 'fdiotools' repositories in order to avoid the rate limit.

Vanessa & Trishan created an fd.io email account and docker account which were then added to all of the Jenkins.fd.io Nomad Plugin configuration templates for all FD.io projects.  Nomad is now successfully issuing docker pull requests and spinning up CI job executors at the request of jenkins.fd.io!

Life is good :)
---- %< ----

On 11/18/2020 6:43 PM, Dave Wallace via lists.fd.io wrote:

Folks,

IT-21051 was resolved by Vanessa's ci-management patch [0] while [nearly] simultaneously two patches [1] [2] from Andrew Y were deployed which remove the artifact publishing from the VPP CI jobs.  These changes were subsequently reverted [3].

Operation of VPP CI jobs has been restored and I have done a 'recheck' on all gerrit changes which previously failed due to the UNSTABLE job completion status.

Unfortunately, there is a new issue caused by hitting the Docker Hub Pull limit [4] which is causing job allocations to fail and the jenkins build queue to back up.  I have opened a new LF Help Desk Ticket [4], sent an email to the TSC, and will bring this up in tomorrow's TSC meeting to get it resolved.

There also appears to be a similar issue with the vpp-csit-verify-device-master-1n-skx job which has jobs failing due to the inability to start containers.

Thank you for your patience during this outage and thanks to Vanessa & the entire LF-IT team who worked on identifying the fix to the log upload issue.  Also a big thank you to Andrew Yourtchenko for his assistance in pushing ci-management patches and Vratko for ci-management patch reviews.

-daw-

[0] https://gerrit.fd.io/r/c/ci-management/+/29986
[1] https://gerrit.fd.io/r/c/ci-management/+/29985
[2]
https://gerrit.fd.io/r/c/ci-management/+/29987
[3] https://gerrit.fd.io/r/c/ci-management/+/29988
[4] https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21063

On 11/17/2020 12:38 PM, Dave Wallace via lists.fd.io wrote:
Folks,

There is an issue with CI jobs being marked as UNSTABLE due to the failure to upload log files to nexus.fd.io.  This is causing the CI job pipeline to be stalled due to checkstyle job not succeeding.

I have opened a case with LF-IT: https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21051

Thanks,
-daw-







Re: FD.io Nomad Infrastructure hitting Docker Hub Rate Limit

Trishan de Lanerolle
 

Hi Jim/Dave,
I'm not aware of the status of the discussion. Will inquire on progress.  
Trishan


On Wed, Nov 18, 2020 at 7:37 PM St Leger, Jim <jim.st.leger@...> wrote:

This new Docker issue has come up more than a few times in more than a few open source places. It was my understanding that the LF was trying to work with Docker to solve this for all of their open source projects, inclusive of LFN/FD.io. If that doesn’t happen then, to Dave’s point, fdiotools will have to go through the singular application to Docker and ask for the open source exemption.

 

Does anyone from the LF have any visibility into where that work stands? (Trishan perhaps?)

 

Jim

 

From: tsc@... <tsc@...> On Behalf Of Dave Wallace
Sent: Wednesday, November 18, 2020 5:12 PM
To: infra-steering@...; tsc@...
Subject: [tsc] FD.io Nomad Infrastructure hitting Docker Hub Rate Limit

 

Folks,

FYI, today the FD.io Nomad CI has been starved of build executors due to hitting the Docker Hub Rate Limit [0] while using the 'fdiotools' account on docker hub.

I have opened an LF Help Desk Ticket [1] requesting that the 'fdiotools' account be upgraded either via Docker's Open Source Application process [2] or by upgrading the account to 'Pro' which costs $5 a month which will eliminate the issue ASAP.  Unfortunately until this is resolved there is likely to be daily outages of the FD.io CI pipeline.

Thanks,
-daw-

[0] https://www.docker.com/increase-rate-limits
[1] https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21063
[2] https://www.docker.com/community/open-source/application






--
Trishan R. de Lanerolle
Technical Program Manager & Community Architect 
Networking, Linux Foundation
voice: +1.203.699.6401
skype: tdelanerolle


Re: FD.io Nomad Infrastructure hitting Docker Hub Rate Limit

St Leger, Jim <jim.st.leger@...>
 

This new Docker issue has come up more than a few times in more than a few open source places. It was my understanding that the LF was trying to work with Docker to solve this for all of their open source projects, inclusive of LFN/FD.io. If that doesn’t happen then, to Dave’s point, fdiotools will have to go through the singular application to Docker and ask for the open source exemption.

 

Does anyone from the LF have any visibility into where that work stands? (Trishan perhaps?)

 

Jim

 

From: tsc@... <tsc@...> On Behalf Of Dave Wallace
Sent: Wednesday, November 18, 2020 5:12 PM
To: infra-steering@...; tsc@...
Subject: [tsc] FD.io Nomad Infrastructure hitting Docker Hub Rate Limit

 

Folks,

FYI, today the FD.io Nomad CI has been starved of build executors due to hitting the Docker Hub Rate Limit [0] while using the 'fdiotools' account on docker hub.

I have opened an LF Help Desk Ticket [1] requesting that the 'fdiotools' account be upgraded either via Docker's Open Source Application process [2] or by upgrading the account to 'Pro' which costs $5 a month which will eliminate the issue ASAP.  Unfortunately until this is resolved there is likely to be daily outages of the FD.io CI pipeline.

Thanks,
-daw-

[0] https://www.docker.com/increase-rate-limits
[1] https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21063
[2] https://www.docker.com/community/open-source/application


FD.io Nomad Infrastructure hitting Docker Hub Rate Limit

Dave Wallace
 

Folks,

FYI, today the FD.io Nomad CI has been starved of build executors due to hitting the Docker Hub Rate Limit [0] while using the 'fdiotools' account on docker hub.

I have opened an LF Help Desk Ticket [1] requesting that the 'fdiotools' account be upgraded either via Docker's Open Source Application process [2] or by upgrading the account to 'Pro' which costs $5 a month which will eliminate the issue ASAP.  Unfortunately until this is resolved there is likely to be daily outages of the FD.io CI pipeline.

Thanks,
-daw-

[0] https://www.docker.com/increase-rate-limits
[1] https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21063
[2] https://www.docker.com/community/open-source/application


Thursday call

Joel Halpern
 

Unless there is something urgent, I will not make the call.  The IETF meeting is midnight to 6am US EST all week.

Yours,

Joel


FD.io - JIRA Maintenance 2020-11-23 at 0300 UTC to 0600 UTC

Vanessa Valderrama
 

What:  JIRA maintenance to migrate users from LDAP to Internal directory

When:   2020-11-23 at 0300 UTC to 0600 UTC

Impact:  JIRA will be unavailable during this time

Why:  This change is related to the JIRA Auth0 migration

Thank you,
Vanessa


[pma_tools-dev] PMA_TOOLS Project Status

Vanessa Valderrama
 

We have approval to archive the PMA_TOOLS project. I've added this topic to the agenda to vote on Thursday.

Thank you,

Vanessa



-------- Forwarded Message --------
Subject: RE: [pma_tools-dev] PMA_TOOLS Project Status
Date: Tue, 10 Nov 2020 22:40:18 +0000
From: Tkachuk, Georgii <georgii.tkachuk@...>
To: Vanessa Valderrama <vvalderrama@...>, pma_tools-dev@... <pma_tools-dev@...>


That works, we can archive it.
-Georgii

-----Original Message-----
From: Vanessa Valderrama <vvalderrama@...> Sent: Tuesday, November 10, 2020 3:40 PM
To: Tkachuk, Georgii <georgii.tkachuk@...>; pma_tools-dev@...
Subject: Re: [pma_tools-dev] PMA_TOOLS Project Status

Yes it can.

On 11/10/20 4:38 PM, Tkachuk, Georgii wrote:
Can it be undone at some point in case we need to?
-Georgii

-----Original Message-----
From: pma_tools-dev@... <pma_tools-dev@...> On Behalf Of Vanessa Valderrama
Sent: Tuesday, November 10, 2020 3:38 PM
To: Tkachuk, Georgii <georgii.tkachuk@...>; pma_tools-dev@...
Subject: Re: [pma_tools-dev] PMA_TOOLS Project Status

When we archive the project we set the repository to read only, stop the GitHub replication and remove the Jenkins Jobs.

Thank you,

Vanessa

On 11/10/20 4:32 PM, Tkachuk, Georgii wrote:
Hi Vanessa, we haven't maintained in a while, but it is there for people to use. What does archiving it entail?
-Georgii

-----Original Message-----
From: pma_tools-dev@... <pma_tools-dev@...> On Behalf Of Vanessa Valderrama
Sent: Tuesday, November 10, 2020 3:28 PM
To: pma_tools-dev@...
Subject: [pma_tools-dev] PMA_TOOLS Project Status

Is the PMA_TOOLS project still active or can it be archived?

Thank you,

Vanessa

261 - 280 of 1727