Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a

Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 

artifacts almost entirely from our projects.

The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 

The hICN project is composed of 6 committers and we do not support multiple releases 

but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 

on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which

is pushed after the VPP project has released the new version. We start catching up

with all VPP updates during the weeks of release preparation made by Andrew.

As soon as Andrew releases the new distribution under "release/" we merge our patches and 

change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be

compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 

"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 

artifacts for each hicn release for archival reasons only. We do not backport anything 

in previous releases nor we support our user base with older releases. Packagecloud has a 

REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 

like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 

of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.

It may work in conjunction with option 3. It may work well if we also get one of our committers 

in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 

 (i) keep host configuration simple, 

 (ii) keep repo size to the right size, 

 (iii) avoid  repo duplication as in 4. 

if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback

Best

Luca

On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:

Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a

Excellent, thanks Luca.

Longer term I still suggest option 4+3, and if you have “golden” releases (which in turn will work with released VPP versions) then you might put them under the fdio/release, thus replicating the behavior of VPP repos.

--a

On 7 Feb 2020, at 11:21, Luca Muscariello <muscariello@...> wrote:



FYI

we have reduced the number of artifacts to 3200.

https://packagecloud.io/app/fdio/release/

This should fix the problem.

Luca

On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:

Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 

artifacts almost entirely from our projects.

The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 

The hICN project is composed of 6 committers and we do not support multiple releases 

but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 

on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which

is pushed after the VPP project has released the new version. We start catching up

with all VPP updates during the weeks of release preparation made by Andrew.

As soon as Andrew releases the new distribution under "release/" we merge our patches and 

change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be

compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 

"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 

artifacts for each hicn release for archival reasons only. We do not backport anything 

in previous releases nor we support our user base with older releases. Packagecloud has a 

REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 

like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 

of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.

It may work in conjunction with option 3. It may work well if we also get one of our committers 

in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 

 (i) keep host configuration simple, 

 (ii) keep repo size to the right size, 

 (iii) avoid  repo duplication as in 4. 

if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback

Best

Luca

On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:

Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a

Dear Luca,

As we are approaching the VPP release 20.05, I wanted to revive this thread.

At the moment of this writing, the fdio/release has 7408 packages.

Could you please give an update where you are with the implementation of the plans we discussed ? (As per the quoted mail thread).

Thanks a lot !

--a

On 7 Feb 2020, at 11:40, Luca Muscariello <muscariello@...> wrote:



Andrew,

Agree, this is just mitigation. 

4+3 is something we could work on in ci-management and plan to deploy for the next release.

Golden releases, as you say, is to be understood for hicn, we don't have muscles to backport bug fixes in golden releases.

We would end up putting (tomb)stone releases :-) 

For the time being we stay cheap and little and maintain one release.

Thanks

Luca

On Fri, Feb 7, 2020 at 11:28 AM Andrew 👽 Yourtchenko <ayourtch@...> wrote:

Excellent, thanks Luca.

Longer term I still suggest option 4+3, and if you have “golden” releases (which in turn will work with released VPP versions) then you might put them under the fdio/release, thus replicating the behavior of VPP repos.

--a

On 7 Feb 2020, at 11:21, Luca Muscariello <muscariello@...> wrote:



FYI

we have reduced the number of artifacts to 3200.

https://packagecloud.io/app/fdio/release/

This should fix the problem.

Luca

On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:

Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 

artifacts almost entirely from our projects.

The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 

The hICN project is composed of 6 committers and we do not support multiple releases 

but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 

on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which

is pushed after the VPP project has released the new version. We start catching up

with all VPP updates during the weeks of release preparation made by Andrew.

As soon as Andrew releases the new distribution under "release/" we merge our patches and 

change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be

compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 

"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 

artifacts for each hicn release for archival reasons only. We do not backport anything 

in previous releases nor we support our user base with older releases. Packagecloud has a 

REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 

like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 

of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.

It may work in conjunction with option 3. It may work well if we also get one of our committers 

in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 

 (i) keep host configuration simple, 

 (ii) keep repo size to the right size, 

 (iii) avoid  repo duplication as in 4. 

if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback

Best

Luca

On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:

Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a

Dear Luca,

My opinion hasn’t changed - it’s possibly a combination of the below two approaches:

3 - we have no need to keep all these artifacts. So we could just keep the latest 

artifacts for each hicn release for archival reasons only. We do not backport anything 

in previous releases nor we support our user base with older releases. Packagecloud has a 

REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 

like to delete obsolete artifacts ad in option 3.

I would suggest starting with the step (3) and the safest option to implement is the two stage process:

- moving outdated hicn project artifacts from fdio/release to fdio/attic (to be created if exists)

- deleting the hicn artifacts from fdio/atticthat were added in the previous cleanup cycle or before that.

This approach in itself is implementable in two phases, and the first phase (release=>attic) is reversible, should any glitch arise and should any unnecessary packages be archived.

Since that work will be useful to verify even after the testing, and since from experience we know we can go to bit higher quantify of packages before the wheels fall off, I would also suggest that you can use the current state of fdio/release repo as a verification that the implementation of step (3) works correctly.

We can then revisit the situation and assess whether the step (4) is even needed, or if (3) is good enough.

What do you think ?

--a

On 6 May 2020, at 14:45, Luca Muscariello <muscariello@...> wrote:

3 - we have no need to keep all these artifacts. So we could just keep the latest 

artifacts for each hicn release for archival reasons only. We do not backport anything 

in previous releases nor we support our user base with older releases. Packagecloud has a 

REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 

like to delete obsolete artifacts ad in option 3.

Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 

artifacts almost entirely from our projects.

The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 

The hICN project is composed of 6 committers and we do not support multiple releases 

but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 

on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which

is pushed after the VPP project has released the new version. We start catching up

with all VPP updates during the weeks of release preparation made by Andrew.

As soon as Andrew releases the new distribution under "release/" we merge our patches and 

change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be

compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 

"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 

artifacts for each hicn release for archival reasons only. We do not backport anything 

in previous releases nor we support our user base with older releases. Packagecloud has a 

REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 

like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 

of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.

It may work in conjunction with option 3. It may work well if we also get one of our committers 

in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 

 (i) keep host configuration simple, 

 (ii) keep repo size to the right size, 

 (iii) avoid  repo duplication as in 4. 

if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback

Best

Luca

On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:

Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a

Luca,

 

Thank you for the excellent write-up on our meeting!

 

@Ed Warnicke, can you please create the hicn repo on packagecloud.io (https://packagecloud/fdio/hicn)?

 

-daw-

 

From: Luca Muscariello <muscariello@...>
Sent: Thursday, June 18, 2020 2:43 AM
To: Andrew 👽 Yourtchenko <ayourtch@...>; Vanessa Valderrama <vvalderrama@...>; Dave Wallace (dwallace) <dwallace@...>; tsc@...
Cc: hicn-dev@...; cicn-dev@...; ci-management-dev@...
Subject: Re: [tsc] fdio/release repository on packagecloud now contains 19900 with some odd artifacts

 

Hi,

 

After meeting with Andrew, Dave and Vanessa, we have decided to

opt for option 4 described in this thread. Tradeoffs have been 

explained in this thread.

 

This decision requires 

- the creation of a new repo https://packagecloud/fdio/hicn;

- update to ci-management on hicn JJB to push artifacts there;

- therefore previous releases are pushed manually to 

   https://packagecloud/fdio/release  via LF tickets.

Plus:

- clean up the release repo to keep actual releases only (there is

  a ci-management patch written by Mauro for this);

- lockdown releases from JJB push;

- https://packagecloud/fdio/attic is used by ci-management to cleanup

  release/ but it may be used to cleanup fdio/hicn at stationary

  regime in the long term.

 

Action points:

- merge the ci-management cleanup job

- create new packagecloud repo

- update hicn ci-management

- lockdown release

 

Thanks

Luca

 

 

 

 

 

 

 

 

On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:

Dear tsc members,

 

(with hicn/cicn PTL's hat on)

 

The current release repo https://packagecloud.io/fdio/release contains 20214 

artifacts almost entirely from our projects.

The current binary distribution is composed of 60 deb packages and 30 rpm packages.

 

Among hicn dependencies we have the latest VPP release, currently 20.01. 

The hICN project is composed of 6 committers and we do not support multiple releases 

but just the master HEAD. 

 

Our git repo has one single master branch (w/o release branches) and releases are based 

on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which

is pushed after the VPP project has released the new version. We start catching up

with all VPP updates during the weeks of release preparation made by Andrew.

As soon as Andrew releases the new distribution under "release/" we merge our patches and 

change git tag in the hicn repo.

 

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be

compatible with "hicn master" which depends on "vpp stable/"

 

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 

"release/" and at each release we need to update ci-management. 

 

3 - we have no need to keep all these artifacts. So we could just keep the latest 

artifacts for each hicn release for archival reasons only. We do not backport anything 

in previous releases nor we support our user base with older releases. Packagecloud has a 

REST API to manage that well.

 

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 

like to delete obsolete artifacts ad in option 3.

 

*Option 1* is unfeasible because it requires our user base to have a complex configuration 

of apt repos, which BTW only works if packages are well created. This is not always the case.

 

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.

It may work in conjunction with option 3. It may work well if we also get one of our committers 

in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

 

*Option 3* seems useful in general and would allow to 

 (i) keep host configuration simple, 

 (ii) keep repo size to the right size, 

 (iii) avoid  repo duplication as in 4. 

if we keep artifacts under release our user-base would be happier as it need not host upgrades.

 

*Option 4 + 3* would be ok as well.

 

Thanks for you feedback

Best

Luca

 

 

 

On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:

Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a