Topics

fdio/release repository on packagecloud now contains 19900 with some odd artifacts


Edward Warnicke
 

Done

Ed

On Thu, Jun 18, 2020 at 9:42 AM Dave Wallace (dwallace) <dwallace@...> wrote:

Luca,

 

Thank you for the excellent write-up on our meeting!

 

@Ed Warnicke, can you please create the hicn repo on packagecloud.io (https://packagecloud/fdio/hicn)?

 

-daw-

 

From: Luca Muscariello <muscariello@...>
Sent: Thursday, June 18, 2020 2:43 AM
To: Andrew 👽 Yourtchenko <ayourtch@...>; Vanessa Valderrama <vvalderrama@...>; Dave Wallace (dwallace) <dwallace@...>; tsc@...
Cc: hicn-dev@...; cicn-dev@...; ci-management-dev@...
Subject: Re: [tsc] fdio/release repository on packagecloud now contains 19900 with some odd artifacts

 

Hi,

 

After meeting with Andrew, Dave and Vanessa, we have decided to

opt for option 4 described in this thread. Tradeoffs have been 

explained in this thread.

 

This decision requires 

- the creation of a new repo https://packagecloud/fdio/hicn;

- update to ci-management on hicn JJB to push artifacts there;

- therefore previous releases are pushed manually to 

   https://packagecloud/fdio/release  via LF tickets.

Plus:

- clean up the release repo to keep actual releases only (there is

  a ci-management patch written by Mauro for this);

- lockdown releases from JJB push;

- https://packagecloud/fdio/attic is used by ci-management to cleanup

  release/ but it may be used to cleanup fdio/hicn at stationary

  regime in the long term.

 

Action points:

- merge the ci-management cleanup job

- create new packagecloud repo

- update hicn ci-management

- lockdown release

 

Thanks

Luca

 

 

 

 

 

 

 

 

On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:

Dear tsc members,

 

(with hicn/cicn PTL's hat on)

 

The current release repo https://packagecloud.io/fdio/release contains 20214 

artifacts almost entirely from our projects.

The current binary distribution is composed of 60 deb packages and 30 rpm packages.

 

Among hicn dependencies we have the latest VPP release, currently 20.01. 

The hICN project is composed of 6 committers and we do not support multiple releases 

but just the master HEAD. 

 

Our git repo has one single master branch (w/o release branches) and releases are based 

on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which

is pushed after the VPP project has released the new version. We start catching up

with all VPP updates during the weeks of release preparation made by Andrew.

As soon as Andrew releases the new distribution under "release/" we merge our patches and 

change git tag in the hicn repo.

 

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be

compatible with "hicn master" which depends on "vpp stable/"

 

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 

"release/" and at each release we need to update ci-management. 

 

3 - we have no need to keep all these artifacts. So we could just keep the latest 

artifacts for each hicn release for archival reasons only. We do not backport anything 

in previous releases nor we support our user base with older releases. Packagecloud has a 

REST API to manage that well.

 

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 

like to delete obsolete artifacts ad in option 3.

 

*Option 1* is unfeasible because it requires our user base to have a complex configuration 

of apt repos, which BTW only works if packages are well created. This is not always the case.

 

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.

It may work in conjunction with option 3. It may work well if we also get one of our committers 

in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

 

*Option 3* seems useful in general and would allow to 

 (i) keep host configuration simple, 

 (ii) keep repo size to the right size, 

 (iii) avoid  repo duplication as in 4. 

if we keep artifacts under release our user-base would be happier as it need not host upgrades.

 

*Option 4 + 3* would be ok as well.

 

Thanks for you feedback

Best

Luca

 

 

 

On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:

Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a


Dave Wallace (dwallace) <dwallace@...>
 

Luca,

 

Thank you for the excellent write-up on our meeting!

 

@Ed Warnicke, can you please create the hicn repo on packagecloud.io (https://packagecloud/fdio/hicn)?

 

-daw-

 

From: Luca Muscariello <muscariello@...>
Sent: Thursday, June 18, 2020 2:43 AM
To: Andrew 👽 Yourtchenko <ayourtch@...>; Vanessa Valderrama <vvalderrama@...>; Dave Wallace (dwallace) <dwallace@...>; tsc@...
Cc: hicn-dev@...; cicn-dev@...; ci-management-dev@...
Subject: Re: [tsc] fdio/release repository on packagecloud now contains 19900 with some odd artifacts

 

Hi,

 

After meeting with Andrew, Dave and Vanessa, we have decided to

opt for option 4 described in this thread. Tradeoffs have been 

explained in this thread.

 

This decision requires 

- the creation of a new repo https://packagecloud/fdio/hicn;

- update to ci-management on hicn JJB to push artifacts there;

- therefore previous releases are pushed manually to 

   https://packagecloud/fdio/release  via LF tickets.

Plus:

- clean up the release repo to keep actual releases only (there is

  a ci-management patch written by Mauro for this);

- lockdown releases from JJB push;

- https://packagecloud/fdio/attic is used by ci-management to cleanup

  release/ but it may be used to cleanup fdio/hicn at stationary

  regime in the long term.

 

Action points:

- merge the ci-management cleanup job

- create new packagecloud repo

- update hicn ci-management

- lockdown release

 

Thanks

Luca

 

 

 

 

 

 

 

 

On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:

Dear tsc members,

 

(with hicn/cicn PTL's hat on)

 

The current release repo https://packagecloud.io/fdio/release contains 20214 

artifacts almost entirely from our projects.

The current binary distribution is composed of 60 deb packages and 30 rpm packages.

 

Among hicn dependencies we have the latest VPP release, currently 20.01. 

The hICN project is composed of 6 committers and we do not support multiple releases 

but just the master HEAD. 

 

Our git repo has one single master branch (w/o release branches) and releases are based 

on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which

is pushed after the VPP project has released the new version. We start catching up

with all VPP updates during the weeks of release preparation made by Andrew.

As soon as Andrew releases the new distribution under "release/" we merge our patches and 

change git tag in the hicn repo.

 

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be

compatible with "hicn master" which depends on "vpp stable/"

 

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 

"release/" and at each release we need to update ci-management. 

 

3 - we have no need to keep all these artifacts. So we could just keep the latest 

artifacts for each hicn release for archival reasons only. We do not backport anything 

in previous releases nor we support our user base with older releases. Packagecloud has a 

REST API to manage that well.

 

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 

like to delete obsolete artifacts ad in option 3.

 

*Option 1* is unfeasible because it requires our user base to have a complex configuration 

of apt repos, which BTW only works if packages are well created. This is not always the case.

 

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.

It may work in conjunction with option 3. It may work well if we also get one of our committers 

in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

 

*Option 3* seems useful in general and would allow to 

 (i) keep host configuration simple, 

 (ii) keep repo size to the right size, 

 (iii) avoid  repo duplication as in 4. 

if we keep artifacts under release our user-base would be happier as it need not host upgrades.

 

*Option 4 + 3* would be ok as well.

 

Thanks for you feedback

Best

Luca

 

 

 

On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:

Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a


Luca Muscariello
 

Hi,

After meeting with Andrew, Dave and Vanessa, we have decided to
opt for option 4 described in this thread. Tradeoffs have been 
explained in this thread.

This decision requires 
- the creation of a new repo https://packagecloud/fdio/hicn;
- update to ci-management on hicn JJB to push artifacts there;
- therefore previous releases are pushed manually to 
   https://packagecloud/fdio/release  via LF tickets.
Plus:
- clean up the release repo to keep actual releases only (there is
  a ci-management patch written by Mauro for this);
- lockdown releases from JJB push;
- https://packagecloud/fdio/attic is used by ci-management to cleanup
  release/ but it may be used to cleanup fdio/hicn at stationary
  regime in the long term.

Action points:
- merge the ci-management cleanup job
- create new packagecloud repo
- update hicn ci-management
- lockdown release

Thanks
Luca








On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:
Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 
artifacts almost entirely from our projects.
The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 
The hICN project is composed of 6 committers and we do not support multiple releases 
but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 
on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which
is pushed after the VPP project has released the new version. We start catching up
with all VPP updates during the weeks of release preparation made by Andrew.
As soon as Andrew releases the new distribution under "release/" we merge our patches and 
change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be
compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 
"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 
of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.
It may work in conjunction with option 3. It may work well if we also get one of our committers 
in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 
 (i) keep host configuration simple, 
 (ii) keep repo size to the right size, 
 (iii) avoid  repo duplication as in 4. 
if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback
Best
Luca



On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a


Andrew 👽 Yourtchenko <ayourtch@...>
 

Dear Luca,

My opinion hasn’t changed - it’s possibly a combination of the below two approaches:

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

I would suggest starting with the step (3) and the safest option to implement is the two stage process:

- moving outdated hicn project artifacts from fdio/release to fdio/attic (to be created if exists)

- deleting the hicn artifacts from fdio/atticthat were added in the previous cleanup cycle or before that.

This approach in itself is implementable in two phases, and the first phase (release=>attic) is reversible, should any glitch arise and should any unnecessary packages be archived.

Since that work will be useful to verify even after the testing, and since from experience we know we can go to bit higher quantify of packages before the wheels fall off, I would also suggest that you can use the current state of fdio/release repo as a verification that the implementation of step (3) works correctly.

We can then revisit the situation and assess whether the step (4) is even needed, or if (3) is good enough.

What do you think ?

--a

On 6 May 2020, at 14:45, Luca Muscariello <muscariello@...> wrote:

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.


Andrew 👽 Yourtchenko <ayourtch@...>
 

Dear Luca,

As we are approaching the VPP release 20.05, I wanted to revive this thread.

At the moment of this writing, the fdio/release has 7408 packages.

Could you please give an update where you are with the implementation of the plans we discussed ? (As per the quoted mail thread).

Thanks a lot !

--a

On 7 Feb 2020, at 11:40, Luca Muscariello <muscariello@...> wrote:


Andrew,

Agree, this is just mitigation. 
4+3 is something we could work on in ci-management and plan to deploy for the next release.

Golden releases, as you say, is to be understood for hicn, we don't have muscles to backport bug fixes in golden releases.
We would end up putting (tomb)stone releases :-) 
For the time being we stay cheap and little and maintain one release.

Thanks
Luca

On Fri, Feb 7, 2020 at 11:28 AM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Excellent, thanks Luca.

Longer term I still suggest option 4+3, and if you have “golden” releases (which in turn will work with released VPP versions) then you might put them under the fdio/release, thus replicating the behavior of VPP repos.

--a

On 7 Feb 2020, at 11:21, Luca Muscariello <muscariello@...> wrote:


FYI

we have reduced the number of artifacts to 3200.


This should fix the problem.

Luca



On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:
Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 
artifacts almost entirely from our projects.
The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 
The hICN project is composed of 6 committers and we do not support multiple releases 
but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 
on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which
is pushed after the VPP project has released the new version. We start catching up
with all VPP updates during the weeks of release preparation made by Andrew.
As soon as Andrew releases the new distribution under "release/" we merge our patches and 
change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be
compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 
"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 
of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.
It may work in conjunction with option 3. It may work well if we also get one of our committers 
in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 
 (i) keep host configuration simple, 
 (ii) keep repo size to the right size, 
 (iii) avoid  repo duplication as in 4. 
if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback
Best
Luca



On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a


Andrew 👽 Yourtchenko <ayourtch@...>
 

Luca,

Excellent, thank you!

So, ballpark timeline-wise, do you think you might get it done to still qualify as a “spring cleaning project” ? :-)

—a

On 6 May 2020, at 16:00, Luca Muscariello <muscariello@...> wrote:


Andrew,

The implementation of (3) is indeed safe and can be done quickly and
does not require any additional decision to be made right now.

Short term, we'll work on (3). There may be some ci-management updates
to force packagecloud to keep the N latest packages only as a FIFO cache
on the currently active release. N can be as small as, say, 5.
For previous releases only one set of packages is kept in the repo.


We are going to do some ci-management updates in any case because we
are currently in the process of catching up with VPP 20.05 so this may
be the right moment.

Thanks
Luca


On Wed, May 6, 2020 at 3:28 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear Luca,

My opinion hasn’t changed - it’s possibly a combination of the below two approaches:

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

I would suggest starting with the step (3) and the safest option to implement is the two stage process:

- moving outdated hicn project artifacts from fdio/release to fdio/attic (to be created if exists)

- deleting the hicn artifacts from fdio/atticthat were added in the previous cleanup cycle or before that.

This approach in itself is implementable in two phases, and the first phase (release=>attic) is reversible, should any glitch arise and should any unnecessary packages be archived.

Since that work will be useful to verify even after the testing, and since from experience we know we can go to bit higher quantify of packages before the wheels fall off, I would also suggest that you can use the current state of fdio/release repo as a verification that the implementation of step (3) works correctly.

We can then revisit the situation and assess whether the step (4) is even needed, or if (3) is good enough.

What do you think ?

--a

On 6 May 2020, at 14:45, Luca Muscariello <muscariello@...> wrote:

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.


Luca Muscariello
 



On Wed, May 6, 2020 at 4:25 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Luca,

Excellent, thank you!

So, ballpark timeline-wise, do you think you might get it done to still qualify as a “spring cleaning project” ? :-)

Message well received :-)


Luca Muscariello
 

Andrew,

The implementation of (3) is indeed safe and can be done quickly and
does not require any additional decision to be made right now.

Short term, we'll work on (3). There may be some ci-management updates
to force packagecloud to keep the N latest packages only as a FIFO cache
on the currently active release. N can be as small as, say, 5.
For previous releases only one set of packages is kept in the repo.


We are going to do some ci-management updates in any case because we
are currently in the process of catching up with VPP 20.05 so this may
be the right moment.

Thanks
Luca


On Wed, May 6, 2020 at 3:28 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear Luca,

My opinion hasn’t changed - it’s possibly a combination of the below two approaches:

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

I would suggest starting with the step (3) and the safest option to implement is the two stage process:

- moving outdated hicn project artifacts from fdio/release to fdio/attic (to be created if exists)

- deleting the hicn artifacts from fdio/atticthat were added in the previous cleanup cycle or before that.

This approach in itself is implementable in two phases, and the first phase (release=>attic) is reversible, should any glitch arise and should any unnecessary packages be archived.

Since that work will be useful to verify even after the testing, and since from experience we know we can go to bit higher quantify of packages before the wheels fall off, I would also suggest that you can use the current state of fdio/release repo as a verification that the implementation of step (3) works correctly.

We can then revisit the situation and assess whether the step (4) is even needed, or if (3) is good enough.

What do you think ?

--a

On 6 May 2020, at 14:45, Luca Muscariello <muscariello@...> wrote:

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.


Luca Muscariello
 

Dear Andrew,

thanks for bringing this to our attention.

As mitigation I'll file another ticket to LF to remove unnecessary packages.
Concerning the long term solution I'm not sure which one has been agreed.

We did not get much feedback on how to proceed except from you.
We had a pending vote to get Mauro approved as committer in ci-management,
as these options require us to have someone to pay attention to hicn patches
during the release phase.

Please share your opinion on what is preferable so that we can implement it.

Thanks and apologies if that is not yet fixed.
Luca





On Wed, May 6, 2020 at 1:03 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear Luca,

As we are approaching the VPP release 20.05, I wanted to revive this thread.

At the moment of this writing, the fdio/release has 7408 packages.

Could you please give an update where you are with the implementation of the plans we discussed ? (As per the quoted mail thread).

Thanks a lot !

--a

On 7 Feb 2020, at 11:40, Luca Muscariello <muscariello@...> wrote:


Andrew,

Agree, this is just mitigation. 
4+3 is something we could work on in ci-management and plan to deploy for the next release.

Golden releases, as you say, is to be understood for hicn, we don't have muscles to backport bug fixes in golden releases.
We would end up putting (tomb)stone releases :-) 
For the time being we stay cheap and little and maintain one release.

Thanks
Luca

On Fri, Feb 7, 2020 at 11:28 AM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Excellent, thanks Luca.

Longer term I still suggest option 4+3, and if you have “golden” releases (which in turn will work with released VPP versions) then you might put them under the fdio/release, thus replicating the behavior of VPP repos.

--a

On 7 Feb 2020, at 11:21, Luca Muscariello <muscariello@...> wrote:


FYI

we have reduced the number of artifacts to 3200.


This should fix the problem.

Luca



On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:
Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 
artifacts almost entirely from our projects.
The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 
The hICN project is composed of 6 committers and we do not support multiple releases 
but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 
on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which
is pushed after the VPP project has released the new version. We start catching up
with all VPP updates during the weeks of release preparation made by Andrew.
As soon as Andrew releases the new distribution under "release/" we merge our patches and 
change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be
compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 
"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 
of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.
It may work in conjunction with option 3. It may work well if we also get one of our committers 
in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 
 (i) keep host configuration simple, 
 (ii) keep repo size to the right size, 
 (iii) avoid  repo duplication as in 4. 
if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback
Best
Luca



On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a


Andrew 👽 Yourtchenko <ayourtch@...>
 

Excellent, thanks Luca.

Longer term I still suggest option 4+3, and if you have “golden” releases (which in turn will work with released VPP versions) then you might put them under the fdio/release, thus replicating the behavior of VPP repos.

--a

On 7 Feb 2020, at 11:21, Luca Muscariello <muscariello@...> wrote:


FYI

we have reduced the number of artifacts to 3200.


This should fix the problem.

Luca



On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:
Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 
artifacts almost entirely from our projects.
The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 
The hICN project is composed of 6 committers and we do not support multiple releases 
but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 
on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which
is pushed after the VPP project has released the new version. We start catching up
with all VPP updates during the weeks of release preparation made by Andrew.
As soon as Andrew releases the new distribution under "release/" we merge our patches and 
change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be
compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 
"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 
of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.
It may work in conjunction with option 3. It may work well if we also get one of our committers 
in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 
 (i) keep host configuration simple, 
 (ii) keep repo size to the right size, 
 (iii) avoid  repo duplication as in 4. 
if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback
Best
Luca



On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a


Luca Muscariello
 

Andrew,

Agree, this is just mitigation. 
4+3 is something we could work on in ci-management and plan to deploy for the next release.

Golden releases, as you say, is to be understood for hicn, we don't have muscles to backport bug fixes in golden releases.
We would end up putting (tomb)stone releases :-) 
For the time being we stay cheap and little and maintain one release.

Thanks
Luca


On Fri, Feb 7, 2020 at 11:28 AM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Excellent, thanks Luca.

Longer term I still suggest option 4+3, and if you have “golden” releases (which in turn will work with released VPP versions) then you might put them under the fdio/release, thus replicating the behavior of VPP repos.

--a

On 7 Feb 2020, at 11:21, Luca Muscariello <muscariello@...> wrote:


FYI

we have reduced the number of artifacts to 3200.


This should fix the problem.

Luca



On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:
Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 
artifacts almost entirely from our projects.
The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 
The hICN project is composed of 6 committers and we do not support multiple releases 
but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 
on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which
is pushed after the VPP project has released the new version. We start catching up
with all VPP updates during the weeks of release preparation made by Andrew.
As soon as Andrew releases the new distribution under "release/" we merge our patches and 
change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be
compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 
"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 
of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.
It may work in conjunction with option 3. It may work well if we also get one of our committers 
in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 
 (i) keep host configuration simple, 
 (ii) keep repo size to the right size, 
 (iii) avoid  repo duplication as in 4. 
if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback
Best
Luca



On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a


Luca Muscariello
 

FYI

we have reduced the number of artifacts to 3200.


This should fix the problem.

Luca



On Fri, Jan 31, 2020 at 9:51 AM Luca Muscariello <muscariello@...> wrote:
Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 
artifacts almost entirely from our projects.
The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 
The hICN project is composed of 6 committers and we do not support multiple releases 
but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 
on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which
is pushed after the VPP project has released the new version. We start catching up
with all VPP updates during the weeks of release preparation made by Andrew.
As soon as Andrew releases the new distribution under "release/" we merge our patches and 
change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be
compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 
"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 
of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.
It may work in conjunction with option 3. It may work well if we also get one of our committers 
in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 
 (i) keep host configuration simple, 
 (ii) keep repo size to the right size, 
 (iii) avoid  repo duplication as in 4. 
if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback
Best
Luca



On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a


Luca Muscariello
 

Dear tsc members,

(with hicn/cicn PTL's hat on)

The current release repo https://packagecloud.io/fdio/release contains 20214 
artifacts almost entirely from our projects.
The current binary distribution is composed of 60 deb packages and 30 rpm packages.

Among hicn dependencies we have the latest VPP release, currently 20.01. 
The hICN project is composed of 6 committers and we do not support multiple releases 
but just the master HEAD. 

Our git repo has one single master branch (w/o release branches) and releases are based 
on git tags. The branch is tagged with the latest VPP release code, i.e. v20.01 which
is pushed after the VPP project has released the new version. We start catching up
with all VPP updates during the weeks of release preparation made by Andrew.
As soon as Andrew releases the new distribution under "release/" we merge our patches and 
change git tag in the hicn repo.

1 - We do not publish hicn binaries under master/ as the vpp dependencies there would not be
compatible with "hicn master" which depends on "vpp stable/"

2 - If we publish under #stable/, we'd create as many artifacts there as currently under 
"release/" and at each release we need to update ci-management. 

3 - we have no need to keep all these artifacts. So we could just keep the latest 
artifacts for each hicn release for archival reasons only. We do not backport anything 
in previous releases nor we support our user base with older releases. Packagecloud has a 
REST API to manage that well.

4 - as an alternative we could create an independent repo for the hicn project. Still we'd 
like to delete obsolete artifacts ad in option 3.

*Option 1* is unfeasible because it requires our user base to have a complex configuration 
of apt repos, which BTW only works if packages are well created. This is not always the case.

*Option 2* does not look like a solution to reduce the number of artifacts in a release folder.
It may work in conjunction with option 3. It may work well if we also get one of our committers 
in ci-management, e.g. Mauro Sardara who's substantially contributed to ci-management already.

*Option 3* seems useful in general and would allow to 
 (i) keep host configuration simple, 
 (ii) keep repo size to the right size, 
 (iii) avoid  repo duplication as in 4. 
if we keep artifacts under release our user-base would be happier as it need not host upgrades.

*Option 4 + 3* would be ok as well.

Thanks for you feedback
Best
Luca



On Thu, Jan 30, 2020 at 7:37 PM Andrew 👽 Yourtchenko <ayourtch@...> wrote:
Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a


Andrew 👽 Yourtchenko <ayourtch@...>
 

Dear TSC,

In the capacity of the VPP 20.01 release manager, I would like to raise your attention to the fact that the fdio/release now contains 19900 files, and the there are some invalid packages, like “honeycomb”, and “-dev”.

This was found as we started CSIT testing for VPP 20.01, and Peter Mikus has a workaround - so it doesn’t impact the schedule for the 20.01 testing. (VPP packages download fine)

However, I wanted to raise this concern.

Are some projects doing so many releases, that it results in almost 20000 files?

Thanks for consideration.

--a