Re: Continuous deployment, in-house PyPi repo and artifact promotion

--===============0805743222==
Content-Type: multipart/alternative; boundary=089e0160d184a8e3dd04dd04d38f

--089e0160d184a8e3dd04dd04d38f
Content-Type: text/plain; charset=ISO-8859-1

Catalog-sig is now dead, but I felt it made sense to tie off this thread by
mentioning our new solution.:
===========================
Defend Against Fruit is focused on providing a pragmatic, continuous
deployment style build system for Python. Current Python build systems do
not properly account for the needs of effective continuous deployment. This
package extends the Python tooling to add the missing pieces, including
integration with Artifactory.

With an eye to agile development principles and fast-feedback, we want a
build system which satisfies the following goals:

* Every SCM change-set committed should result in a potentially shippable
release candidate.

* When a defect is introduced, we want to immediately detect and isolate the
offending SCM change-set. This is true  even if the defect was introduced
into a library we depend upon.

* Library management should be so easy as to never impede code changes, even
in multi-component architecture.

More details available at:  http://teamfruit.github.io/defend_against_fruit/
<http://teamfruit.github.io/defend_against_fruit/>

License: Apache Public License v2

Authors:

James Carpenter
jcarpenter621 at yahoo.com
LinkedIn:  http://www.linkedin.com/in/jamescarpenter1
<http://www.linkedin.com/in/jamescarpenter1>

Matthew Tardiff
mattrix at gmail.com
LinkedIn:  http://www.linkedin.com/in/matthewtardiff
<http://www.linkedin.com/in/matthewtardiff>


On Thu, Feb 21, 2013 at 5:21 PM, James Carpenter <[email protected]> wrote:

> 1) Which PyPi repository servers are the most mature and feature rich, and
> therefore the best choice at the moment?
>
> 2) Do any of the PyPi repository servers support managing multiple local
> repositories? If yes, which ones? If no, do you have any recommendations on
> how to most easily support the artifact promotion needs of continuous
> deployment within the Python build ecosystem?
>
> =================================
> Background:
> As part of an effort to implement continuous deployment a fellow traveler
> and I are attempting to workout a good solution for an in-house PyPI
> repository manager. Support of the PyPI XML-RPC API and artifact promotion
> appear to be critical requirements of any solution.
>
> We have tried using Artifactory, but unfortunately this only gives us
> support for "simple" PyPI, without the XML-RPC API distutils/pip uses to
> support searches.
> This is disappointing, because in other respects Artifactory and Nexus
> provide very complete mature support. Most importantly Artifactory and
> Nexus support the ability to host multiple local repositories and proxy
> external ones. By leveraging virtual repositories within Artifactory or
> Nexus one can easily inter-weave results from various local repositories as
> per configurable precedence rules.
>
> The ability to have the build tooling pull artifacts from multiple
> internal repositories turns out to be very important from a continuous
> deployment standpoint. Effective continuous deployment typically treats
> every build artifact as a potential release candidate. As a release
> candidate marches through each gauntlet of tests (unit, integration, load,
> manual, etc.) it is promoted to the next internal repository. This is in
> contrast to the approach of using SNAPSHOTS or the like which fails the
> rule of giving each code check-in the traceability necessary to be a
> potential release candidate. Whether the build tool itself (distutils, Ivy,
> Maven, etc.), the repository manager (Artifactory, Nexus, CheeseShop, etc.)
> or something else knows how to overlay artifacts is an implementation
> detail, but some tool has to do the job. Similarly the details of how a
> build artifact is promoted from one local repository to another is also an
> implementation detail with a variety of possible solutions.
>
> Recognizing support for the PyPI XMLRPC API tends to be a big deal at
> scale, we are looking into using CheeseShop,  Crate.io, or similar to
> address our needs. Another choice may be to place a read-only solution in
> front of Artifactory that adds support for the PyPI XMLRPC API. I am hoping
> members of this reading list can help point us in the right direction.
>
> Thanks for the time and effort you have spent reading this rather long
> post. I look forward to your responses.
>
> Sincerely,
> James Carpenter
> jcarpenter621 at yahoo dot com
>
> P.S.: As you may have guessed I am a visitor from the land of Java where
> the Sun has now set and the Seth lord reigns. (I know a Monty Python
> reference would be better, but I am a bit ignorant in that regard.)
>

--089e0160d184a8e3dd04dd04d38f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div style><font face=3D"arial, sans-serif">Catalog-sig is=
 now dead, but I felt it made sense to tie off this thread by mentioning ou=
r new solution.:</font></div><div style><font face=3D"arial, sans-serif">=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D</font></div>
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>Defend Against Fruit is focused on providing a pragmatic, continuous</span=
><br style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>deployment style build system for Python. Current Python build systems do<=
/span><br style=3D"font-family:arial,sans-serif;font-size:12.80000019073486=
3px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>not properly account for the needs of effective continuous deployment. Thi=
s</span><br style=3D"font-family:arial,sans-serif;font-size:12.800000190734=
863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>package extends the Python tooling to add the missing pieces, including</s=
pan><br style=3D"font-family:arial,sans-serif;font-size:12.800000190734863p=
x">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>integration with Artifactory.</span><br style=3D"font-family:arial,sans-se=
rif;font-size:12.800000190734863px"><br style=3D"font-family:arial,sans-ser=
if;font-size:12.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>With an eye to agile development principles and fast-feedback, we want a</=
span><br style=3D"font-family:arial,sans-serif;font-size:12.800000190734863=
px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>build system which satisfies the following goals:</span><br style=3D"font-=
family:arial,sans-serif;font-size:12.800000190734863px"><br style=3D"font-f=
amily:arial,sans-serif;font-size:12.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>* Every SCM change-set committed should result in a potentially shippable<=
/span><br style=3D"font-family:arial,sans-serif;font-size:12.80000019073486=
3px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>release candidate.</span><br style=3D"font-family:arial,sans-serif;font-si=
ze:12.800000190734863px"><br style=3D"font-family:arial,sans-serif;font-siz=
e:12.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>* When a defect is introduced, we want to immediately detect and isolate t=
he</span><br style=3D"font-family:arial,sans-serif;font-size:12.80000019073=
4863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>offending SCM change-set. This is true =A0even if the defect was introduce=
d</span><br style=3D"font-family:arial,sans-serif;font-size:12.800000190734=
863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>into a library we depend upon.</span><br style=3D"font-family:arial,sans-s=
erif;font-size:12.800000190734863px"><br style=3D"font-family:arial,sans-se=
rif;font-size:12.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>* Library management should be so easy as to never impede code changes, ev=
en</span><br style=3D"font-family:arial,sans-serif;font-size:12.80000019073=
4863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>in multi-component architecture.</span><br style=3D"font-family:arial,sans=
-serif;font-size:12.800000190734863px"><br style=3D"font-family:arial,sans-=
serif;font-size:12.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>More details available at: =A0</span><a href=3D"http://teamfruit.github.io=
/defend_against_fruit/" target=3D"_blank" style=3D"font-family:arial,sans-s=
erif;font-size:12.800000190734863px">http://teamfruit.github.io/defend_agai=
nst_fruit/</a><br style=3D"font-family:arial,sans-serif;font-size:12.800000=
190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>&lt;</span><a href=3D"http://teamfruit.github.io/defend_against_fruit/" ta=
rget=3D"_blank" style=3D"font-family:arial,sans-serif;font-size:12.80000019=
0734863px">http://teamfruit.github.io/defend_against_fruit/</a><span style=
=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">&gt;</span=
><br style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">
<br style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"><=
span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">=
License: Apache Public License v2</span><br style=3D"font-family:arial,sans=
-serif;font-size:12.800000190734863px">
<br style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"><=
span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">=
Authors:</span><br style=3D"font-family:arial,sans-serif;font-size:12.80000=
0190734863px">
<br style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"><=
span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">=
James Carpenter</span><br style=3D"font-family:arial,sans-serif;font-size:1=
2.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>jcarpenter621 at=A0</span><a href=3D"http://yahoo.com/" target=3D"_blank" =
style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">yahoo=
.com</a><br style=3D"font-family:arial,sans-serif;font-size:12.800000190734=
863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>LinkedIn: =A0</span><a href=3D"http://www.linkedin.com/in/jamescarpenter1"=
 target=3D"_blank" style=3D"font-family:arial,sans-serif;font-size:12.80000=
0190734863px">http://www.linkedin.com/in/jamescarpenter1</a><br style=3D"fo=
nt-family:arial,sans-serif;font-size:12.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>&lt;</span><a href=3D"http://www.linkedin.com/in/jamescarpenter1" target=
=3D"_blank" style=3D"font-family:arial,sans-serif;font-size:12.800000190734=
863px">http://www.linkedin.com/in/jamescarpenter1</a><span style=3D"font-fa=
mily:arial,sans-serif;font-size:12.800000190734863px">&gt;</span><br style=
=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">
<br style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"><=
span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">=
Matthew Tardiff</span><br style=3D"font-family:arial,sans-serif;font-size:1=
2.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>mattrix at=A0</span><a href=3D"http://gmail.com/" target=3D"_blank" style=
=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">gmail.com<=
/a><br style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px=
">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>LinkedIn: =A0</span><a href=3D"http://www.linkedin.com/in/matthewtardiff" =
target=3D"_blank" style=3D"font-family:arial,sans-serif;font-size:12.800000=
190734863px">http://www.linkedin.com/in/matthewtardiff</a><br style=3D"font=
-family:arial,sans-serif;font-size:12.800000190734863px">
<span style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px"=
>&lt;</span><a href=3D"http://www.linkedin.com/in/matthewtardiff" target=3D=
"_blank" style=3D"font-family:arial,sans-serif;font-size:12.800000190734863=
px">http://www.linkedin.com/in/matthewtardiff</a><span style=3D"font-family=
:arial,sans-serif;font-size:12.800000190734863px">&gt;</span><br>
<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Thu, Feb 2=
1, 2013 at 5:21 PM, James Carpenter <span dir=3D"ltr">&lt;<a href=3D"mailto=
:[email protected]" target=3D"_blank">[email protected]</a>&gt;</span> wrot=
e:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex"><div><div>1) Which PyPi repository servers are the most ma=
ture and feature rich, and therefore the best choice at the moment?</div>

<div><br></div><div>2) Do any of the PyPi repository servers support managi=
ng multiple local repositories? If yes, which ones? If no, do you have any =
recommendations on how to most easily support the artifact promotion needs =
of continuous deployment within the Python build ecosystem?</div>


</div><div><br></div><div>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D</div><div>Background:</=
div><div>As part of an effort to implement continuous deployment a fellow t=
raveler and I are attempting to workout a good solution for an in-house PyP=
I repository manager. Support of the PyPI XML-RPC API and artifact promotio=
n appear to be critical requirements of any solution.</div>


<div><br><div>We have tried using Artifactory, but unfortunately this only =
gives us support for &quot;simple&quot; PyPI, without the XML-RPC API distu=
tils/pip uses to support searches.</div><div>This is disappointing, because=
 in other respects Artifactory and Nexus provide very complete mature suppo=
rt. Most importantly Artifactory and Nexus support the ability to host mult=
iple local repositories and proxy external ones. By leveraging virtual repo=
sitories within Artifactory or Nexus one can easily inter-weave results fro=
m various local repositories as per configurable=A0precedence=A0rules.</div=
>


<div><br></div><div>The ability to have the build tooling pull artifacts fr=
om multiple internal repositories turns out to be very important from a con=
tinuous deployment standpoint. Effective continuous deployment typically tr=
eats every build artifact as a potential release candidate. As a release ca=
ndidate marches through each gauntlet of tests (unit, integration, load, ma=
nual, etc.) it is promoted to the next internal repository. This is in cont=
rast to the approach of using SNAPSHOTS or the like which fails the rule of=
 giving each code check-in the=A0traceability=A0necessary to be a potential=
 release=A0candidate. Whether the build tool itself (distutils, Ivy, Maven,=
 etc.), the repository manager (Artifactory, Nexus, CheeseShop, etc.) or so=
mething else knows how to overlay artifacts is an implementation detail, bu=
t some tool has to do the job. Similarly the details of how a build artifac=
t is promoted from one local repository to another is also an implementatio=
n detail with a variety of possible solutions.</div>


<div><br></div><div>Recognizing support for the PyPI XMLRPC API tends to be=
 a big deal at scale, we are looking into using CheeseShop, =A0Crate.io, or=
 similar to address our needs. Another choice may be to place a read-only s=
olution in front of Artifactory that adds support for the PyPI XMLRPC API. =
I am hoping members of this reading list can help point us in the right dir=
ection.</div>


<div><br></div><div>Thanks for the time and effort you have spent reading t=
his rather long post. I look forward to your responses.</div></div><div><br=
></div><div>Sincerely,</div><div>James Carpenter</div><div>jcarpenter621 at=
 yahoo dot com</div>


<div><br></div><div>P.S.: As you may have guessed I am a visitor from the l=
and of Java where the Sun has now set and the Seth lord reigns. (I know a M=
onty Python reference would be better, but I am a bit ignorant in that rega=
rd.)</div>


</blockquote></div><br></div></div>

--089e0160d184a8e3dd04dd04d38f--

--===============0805743222==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Catalog-SIG mailing list
[email protected]
http://mail.python.org/mailman/listinfo/catalog-sig

--===============0805743222==--