Re: COPY from STDIN vs file with large CSVs

--000000000000893401059ba48811
Content-Type: text/plain; charset="UTF-8"

Thanks, I had looked into pg_bulkload a bit but it does not seem to be
available for PG 12? It's not in the extension directory, it's not
available through apt-cache search, and I have all kinds of issues with it
finding pgcommon and pgport when I try to build from source. Using Ubuntu
18 LTS...

On Wed, Jan 8, 2020 at 9:09 AM Ron <[email protected]> wrote:

> On 1/8/20 10:54 AM, Wells Oliver wrote:
> > I have a CSV that's ~30GB. Some 400m rows. Would there be a meaningful
> > performance difference to run COPY from STDIN using: cat f.csv | psql
> > "COPY .. FROM STDIN WITH CSV" versus just doing "COPY ... FROM 'f.csv'
> > WITH CSV"?
> >
> > Thanks. It took about four hours to copy one and I felt that was a
> little
> > much.
>
> catting the file starts another process, and opens a pipe.  That can't be
> faster than "COPY ... FROM ... WITH CSV".
>
> pg_bulkload (which might be in your repository) is probably what you
> really
> want.
>
> --
> Angular momentum makes the world go 'round.
>
>
>

-- 
Wells Oliver
[email protected] <[email protected]>

--000000000000893401059ba48811
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks, I had looked into pg_bulkload a bit but it does no=
t seem to be available for PG 12? It&#39;s not in the extension directory, =
it&#39;s not available through apt-cache search, and I have all kinds of is=
sues with it finding pgcommon and pgport when I try to build from source. U=
sing Ubuntu 18 LTS...<br></div><br><div class=3D"gmail_quote"><div dir=3D"l=
tr" class=3D"gmail_attr">On Wed, Jan 8, 2020 at 9:09 AM Ron &lt;<a href=3D"=
mailto:[email protected]">[email protected]</a>&gt; wrote:<br><=
/div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bo=
rder-left:1px solid rgb(204,204,204);padding-left:1ex">On 1/8/20 10:54 AM, =
Wells Oliver wrote:<br>
&gt; I have a CSV that&#39;s ~30GB. Some 400m rows. Would there be a meanin=
gful <br>
&gt; performance difference to run COPY from STDIN using: cat f.csv | psql =
<br>
&gt; &quot;COPY .. FROM STDIN WITH CSV&quot; versus just doing &quot;COPY .=
.. FROM &#39;f.csv&#39; <br>
&gt; WITH CSV&quot;?<br>
&gt;<br>
&gt; Thanks. It took about four hours to copy one and I felt that was a lit=
tle <br>
&gt; much.<br>
<br>
catting the file starts another process, and opens a pipe.=C2=A0 That can&#=
39;t be <br>
faster than &quot;COPY ... FROM ... WITH CSV&quot;.<br>
<br>
pg_bulkload (which might be in your repository) is probably what you really=
 <br>
want.<br>
<br>
-- <br>
Angular momentum makes the world go &#39;round.<br>
<br>
<br>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g=
mail_signature"><div dir=3D"ltr"><div>Wells Oliver<br><a href=3D"mailto:wel=
[email protected]" target=3D"_blank">[email protected]</a></div></div=
></div>

--000000000000893401059ba48811--