Re: COPY from STDIN vs file with large CSVs

--000000000000ee7242059ba6ad9f
Content-Type: text/plain; charset="UTF-8"

On Wed, Jan 8, 2020 at 8:55 AM Wells Oliver <[email protected]> wrote:

> I have a CSV that's ~30GB. Some 400m rows. Would there be a meaningful
> performance difference to run COPY from STDIN using: cat f.csv | psql "COPY
> .. FROM STDIN WITH CSV" versus just doing "COPY ... FROM 'f.csv' WITH CSV"?
>

If you're looking to speed up the loading - and your disk subsystem is
decent - consider running your csv through the "split" command to decompose
it into smaller CSV files. You can then load them in parallel using
multiple psql sessions.

--000000000000ee7242059ba6ad9f
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"col=
or:#000000"><br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr"=
 class=3D"gmail_attr">On Wed, Jan 8, 2020 at 8:55 AM Wells Oliver &lt;<a hr=
ef=3D"mailto:[email protected]">[email protected]</a>&gt; wrote:<=
br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e=
x;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"=
><div>I have a CSV that&#39;s ~30GB. Some 400m rows. Would there be a meani=
ngful performance difference to run COPY from STDIN using: cat f.csv | psql=
 &quot;COPY .. FROM STDIN WITH CSV&quot; versus just doing &quot;COPY ... F=
ROM &#39;f.csv&#39; WITH CSV&quot;?</div></div></blockquote><div><br></div>=
<div style=3D"color:rgb(0,0,0)" class=3D"gmail_default">If you&#39;re looki=
ng to speed up the loading - and your disk subsystem is decent - consider r=
unning your csv through the &quot;split&quot; command to decompose it into =
smaller CSV files. You can then load them in parallel using multiple psql s=
essions.<br></div></div></div>

--000000000000ee7242059ba6ad9f--