Re: Copy_from gets stuck with bigger input file

I encountered this problem some time ago. I don't think I found a  
solution. See:
http://lists.initd.org/pipermail/psycopg/2007-June/005552.html
http://lists.initd.org/pipermail/psycopg/2007-June/005553.html

On Feb 12, 2008, at 15:07 , Christian J Dietrich wrote:

> Dear list,
>
> I am experiencing some problems with copy_from() when feeding it with
> input files that are quite large (spoken in number of lines, not file
> size). If a file exceeds a certain number of lines, copy_from simply
> hangs and does not return. I am using psycopg2 2.0.6. PostgreSQL is
> version 8.2.4.
>
> In detail, I tried to feed the table as defined below:
> create table test_2 (my_num int);
>
> a) - created a file of 118k lines, each line is just "30000\n" (could
>       be any string)
>     - fed it into the DB via psycopg2
>
> b) - added 1k lines, each line is just "30000\n" to the file before
>     - fed it to the DB via psycopg2
>
> Although a) works fine for me, b) just stucks. To me it's not clear
> what happens here, since neither CPU nor memory is wasted during this
> time. SIGINT does not interrupt the copy_from() call, first SIGTERM
> actually stops the computation. Looks like a memory story to me, or  
> some
> constraints of stdin. Apparently the number of lines is not the only
> criteria, since copying tables with multiple columns leads to the same
> problem with less rows.
>
> Would be great if someone could help me out. Just for completeness,  
> both
> scenarios worked well using the cmd line client psql combined with  
> COPY.
> Here are the file sizes:
> 692K  /tmp/input118k
> 698K  /tmp/input119k
>
> Thanks in advance!
>
> Cheers,
> Christian
>
> _______________________________________________
> Psycopg mailing list
> Psycopg-IAPFreCvJWPBWskQ1e/[email protected]
> http://lists.initd.org/mailman/listinfo/psycopg
>