Copy_from gets stuck with bigger input file

Christian J Dietrich <christian.dietrich-jNDFPZUTrfRobmy5twv+e3F3/sHOsa9/[email protected]>
Newsgroups gmane.comp.python.db.psycopg.devel
Message-ID <[email protected]>
Dear list,

I am experiencing some problems with copy_from() when feeding it with
input files that are quite large (spoken in number of lines, not file 
size). If a file exceeds a certain number of lines, copy_from simply 
hangs and does not return. I am using psycopg2 2.0.6. PostgreSQL is 
version 8.2.4.

In detail, I tried to feed the table as defined below:
create table test_2 (my_num int);

a) - created a file of 118k lines, each line is just "30000\n" (could
      be any string)
    - fed it into the DB via psycopg2

b) - added 1k lines, each line is just "30000\n" to the file before
    - fed it to the DB via psycopg2

Although a) works fine for me, b) just stucks. To me it's not clear
what happens here, since neither CPU nor memory is wasted during this
time. SIGINT does not interrupt the copy_from() call, first SIGTERM
actually stops the computation. Looks like a memory story to me, or some
constraints of stdin. Apparently the number of lines is not the only
criteria, since copying tables with multiple columns leads to the same
problem with less rows.

Would be great if someone could help me out. Just for completeness, both
scenarios worked well using the cmd line client psql combined with COPY. 
Here are the file sizes:
692K  /tmp/input118k
698K  /tmp/input119k

Thanks in advance!

Cheers,
Christian
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.