Re: mnogosearch URL file import

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Actually I gave up on this and moved on, this just was to seed the URL
database but it will find all of the same URLs on its own.  I don't
have the Server commands I was using at the time but the import
command I still remember, I tried two if memory serves, passing each
one to a new instance as indexer -i -u URL and indexer -i -f FILENAME.

On 8/31/11 9:28 AM, Alexander Barkov wrote:
> Hi Glenn,
> 
> How do your Server/Real/URL commands look like? What is the exact
> command you use to import?
> 
> 
> On 08/05/2011 11:57 PM, Glenn McGurrin wrote: I have a file with a
> large number of URLS I am trying to import into mnogosearch to seed
> my indexing (not servers, urls)  I have defined my servers in my
> configuration file and have started the command to import all of
> the URLs in the file but it has been hours and it still has not 
> made one change to mysql and I'm worried when it finally gets to
> that point it will try to insert them all at once which will fail.
> The URL list file is 275MB and has ~8 million URLs if memory
> serves.  It has been taking 99-100% of a core for about 3.25-4
> hours of cpu time (so for about 2.25-4.25 hours).  Does anyone have
> any good suggestions for importing these URLs faster?  Is it safe
> to run an indexing process at the same time as this?  I don't think
> a server table is practical for this for a number of reasons while
> I won't fully go into but it mainly comes down to performance.  How
> long can I expect this to take?  Why is it taking so long, my
> program to generate the file written quickly in PHP took only
> seconds to generate the whole thing, why is it taking so long for
> this program in C to import them?
>> 
>> ---------------------------------------------------------------------
>>
>> 
To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQQVAwUBTl47fj5noacwLKxwAQIfgh//Y+hOkW5DELm8Y8qn6/yTq0Es62s5FVqQ
tZquYH+U7gngvmtBOIfcSCVtBoxumqeIgEEEsWF4WkSLh5eRdToulNKQIDp1qM1/
7RxWIQmk77Wmi/IhO4XSnw8GwWAGqYsx4RJC4HVr5l/NVETvAJjP3MDI3eZ3bo7C
hBb/os1BvnNNnD/NQxlPTG2ZCqh+jb7VFEx0kJxk9mWa2GUqPew60CgQge+GjG8m
/Kbgfv/p92ohOe2oaItb9BQRwhxEkTII+/iC2NdOlKIU2AO/wuCt+JItrwnVJv5O
NnRpQhMGD+UwM5FXzobMsj+s/nKVlxeJffxC0WCvb9gD2NwBqpVRz9qxcIxvnnnd
Cc/+EFXJBF0RCaey5eTJtoo3ax0td6tAn6MUE3YrESyv3NKisH51zXyZCgwWvE1k
/3yoiNyXKyYXBQvXyy9DAE5+mH+308tK7au96gNm1WqnweJfJwYlJnRvxp0cHND1
C5jyE5H8eMcil11zhQBUg+sqokPEq9URiDa8rdN9muq7RII1J03vHHaNqTL46GJK
O+XuKluT1+JfiHBJnBaBoudKJ4/cvPkiRLaw5zteRF510wx6R7wx9HIwLy+SCGbw
HRRWp0XMGhR0yJdiPSMPNZJsw3Pm3xEx0djHXluDP116+2zCKty44P6vh+BXW2Jo
WDtQ5fT0HuP4Q/HVEJHbE7LeL4LD1Wr4JQFHavm/8ZVmxgqaOR9nitQ1vuAc5I/s
t0vm7u8Crvf1+8A0RHLG5lqaMsBk+xBQMWpL2LawflWNPrWzR2Z0J8I8ehXhHdyU
nP8jCCmvDN8og18FYrPgeJWZ1BI3aWoe6973D3gULMKP2QRmNSiliiBm2rqgH9fw
GJPrg7UyUjLIpr85e2yLJRAYjCg0pK+0o98wSHz7kG8mgVuLw3I7csgsrtJaFHs9
j50FtI2mcyk1DU/adZ+yZBeR1xVw0toBJz5q/wE/+otnkPe9sZ/IVDSOaEonQ9bA
+FG7wIOrQQJMEDzKZ9bkjLSj2948Q3Z9N7BajxucrFVDsHqAwSXJJyaZ5hQwHrbM
Ka+7YG+80tnSI7lczAzHEiuNWk/FNquaCJN9KP/TRYJZb8Kg6r9DNC8U2Wi2WCAE
LbrfQA2UgPOSe2uh7JyosNM1f7fF9wEFnprD3jganIfFde8Jx3S2UAy3jj8vc77F
UhPSkenTQBSwQQuwHZQnzbSFaZUgeHT1RPlEWZ591O1U47lkd7+YOGY7+H3SQr7c
9NZYZ8p10fuwzv66zbS/iD7w3M15ceX+jCIwxu5TecB4YpkNBVd07pgqZyEL8+by
C93qeeE+J+iIr968DXglzu/TxPjnJkoURoP2sHxMMYNPARPIDokQog==
=cPTW
-----END PGP SIGNATURE-----