mnogosearch URL file import

Glenn McGurrin <[email protected]>
Newsgroups gmane.comp.web.mnogosearch.general
Message-ID <[email protected]>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have a file with a large number of URLS I am trying to import into
mnogosearch to seed my indexing (not servers, urls)  I have defined my
servers in my configuration file and have started the command to import
all of the URLs in the file but it has been hours and it still has not
made one change to mysql and I'm worried when it finally gets to that
point it will try to insert them all at once which will fail.  The URL
list file is 275MB and has ~8 million URLs if memory serves.  It has
been taking 99-100% of a core for about 3.25-4 hours of cpu time (so for
about 2.25-4.25 hours).  Does anyone have any good suggestions for
importing these URLs faster?  Is it safe to run an indexing process at
the same time as this?  I don't think a server table is practical for
this for a number of reasons while I won't fully go into but it mainly
comes down to performance.  How long can I expect this to take?  Why is
it taking so long, my program to generate the file written quickly in
PHP took only seconds to generate the whole thing, why is it taking so
long for this program in C to import them?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQQVAwUBTjxLFj5noacwLKxwAQKOmSAAomFsQY5OM32C6kfZbjNYBSLhWzUgUn0q
3Y/ByRcGVuvPAHJbKLEup7qllNzhiDt8QYZEl7XRKBlClgddOMKhm7JBcXAyyGeI
2NSxuAO6Se9OCyWsgZ3ppgd9kwshnrEwAYITcEAmr+7vRAekxmVOrgkS8bjZOtpe
ZCOK5gc1mOBY2ejx3sm5gqSe+dI5JBkPld1uFhjxvsLlY+lVqrgyNu8FJ7Z0UHYC
GzFM+eljiZNhNVCNH1A4qMRMMXOo3oNB7t2SKYNwOgZ+G5vtWPXAzu7LVe4qa2iw
1M8cPr4Zvg511MIuu2LJJZgS/6M7+4LlK+ruWuEvRqFfXIoAomwhABZs3NqSMJ0x
OStJaTCUnIh5zj6ZO56rWXkoSgICA/o/Bntnvt4s8WgPg5jFManJwY3SUCDKlT1L
Mhu2UgMyxziUNP+G8WH0z/1eW+a61cGYOlmfHLKRQXbfB/VZnd7HYZta+kUOHQE6
DNWaCpCH0URk9ZpiPUKI7hHb8dqow47UUbkxbS5zmQNFxZRbgJ1fHGEHuvlYEUdK
n+4lovm9RjiKdCqF6kbTR1Ogr7JHgIairxkHL6pgKBUGF2YagkrlANy5hNbnabZL
g3m/WcdweJn/LmTb4Ap7V5bZGIVXF7IbP8hMC91LDcFIkqoUWBRwslFbA9PE3nNA
5EUEzo1gRmFbu6EtYG4p8OV2UF0L6uyxY5ISSwRGgkUP3pBIEa8NGdvGCLgWyHyv
BS1posrcGWkBMGfNGC3Z+cKzA3nTG1k23lHd3IZnrshrDFykKGcUz1jfQKRARVF0
sCCZU5bEfVml41MeaAacO2xybUvwVDT8Sny77ANrMTL3zOeRAm/bhedajaEkx/dZ
ye6MXzejfnQwlrg/FlbUt+RuLpq8dKnSXeQGAu/pk2e6UBuM8XNosSZrliF38e9D
qyYw8ffsmJs14pEyccSLVnVfF4sPEr9X0pHUEA84vt5RNYWBJfMwLvcwAUAymfGC
+pwr8Oo4wXdrbvad+ztnAorsOh9SkhTCPGFLtOnH1folM91C910DBpX3tCMIng/6
MxGpBWOOrqznHwCL0fagIMC3ctWOsY3m04lL7BwC0Tm6MnHh7TwupaQJ8N9mmsaI
1bXuHsruL2hgvRsq2vI5JDHPLp96mbSIK8zBP+hkhsoUDuq0JPswPEITvnJ6AXCd
1+RwCBnt6LXrPPWDT47Jfm7/xTqXGv1vvt+Ats2/NDPteicMdNCcDx4yA9vv1XUI
Xfyirpt9xdEcD5Hhh0hDXibRPr0p+DwiuxhI9REcG7WEBiM7o7vLjzN20n7T/MwS
6y0GDy1U2c2T/zyPVVAgVwytWVq6X95Sk0KOhBKO+M8M4quUu1UAsw==
=hygA
-----END PGP SIGNATURE-----
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.