Re: mnogosearch URL file import

Hi Glenn,

How do your Server/Real/URL commands look like?
What is the exact command you use to import?


On 08/05/2011 11:57 PM, Glenn McGurrin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I have a file with a large number of URLS I am trying to import into
> mnogosearch to seed my indexing (not servers, urls)  I have defined my
> servers in my configuration file and have started the command to import
> all of the URLs in the file but it has been hours and it still has not
> made one change to mysql and I'm worried when it finally gets to that
> point it will try to insert them all at once which will fail.  The URL
> list file is 275MB and has ~8 million URLs if memory serves.  It has
> been taking 99-100% of a core for about 3.25-4 hours of cpu time (so for
> about 2.25-4.25 hours).  Does anyone have any good suggestions for
> importing these URLs faster?  Is it safe to run an indexing process at
> the same time as this?  I don't think a server table is practical for
> this for a number of reasons while I won't fully go into but it mainly
> comes down to performance.  How long can I expect this to take?  Why is
> it taking so long, my program to generate the file written quickly in
> PHP took only seconds to generate the whole thing, why is it taking so
> long for this program in C to import them?
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.12 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQQVAwUBTjxLFj5noacwLKxwAQKOmSAAomFsQY5OM32C6kfZbjNYBSLhWzUgUn0q
> 3Y/ByRcGVuvPAHJbKLEup7qllNzhiDt8QYZEl7XRKBlClgddOMKhm7JBcXAyyGeI
> 2NSxuAO6Se9OCyWsgZ3ppgd9kwshnrEwAYITcEAmr+7vRAekxmVOrgkS8bjZOtpe
> ZCOK5gc1mOBY2ejx3sm5gqSe+dI5JBkPld1uFhjxvsLlY+lVqrgyNu8FJ7Z0UHYC
> GzFM+eljiZNhNVCNH1A4qMRMMXOo3oNB7t2SKYNwOgZ+G5vtWPXAzu7LVe4qa2iw
> 1M8cPr4Zvg511MIuu2LJJZgS/6M7+4LlK+ruWuEvRqFfXIoAomwhABZs3NqSMJ0x
> OStJaTCUnIh5zj6ZO56rWXkoSgICA/o/Bntnvt4s8WgPg5jFManJwY3SUCDKlT1L
> Mhu2UgMyxziUNP+G8WH0z/1eW+a61cGYOlmfHLKRQXbfB/VZnd7HYZta+kUOHQE6
> DNWaCpCH0URk9ZpiPUKI7hHb8dqow47UUbkxbS5zmQNFxZRbgJ1fHGEHuvlYEUdK
> n+4lovm9RjiKdCqF6kbTR1Ogr7JHgIairxkHL6pgKBUGF2YagkrlANy5hNbnabZL
> g3m/WcdweJn/LmTb4Ap7V5bZGIVXF7IbP8hMC91LDcFIkqoUWBRwslFbA9PE3nNA
> 5EUEzo1gRmFbu6EtYG4p8OV2UF0L6uyxY5ISSwRGgkUP3pBIEa8NGdvGCLgWyHyv
> BS1posrcGWkBMGfNGC3Z+cKzA3nTG1k23lHd3IZnrshrDFykKGcUz1jfQKRARVF0
> sCCZU5bEfVml41MeaAacO2xybUvwVDT8Sny77ANrMTL3zOeRAm/bhedajaEkx/dZ
> ye6MXzejfnQwlrg/FlbUt+RuLpq8dKnSXeQGAu/pk2e6UBuM8XNosSZrliF38e9D
> qyYw8ffsmJs14pEyccSLVnVfF4sPEr9X0pHUEA84vt5RNYWBJfMwLvcwAUAymfGC
> +pwr8Oo4wXdrbvad+ztnAorsOh9SkhTCPGFLtOnH1folM91C910DBpX3tCMIng/6
> MxGpBWOOrqznHwCL0fagIMC3ctWOsY3m04lL7BwC0Tm6MnHh7TwupaQJ8N9mmsaI
> 1bXuHsruL2hgvRsq2vI5JDHPLp96mbSIK8zBP+hkhsoUDuq0JPswPEITvnJ6AXCd
> 1+RwCBnt6LXrPPWDT47Jfm7/xTqXGv1vvt+Ats2/NDPteicMdNCcDx4yA9vv1XUI
> Xfyirpt9xdEcD5Hhh0hDXibRPr0p+DwiuxhI9REcG7WEBiM7o7vLjzN20n7T/MwS
> 6y0GDy1U2c2T/zyPVVAgVwytWVq6X95Sk0KOhBKO+M8M4quUu1UAsw==
> =hygA
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]