mnogosearch URL file import
Glenn McGurrin <[email protected]>
| Newsgroups | gmane.comp.web.mnogosearch.general |
|---|---|
| Message-ID | <[email protected]> |
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I have a file with a large number of URLS I am trying to import into mnogosearch to seed my indexing (not servers, urls) I have defined my servers in my configuration file and have started the command to import all of the URLs in the file but it has been hours and it still has not made one change to mysql and I'm worried when it finally gets to that point it will try to insert them all at once which will fail. The URL list file is 275MB and has ~8 million URLs if memory serves. It has been taking 99-100% of a core for about 3.25-4 hours of cpu time (so for about 2.25-4.25 hours). Does anyone have any good suggestions for importing these URLs faster? Is it safe to run an indexing process at the same time as this? I don't think a server table is practical for this for a number of reasons while I won't fully go into but it mainly comes down to performance. How long can I expect this to take? Why is it taking so long, my program to generate the file written quickly in PHP took only seconds to generate the whole thing, why is it taking so long for this program in C to import them? -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.12 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQQVAwUBTjxLFj5noacwLKxwAQKOmSAAomFsQY5OM32C6kfZbjNYBSLhWzUgUn0q 3Y/ByRcGVuvPAHJbKLEup7qllNzhiDt8QYZEl7XRKBlClgddOMKhm7JBcXAyyGeI 2NSxuAO6Se9OCyWsgZ3ppgd9kwshnrEwAYITcEAmr+7vRAekxmVOrgkS8bjZOtpe ZCOK5gc1mOBY2ejx3sm5gqSe+dI5JBkPld1uFhjxvsLlY+lVqrgyNu8FJ7Z0UHYC GzFM+eljiZNhNVCNH1A4qMRMMXOo3oNB7t2SKYNwOgZ+G5vtWPXAzu7LVe4qa2iw 1M8cPr4Zvg511MIuu2LJJZgS/6M7+4LlK+ruWuEvRqFfXIoAomwhABZs3NqSMJ0x OStJaTCUnIh5zj6ZO56rWXkoSgICA/o/Bntnvt4s8WgPg5jFManJwY3SUCDKlT1L Mhu2UgMyxziUNP+G8WH0z/1eW+a61cGYOlmfHLKRQXbfB/VZnd7HYZta+kUOHQE6 DNWaCpCH0URk9ZpiPUKI7hHb8dqow47UUbkxbS5zmQNFxZRbgJ1fHGEHuvlYEUdK n+4lovm9RjiKdCqF6kbTR1Ogr7JHgIairxkHL6pgKBUGF2YagkrlANy5hNbnabZL g3m/WcdweJn/LmTb4Ap7V5bZGIVXF7IbP8hMC91LDcFIkqoUWBRwslFbA9PE3nNA 5EUEzo1gRmFbu6EtYG4p8OV2UF0L6uyxY5ISSwRGgkUP3pBIEa8NGdvGCLgWyHyv BS1posrcGWkBMGfNGC3Z+cKzA3nTG1k23lHd3IZnrshrDFykKGcUz1jfQKRARVF0 sCCZU5bEfVml41MeaAacO2xybUvwVDT8Sny77ANrMTL3zOeRAm/bhedajaEkx/dZ ye6MXzejfnQwlrg/FlbUt+RuLpq8dKnSXeQGAu/pk2e6UBuM8XNosSZrliF38e9D qyYw8ffsmJs14pEyccSLVnVfF4sPEr9X0pHUEA84vt5RNYWBJfMwLvcwAUAymfGC +pwr8Oo4wXdrbvad+ztnAorsOh9SkhTCPGFLtOnH1folM91C910DBpX3tCMIng/6 MxGpBWOOrqznHwCL0fagIMC3ctWOsY3m04lL7BwC0Tm6MnHh7TwupaQJ8N9mmsaI 1bXuHsruL2hgvRsq2vI5JDHPLp96mbSIK8zBP+hkhsoUDuq0JPswPEITvnJ6AXCd 1+RwCBnt6LXrPPWDT47Jfm7/xTqXGv1vvt+Ats2/NDPteicMdNCcDx4yA9vv1XUI Xfyirpt9xdEcD5Hhh0hDXibRPr0p+DwiuxhI9REcG7WEBiM7o7vLjzN20n7T/MwS 6y0GDy1U2c2T/zyPVVAgVwytWVq6X95Sk0KOhBKO+M8M4quUu1UAsw== =hygA -----END PGP SIGNATURE-----