Using PostgreSQL to store URLs for a web crawler

Simon Connah <[email protected]>
Newsgroups gmane.comp.db.postgresql.novice
Message-ID <[email protected]>
First of all apologies if this is a stupid question but I've never used 
a database for something like this and I'm not sure if what I want is 
possible with PostgreSQL.

I'm writing a simple web crawler in Python and the general task it needs 
to do is first get a list of domain names and URL paths from the 
database and download the HTML associated with that URL and save it in 
an object store.

Then another process goes through the downloaded HTML and extracts all 
the links on the page and saves it to the database (if the URL does not 
already exist in the database) so the next time the web crawler runs it 
picks up even more URLs to crawl. This process is exponential so the 
number of URLs that are saved in the database will grow very quickly.

I'm just a bit concerned that saving so much data to PostgreSQL quickly 
would cause performance issues.

Is this something PostgreSQL could handle well? I'm not going to be 
running PostgreSQL server myself. I'll be using Amazon RDS to host it.
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.