Re: how big is a big database?

Gildor Oronar wrote:
> As said, starting with 360,000 xml documents (all submitted by XForm) is 
> not a big amount, any NxD should work.

It's not small either :) It's a good sized database.


> Then how much documents can be a 
> problem that makes the project have to think which NxD to choose carefully?

I'd put cut offs first at 1 million documents, next at 10 millions, etc. 
Approximately, of course - there are no hard numbers here since a lot depends on 
document sizes, indeces, etc.


> I seriously lack background information on database scalability. I only 
> know for one time when a database capacity become a problem, that a 
> colleague of mine has worked on a project with Microsofot SQL server and 
> database crash at 3000 stimulated connection (and upgrade hardware 
> doesn't improve anything).

3000 simultaneous connections? It's not possible on windows! ;-)

Seriously, server should limit count of simultaneous connections to a reasonable 
number (which is related to count of cores in the server. Say, if you get 16 way 
box, you can set thread count to anywhere in 64-128 range, but probably not higher).


> I mean I just need to be careful, knowing 
> 360k documents is small thing is good, but what if we end up with 3600k 
> documents? I just wish to be aware what's the next level (of document 
> numbers) that I should be aware.
> 
> If the project I am working on is really successfull, it might reach 1 
> million documents in coming 3 years. I'd like to say this should be the 
> max I can guess.

You can read archives of xindice-users, -dev mailing lists. I remember there 
were reports on Xindice scalability. I think somebody had success running 
database with 10 millions documents. But I don't recall details.

Vadim