Re: Over 1.5GB file limit?
Thadeus Burgess <[email protected]> Tue, 29 Mar 2011 12:21:11 -0500
| Newsgroups | gmane.comp.db.metakit |
|---|---|
| Message-ID | <[email protected]> |
--0016e6d77df6d0256b049fa24b9d Content-Type: text/plain; charset=ISO-8859-1 3rd tries a charm to post this message to the group ! ---------- It is glad to know metakit is being actively used. We are using it to log data that comes in every second from many nodes, all time stamped and need to be able to store potentially millions of rows. We do plan on running metakit on some cloud servers, however this also has to run on some Atom class processor typically found in netbooks (since this is where the data is coming from). Currently we are using PostgreSQL in production, yet unfortunately it is VERY slow and is unable to keep up with the data demand. Yes Postgres is configured, has proper indexing etc etc.... In preliminary tests... We can stuff about 11million records into 2GB. There shouldn't be more than 2million records in a given day. Using a time-series based file partitioning (year -> month -> day -> data.mk) the 2GB limit should not occur. Performing a select on a range in the 11M records takes about .12 seconds (SQL -> 15 minutes) Performing the above select and sorting it by another column .40 seconds (SQL -> 19 minutes) Performing an avg on a column in .96 seconds (SQL -> 17 minutes) Finding min/max on a column 1 second (SQL -> 23 minutes... yeah I don't get that either) It may not be the fastest out there, but its still trillions of times faster than any SQL would ever be. I have read in certain places that HDF5 outperforms metakit. However HDF5 looks more complicated to get installed and running. We will also continue to be using SQL for our metadata, since the metadata is relational a relational database makes sense there. -- Thadeus On Tue, Mar 29, 2011 at 3:10 AM, Steve Landers <[email protected]>wrote: > > On 29/03/2011, at 3:51 PM, [email protected] wrote: > > > Steve Landers <[email protected]> writes: > > > >> On 29/03/2011, at 2:54 PM, [email protected] wrote: > >> > >>> It's still 2GB. The file format itself doesn't allow bigger databases. > >>> > >>> My advise is to stay away from metakit, it's dead. > >> > >> That is a ridiculous assertion, Metakit is stable and maintained. > > > > The last release is from 2007, it's probably just finished? > > Or perhaps just not broken? Pat Thoyts earlier today referred to bug > fixes in the mk::channel support. Did you miss that? > > > A 2 GB file limit just doesn't seem appropriate any more, and there is no > plan to lift this limit. > > There are a number of devices where 2Gb is more than appropriate. There > are a number of devices where slot-based access to memory mapped data > structures is very appropriate and significantly more efficient (and > predictable) than other approaches. > > Of course, it may not be appropriate for your needs. But for you to say > Metakit is dead is unfortunate and quite misleading. > > > That's why I call metakit a dead project. > > Never let the facts get in the way of personal opinion. > > Now, back to the original topic. It may well be that Thadeus would be > better served by considering another database library, depending on his > needs. But neither you nor I know that based on the information he provided. > > Steve > > -- > You received this message because you are subscribed to the "metakit" > group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/metakit?hl=en > -- You received this message because you are subscribed to the "metakit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/metakit?hl=en --0016e6d77df6d0256b049fa24b9d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable 3rd tries a charm to post this message to the group !<br><br>----------<br>= <br><div class=3D"im">It is glad to know metakit is being actively used. <b= r><br></div>We are using=20 it to log data that comes in every second from many nodes, all time=20 stamped and need to be able to store potentially millions of rows. We do plan on running metakit on some cloud servers, however this also=20 has to run on some Atom class processor typically found in netbooks=20 (since this is where the data is coming from).<br> <br>Currently we are using PostgreSQL in production, yet=20 unfortunately it is VERY slow and is unable to keep up with the data=20 demand. Yes Postgres is configured, has proper indexing etc etc....<br><br> In preliminary tests...<br><br>We can stuff about 11million records into 2GB. There shouldn't be more than 2million records in a given day.=20 Using a time-series based file partitioning (year -> month -> day=20 -> <a href=3D"http://data.mk/" target=3D"_blank">data.mk</a>) the 2GB li= mit should not occur.<br> <br>Performing a select on a range in the 11M records takes about .12 secon= ds (SQL -> 15 minutes)<br>Performing the above select and sorting it by = another column .40 seconds (SQL -> 19 minutes)<br>Performing an avg on a= column in .96 seconds (SQL -> 17 minutes)<br> Finding min/max on a column 1 second (SQL -> 23 minutes... yeah I don= 9;t get that either)<br><br>It may not be the fastest out there, but its st= ill trillions of times faster than any SQL would ever be.<br><br>I have rea= d in certain places that HDF5 outperforms metakit. However HDF5 looks more = complicated to get installed and running.<br> <br>We will also continue to be using SQL for our metadata, since the=20 metadata is relational a relational database makes sense there.<br><br><br = clear=3D"all">--<br>Thadeus<br><br><br> <br><br><div class=3D"gmail_quote">On Tue, Mar 29, 2011 at 3:10 AM, Steve L= anders <span dir=3D"ltr"><<a href=3D"mailto:[email protected]">s= [email protected]</a>></span> wrote:<br><blockquote class=3D"gmai= l_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204= , 204, 204); padding-left: 1ex;"> <div class=3D"im"><br> On 29/03/2011, at 3:51 PM, <a href=3D"mailto:[email protected]">schmir@gmail= .com</a> wrote:<br> <br> > Steve Landers <<a href=3D"mailto:[email protected]">steve@d= igitalsmarties.com</a>> writes:<br> ><br> >> On 29/03/2011, at 2:54 PM, <a href=3D"mailto:[email protected]">sch= [email protected]</a> wrote:<br> >><br> >>> It's still 2GB. The file format itself doesn't allow b= igger databases.<br> >>><br> >>> My advise is to stay away from metakit, it's dead.<br> >><br> >> That is a ridiculous assertion, Metakit is stable and maintained.<= br> ><br> > The last release is from 2007, it's probably just finished?<br> <br> </div>Or perhaps just not broken? =A0 Pat Thoyts earlier today referred to = bug fixes in the mk::channel support. Did you miss that?<br> <div class=3D"im"><br> > A 2 GB file limit just doesn't seem appropriate any more, and ther= e is no plan to lift this limit.<br> <br> </div>There are a number of devices where 2Gb is more than appropriate. =A0= There are a number of devices where slot-based access to memory mapped data= structures is very appropriate and significantly more efficient (and predi= ctable) than other approaches.<br> <br> Of course, it may not be appropriate for your needs. =A0But for you to say = Metakit is dead is unfortunate and quite misleading.<br> <div class=3D"im"><br> > That's why I call metakit a dead project.<br> <br> </div>Never let the facts get in the way of personal opinion.<br> <br> Now, back to the original topic. It may well be that Thadeus would be bette= r served by considering another database library, depending on his needs. B= ut neither you nor I know that based on the information he provided.<br> <font color=3D"#888888"><br> Steve<br> </font><div><div></div><div class=3D"h5"><br> --<br> You received this message because you are subscribed to the "metakit&q= uot; group.<br> To post to this group, send email to <a href=3D"mailto:metakit@googlegroups= .com">[email protected]</a><br> To unsubscribe from this group, send email to <a href=3D"mailto:metakit-uns= [email protected]">[email protected]</a><br> For more options, visit this group at <a href=3D"http://groups.google.com/g= roup/metakit?hl=3Den" target=3D"_blank">http://groups.google.com/group/meta= kit?hl=3Den</a><br> </div></div></blockquote></div><br> <p></p> -- <br /> You received this message because you are subscribed to the "metakit&q= uot; group.<br /> To post to this group, send email to [email protected]<br /> To unsubscribe from this group, send email to metakit-unsubscribe@googlegro= ups.com<br /> For more options, visit this group at <a href=3D"http://groups.google.com/g= roup/metakit?hl=3Den">http://groups.google.com/group/metakit?hl=3Den</a> --0016e6d77df6d0256b049fa24b9d--