<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=utf-8">
<META content="MSHTML 6.00.6000.16788" name=GENERATOR></HEAD>
<BODY style="MARGIN: 4px 4px 1px; FONT: 10pt Tahoma">How big and complex are the records, and how fast are you reading them? Can you load them into some other data structure say, a sparse array, during data aq and then process the recorded data offline?<BR><BR>>>> "Bill Moran" <wmoran@potentialtech.com> 3/6/2009 9:37 AM >>><BR>In response to DK <wplug@curlynoodle.com>:<BR>> <BR>> I need to implement a data acquisition system which will sample and<BR>> store large amounts of "time-series" data, that is hundred of millions<BR>> of records. I would like to investigate using an open source<BR>> database. Does anyone have suggestions?<BR><BR>PostgreSQL is frequently used for datasets that size.<BR><BR>However, whatever you choose, you're going to have to take some time to<BR>tune your methodology if you're dealing with that much data. You're<BR>crossing a line between what ordinary people expect computers to do and<BR>what _REAL_ number crunching is like.<BR><BR>You will _NEVER_ get the kind of performance out of a database that size<BR>that the average user gets out of his MS Access database that he uses to<BR>catalog his library of mp3s. You will have queries that take hours to<BR>run. This will be the case regardless of what DB system you use ... pay<BR>$infinity for Oracle and the performance won't be any better. It's just<BR>the realities of scale.<BR><BR>You will probably also find that collecting said data will require various<BR>forms of "trickery". If you just try to insert it one row at a time, you'll<BR>spend weeks getting it into the database. If you're trying to capture it<BR>"real time", you'll probably find that it's impossible to do. There are<BR>LOTS of tricks to getting this to work, but the standard methods of<BR>data processing usually don't scale up to recordsets that size. Again,<BR>it doesn't matter what DB you use, you'll come across these same problems.<BR>Keep in mind, that depending on the size of each row, you're talking about<BR>10s of G of data ... possibly 100s of G.<BR><BR>With that warning in mind, I further recommend PostgreSQL. I've talked to<BR>many people dealing with datasets that size, and PG handles them as well<BR>as anything you'll pay money for. Personally, I have a few databases with<BR>10s of millions of rows and have had no problems with PG's performance, but<BR>I've yet to push a database into the 100s of millions. I'll be interested<BR>to see how it works for your project.<BR><BR>-- <BR>Bill Moran<BR><A href="http://www.potentialtech.com">http://www.potentialtech.com</A><BR><A href="http://people.collaborativefusion.com/~wmoran/">http://people.collaborativefusion.com/~wmoran/</A><BR>_______________________________________________<BR>wplug mailing list<BR>wplug@wplug.org<BR><A href="http://www.wplug.org/mailman/listinfo/wplug">http://www.wplug.org/mailman/listinfo/wplug</A><BR><BR></BODY></HTML>