[wplug] Large Database

[Michael Semcheski]

On Fri, Mar 6, 2009 at 2:53 PM, Bill Moran <wmoran at potentialtech.com> wrote:
> If you're just going to be looking at individual records, or generating
> graphs over specified plots of time, you might be much better off with
> flat files.  I'm not crazy about binary formats, as they're difficult
> to parse with tools like sed/awk/sh, but flat files are going to have
> lots of performance benefits over a RDBMS if you're accessing data
> sequentially, or only looking at individual records and the data is
> of predictable size.

One thing about high sampling rates is they tend to reduce the change
from one sample to the next.  For some applications, you can achieve
really significant compression by recording the delta rather than the
value.

Compression has trade-off's of course, but one benefit _can_ be that
you can read in and write out data faster because you have smaller
datasets.