[wplug] Large Database

[terry mcintyre]

A lot of the system design is going to hinge upon "what does the user want to do with this data?"

Are you ever going to need to operate on all 3 billion samples for a given year, or is it more likely that you want to graph the rolling average? Are you interested in detecting outliers or exceptional conditions? Do you want to be able to drill down to "day 37, hour 12, minute six, process X,Y,Z?"

For a dataset this big, I don't know if I'd want to use awk/sed/vi -- it takes a long time to process billions of text records.

Terry McIntyre <terrymcintyre at yahoo.com>


-- Libertarians Do It With Consent!