You are here

Home

Data Aggregation Options

In recent years, the buzz around the internet of things and smart meters has prompted several sites to pop up dedicated to online data aggregation. They offer a centralized repository for your data, visualization, and some level of long term data storage. You can view your data from anywhere in the world, and share it easily.

A nice site with a quick summary of the multitude of data sites is here.

Each site has a learning curve before you can master posting data. In some cases it can be rather steep (especially with the newer sites). To make it easier to sort through them, choose the best option for you, and get started, I thought I would make up this guide. I based the guide on the above link, and my own experiences. Here you will only find sites I have at least some direct experience with.

The following is a list of online data aggregation sites, their current status, and my first impressions.

(The guide is in beta. I am playing with tab formatting to get the most out of it. Dont expect much just yet) 

 

Aggregators

 

Patchube

Status: Public release

Server options: Their servers only

Fee structure: Free

Maturity: High

Documentation: Excellent

Visualizations: Line chart only

Plug ins available: None(?)

Link: Patchube.com

Impressions:

Patchube is the big boy on the block. It originally used a pay rate structure in which the basic account was free. With it, Patchube would store your data for a maximum of one year, allow 12 posts per minute, and 6(?) sensor streams. Value added services included longer storage times, more steams and  higher post rates.

Eventually the rate structure was simplified, finally to be eradicated all together in favor of a free service. One can only assume that they intend to earn money by reselling your sensor data to corporations and the scientific communities in bulk. Anyone can search and RSS data from your feeds, but this may prove cumbersome, which would explain why anyone would be willing to pay for data that is freely available.

Opting for a totally free service also alienated some long term customers, whom have been paying for total service, only to have it given away for free to new customers. As far as I know, a pro-rated refund to paying customers has not yet been offered.

Gripes about money aside, Patchube is a very strongly tested data engine, with excellent community support and a high level of documentation. API examples exist for Arduino (via ethernet shield), Processing, Arduino through Processing Java, C, Visual Studio.. and on and on.  Once past the connection hurdles, the user interface is mostly intuitive, including rendered maps of sensor locations (assuming you post GPS data along with sensor data), easy sensor configuration and applications for both Android and iPhone.

My only gripe on the usage side would be the lack of graph options. Certainly the strip chart works well. Sadly, it is the only visualization option. I would hope that in the very near future, Patchube will be able to offer much more in the visualization and plug-ins department. Also, the ability to download my own data as an XLS would be highly appreciated.

Nimbits

 Status: Public release

Server options: Uses Google App engine. Private installation available

Fee structure: Free

Maturity: Low

Documentation: Minimal

Visualizations: ?

Plug ins available: ?

Link: http://www.nimbits.com/

 

Impressions:

Nimbits is a newer service, which replaces GooglePowerMeter in a big way. GPM was limited to smart utility meter owners only, and only in certain states and counties. Nimbits is a whole new system, unrelated to GPM but utilizing the Google App Engine to handle data.

I commend Nimbits for attempting to offer both a free alternative to the other sites (at the time the Nimbits project started, Patchube was still a pay service), as well as offering a private server option. The hope that us net novices could eventually just go to our Fantastico web server control panel and install our very own instance of a high end data aggregator is an idea that keeps me up at night!

Nimbits is a system still in it's "potential" stage. It has a lot of promise, but as of yet, is a bit unrealized. Documentation is very poor at the moment, relying mostly on a google group discussion to get things up and running.

To be honest, I have yet to figure out how to get my Arduino to post through a Processing sketch. Fundamentally, the theory is simple: its a POST. I have poked around the demo code for an Arduino with Ethernet shield, but have not connected one. 

I hope the nimbits team will charge ahead and get documentation and examples out more quickly.

Open.Sen.se

Status: Invitation beta

Fee structure: Free (for now?)

Maturity: Intermediate

Documentation: intermediate

Visualizations: Line chart only

Plug ins available: Many

Link: http://open.sen.se/

Impressions:

Open.Sen.se aims to be be the everyman's data aggregator. The user interface is built on dragging and dropping, filling in simple forms, and flipping switches. Much of the behind the scenes stuff is ment to be as easy as possible. At first it would appear that this takes away much of the power from the user. However, the considerable list of plug ins for items such as sensor device platform, data types, and visualization preferences means that out of the box, Open.Sen.Se intends to provide much more usability to the customer without much fuss. There are even modules to script control actions among devices (a remote switch in Australia could turn on the porch light in London, depending on the value of a geiger counter in Japan for example).

While OpenSen.Se can offer quite a lot to the user, it is still in invitation only beta. You will need to request an invite, then wait for approval (which could take a long time). Thus, it is not for casual users. There are still a lot of bugs to work out, panels to fix, and fields that go no where.

 

 

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer