Big Data Standards and Provenance

The National Institute of Standards and Technology (NIST) has a fresh and important perspective on standards for big data that business technologists should consider seriously. The key quote from a recent NIST workshop is from Howard Wactlar of the National Science Foundation: “[T]he real breakthroughs will come when we are able to use someone else’s data.”

How many businesses are investing in their own proprietary data models intended to facilitate data integration across their own sources... without considering the need to integrate other sources? It's a natural viewpoint to take – but a limiting one. It ignores the fact that the development of technical standards for big data and cloud services are driving the wider shareability of data.

 The scientists at the workshop recognized three challenges to standards for big data in their area of interest. These are provenance, cross-domain standards, and security. I guess cross-domain standards naturally come top of my own list, because creating definitions that enable people to share data across boundaries is what industry standards are all about – from the department level right up to international trading. But the scientists put “provenance” first: for them, knowing where a piece of data came from is on a par with knowing what it represents.

As business organizations come to share, trade and aggregate data on a wider scale, the question of provenance will assume a greater priority for us too. Would you buy a ten-year price data series without some assurance that it comes from a reputable source?

Now, scientists guard reputation through institutions – principally universities, research institutions and learned journals. These bodies act as the guarantors of data quality. Business does not have the same reputation infrastructure. Data providers with trusted brands may well act as sourcers and packagers of raw big data, underwriting its quality before selling it on to users. If so, they too will need their quality standards – ideally, open, transparent ones. Arguably, within specific industries, trade associations might be better placed to fulfil this function on behalf of the communities they serve.

Standards for Big Data

Comments

[Donna]

Are you aware of the National Information Exchange Model and it's role in helping govt to govt cross domain standards? I am interested in understanding more about your view on Standards for Big Data.

Follow me at @NIEMExecDir