Sunday, August 4, 2013

Framework for metadata management for big data analytics

Big data metadata framework layers
Image above depicts a good solid framework for big data metadata. Note that if metadata for various data streams cannot be reconciled neither can the data.


Metadata Discovery

Discovering metadata is critical in Big data for the reasons of context and intent noted in the prior section. Social data is typically sourced from multiple sources. Allthese sources will have different formats. Once metadata for a certain entity is discovered for one source it needs to be harmonized across all sources of interest.

Metadata Collection

A metadata collection mechanism should be implemented. A robust collection mechanism should aim to minimize or eliminate metadata silos.

Metadata Governance

Metadata creation and maintenance needs to be governed. Governance should include resources from both the business and IT teams. A collaborative framework between business and IT should be established to provide this governance.

Metadata Storage

Multiple models for enterprise metadata storage exist.The Common Warehouse Meta-model (CWM) is one example.

Metadata Distribution

This is the final component. Metadata, once stored will need to be distributed to consuming applications. A formal distribution model should be put into place to enable this distribution.