Framework for metadata management for big data analytics
Image above depicts a good solid framework for big data metadata. Note that if metadata for various data streams cannot be reconciled neither can the data.
Discovering metadata is critical in Big data for the reasons of context and intent noted in the prior section. Social data is typically sourced from multiple sources. Allthese sources will have different formats. Once metadata for a certain entity is discovered for one source it needs to be harmonized across all sources of interest.
A metadata collection mechanism should be implemented. A robust collection mechanism should aim to minimize or eliminate metadata silos.
Metadata creation and maintenance needs to be governed. Governance should include resources from both the business and IT teams. A collaborative framework between business and IT should be established to provide this governance.
Multiple models for enterprise metadata storage exist.The Common Warehouse Meta-model (CWM) is one example.
This is the final component. Metadata, once stored will need to be distributed to consuming applications. A formal distribution model should be put into place to enable this distribution.