If you are not able to come up with the resources, software, money to ingest big data, then do the obvious--compress it to submission. The previous statement maybe over the top but it pretty much sums up the fascinating paper authored by J. Gerard Wolff titled "The management and analysis of big data, and the SP theory of intelligence". It is an academic research paper and its premise is crazy at first blush but amazing in scope and promise when you dig a little deeper. I will use the following text from the paper itself to help the reader understand the essence of the proposal:
"For example, when we view a scene with two eyes, the image on the
retina of the left eye is almost exactly the same as the image on the retina
of right eye, but our brains merge the two images into a single percept, and
thus compress the information..."
Some of the problems in his proposed machine as noted by the author are related to image and sound processing on one end to math and understanding of numbers on the other. Till some of these issues are resolved, I do not expect any major, large scale applications from this body of research to come out anytime soon. But once they are the applications could truly be amazing.
The image below from the paper captures the range of applications that will be possible in the future.
Encourage all to read at least the first two pages of the paper.
"For example, when we view a scene with two eyes, the image on the
retina of the left eye is almost exactly the same as the image on the retina
of right eye, but our brains merge the two images into a single percept, and
thus compress the information..."
The image below from the paper captures the range of applications that will be possible in the future.
