Aukio Data - Data storage and computing engine
Table of Contents
1. General
- This program is free software: released under Creative Commons Zero (CC0) license
- Program author:
- Svjatoslav Agejenko
- Homepage: https://svjatoslav.eu
- Email: mailto://svjatoslav@svjatoslav.eu
- Other software projects hosted at svjatoslav.eu
1.1. Source code
- Download latest snapshot in TAR GZ format
- Browse Git repository online
Clone Git repository using command:
git clone https://www2.svjatoslav.eu/git/aukio-data.git
- See JavaDoc.
2. Vision / goal
Provide hackable, versioned, optimized, distributed, geometrical, arbitrary dimensional hypercube based data storage and computation engine (as inspired by the brain) for general purpose visual computing environment called Aukio.
3. Ideas / inspiration
3.1. Brain
- Brain appears to be natural geometrical/parallel data storage and computational engine:
- Even more awesome is that brain appears to operate and is wired as arbitrary/variable dimensional structure: https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/
- On top of this, this multidimensional space that brain represents has dynamic/variable resolution/density:
- Such properties allow parallel Geometrical computation and beautifully fits CM-1 Connection Machine architecture (for extra hardware accelerated solution).
3.2. CM-1 Connection Machine
https://en.wikipedia.org/wiki/Connection_Machine
Massively parallel (thousands of CPUs) connected via machine's internal 12-dimensional hypercube network allows to efficiently simulate arbitrary dimensional hypercube and network topology between computational units. So that when we are solving/simulating for example 5 dimensional problem, we can arrange computational units into virtual 5D network. See: http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html
we can pre-distribute data across computation units and perform parallel geometrical computation.
3.3. Integrate with lossless diamond square algorithm based multidimensional data compressor
- Demonstrate using 3D UI for debugging algorithm internals.
- Separate RGB/YUV channels into separate encoders/decoders
- Make demo lossless image and sound CODECS.
3.4. Distributed computation and data storage
Lots of problems can be translated to geometry (use any shapes and as many dimensions as you need). Solution(s) to such problems could be then found via geometrical search/comparison/lookup results. As a bonus, such geometrical data storage AND computation can be naturally made in parallel and distributed.
3.5. The "Datacube" is a Fundamental Data Structure
The world is not just Sets (SQL), Hierarchies (XML/JSON), or Graphs (SPARQL). Arrays are the fourth pillar. Treating them as "just another file format" or "a blob in a table" is a fundamental error that leads to performance collapse.
4. Current status
- More or less defined Vision / goal.
- Collected some inspiring ideas.
- Implemented very simple persistent key-value map.
- Long term goal is to use it as a backing storage engine and implement more advanced features on top of this via layered architecture.