Lambda Architecture
Lambda architecture (read more about it here) has three layers:
- Batch;
- Serving; and
- Speed.
Batch Layer
Stores the master immutable set, and computes the views of the data.
It is always out of date, because it gets updated in batches.
Serving Layer
Indexes the Batch Layer's output (which is likely to be flat files).
Loads into database for fast querying.
Doesn't do any random writes (it always writes sequentially into database), which makes it faster!
Speed Layer
Batch layer is always out of date, perhaps even by a few hours!
The Speed Layer only works on new data as it arrives.
When the Batch Layer catches up, throw it away.
In the end...
It looks something like this:
Lambda Technologies
- Batch Layer
- Serving Layer
- ElephantDB (now seems to be a dead project)
- Speed Layer
- Storm
- Cassandra
- Heron (new Twitter technology)