In an enterprise system, populating a data lake relies heavily on interdependent batch processes. Typically these data lakes are updated at a frequency set to a few hours. Today’s business demands high-quality data not in a matter of hours or days but in minutes or seconds.
The typical steps to update the data lake are (a) build incremental data (b) read the existing data lake files, update incremental changes, and rewrite the data lake files (note: S3 files are immutable). This also brings in the challenge of ACID compliance between readers and writers of a data lake.
from DZone.com Feed https://ift.tt/2RSWTio
No comments:
Post a Comment