Manage Data

Every time new data is appended to an existing Delta table, a new Parquet file is generated. This allows data to be ingested incrementally without having to rewrite the entire dataset.

As files accumulate, read operations may surge. The compact function merges small files into larger ones, enhancing scanning performance.

Combining incremental processing with the compact function enables efficient data updates and scans as your dataset expands.

Appending mismatched data to a Parquet table involves reading the existing data, concatenating it with the new data, and overwriting the existing Parquet file.

This approach can be expensive and may lead to schema inconsistencies.

With Delta Lake, you can effortlessly append DataFrames with extra columns while ensuring the preservation of your data’s schema.

Merging two datasets and performing both insert and update operations can be a complex task. Delta Lake makes it easy to perform multiple data manipulation operations during a merge operation.

The following code demonstrates merging two datasets using Delta Lake:

✔️ If a match is found, the last_talk value in people_table is updated with the corresponding value from new_df 

✖️ If a match is not found and the last_talk value in people_table is older than 30 days, the status column is updated to ‘rejected’.