Overwrite Partitions of a pandas DataFrame with Delta Lake

Overwrite Partitions of a pandas DataFrame with Delta Lake

If you need to modify a specific subset of your pandas DataFrame, such as yesterday’s data, it is not possible to overwrite only that partition. Instead, you have to load the entire DataFrame into memory as a workaround solution.

Delta Lake makes it easy to overwrite partitions of a pandas DataFrame.

Link to delta-rs.

Full code.

Search

Related Posts

Scroll to Top

Work with Khuyen Tran

Work with Khuyen Tran