Yousry Mohamed
Oct 25, 2022

Hi Vineet,

In delta lake, you cannot have the transaction log file written while data writing fails. Apparently, creating the transaction log file would be the last step in the operation.

If delta creates parquet files but the transaction log fails, nothing will broken here as the mere effect is some orphan files on disk that could be cleaned up later using Vaccum or something.

Regards,

Yousry

Yousry Mohamed
Yousry Mohamed

Written by Yousry Mohamed

Yousry is a principal data engineer working for Mantel Group. He is very passionate about all things data including Big Data, Machine Learning and AI.

No responses yet