Introduction Modern data engineering is more than just moving data from one place to another. A production-ready pipeline must be scalable, reliable, secure, cost-efficient, and capable of handling failures gracefully. To explore these concepts, I built an end-to-end serverless Data Engineering pipeline on AWS that ingests live YouTube Trending data through the YouTube Data API … Read more

How Hash Tables Silently Help Modern Database Engines Join Tables Under the Hood When developers write a SQL query like: it looks simple. But behind the scenes, the database engine may process millions — sometimes billions — of rows.So how does it perform these joins so quickly? The answer is often a powerful data structure … Read more

In the world of data engineering, “scale” is often a relative term. But at Uber, scale means managing a multi-hundred-petabyte repository that handles 6 trillion rows ingested daily. To manage this tidal wave of information, Uber moved away from traditional append-only data lakes to create Apache Hudi™, a storage engine that brings database-like primitives to … Read more