Reservoir Sampling
The method for drawing a fair random sample from a stream of unknown length, the elegant algorithm R, and its use in large-scale data systems.
15 posts about data engineering
The method for drawing a fair random sample from a stream of unknown length, the elegant algorithm R, and its use in large-scale data systems.
Large Language Models can extract structured data like JSON from unstructured text with surprising reliability, but the mechanism and tradeoffs determine when this approach is superior to traditional parsers.
Web crawling and web scraping are often confused, but they serve fundamentally different purposes: crawling is a broad search for URLs to index the web, while scraping is a targeted extraction of specific data from a known set of pages.
A comprehensive overview of Fivetran, an automated data movement platform that simplifies the process of consolidating data from disparate sources into a central data warehouse or lakehouse for analytics.
You were told there is no API and got stuck. Here is the ladder for pulling the data anyway, and what to hand a developer to get it clean.
The algorithm that counts billions of distinct items in kilobytes, from Flajolet and Martin's 1984 work to the Flajolet 2007 refinement used in databases.
A deep, technical explanation of Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP) databases, covering their internal mechanisms, core tradeoffs, and the decision framework for when to use each.
A compact structure for frequency estimation on data streams, introduced by Cormode and Muthukrishnan in 2005, and its accuracy guarantees.
What investors and agents actually want from real estate data, and how to turn scattered scraped listings into one clean, current feed.
How we built an AI-powered data extraction system that transforms unstructured documents into structured data with 95% accuracy, and the architectural decisions that made it scalable.
If someone on your team spends hours copying data into spreadsheets, here's how to automate that manual work from collection to delivery.
A comprehensive, balanced overview of Snowflake's cloud data platform, covering its architecture, business model, market position, and key considerations for technical leaders.
dbt Labs is the company behind dbt (data build tool), an open-source command-line tool and commercial SaaS product that has become a central component in the modern data stack for transforming data within cloud warehouses.
Entity Resolution is the process of identifying and linking records that refer to the same real-world entity across different data sources, a critical task for data quality and creating a single source of truth.
Airbyte is a data integration company that provides an open-source platform for moving data between applications, APIs, and databases into data warehouses and other destinations, using a large library of pre-built connectors.