BigQuery under the hood: Short query optimizations in the advanced runtime

In a prior blog post, we introduced BigQuery’s advanced runtime, where we detailed enhanced vectorization and discussed techniques like dictionary and run-length encoded data, vectorized filter evaluation and compute pushdown, and parallelizable execution.

This blog post dives into short query optimizations, a key component of BigQuery's advanced runtime. These optimizations can significantly speed up the "short" queries all while using fewer BigQuery "slots" (our term for computational capacity). They are commonly used by business intelligence (BI) tools such as LookerStudio or custom applications powered by BigQuery.

Similar to other BigQuery optimization techniques, the system uses a set of internal rules to determine if it should consolidate a distributed query plan into a single, more efficient step for short queries. These rules consider factors like:

• The estimated amount of data to be read
• How effectively the filters are reducing the data size
• The type and physical arrangement of the ...