Here's a smell that your Big Query query ain't that big - lots of 1 row tables getting used to load config-like values, which are then used for JOINs later.
So much cruft to clean up 😥 .
My brain says: - maybe couldda just done it in a PAndAs?! Let's definitely not learn about CTEs right?! 😬
@adamchainz I like this phrase very much and I’m going to keep it and reuse it. ☺️
Last year we started migrating not-big things into DataFrame-driven tasks and it’s gone very well. Quicker tests and fewer test failures have been immediate wins.
However, these 1-row config-like tables aren’t going into Pandas… we’ll template their values into the BQ queries that need them, the way it should have been done in the first place.
@adamchainz Lol no have not tried Polars. Because I'm old this has been scheduled for 2029 - this allows me to remain approx 5 years behind "THE CURVE". 🧓
Add comment