Building robust data pipelines with dbt, Airflow, and Great Expectations 🚀
I started to dive into great expectations - a Python library for data quality checks, and I found this great talk by Sam Bail about building data pipelines with dbt, Airflow, and great expectations.
I'm far too fascinated with far too many topics. I've compiled an exhaustive list of hashtags for some of those interests; I'll try to do better about keeping it current and updated.
At #fosdem2024 , Jarek Potiuk made¹ an interesting point about why the #AirFlow pipeline manager uses a semi-declarative approach: semantic scalability. When defining complex pipelines, you often end up generating the declaration with a script, which essentially discards the benefit of the approach.
What's fun is that it is the same reasoning I used in my article about #liquidprompt's design², advocating for its procedural theming approach.
Tatiana Al-Chueyr also presented¹ some ideas around this question, while discussing integration of #AirFlow and #dbt (#airflow_dbt).
This "declarative VS procedural" debate is extremely pervasive to computer science (think of build systems, for instance). And I keep encountering it in my work.
I wonder if some people gave it a good think? And would have produced some tooling for a progressive (declarative → semi-declarative → procedural) generic engine?
I'm looking for a role leading an engineering team. I've got several years both managing teams as well as an individual contributor. Strong recent #BigData and #DataEngineering experience.
Dialectical Behavior Therapy #DBT is a godsend for folks with personality disorders, and I really think just about anyone - mentally ill or not - could benefit from its concepts.
As someone who simultaneously struggles with Borderline Personality Disorder #BPD and Attention Deficit Disorder #ADD/#ADHD, being able to consistently implement and maintain the principles of DBT feels like a monumental and discouraging challenge, not the least of which is the cost of attending traditional #therapy when my partner and I are struggling just to survive. And in order to become stable enough to survive, we need to develop the #CopingSkills taught in DBT.. but we can't afford it!
My ADHD augments my BPD, leaving me with zero self-discipline. For example, I've dabbled with Duolingo and have a 42 day streak going, it's probably the longest amount of time I've maintained a healthy habit in my entire life. I need help to focus and I'm tired of being ashamed to say so.
I realize people need to be compensated for their pioneering work in behavioral therapy; but I wonder if there's some way to "open source" the teaching and practicing of DBT for people like me who otherwise couldn't afford it, and will never be able to afford it without the skills therein!
At this point I'm essentially self-medicating with cannabis most of my waking hours to cope with.. well, the shitshow that is my personal life with mental illness and the burning shitshow that is the world. I can't afford to do that long-term.
I truly, honestly believe DBT is a miracle but we need a way to share the knowledge as a matter of improving overall humanity.
Thoughts? Ideas? Slipped into a coma trying to finish this? I understand.
"And all them politicians
They all lyin' sacks of shit
They say better days upon us
But it's sucking left hind tit
And the preacher on the TV
Says it ain't too late for me
But I bet he drives a Cadillac
And I'm broke with hungry mouths to feed
And I wish I'z still an outlaw
Was a better way of life
I could clothe and feed my family
Still have time to love my pretty wife
If you say I'm being punished
Ain't he got better things to do?
Turnin' mountains into oceans
Puttin' people on the moon"
One of the things I sometimes chuckle with on #DBT is how clearly it calls out what is sometimes obvious, but is helpful to have written out. Like their "option for solving any problem" is:
Solve the Problem
Feel Better about the Problem
Tolerate the Problem
Stay Miserable Or possibly make it worse!
Which, yes, is all totally true. And also the devil is in the details for 1, 2, and 3.
Whelp.. I finally made too many hashtags to fit on my profile. I tried to pick the ones most applicable to me, or ones I feel most passionate about for my page, and I'll attach an exhaustive, absurdly long list of hashtags below. It was surprisingly difficult to decide which ones made the cut. Drumroll please: