Thanks for sharing that perspective. I do think jq is great for slicing up JSON before handing it off to Python—especially if your transformations are primarily stateless and can be expressed as simple map/filters. And yes, a lot of frameworks can turn into "magic black boxes" that complicate what should be simple.
That said, my gripe is that when you move beyond "transform this data" and step into "transform and then push to some API, handle partial success, retries, etc.," the line-by-line or chunk-by-chunk side-effects logic in Python can get gnarly fast. That’s the part I wish there was a standard, declarative approach for—something that doesn’t require a big workflow orchestrator (like Dagster) but also doesn’t devolve into tons of imperative glue code.
I agree no tool is a silver bullet. In many cases, plain, well-structured Python is enough. But once you need concurrency, chunk-based error handling, or incremental streaming with side effects, you end up coding your own partial solution anyway—and that’s where I start wishing for a more composable pattern in the standard Python ecosystem. Until then, jq + Python is definitely a solid approach for the simpler jobs!
Thanks for laying that out. You raise great points: Pandas (or Polars) chaining is indeed closer to what we want than raw imperative loops, and it handles a lot of the transformations. But it still assumes an in-memory, columnar approach, which doesn’t always gel with row-by-row side effects like pushing data to an external API. Also, real-world systems do require robust error handling and partial retries, which is exactly the scenario we’d love a functional pipeline to address by design, rather than patching in big try/except blocks. Tools like DAG orchestrators solve that at a workflow level, but not so much at the record/stream level.
So that’s the core frustration: yes, we can hack it in Python with a pipeline style, yes we can load data into DataFrames and transform it, but as soon as we do side-effectful calls (like collection.add(...) per record or batch), we have to drop out of the nice .pipe() flow. For me, it’s less about syntax and more about a missing standard approach to composable streams + side effects. Elixir Broadway, or even a DSL in Lisp/Clojure, addresses that more natively.
“Just use batch jobs / just use SQL transactions” works only if your external system supports transactions or if you keep your data in a single database. But many people use specialized systems (e.g. vector DBs, text embeddings) that don’t have an ACID-like model.
Tools like Elixir Broadway do exactly that for streaming: concurrency, batching, backpressure, built-in retry, etc. That’s part of the reason we’re exploring new paradigms, rather than simply reading into Pandas or writing a standard “for line in file” script.
In short, I’m not saying Pandas can’t be coerced into something close to a pipeline. It’s more that we lack a standard, built-in, composable “stream of records -> transform -> side-effect-> handle errors” solution. Once you leave the purely tabular, in-memory paradigm, or need partial success handling, you quickly outgrow the typical DataFrame approach. That’s why I find frameworks like Elixir Broadway, or even an RxJS-like pipeline, so appealing — they make chunk-based side effects a first-class citizen.
So the question is: do we keep building one-off solutions in Python, or do we push for a more standardized, composable approach that can handle both transformations and side effects gracefully?
I think you’re right to point out the inertia in the labor market and the structural energy required for paradigm shifts.
What worries me most, though, is that because of exactly the reasons you’ve described — and now, with the rise of GenAI that makes generating boilerplate imperative code even easier — we might be permanently missing the opportunity to invent a better, more elegant solution.
Instead of challenging the underlying paradigm, we’re accelerating its replication — just with fewer keystrokes. GenAI is helping us pave cow paths, not design highways.
My fear is that the convenience of generating "good enough" code at scale will solidify suboptimal workflows, and the space for reimagining them might quietly close forever.
Hey, thanks for your feedback. That is great input, because I just realized that users might prefer to have a command printed out (inserted into clipboard) for them, instead of it being executed immediately. LocalLLMs are on my todo list! Also, what I want to work on next, is to make it possible to pipe errors / logs into the application, so it could help you troubleshoot and debug faster
I saw your very recent comment on EMRs /EHRs and that you would like to build a new one.
we are actually in the process of building one, we have a doctor on board and we also have our first users.
Our goal is to create the most fantastic user experience for primary Care practitioners.
Our existing MVP uses AI to to speed up clinical documentation process using speech text to speech. Transcribing the conversation, summarizing it, generating a doctor's letter.
We try to identify what actions the doctor recommended to the patients while talking to them. For instance, regenerate a laboratory referral in real time.
We are writing our yc application atm.
I can't see your email in your profile, but you should be able to see mine. I would love to talk to you.
Please drop me a message if you are still interested in building a new EMR.
That said, my gripe is that when you move beyond "transform this data" and step into "transform and then push to some API, handle partial success, retries, etc.," the line-by-line or chunk-by-chunk side-effects logic in Python can get gnarly fast. That’s the part I wish there was a standard, declarative approach for—something that doesn’t require a big workflow orchestrator (like Dagster) but also doesn’t devolve into tons of imperative glue code.
I agree no tool is a silver bullet. In many cases, plain, well-structured Python is enough. But once you need concurrency, chunk-based error handling, or incremental streaming with side effects, you end up coding your own partial solution anyway—and that’s where I start wishing for a more composable pattern in the standard Python ecosystem. Until then, jq + Python is definitely a solid approach for the simpler jobs!