thanks for the feedback! the first version had a lot more detailed code but decided to go with linking to our GitHub than copying all the code. Wanted to illustrate the core touch points involved in extending DF.

Building Databases over a Weekend

DataFusion is primarily a batch OLAP system, so we should be able to support hybrid workloads as well. And definitely agree with you re: Polars dev exp. That is something we are aiming for with our forthcoming Python sdk.

> It'd also be important/useful to support Python udfs (think numpy/jax/etc.).

Yep that's our longterm gameplan.

> It'd be very cool if you could collaborate with or even tap into the polars frontend. If you could execute polars logical plans but with a streaming source, that would be huge.

Are there examples of project that do this? I'd be very much interested in looking into this.

we absolutely do, the library itself is designed to be extensible. we are currently working on adding webhooks as one of our sources. are there are any specific connectors/sources you'd be interested in?

while haven't checked out Fluvio yet, we are fans of Arroyo. regarding latter my understanding is that the team is going for a SQL first complete replacement for Flink. Denormalized is meant to be an embeddable engine you can import within your project. Our plan is to focus on the developer experience for users building with Python and Typescript in particular.