Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Datafusion is undoubtedly one of the best open source projects of all time, it's so incredibly powerful and well designed.

The extensibility is insane, you can create your own query language that compiles to logical plans.



I agree 100%! DataFusion is beautiful and easy to extend in any direction. For the second version of my "out-of-core" graph algorithms project, for example, I implemented my own "co-partitioning" to speed up joins and achieved a performance improvement of two times! It was also easy to modify the physical plan and declare partitioning.


Co-partitioning of range-partitioned data is now supported on `main` too: see https://github.com/apache/datafusion/issues/23183 and the epic https://github.com/apache/datafusion/issues/22395




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: