Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Does it support out-of-core or multi-processor processing?


Yes, it is both out-of-core and multi-processor. I created this toy project to process graphs that cannot fit into memory (in CSR format) using all available cores.

The multi-processing relies on DataFusion's Tokyo workers. The out-of-core aspect is achieved through a combination of DataFusion FairSpillPool, Sort-Merge-Join, and manually offloading everything to temporary Parquet files on disk.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: