Home research People General Info Seminars Resources Intranet

Harshvardhan, Nancy M. Amato, Lawrence Rauchwerger, "Processing Big Data Graphs on Memory-Restricted Systems," In Proc. IEEE Int.Conf. on Parallel Architectures and Compilation Techniques (PACT), pp. 517-518 (SRC Poster), Edmonton, AB, Canada, Aug 2014.
Proceedings(pdf, abstract)

With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques, which can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. Contribution: In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into subgraphs that fit in RAM and uses a paging-like technique to load subgraphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16,000+ cores.