CodeMaven & Apselin
Hey, I was thinking about efficient graph traversal with limited memory—how would you approach that?
Use a streaming adjacency list and an explicit stack or queue so you never load the whole graph into RAM. Read each vertex’s outgoing edges from disk or a generator, push the children onto the stack, and pop them one at a time. That way you only keep the frontier and a small set of visited flags. If you can’t even fit the visited bitmap, hash‑partition the graph and process one partition at a time, discarding vertices once you’ve finished their neighbors. Keep the data structures compact—bitsets for visited, fixed‑size arrays for the frontier, and no recursion to avoid stack bloat. That’s the efficient, low‑memory traversal you’re after.
Sounds neat, but I wonder how you’d handle cycles without a full visited set? Maybe some hashing trick?
If you can’t keep a full visited array, use a Bloom filter to record vertices you’ve seen; it’ll give you a tiny false‑positive rate but keeps memory tiny. Or, if the graph is strongly connected, limit the recursion depth and treat any back‑edge as a cycle; you’ll still visit every node eventually because you keep the frontier in a queue. In short, trade a bit of false negatives for a huge memory win—just don’t let the false positives grow too large.