Skip to content

How scalable is this? #34

Open
Open
@uriva

Description

@uriva
  1. Would it support a chunk of 5 billion nodes/edges?
  2. If each has minimal payload, how much time would the process take?

Activity

jeffreylovitz

jeffreylovitz commented on Aug 10, 2020

@jeffreylovitz
Contributor

Hi @uriva,

  1. If your server has enough RAM to store and query a graph with 5 billion entities, you should not have an issue running the bulk loader. It will automatically divide your input into batches to populate a buffer of up to 2 gigabytes, and maintains a dictionary mapping all nodes to their identifiers.
  2. I'd expect this to take dozens of hours, but there are too many factors in play to be very precise. Generally, load time will scale linearly with the input size. Building a graph with about 5 million nodes, 5 million edges, and 20 million properties on my system takes 220 seconds, so increasing that by a factor of 500 gives about 30 hours as a very very rough estimate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @uriva@jeffreylovitz

        Issue actions

          How scalable is this? · Issue #34 · RedisGraph/redisgraph-bulk-loader