Skip to content

Move db writer to a separate process with a single queue #32

Open

Description

@robobenklein

All processes will append docs to insert to a queue which will be consumed by the writer process.

This helps prevent the write-write conflict / locking over the db updates which could result in dedup conflicts.

Would also simplify writing to file if all docs written pass through the queue.

Writer process will need to be extremely efficient though in order to handle thousands of docs/second.

Activity

  1. self-assigned this
    on May 28, 2021
  2. robobenklein commented on May 28, 2021

    @robobenklein
    MemberAuthor

    Inserts that fail will be re-added to a retry queue along with a list of the errors, when that list for some doc becomes too large, stop the entire procedure since writes are failing.

  3. robobenklein commented on Jul 16, 2021

    @robobenklein
    MemberAuthor

    This is a bad idea.

    Don't use a single IPC queue for passing documents, performance is worse than my house connection to the DB...

    Need some method to write documents from each worker in parallel, maybe FileLock is performant enough? Until then, manual checks on arango it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions