Skip to content

Database optimize - #291

Open
zly2006 wants to merge 10 commits into
QuiltServerTools:masterfrom
zly2006:database-optimize
Open

zly2006 wants to merge 10 commits into
QuiltServerTools:masterfrom
zly2006:database-optimize

Conversation

@zly2006

@zly2006 zly2006 commented Sep 29, 2024 •

Copy link
Copy Markdown
Contributor

This PR optimize database size in various ways:

  • Use long instead of string to store timestamp
  • Use a separate string table to store strings, also use java hash code to optimize searching.

I finally found it very hard to support both old and new scheme. Currently only auto purge and muanually purge are supported. I made a /ledger convert command to convert data from the old scheme.

@zly2006 zly2006 closed this Sep 29, 2024
@zly2006 zly2006 reopened this Sep 29, 2024
@zly2006

zly2006 commented Sep 29, 2024

Copy link
Copy Markdown
Contributor Author

I think we can only support searching of the old scheme, and implement a converting program for it.

@zly2006

zly2006 commented Sep 29, 2024

Copy link
Copy Markdown
Contributor Author

old scheme on sqlite, it stores timestamp in a string

-- auto-generated definition
create table actions
(
    id              INTEGER
        primary key autoincrement,
    action_id       INT     not null
        constraint fk_actions_action_id__id
            references ActionIdentifiers
            on update restrict on delete restrict,
    time            TEXT    not null,
    x               INT     not null,
    y               INT     not null,
    z               INT     not null,
    world_id        INT     not null
        constraint fk_actions_world_id__id
            references worlds
            on update restrict on delete restrict,
    object_id       INT     not null
        constraint fk_actions_object_id__id
            references ObjectIdentifiers
            on update restrict on delete restrict,
    old_object_id   INT     not null
        constraint fk_actions_old_object_id__id
            references ObjectIdentifiers
            on update restrict on delete restrict,
    block_state     TEXT,
    old_block_state TEXT,
    source          INT     not null
        constraint fk_actions_source__id
            references sources
            on update restrict on delete restrict,
    player_id       INT
        constraint fk_actions_player_id__id
            references players
            on update restrict on delete restrict,
    extra_data      TEXT,
    rolled_back     BOOLEAN not null
);

create index actions_action_id
    on actions (action_id);

create index actions_by_location
    on actions (x, y, z, world_id);

create index actions_object_id
    on actions (object_id);

create index actions_old_object_id
    on actions (old_object_id);

create index actions_player_id
    on actions (player_id);

create index actions_source
    on actions (source);

@zly2006
zly2006 marked this pull request as ready for review October 6, 2024 07:05
@zly2006
zly2006 requested a review from a team as a code owner October 6, 2024 07:05
@zly2006

zly2006 commented Oct 6, 2024

Copy link
Copy Markdown
Contributor Author

On my real game server testing:
new scheme:
20140591 unique strings, 46653892 actions, timestamp: 1727628523588-1728200973330 (159 hour), 26.1 GB, 601 bytes/action

old:
6308393 actions, time: 2024-07-01 11:54:21.086 -> 2024-10-04 03:34:17.058, 12.0 GB, 2058 bytes/action

70% database size saved.

@DrexHD

@DrexHD

DrexHD commented Oct 9, 2024 •

Copy link
Copy Markdown
Member

Impressive!, but...
Maintaining two different formats feels very jank, because it introduces lots of code duplication, maintenance burden and gets worse if we want to make more changes to the schema.
We have already discussed versioned sql, with automatic schema updates on discord to give us the ability to improve the database format over time, but that is a lot of effort to implement properly.

@zly2006

zly2006 commented Oct 10, 2024

Copy link
Copy Markdown
Contributor Author

I tried to run this sql on a 100k sample of our real data:

select action_id,count(*) as c from actions
group by action_id,x,y,z
order by  c desc ;

here's the top 10 result:

6,47566
6,27051
6,5015
1,4880
1,4880
6,4196
1,2442
6,713
6,523
6,237

So we can implement a better "auto purge", that purges actions at the same locations over 100 times, this saves 97% data from our database. command:

select sum(c) - 100*count(*)
from (select action_id,count(*) as c from actions
group by action_id,x,y,z
order by  c desc)
where c>100;

@zly2006

zly2006 commented Oct 10, 2024

Copy link
Copy Markdown
Contributor Author

My approach will be, add some new config options: smart_purge_enabled and smart_purge_filters(default=action_id,object_id,world_id,x,y,z) and smart_purge_max(default=100).

If some actions, their all smart_purge_filters are same, then we will purge old actions if their count >= smart_purge_max.

We can also encourage server admins to enable it since it almost never make you lose important data and save a lot of space.

Pau1am pushed a commit to Pau1am/Ledger-Reoptimization that referenced this pull request Oct 7, 2026
…tibility export

Upstream PR QuiltServerTools#291 was right about which field matters: `extra_data` holds serialised NBT
for container contents and entity kills, and it is the largest column on a real server.
This takes that idea and implements it without the part that got the PR rejected.

What the PR proposed and what changed here:

  * its `actions_v2` plus a retained legacy table -> additive only: one nullable column
    and one new table, created by the same instant ALTER path as `block_state_ref`
  * its 920-character gzip floor -> 92 bytes. Every NBT value we measured on a live
    server was under 920 characters, so that floor would have compressed nothing
  * its per-row `gzip` boolean -> a header byte inside the payload, saving a column
  * its 32-bit javaHash plus a payload comparison on every hit -> a 128-bit content hash
    as a unique key, so a lookup is one index probe and never reads the blob. 64 bits
    would collide ~1 in 10^5 across 20M entries; 128 bits is ~10^-21
  * its `LongIdTable` (64-bit ids) -> 32-bit, which is ample and keeps three references
    12 bytes per row narrower

Measured on real Minecraft entity NBT taken from a running server: 587-624 bytes raw
became 438-450 bytes stored, i.e. 72-75%.

Compatibility, which is the reason for the second half of this commit. Rows this fork
writes populate only the `*_ref` columns, because storing both forms would cost exactly
the bytes the dictionary saves. Unmodified upstream Ledger would therefore read NULL for
them. `/ledger exportlegacy` writes the text back on demand, after which an upstream
build sees a complete database - verified byte-identical (matching SHA-256 of the NBT
string) against the dictionary contents. That makes the compatibility cost a one-time
opt-in expense rather than a permanent one, and it is why this does not repeat the
two-format problem that PR QuiltServerTools#291 was blocked on.

Verified live: 4 entity kills written, 4 dictionary rows created, all gzip-compressed,
`extra_data` NULL on every row as intended, then exportlegacy restored all four
byte-identically with the references left in place. Zero exceptions, zero tick warnings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants