Repository navigation
Conversation
|
I think we can only support searching of the old scheme, and implement a converting program for it. |
|
old scheme on sqlite, it stores timestamp in a string -- auto-generated definition
create table actions
(
id INTEGER
primary key autoincrement,
action_id INT not null
constraint fk_actions_action_id__id
references ActionIdentifiers
on update restrict on delete restrict,
time TEXT not null,
x INT not null,
y INT not null,
z INT not null,
world_id INT not null
constraint fk_actions_world_id__id
references worlds
on update restrict on delete restrict,
object_id INT not null
constraint fk_actions_object_id__id
references ObjectIdentifiers
on update restrict on delete restrict,
old_object_id INT not null
constraint fk_actions_old_object_id__id
references ObjectIdentifiers
on update restrict on delete restrict,
block_state TEXT,
old_block_state TEXT,
source INT not null
constraint fk_actions_source__id
references sources
on update restrict on delete restrict,
player_id INT
constraint fk_actions_player_id__id
references players
on update restrict on delete restrict,
extra_data TEXT,
rolled_back BOOLEAN not null
);
create index actions_action_id
on actions (action_id);
create index actions_by_location
on actions (x, y, z, world_id);
create index actions_object_id
on actions (object_id);
create index actions_old_object_id
on actions (old_object_id);
create index actions_player_id
on actions (player_id);
create index actions_source
on actions (source); |
|
On my real game server testing: old: 70% database size saved. |
|
Impressive!, but... |
|
I tried to run this sql on a 100k sample of our real data: here's the top 10 result: So we can implement a better "auto purge", that purges actions at the same locations over 100 times, this saves 97% data from our database. command: |
|
My approach will be, add some new config options: If some actions, their all We can also encourage server admins to enable it since it almost never make you lose important data and save a lot of space. |
…tibility export Upstream PR QuiltServerTools#291 was right about which field matters: `extra_data` holds serialised NBT for container contents and entity kills, and it is the largest column on a real server. This takes that idea and implements it without the part that got the PR rejected. What the PR proposed and what changed here: * its `actions_v2` plus a retained legacy table -> additive only: one nullable column and one new table, created by the same instant ALTER path as `block_state_ref` * its 920-character gzip floor -> 92 bytes. Every NBT value we measured on a live server was under 920 characters, so that floor would have compressed nothing * its per-row `gzip` boolean -> a header byte inside the payload, saving a column * its 32-bit javaHash plus a payload comparison on every hit -> a 128-bit content hash as a unique key, so a lookup is one index probe and never reads the blob. 64 bits would collide ~1 in 10^5 across 20M entries; 128 bits is ~10^-21 * its `LongIdTable` (64-bit ids) -> 32-bit, which is ample and keeps three references 12 bytes per row narrower Measured on real Minecraft entity NBT taken from a running server: 587-624 bytes raw became 438-450 bytes stored, i.e. 72-75%. Compatibility, which is the reason for the second half of this commit. Rows this fork writes populate only the `*_ref` columns, because storing both forms would cost exactly the bytes the dictionary saves. Unmodified upstream Ledger would therefore read NULL for them. `/ledger exportlegacy` writes the text back on demand, after which an upstream build sees a complete database - verified byte-identical (matching SHA-256 of the NBT string) against the dictionary contents. That makes the compatibility cost a one-time opt-in expense rather than a permanent one, and it is why this does not repeat the two-format problem that PR QuiltServerTools#291 was blocked on. Verified live: 4 entity kills written, 4 dictionary rows created, all gzip-compressed, `extra_data` NULL on every row as intended, then exportlegacy restored all four byte-identically with the references left in place. Zero exceptions, zero tick warnings.
This PR optimize database size in various ways:
I finally found it very hard to support both old and new scheme. Currently only auto purge and muanually purge are supported. I made a
/ledger convertcommand to convert data from the old scheme.