Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .cursorignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@
# limitations under the License.

# Add directories or file patterns to ignore during indexing (e.g. foo/ or *.csv)
contrib/
contrib/aws*
contrib/grpc
2 changes: 1 addition & 1 deletion contrib/cloud-storage-engine
21 changes: 20 additions & 1 deletion contrib/tiflash-columnar-hub/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,25 @@ kvengine = { path = "../../../cloud-storage-engine/components/kvengine" }

Adjust paths relative to `contrib/tiflash-columnar-hub/`. Then run `cargo check` / `make release` and rebuild TiFlash. Remember to revert path patches before committing lockfile changes meant for upstream CI.

### Lockfile out of sync after cloud-storage-engine commit bump

When the cloud-storage-engine commit is bumped, `Cargo.toml` files inside it may change (new / removed dependencies). The committed `Cargo.lock` then no longer matches the resolved dependency graph, and the build fails with:

```
Comment thread
JaySon-Huang marked this conversation as resolved.
error: cannot update the lock file contrib/tiflash-columnar-hub/Cargo.lock because --locked was passed to prevent this
```

To fix:

```bash
cd contrib/tiflash-columnar-hub
cargo update -p workspace-hack # re-resolve; commit submodule pointer + Cargo.lock together:
```

Notes:

- `cargo update -p workspace-hack` is enough in most cases; the resolver may also pick up a few compatible upgrades because it re-solves the whole graph — that is fine as long as `make release` builds.
Comment thread
JaySon-Huang marked this conversation as resolved.

## Standalone commands

```bash
Expand All @@ -140,6 +159,6 @@ make release ENABLE_FEATURES=external-jemalloc

## Notes

- `workspace-hack/` intentionally overrides cloud-storage-engine's generated workspace-hack so the cdylib does not export a second jemalloc into the TiFlash process.
- `workspace-hack/` exists to override cloud-storage-engine's generated workspace-hack so the cdylib does not export a second jemalloc into the TiFlash process. Note the `[patch]` in `Cargo.toml` does not actually take effect (path dependencies inside the submodule cannot be patched), yet the built `libtiflash_proxy.so` contains no jemalloc symbols, because the linker drops un-referenced allocator code from the cdylib.
- Changing `Cargo.lock` alone is not enough for runtime behavior: always rebuild and redeploy `libtiflash_proxy.so`.
- Columnar integration tests under `tests/fullstack-test-next-gen-columnar/` also use separate TiKV / TiDB binaries from `_env.sh`; upgrading proxy kvengine does not upgrade TiKV. For end-to-end columnar behavior, consider keeping proxy and TiKV cloud-storage-engine commits compatible.
24 changes: 15 additions & 9 deletions contrib/tiflash-columnar-hub/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion metrics/grafana/tiflash_summary.dashboard.py
Original file line number Diff line number Diff line change
Expand Up @@ -1855,9 +1855,11 @@ def ColumnarStorage() -> RowPanel:
instance_selector=PROXY_LABEL_SELECTORS,
),
duration_panel(
"Columnar Prefetch Cache Hit Duration",
"Columnar Prefetch Cache Hit Ratio",
"tiflash_proxy_kv_engine_columnar_prefetch_cache_hit",
instance_selector=PROXY_LABEL_SELECTORS,
# Observe hit_ratio*100 with linear percent buckets (not duration).
unit="percent",
),
]
)
Expand Down
4 changes: 2 additions & 2 deletions metrics/grafana/tiflash_summary.json
Original file line number Diff line number Diff line change
Expand Up @@ -13033,7 +13033,7 @@
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Columnar Prefetch Cache Hit Duration",
"title": "Columnar Prefetch Cache Hit Ratio",
"tooltip": {
"msResolution": true,
"shared": true,
Expand All @@ -13052,7 +13052,7 @@
"yaxes": [
{
"decimals": null,
"format": "s",
"format": "percent",
"label": null,
"logBase": 1,
"max": null,
Expand Down
2 changes: 1 addition & 1 deletion metrics/grafana/tiflash_summary.json.sha256
Original file line number Diff line number Diff line change
@@ -1 +1 @@
6ed50a4a0e9d4dc79211ec7fa1a4a23e412ae04a82ba52c3a6b073b847afdd7d ./metrics/grafana/tiflash_summary.json
c16590c8f5fe7a7c777e0e5f7bfa814c2d398f66e0d0d0a0f79be500d2619866 ./metrics/grafana/tiflash_summary.json