Use Storage Scale pools and filesets: storage totals and find filters

CHANGELOG.md

@@ -89,6 +89,10 @@ commit. It becomes `0.1.0` when that is tagged.
   `relatime` and `no` give each file's last read and "never opened since it
   changed" (dated to the listing); `yes` and unrecognised values give unknown.
   `doctor` shows the setting for each imported root.
+- `cdm storage` and the MCP `storage` tool: files and bytes by Storage Scale
+  storage pool and fileset; `find --fileset` / `--pool` and the same filters on
+  MCP `find`; `stat` shows both. Over MCP, pools are named by default and
+  filesets ranked by size, never named, unless names are exposed (ADR 0007).
 - `cdm policy`: prints a script that lists an IBM Storage Scale fileset (or,
   with `--whole-filesystem`, a whole filesystem) with `mmapplypolicy -I defer`,
   for an administrator to run as root; cdm never runs it. The listing carries

README.md

@@ -206,6 +206,11 @@ for files that are gone. A fileset listing covers only that fileset — Storage
 Scale leaves out other filesets even when they are linked under its junction —
 so re-importing a fileset never removes a nested fileset's rows.
 
+Each file's fileset and storage pool are kept: `cdm storage` totals the space
+by pool and fileset, and `cdm find --pool data1` or `--fileset proj` narrows a
+search. Over MCP, pools are named but filesets are reported by size rank unless
+names are exposed, because fileset names are often a person's or a project's.
+
 Access times come with the listing, under the same rules as on your own disks:
 with Storage Scale's default `-S relatime` (or `no`), each file's last read and
 whether it has been opened since it changed are recorded, so `find
@@ -349,6 +354,7 @@ the server answers from the *shape* of your data, not its names:
 | `duplicates_summary` | how much looks duplicated, how much is confirmed |
 | `suggest` | what is worth doing, ranked, with risk and command (no paths) |
 | `guide` | the getting-started checklist, with the next step |
+| `storage` | Storage Scale space by pool (named) and fileset (ranked, not named) |
 
 To make it easy to start, the server also offers **prompts** — *Getting
 started* (a step-by-step walk through `guide`), *What's using my disk?*, *What can I clean up?*, *How much is duplicated?*, *What changed
@@ -480,7 +486,8 @@ What comes next, and in what order, is in [docs/roadmap.md](docs/roadmap.md).
 - **[docs/adr/](docs/adr/)** — decision records: why names are opt-in over MCP
   (0002), why suggestions come from one advisory engine (0003), and why the
   getting-started guide takes its status from the index (0004), when an access
-  time is trusted (0005), and how "never opened" is dated (0006).
+  time is trusted (0005), how "never opened" is dated (0006), and why pool
+  names reach the AI by default but fileset names don't (0007).
 - **[CONTRIBUTING.md](CONTRIBUTING.md)** — setup, and the list of choices that
   are deliberate rather than accidental.
 - **[docs/releasing.md](docs/releasing.md)** — how a release happens, and the

cdm/cli.py

@@ -1,7 +1,7 @@
 """The command line.
 
-Verbs: scan, rescan, import, roots, forget, find, du, dupes, suggest, guide, policy,
-stat, doctor, mcp.
+Verbs: scan, rescan, import, roots, forget, find, du, dupes, storage, suggest, guide,
+policy, stat, doctor, mcp.
 
 Output goes to stdout as plain columns; anything the user did not ask for --
 skip counts, warnings, timings -- goes to stderr, so `cdm find ... | xargs` and
@@ -20,7 +20,20 @@
 from contextlib import closing
 from pathlib import Path
 
-from . import atime, db, guide, hashing, importer, paths, policy, probe, query, roots, suggest
+from . import (
+    atime,
+    db,
+    guide,
+    hashing,
+    importer,
+    paths,
+    policy,
+    probe,
+    query,
+    roots,
+    shape,
+    suggest,
+)
 from .exclude import Excluder
 from .scan import scan_root
 
@@ -297,6 +310,8 @@ def cmd_find(args, conn) -> int:
             accessed_before=(query.parse_when(args.accessed_before)
                              if args.accessed_before else None),
             unopened=args.unopened,
+            fileset=args.fileset,
+            pool=args.pool,
             order=args.order,
             limit=args.limit,
         )
@@ -455,6 +470,31 @@ def cmd_import(args, conn) -> int:
     return 0
 
 
+@with_index
+def cmd_storage(args, conn) -> int:
+    root = str(Path(args.root).expanduser().resolve()) if args.root else None
+    # The person at the terminal owns the index, so names are shown here.
+    out = shape.storage(conn, host=_visible(conn), root=root, names=True)
+    if args.json:
+        print(json.dumps(out, indent=2))
+        return 0
+    if not out["pools"] and not out["filesets"]:
+        _err("nothing from a Storage Scale listing yet: see `cdm policy` and "
+             "`cdm import --policy`")
+        return 0
+    for title, rows, key in (("pool", out["pools"], "pool"),
+                             ("fileset", out["filesets"], "fileset")):
+        print(f"{title:<8} {'size':>8}  {'files':>10}")
+        for r in rows:
+            print(f"{r[key]:<8} {r['size']:>8}  {r['files']:>10,}")
+        print()
+    rest = out["not_from_a_listing"]
+    if rest["files"]:
+        _err(f"also {rest['files']:,} scanned file(s), {rest['size']}, with no pool or "
+             f"fileset")
+    return 0
+
+
 def cmd_policy(args) -> int:
     """Print the listing script; cdm never runs mmapplypolicy itself."""
     fileset = None if args.whole_filesystem else args.fileset
@@ -561,6 +601,8 @@ def cmd_stat(args, conn) -> int:
     elif row["type"] == "file":
         print("accessed  not recorded (the filesystem does not update access times, "
               "or it has not been read since cdm last hashed it)")
+    if row["fileset"] is not None:
+        print(f"fileset   {row['fileset']}  (pool {row['pool']})")
     if row["unopened_until"] is not None:
         print(f"opened    not since it last changed, as of "
               f"{human_time(row['unopened_until'])}")
@@ -763,6 +805,10 @@ def build_parser() -> argparse.ArgumentParser:
     f.add_argument("--accessed-after", metavar="WHEN",
                    help="last read after WHEN; files with no trusted access time never match")
     f.add_argument("--accessed-before", metavar="WHEN")
+    f.add_argument("--fileset", metavar="NAME",
+                   help="Storage Scale fileset (imported listings only)")
+    f.add_argument("--pool", metavar="NAME",
+                   help="Storage Scale storage pool (imported listings only)")
     f.add_argument("--unopened", action="store_true",
                    help="only files known not to have been opened since they last changed")
     f.add_argument("--root", metavar="PATH", help="restrict to one root (and roots nested in it)")
@@ -814,6 +860,15 @@ def build_parser() -> argparse.ArgumentParser:
                         "even when stderr is not a terminal")
     i.set_defaults(func=cmd_import)
 
+    st = sub.add_parser(
+        "storage", help="files and bytes by Storage Scale pool and fileset",
+        description="Totals by storage pool and fileset, from imported Storage Scale "
+                    "listings.")
+    st.add_argument("--root", metavar="PATH",
+                    help="restrict to one root (and roots nested in it)")
+    st.add_argument("--json", action="store_true")
+    st.set_defaults(func=cmd_storage)
+
     o = sub.add_parser(
         "policy", help="print a script that lists a Storage Scale fileset for import",
         description="Print a script for an administrator to run as root: an "

cdm/query.py

@@ -47,7 +47,7 @@ def parse_when(text: str, *, now: float | None = None) -> float:
 def find(conn, *, host=None, root=None, name=None, iname=None, kind=None,
          larger_than=None, smaller_than=None, modified_after=None,
          modified_before=None, accessed_after=None, accessed_before=None,
-         unopened=False, order="size", limit=100):
+         unopened=False, fileset=None, pool=None, order="size", limit=100):
     # Every filter is one optional AND-ed clause over one column. Keeping them
     # in a table rather than a run of ifs is what makes the set easy to extend
     # -- and easy for a future NL layer to enumerate.
@@ -76,6 +76,9 @@ def find(conn, *, host=None, root=None, name=None, iname=None, kind=None,
         # matches neither, so "not accessed since" is never guessed.
         ("atime > ?", accessed_after),
         ("atime < ?", accessed_before),
+        # Storage Scale, from an imported listing; scanned rows have neither.
+        ("fileset = ?", fileset),
+        ("pool = ?", pool),
     )
     clauses, params = [], []
     for sql_fragment, value in specs:

cdm/shape.py

@@ -282,3 +282,31 @@ def duplicates_summary(conn, *, host: str | None = None,
                      "full_hashed": hashed.get("full", 0),
                      "stale_hashes": stale},
     }
+
+
+def storage(conn, *, host=None, root: str | None = None, names: bool = False) -> dict:
+    """Files and bytes by Storage Scale storage pool and by fileset.
+
+    Pool names are shape: storage tiers named by an administrator (system,
+    data1, ssd). Fileset names are not, by default: they are often a user's
+    name or a project's codename. Without `names`, filesets are reported by
+    rank -- "fileset #1" is the largest -- so the shape survives and the names
+    do not. See docs/adr/0007. Files that came from a scan, not a listing,
+    have neither and are counted apart.
+    """
+    where, params = _scope(host, root)
+    base = f"FROM files WHERE type = 'file' AND {where}"
+    pools = [{"pool": r[0], "files": r[1], "bytes": r[2], "size": human(r[2])}
+             for r in conn.execute(
+                 f"SELECT pool, COUNT(*), COALESCE(SUM(size), 0) {base} AND pool IS NOT "
+                 f"NULL GROUP BY pool ORDER BY 3 DESC", params)]
+    filesets = []
+    for rank, r in enumerate(conn.execute(
+            f"SELECT fileset, COUNT(*), COALESCE(SUM(size), 0) {base} AND fileset IS "
+            f"NOT NULL GROUP BY fileset ORDER BY 3 DESC, 1", params), 1):
+        filesets.append({"fileset": r[0] if names else f"fileset #{rank}",
+                         "files": r[1], "bytes": r[2], "size": human(r[2])})
+    n, b = conn.execute(f"SELECT COUNT(*), COALESCE(SUM(size), 0) {base} AND pool IS "
+                        f"NULL AND fileset IS NULL", params).fetchone()
+    return {"pools": pools, "filesets": filesets, "fileset_names": names,
+            "not_from_a_listing": {"files": n, "bytes": b, "size": human(b)}}

cdm/tools.py

@@ -55,6 +55,11 @@
     "duplicates_summary": "How much space looks duplicated, how much is "
                           "confirmed, and how much of the tree was hashed at "
                           "all. Returns no filenames.",
+    "storage": "Files and bytes per IBM Storage Scale storage pool (by name) and "
+               "per fileset, from imported policy listings. Fileset names are "
+               "only included when names are exposed; otherwise filesets are "
+               "ranked by size (fileset #1 is the largest). Answers 'which "
+               "tier or fileset holds the space'.",
     "guide": "Step-by-step setup and upkeep: index a folder, hash it, look at "
              "suggestions, clean up and rescan, keep it current, connect an AI "
              "client. Each step's status is read from the index; `next` says "
@@ -73,7 +78,8 @@
             "2026-08-01. accessed_before/after only match files with a trusted "
             "access time. unopened=true lists files known not to have been opened "
             "since they last changed (e.g. downloads never used); each result's "
-            "unopened_as_of says as of when.",
+            "unopened_as_of says as of when. fileset and pool filter imported "
+            "Storage Scale listings.",
     "du": "Disk usage by subdirectory under a path, answered from the index. "
           "Returns directory paths.",
     "dupes": "Groups of files that share a hash. Returns paths. Partial-hash "
@@ -179,7 +185,9 @@ def _file_row(r) -> dict[str, Any]:
             "size": shape.human(r["size"]), "modified": _iso(r["mtime"]),
             "accessed": _iso(r["atime"]),
             # Not opened between its last change and this date; see ADR 0006.
-            "unopened_as_of": _iso(r["unopened_until"])}
+            "unopened_as_of": _iso(r["unopened_until"]),
+            # Storage Scale, from an imported listing; None for scanned files.
+            "fileset": r["fileset"], "pool": r["pool"]}
 
 
 class Catalog:
@@ -280,6 +288,20 @@ def duplicates_summary(self, root: str | None = None) -> dict[str, Any]:
                      "terminal confirms them before anything is deleted.")
         return _next(out, steps)
 
+    def storage(self, root: str | None = None) -> dict[str, Any]:
+        with self._open() as conn:
+            out = shape.storage(conn, host=self._hosts(conn),
+                                root=self._root(conn, root), names=self.expose_names)
+        steps = ["`suggest` flags data that is cold on the fastest pool, once "
+                 "access times are known."]
+        if not out["pools"]:
+            steps = ["Nothing here comes from a Storage Scale listing: see `cdm policy` "
+                     "and `cdm import --policy`."]
+        elif not self.expose_names:
+            steps.append("Fileset names are withheld; the user can see them with "
+                         "`cdm storage` in a terminal.")
+        return _next(out, steps)
+
     def guide(self, root: str | None = None) -> dict[str, Any]:
         with self._open() as conn:
             out = guide_mod.guide(conn, host=self._hosts(conn), root=self._root(conn, root))
@@ -309,6 +331,7 @@ def find(self, name: str | None = None, iname: str | None = None,
              smaller_than: str | None = None, modified_after: str | None = None,
              modified_before: str | None = None, accessed_after: str | None = None,
              accessed_before: str | None = None, unopened: bool = False,
+             fileset: str | None = None, pool: str | None = None,
              root: str | None = None, order: str = "size",
              limit: int = 50) -> dict[str, Any]:
         if kind not in (None, "file", "dir", "link"):
@@ -330,7 +353,7 @@ def find(self, name: str | None = None, iname: str | None = None,
                                 if accessed_after else None),
                 accessed_before=(query.parse_when(accessed_before)
                                  if accessed_before else None),
-                unopened=bool(unopened),
+                unopened=bool(unopened), fileset=fileset, pool=pool,
                 order=order, limit=limit + 1)
         return _next({"results": [_file_row(r) for r in rows[:limit]],
                       "truncated": len(rows) > limit},

docs/adr/0007-pool-names-are-shape-fileset-names-are-names.md

@@ -0,0 +1,30 @@
+# 0007 — Storage pool names are shape; fileset names are names
+
+**Context.** Imported Storage Scale listings carry each file's storage pool and
+fileset, and the MCP server can now report totals by both. ADR 0002 decided
+that by default the server returns the *shape* of the data — sizes, ages, file
+types — and no name below a root, because with a cloud-hosted model everything
+it returns goes to the provider. The Phase 2 plan assumed pool and fileset
+names were both shape, being chosen by an administrator rather than by users.
+On a real cluster that held for pools and not for filesets: the fileset in use
+was named after its user. Fileset names are commonly a person's name, a
+project's, or a customer's codename — exactly what ADR 0002 keeps back.
+
+**Decision.** Storage pool names are shape and are returned by default: they
+name storage tiers (`system`, `data1`, `ssd`) and say where data sits, not whose
+it is. Fileset names are names: without `--expose-names` the MCP `storage` tool
+reports filesets by rank of size (`fileset #1` is the largest), with their
+file counts and bytes, and never their names; with `--expose-names` it names
+them, and `find` (a name tool) can filter on them. The command line always
+shows both, since its user owns the index.
+
+**Why.** Ranking keeps the useful shape — how many filesets, how lopsided —
+while dropping the part that identifies people. Pools carry no such risk and
+are the point of tiering advice (the pool-aware suggestion in #14), so hiding
+them would cost usefulness for no privacy gain. This follows ADR 0002's rule
+that the default must never be loosened silently: a fileset name reaching the
+model is something a user opts into, never something that starts happening.
+
+**Consequences.** Anything new that reports filesets through a shape tool must
+rank, not name, them unless names are exposed. If a site's pool names turn out
+to be sensitive, that needs revisiting here, not a quiet exception in code.

man/cdm.1

@@ -96,6 +96,15 @@ for an administrator to run. See
 Show the getting\-started checklist and what to do next. See
 .BR GUIDE .
 .TP
+.BI "storage " [ options ]
+Files and bytes by Storage Scale storage pool and by fileset, from imported
+listings, with names. See
+.BR "STORAGE SCALE" .
+Takes
+.BI \-\-root " path"
+and
+.BR \-\-json .
+.TP
 .BI "stat " path
 Print everything the index records about one path, including whether its hash
 has gone stale and, where they are known, when the file was last read and
@@ -195,6 +204,11 @@ either. See
 Only files known not to have been opened since they last changed. See
 .BR "ACCESS TIMES" .
 .TP
+.BI \-\-fileset " name" ", " \-\-pool " name"
+Only files in this Storage Scale fileset, or on this storage pool. Only
+imported listings have either. See
+.BR "STORAGE SCALE" .
+.TP
 .BI \-\-root " path"
 Restrict results to one root, including any roots nested inside it.
 .TP
@@ -598,6 +612,16 @@ does not recognise. The policy engine reports times to the microsecond.
 .B "cdm doctor"
 shows the setting for each imported root. See
 .BR "ACCESS TIMES" .
+.PP
+Each file's fileset and storage pool are recorded.
+.B "cdm storage"
+totals them,
+.B "cdm find \-\-fileset"
+and
+.B \-\-pool
+filter on them, and the MCP
+.B storage
+tool reports pools by name and filesets by rank unless names are exposed.
 .TP
 .BI \-\-policy " file" .raw
 The listing. Required.
@@ -633,11 +657,17 @@ Out of the box the server offers only
 .BR age_histogram ,
 .BR extensions ,
 .BR duplicates_summary ,
+.BR storage ,
 .B suggest
 and
 .BR guide :
 totals, histograms, file types and ranked suggestions, with no file or
-directory name below a root. With a cloud\-hosted model every answer goes to
+directory name below a root.
+.B storage
+names storage pools, which name tiers rather than people, but reports
+filesets by rank of size (\(lqfileset #1\(rq is the largest) unless names are
+exposed, because fileset names are often a person's or a project's.
+With a cloud\-hosted model every answer goes to
 its provider, and filenames are often sensitive on their own. Extensions are
 shown only when they look like a file type \(em short, alphanumeric, and shared
 by at least two files \(em so a suffix cannot easily carry part of a name.