Make the tutorial notebook self-checking and host-configurable#4
Closed
estebanzimanyi wants to merge 21 commits into
Closed
Make the tutorial notebook self-checking and host-configurable#4estebanzimanyi wants to merge 21 commits into
estebanzimanyi wants to merge 21 commits into
Conversation
A compiled, pooled (pgxpool) parallel-path counterpart to the PyMEOS server. All temporal work and (de)serialization run in MobilityDB (asMFJSON read path assembled in SQL and streamed; tgeompointFromMFJSON write path); the tier holds no MEOS (no cgo, no PyMEOS) and only applies the OGC<->MobilityDB convention mapping (crs URN<->EPSG name, interpolation Stepwise<->Step). Endpoints: landing, conformance, collections, items GET/POST, single item GET.
Controlled head-to-head over the same MobilityDB (identical SQL): Go (net/http + pgxpool) vs a threaded Python http.server + psycopg2 pool. Tier-bound endpoints favour Go ~47-64x; the heavy asMFJSON endpoint converges on the DB but the Python tier sheds load under concurrency while Go stays reliable. Includes run_compare.sh (retargetable at the production PyMEOS server) and the DB-free /health tier probe.
Rework the items path to write the FeatureCollection row-by-row from a server-side cursor (bounded memory: a 393 MB export holds ~18 MB RSS), add keyset pagination with OGC next links (no OFFSET), push bbox and datetime filters to the MobilityDB GiST index via &&, clip with atTime for subTrajectory (gap-aware), and add a streaming NDJSON /export feed the lakehouse (DuckDB / MobilityDuck / Spark) ingests directly. Add graceful shutdown, env-configurable pool/port/limits, and /health. All temporal work stays in MobilityDB (asMFJSON / tgeompointFromMFJSON); no MEOS in the tier.
GET /collections/{cid}/export?format=parquet emits the columnar lakehouse
form: trajectory WKB plus xmin..tmax scalar columns so a lake consumer prunes
row groups by space and time before decoding geometry. Streamed from a
server-side cursor and flushed one row group per MFAPI_PARQUET_ROWGROUP rows,
so peak memory is set by the row group, not the table size (rowgroup=128 holds
~148 MB vs ~345 MB at 1024 on the same 2,833-trip set). The geometry
materialises once per row so the sidecar accessors do not re-clip.
…properties
Adds the moving-feature sub-resources of OGC API - Moving Features Part 1:
GET .../tgsequence (the temporal geometry as MF-JSON, datetime/leaf clipped),
GET .../tgsequence/{tgid}/{distance|velocity} and GET .../tproperties[/{name}]
for the derived temporal properties velocity (speed), distance (cumulative
length) and heading (azimuth), each an exact MobilityDB function reshaped into
an OGC temporalProperty whose valueSequence carries the function's true
interpolation. velocity and heading are reported piecewise-constant (Stepwise)
because position is linearly interpolated, so acceleration returns 501: the
derivative of a piecewise-constant speed is zero within segments and undefined
at the vertices, and resampling speed to linear would fabricate it. Declares the
three Moving Features conformance classes plus OGC Common core and collections,
serves a minimal OpenAPI definition at /api (service-desc), and accepts the
canonical lowercase subtrajectory parameter and the leaf instant selector.
…sub-trajectory
PUT .../items/{fid} replaces a moving feature's temporal geometry and name,
DELETE .../items/{fid} removes it (204), and POST .../items/{fid}/tgsequence
appends a temporally-disjoint sub-trajectory via MobilityDB merge, which rejects
time overlap (returned as 409). Writes to the derived temporal properties and to
an individual temporal primitive return 501 with the reason: the properties are
computed from the trajectory and the feature carries a single inseparable
temporal geometry, so both are modified through the geometry itself.
MobilityAPIV5.pptx presents the thin compiled OGC API - Moving Features server over MobilityDB as the production solution: a 25-slide full walkthrough of the real endpoints and responses (collections, streamed items, bbox/datetime/ subtrajectory/leaf, writes with 409 on overlap, tgsequence, derived measures with acceleration returning 501, temporal properties, conformance, and the streaming NDJSON/Parquet lakehouse export), with no Python tier and faithful piecewise-constant velocity semantics. MobilityAPIV4.pptx is retained unchanged for presenting the PyMEOS-based solution.
Drops unearned adjectives from V5 (no 'production', 'complete', 'built for very large databases', 'honest'): the title reads 'An OGC API - Moving Features server over MobilityDB', the surface slide 'The implemented endpoint surface', the scale slide 'Bounded memory at any result size', and the recap notes acceleration returns 501 as 'not derivable'. Adds MobilityAPIV4.pdf (29 pp) and MobilityAPIV5.pdf (25 pp) rendered with LibreOffice for viewing without opening PowerPoint.
Bullets now read 'Keyword: description' across every slide (one change in the shared bullets helper). The OGC Associate Member badge on the title slide has its white corners knocked out to transparent (border-connected white only, so the white OGC panel and Associate Member text stay solid), removing the white squares against the navy background. PDF re-rendered.
…s, future-work slide Bakes the navy slide colour into the OGC badge corners (opaque), so the white corners are gone in both PowerPoint and the LibreOffice PDF. Replaces the three reused figures with real-data plots in one plain style: an overview of the 2026 AIS network, a bounding-box query showing the trajectories that cross the box (teal) over other traffic (grey) with the box in red, and a subtrajectory plot showing the one-hour clipped portions (green) over full trajectories (grey). Captions now sit directly under each image (no gap). Adds a future-work slide generalising the thin-tier pattern across MEOS engines: MobilityDB today, with MobilityDuck, MobilitySpark and MobilityFlink reusing the same MF-JSON/WKB tier.
…m, basemaps Splits the strict OGC surface from beyond-standard features: the endpoint-surface slide lists only standard resources with full template paths and OGC parameter names; a new Additional features slide holds the lakehouse export and the in-place replace (PUT), banner-marked as excluded from conformsTo. Walkthrough requests use full concrete paths instead of leading ellipses. Slide 2 becomes a three-tier diagram (client boxes, MobilityAPI, MobilityDB) with the link languages labelled (HTTP / MF-JSON, then SQL / pgx). The overview, bbox and subtrajectory figures regain an OSM XYZ-tile basemap (data reprojected to Web Mercator). Fixes the overflowing INSERT, the asMFJSON wording (MF-JSON is the format), doubled colons after bold keywords, and the ships table box title.
Features (GET items, GET items/{id}) now carry crs, trs, bbox, time and a self
link in addition to id/type/properties/temporalGeometry; the single-feature
response also includes the GeoJSON geometry. bbox and time come from the inline
STBOX, crs from the collection SRID (rewritten to the OGC URN), trs is the
Gregorian calendar link. Collections now carry crs and a self/items/enclosure
link set, and a single collection includes its spatial and temporal extent
computed from extent(trip).
…, polish Backing-model slide now lists every OGC field for the Collection and MovingFeature resources with its provenance (stored column, derived from the tgeompoint, from the registry, constant, or generated), in a clean two-column layout. The scale and extensions slides merge into one Scale and the lakehouse slide that shows the bounded-memory figure once, the standard items resource on top, and the export and in-place replace demarcated as not in conformsTo. The architecture NO MEOS panel becomes short bullets. Future-work tags read Planned with no trailing periods, and the acceleration request uses its full path.
…p; uniform diagram borders The collection, items and single-feature panels now show the full standard responses (collection crs/extent/links; feature crs/trs/bbox/time/geometry/ links) consistent with the backing-model slide and the server. Reflows the CRS object and widens the single-feature panel so no line wraps. The slide-2 tier diagram uses one border width and corner radius across the client, API and database boxes.
All client, API and database boxes (and the container) share one border colour, width and corner radius, and the inherited drop shadow is removed so the boxes read as a single uniform style.
…hes the sides The panel ended well below the last box, leaving an oversized gap under MobilityDB; shrink it so the margin under the bottom box equals the left and right margins.
…ident SQL-vs-REST caption The bottom panel on the architecture slide reads No MEOS in the tier, matching the sentence-case column sub-headers beside it instead of an all-caps eyebrow. Removes the SQL-vs-REST caption, whose point is already carried by the slide kicker, title, and the side-by-side panels.
…cing via one helper Adds a notebox helper (panel + label + bullets, matching the No MEOS in the tier box) and routes the in-content paragraph notes through it: the endpoint-surface notes, the backing-model provenance note, and the temporal-geometry note. Adds an ep helper for the method-pill-plus-descriptor row so the descriptor sits below the pill with one consistent offset everywhere (writes delete, the export and replace rows), fixing the descriptor/pill overlap globally rather than per instance.
Transposes the OGC API - Moving Features walkthrough to the Go MobilityAPI: plain HTTP against the server (no MEOS in the notebook), adapted to the Go responses. Covers collections (crs/extent/links), streamed keyset-paged items, a single feature with the full standard fields, the bbox and subtrajectory/datetime filters, the temporal-geometry sequence, derived velocity/distance (acceleration returns 501, not derivable for piecewise-linear motion), temporal properties with the leaf selector, the POST/PUT/DELETE write lifecycle, and the NDJSON/Parquet export. Maps use a self-contained OSM XYZ-tile basemap (matplotlib + pyproj + Pillow), so geopandas/contextily are not required.
The notebook is a plain-HTTP client against the Go tier, so a missing server or unset Python environment surfaced as a deep ConnectionError or ImportError. The first cell now installs its packages (requests, numpy, matplotlib, pyproj, pillow), HOST and the collection id read from the MFAPI_HOST / MFAPI_COLLECTION environment variables, and a preflight cell checks the tier is reachable and the ships collection is present — printing what to start otherwise — before the walkthrough runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Go tier tutorial is a plain-HTTP client, so a missing server or an unset Python environment surfaced as a deep ConnectionError or ImportError partway through. The notebook's first cell installs its packages (requests, numpy, matplotlib, pyproj, pillow), HOST and the collection id read from the MFAPI_HOST / MFAPI_COLLECTION environment variables, and a preflight cell checks the tier is reachable and the ships collection is present — printing what to start otherwise — before the walkthrough runs. The tutorial README matches the new behaviour.