Replace pull-config with TypeScript agent by cmyers-mieweb · Pull Request #407 · mieweb/opensource-server

cmyers-mieweb · 2026-07-15T14:45:20Z

Resolves: #357

Replaces the cron-driven bash pull-config system with a TypeScript Node.js agent that runs every 30 seconds via a systemd timer.

Key changes:

New agent/ package: TypeScript sources (config, state, api, system, apply), EJS templates moved from create-a-container/views/, systemd service+timer units
Manager: new Agents DB model/migration, POST /api/v1/agents check-in endpoint with ETag/304 support, GET /api/v1/agents admin list, utils/agent-config.js config snapshot builder
Client: AgentsListPage with service health badges, sidebar nav entry, query/types
Removes: pull-config bash scripts, EJS template rendering from manager (templates router, ejs dependency, views engine setup)
Updates docs, Dockerfiles, Makefiles, and OpenAPI spec

Resolves: #357 Replaces the cron-driven bash pull-config system with a TypeScript Node.js agent that runs every 30 seconds via a systemd timer. Key changes: - New `agent/` package: TypeScript sources (config, state, api, system, apply), EJS templates moved from create-a-container/views/, systemd service+timer units - Manager: new `Agents` DB model/migration, `POST /api/v1/agents` check-in endpoint with ETag/304 support, `GET /api/v1/agents` admin list, `utils/agent-config.js` config snapshot builder - Client: AgentsListPage with service health badges, sidebar nav entry, query/types - Removes: pull-config bash scripts, EJS template rendering from manager (templates router, ejs dependency, views engine setup) - Updates docs, Dockerfiles, Makefiles, and OpenAPI spec

Copilot

Pull request overview

Replaces the legacy cron-driven pull-config bash system with a TypeScript/Node-based site agent that checks in to the manager every 30 seconds (systemd timer), receives a JSON config snapshot (ETag/304), and applies nginx/dnsmasq configuration locally. The manager gains an Agents model + API endpoints, and the web client gains an admin Agents status page.

Changes:

Introduces new agent/ package (TypeScript agent, local EJS templates, systemd service/timer, packaging).
Adds manager-side agent check-in + status listing (/api/v1/agents), config snapshot builder + ETag, and Agents DB model/migration.
Removes manager-side EJS template rendering routes and deletes the old pull-config/ cron/bash implementation; updates docs and images accordingly.

Reviewed changes

Copilot reviewed 54 out of 62 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
README.md	Updates top-level component list to replace `pull-config/` with `agent/`.
pull-config/README.md	Removes legacy pull-config docs stub (component retired).
pull-config/etc/pull-config.d/nginx	Removes legacy nginx pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-servers	Removes legacy dnsmasq forwarders pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-hosts	Removes legacy dnsmasq hosts pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-dhcp-opts	Removes legacy dnsmasq DHCP opts pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-dhcp-hosts	Removes legacy dnsmasq DHCP hosts pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-conf	Removes legacy dnsmasq main config pull-config instance script.
pull-config/etc/cron.d/pull-config	Removes cron schedule for pull-config run-parts.
pull-config/bin/pull-config	Removes legacy bash pull-config engine.
pull-config/.gitignore	Removes pull-config packaging artifact ignores (component retired).
mie-opensource-landing/zensical.toml	Renames docs nav entry from Pull Config to Agent.
mie-opensource-landing/docs/developers/release-pipeline.md	Updates release pipeline docs to build/package `agent` instead of `pull-config`.
mie-opensource-landing/docs/developers/pull-config.md	Removes legacy developer reference for pull-config (replaced by agent docs).
mie-opensource-landing/docs/developers/docker-images.md	Updates agent image description to reflect check-in agent.
mie-opensource-landing/docs/developers/database-schema.md	Extends ER diagram and schema docs to include new `Agents` entity.
mie-opensource-landing/docs/developers/agent.md	Adds new developer reference for agent architecture and protocol.
mie-opensource-landing/docs/admins/deploying-agents.md	Updates admin deployment guide from pull-config verification to agent check-in verification.
Makefile	Swaps top-level component list from `pull-config` to `agent`.
images/manager/Dockerfile	Updates comments around default env to refer to the new agent behavior.
images/agent/Dockerfile	Updates comments for dependencies/config to refer to the new agent.
create-a-container/utils/agent-config.js	Adds manager-side config snapshot builder + strong ETag computation.
create-a-container/server.js	Removes view engine + templates router wiring; simplifies SPA fallback routing.
create-a-container/routers/templates.js	Removes legacy template-rendering routes for nginx/dnsmasq.
create-a-container/routers/api/v1/index.js	Mounts new `/agents` router before CSRF guard.
create-a-container/routers/api/v1/agents.js	Adds agent check-in endpoint + admin listing with “online” computed status.
create-a-container/package.json	Removes `ejs` dependency from manager (no longer renders templates).
create-a-container/package-lock.json	Removes `ejs` transitive deps and updates lock metadata.
create-a-container/openapi.v1.yaml	Documents new `/agents` endpoints under API v1.
create-a-container/models/agent.js	Adds Sequelize `Agent` model with `(siteId, hostname)` uniqueness.
create-a-container/migrations/20260714120000-create-agents.js	Adds `Agents` table and unique composite index migration.
create-a-container/middlewares/index.js	Extracts and exports `isLocalhostRequest` helper used by agent auth.
create-a-container/Makefile	Stops packaging `views/` since manager no longer renders templates.
create-a-container/client/src/pages/agents/AgentsListPage.tsx	Adds admin Agents list page with online/offline and last check-in display.
create-a-container/client/src/pages/agents/AgentServiceBadges.tsx	Adds per-service status badges for agent-reported service states.
create-a-container/client/src/lib/types.ts	Adds typed Agent/AgentServiceStatus models to match new API responses.
create-a-container/client/src/lib/queries.ts	Adds React Query key + `listAgents()` query to call `/api/v1/agents`.
create-a-container/client/src/app/Sidebar.tsx	Adds admin-only “Agents” navigation entry.
create-a-container/client/src/app/router.tsx	Registers `/agents` route to render Agents list page.
agent/tsconfig.json	Adds TypeScript compiler configuration for the agent package.
agent/templates/nginx.conf.ejs	Updates nginx template to consume the new JSON snapshot shape.
agent/templates/dnsmasq/servers.ejs	Adds agent-side dnsmasq forwarders template.
agent/templates/dnsmasq/hosts.ejs	Adds agent-side dnsmasq hosts template.
agent/templates/dnsmasq/dhcp-opts.ejs	Adds agent-side dnsmasq DHCP options template.
agent/templates/dnsmasq/dhcp-hosts.ejs	Adds agent-side dnsmasq DHCP hosts template.
agent/templates/dnsmasq/conf.ejs	Adds agent-side dnsmasq main config template.
agent/src/types.ts	Adds shared request/response/config snapshot types for agent logic.
agent/src/system.ts	Adds hostname/IP/systemd ActiveState collection helpers.
agent/src/state.ts	Adds persisted state (ETag + per-service lastApply) storage helpers.
agent/src/index.ts	Adds agent oneshot main loop: check-in, apply config, repeat until 304.
agent/src/config.ts	Adds environment-based agent config loader (SITE_ID/MANAGER_URL/etc).
agent/src/apply.ts	Adds apply/test/reload orchestration for nginx and dnsmasq configs.
agent/src/api.ts	Adds HTTP client for POST `/api/v1/agents` with If-None-Match + auth.
agent/README.md	Adds agent package overview and links to admin/dev docs.
agent/package.json	Adds agent package manifest with build/start scripts and dependencies.
agent/package-lock.json	Adds agent lockfile for reproducible builds.
agent/Makefile	Adds build/package install flow (npm ci, tsc build, stage systemd units).
agent/contrib/systemd/opensource-agent.timer	Adds 30-second systemd timer unit.
agent/contrib/systemd/opensource-agent.service	Adds oneshot systemd service unit for the agent.
agent/contrib/postinstall.sh	Enables timer on package install and restarts when systemd is running.
agent/.gitignore	Ignores node/TS build and packaging artifacts for agent component.
agent/.fpm	Updates packaging metadata/dependencies and adds postinstall hook for systemd timer.

Files not reviewed (2)

agent/package-lock.json: Generated file
create-a-container/package-lock.json: Generated file

+    async render(config) {
+      if (!config.site) return null;
+      const data = { site: config.site };
+      return [


+  internalDomain: string;
+  dhcpRange: string;
+  subnetMask: string;
+  gateway: string;
+  dnsForwarders: string;


+  for (let pass = 0; pass < MAX_PASSES; pass++) {
+    const result = await checkin(cfg, buildCheckinBody(cfg, state), state.etag);
+    if (result.notModified) return;
+
+    for (const svc of services) {
+      state.lastApply[svc.unit] = await applyService(svc, result.config);
+    }
+
+    // The ETag is saved even after a failed apply: a rejected config won't
+    // fix itself without a server-side change (which changes the ETag), and
+    // the failure has been reported via lastApply.
+    state.etag = result.etag;
+    saveState(cfg.stateDir, state);
+  }
+}


runleveldev · 2026-07-16T13:55:49Z

+    siteId,
+    managerUrl: managerUrl.replace(/\/+$/, ''),
+    apiKey: env.API_KEY || undefined,
+    stateDir: env.STATE_DIR || '/var/lib/opensource-agent',


Use STATE_DIRECTORY and set StateDirectory=opensource-agent in the service file to let SystemD automatically manage it's lifecycle. This would also remove the need to make the directory in the Makefile install target.

runleveldev · 2026-07-16T13:57:40Z

+  try {
+    const raw = JSON.parse(fs.readFileSync(stateFile(stateDir), 'utf8'));
+    return { lastApply: {}, ...raw };
+  } catch {


I don't like unqualified catch. Only catch known errors (file missing, failed to parse, etc) and throw anything unexpected.

runleveldev · 2026-07-16T13:58:44Z

+  }
+}
+
+export function saveState(stateDir: string, state: AgentState): void {


Just a thought: why don't we make a State class that has load and save methods rather than handling this procedurally?

runleveldev · 2026-07-16T14:00:42Z

+/** systemd ActiveState for a unit (active, inactive, failed, ...). */
+export function getServiceState(unit: string): string {
+  try {
+    const out = execFileSync('systemctl', ['show', '--property=ActiveState', '--value', unit], {


Do not exec subprocesses: SystemD supposts a Unix-socket based API (D-Bus) which should be used for these kinds of operations.

runleveldev · 2026-07-16T14:14:03Z

+    },
+    test: ['nginx', '-t'],
+    reload() {
+      run(['systemctl', 'reload-or-restart', 'nginx']);


I won't complain about the test command, but the reload shoudl use the SystemD DBus API rather than subcommands.

runleveldev · 2026-07-16T14:20:35Z

+// The manager's own agent checks in over localhost without credentials
+// (bootstrap: no site, no API key exist yet). Remote agents authenticate
+// with an admin API key.
+function agentAuth(req, res, next) {
+  if (isLocalhostRequest(req)) return next();
+  return apiAuth(req, res, (err) => {
+    if (err) return next(err);
+    return apiAdmin(req, res, next);
+  });
+}


Was the requireLocalhostOrAdmin middleware not sufficient?

runleveldev · 2026-07-16T14:23:00Z

+  const etag = computeConfigEtag(config);
+  res.set('ETag', etag);
+  if (req.get('If-None-Match') === etag) {
+    return res.status(304).end();
+  }


Fact check me on thin, but I'm pretty sure Express already handles ETag logic. I was using it for the original pull-config setup but that might only work with rendering templates.

runleveldev · 2026-07-16T14:25:07Z

+    ipv4Address: a.ipv4Address,
+    services: a.services,
+    lastCheckinAt: a.lastCheckinAt,
+    online: !!a.lastCheckinAt && now - new Date(a.lastCheckinAt).getTime() <= ONLINE_WINDOW_MS,


I do want the time-since-checkin to be computed serverside to avoid client-side clock drift in the UI, but I think this field should just be that delta rather than a boolean is-online and the UI (or even just the human admin) could decide whether there's an issue based on that.

runleveldev · 2026-07-16T14:26:54Z

-/etc/pull-config.d/dnsmasq-dhcp-opts
-/etc/pull-config.d/dnsmasq-servers
+# Run a check-in manually
+node /opt/opensource-server/agent/dist/index.js


systemctl start opensource-agent to ensure the systemd environment is respected.

runleveldev · 2026-07-16T14:27:43Z

+```
+agent/
+├── src/                     # TypeScript sources (compiled to dist/)
+│   ├── index.ts             # Oneshot entry point: check-in loop
+│   ├── config.ts            # Environment configuration
+│   ├── state.ts             # Persisted ETag + last apply results
+│   ├── system.ts            # Hostname/IP/systemd state collection
+│   ├── api.ts               # Check-in HTTP client
+│   └── apply.ts             # Managed services: render/test/apply/reload
+├── templates/               # EJS templates rendered locally
+│   ├── nginx.conf.ejs
+│   └── dnsmasq/             # conf, dhcp-hosts, hosts, dhcp-opts, servers
+├── contrib/
+│   ├── systemd/             # opensource-agent.service + .timer (30s)
+│   └── postinstall.sh       # Enables the timer on package install
+└── Makefile                 # Builds the opensource-agent package (see Release Pipeline)
+```


Don't do this, sucks to keep up-to-date on changes.

cmyers-mieweb requested a review from runleveldev July 15, 2026 14:45

cmyers-mieweb mentioned this pull request Jul 15, 2026

Agent check-in #357

Open

runleveldev requested a review from Copilot July 16, 2026 13:51

Copilot started reviewing on behalf of runleveldev July 16, 2026 13:52 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

runleveldev requested changes Jul 16, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replace pull-config with TypeScript agent#407

Replace pull-config with TypeScript agent#407
cmyers-mieweb wants to merge 1 commit into
mainfrom
cmyers_issue357

cmyers-mieweb commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

runleveldev Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

cmyers-mieweb commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants