Skip to content

ARCHITECTURE_DEEP_DIVE: The Smart De-duplicated Poller ​

This document explains the technical implementation and reasoning behind the core scanning engine of camply.

Python package boundaries ​

The Python backend is one camply distribution with internal camply.core, camply.providers, camply.db, camply.server, and camply.worker subpackages. Base includes providers and SQLite; server, distributed, postgres, and sentry independently add runtime dependencies. Server and worker application bootstrap must remain independent. The foreground CLI discovers locations and monitors live availability without application state or service bootstraps. Existing Celery JSON task names, beat schedule names, and database revisions are preserved through the module migration.

The enforced layer order is cli | server | worker → core → providers → db → __about__. camply.core.search normalizes local date windows, constructs exact-length stays from provider observations, and runs cancellable foreground checks. It reuses camply.db.availability.matching_available_dates, the same consecutive-night matcher used by saved web scans. Database adapters retain subscriber filtering and persistence; the CLI never creates subscribers, targets or notification deliveries. The core imports no optional service dependencies.

The CLI owns one provider client across discovery and monitoring, closes it on success, failure or cancellation, and awaits checks sequentially so the same target cannot overlap. Requests use existing provider pacing, retries and circuit breakers. An internal 60-second wait separates successful cycles; this provisional implementation default is not the deferred public scheduling/mode contract. Provider failures remain errors rather than empty availability. CLI output is console-only. See the CLI guide and feature 026.

CLI organization ​

camply.cli.main owns the root command and lazily loads each command module. serve.py, db.py, providers.py, recreation_areas.py, campgrounds.py and campsites.py own their respective commands; healthcheck.py owns the nested serving health checks. Serving helpers remain with the serving group.

options.py defines reusable Click options, strict calendar-date parsing and debug precedence. It imports no providers or application configuration, keeping root imports and version reporting lightweight. Shared discovery validation and target resolution live in discovery.py; terminal rendering lives in output.py; async execution, diagnostic sanitization and signal cleanup live in execution.py. runtime.py checks optional runtime dependencies before serving side effects.

Command modules compose these helpers without importing the root router. New commands should reuse the option decorators rather than duplicate their defaults, repeatability or help text. User-visible options and provider precedence remain defined by the initial command contract.

Optional local execution ​

camply.worker.engine contains shared asynchronous discovery, scanning/filtering, catalog, and notification functions. Distributed Celery adapters keep locks, retries, and metrics; camply.worker.runtime runs the same engine in the API lifespan for standalone. Discovery begins immediately, pages 100 targets, honors subscriber eligibility and cooldown, and excludes queued/running/deferred targets. One scan consumer retries exceptions after 60/120/240 seconds, then records a failure and waits for the next heartbeat cycle. Separate consumers handle catalog work and durable delivery without creating one task per target.

Manual catalog sync coalesces under one job ID; daily refresh runs at midnight UTC. Each provider has an independent outcome, and later providers run after a failure. Recreation.gov archive parsing is offloaded from the event loop. Existing provider commit boundaries are preserved. Shutdown stops discovery, allows up to five seconds to drain scan/catalog queues, then cancels and reaps remaining loops. Any unexpectedly terminated supervised loop makes readiness unhealthy.

Standalone result replacement, successful-check time, and matching notification intents commit in one transaction. The delivery consumer revalidates current subscriptions, credentials, dates, and filters before sending; see notification recovery. Local queues/history are bounded; only pending delivery and application state persist. This runtime supports one process, one host, and a writable local SQLite file. Distributed scheduling and deployment remain available for larger instances.

🎯 The Problem: API Bans & Inefficiency ​

In a multi-user system, multiple users often want to watch the same popular campgrounds (e.g., Lower Pines in Yosemite) for the same dates.

  • Legacy Approach: 10 users = 10 API calls every minute. Result: IP ban from Recreation.gov.
  • Smart Approach: 10 users = 1 API call every minute. Result: High performance and API safety.

🏗️ The Data Flow (User to Alert) ​

mermaid
graph TD
    A[User in React UI] -->|POST /scans| B(FastAPI Backend)
    B -->|Calculate Hash| C{Unique Target Exists?}
    C -->|No| D[Create New UniqueTarget]
    C -->|Yes| E[Link UserScan to Existing]

    F[Celery Beat Scheduler] -->|Every 5m (default)| G[Identify Targets to Check]
    G -->|Enqueue| H[camply.search.check_campsite_availability Task]

    H -->|Hit Provider API| I[Raw Results]
    I -->|Update Cache| J[scan_results Table]

    J -->|New Opening Found| K[Fan-out: Match UserScans]
    K -->|User A matches| L[Enqueue camply.notifications.send_pushover_notification]
    K -->|User B matches| M[Enqueue camply.notifications.send_pushover_notification]

    L -->|Pushover API| N[User Mobile Alert]

🔄 The UniqueTarget State Machine ​

A UniqueTarget represents a background polling job. Its lifecycle is:

  1. Initialized: Created when the first user subscribes to a park/date range.
  2. Active: is_active=True. Polled by the worker as long as at least one linked UserScan is active.
  3. Cooldown: After a check, last_checked_at is updated. It won't be checked again until the provider's rate_limit_interval has passed.
  4. Dormant: If all users delete their scans for this target, is_active becomes False. The record remains for history but the worker skips it.

📣 The "Fan-out" Notification Logic ​

We separate Checking from Notifying to ensure high throughput.

  1. The Poller is "Blind": It just wants to know "What sites are open for Park X between Date A and B?".
  2. Standardization: Raw results are converted into CampsiteDTOs.
  3. Matching: The worker queries the user_scans table for that target_id.
  4. Filtering:
    • User A only wants Electric sites.
    • User B wants any site but for 3+ nights.
    • The worker checks the CampsiteDTO list against these filters.
  5. Individual Alerts: A user is only alerted if their specific criteria are met, even if other sites are open.

🔒 Concurrency & Locking ​

To prevent race conditions (two workers checking the same park simultaneously):

  • We use Valkey Distributed Locks.
  • Before a camply.search.check_campsite_availability (check_target_availability) task starts, it attempts to acquire a lock: lock:target_id.
  • If the lock is held, the task exits immediately.