ServiceLink EXOS | Analytics

TECHNICAL REFERENCE

System Architecture

Full technical breakdown of the observability pipeline. Component inventory, data flow, deployment topology, security posture, and schema reference.

On this page

Data Flow Component Stack Deployment Security & Compliance Data Schema Capacity & Performance

Data Flow

End-to-end telemetry pipeline from browser instrumentation to queryable storage. All arrows represent real, production data paths.

flowchart LR
    subgraph Client["Client (Browser)"]
        SDK["HyperDX Browser SDK\n@hyperdx/browser"]
    end

    subgraph Tunnel["Cloudflare"]
        CF["Cloudflared Tunnel\n(ingress)"]
    end

    subgraph SelfHost["Self-hosted VM (Cloudflare-tunneled)"]
        Collector["OTEL Collector\n:4318 OTLP/HTTP"]
        HDX["HyperDX App\n:8080"]
        GF["Grafana\n:3000"]
    end

    subgraph Storage["ClickHouse Cloud (GCP us-central1, managed)"]
        CH[("otel_traces\notel_logs\notel_metrics")]
    end

    subgraph Consumers["Query Interfaces"]
        Vercel["Vercel Portal + API\n(serverless functions)"]
        SQL["SQL Console\n(ad-hoc)"]
    end

    SDK -->|"OTLP/HTTP\ntraces, logs, RUM"| CF
    CF -->|"tunnel"| Collector
    Collector -->|"INSERT (batch)"| CH
    HDX -->|"SELECT"| CH
    GF -->|"SELECT"| CH
    Vercel -->|"SELECT :8443"| CH
    SQL -->|"SELECT"| CH

    style SDK fill:#3b82f6,color:#fff,stroke:#60a5fa
    style CF fill:#f97316,color:#fff,stroke:#fb923c
    style Collector fill:#06b6d4,color:#fff,stroke:#22d3ee
    style HDX fill:#7c3aed,color:#fff,stroke:#a78bfa
    style GF fill:#ea580c,color:#fff,stroke:#f97316
    style CH fill:#ca8a04,color:#fff,stroke:#facc15
    style Vercel fill:#0ea5e9,color:#fff,stroke:#38bdf8
    style SQL fill:#64748b,color:#fff,stroke:#94a3b8

Capture

Browser SDK auto-instruments clicks, navigation, network requests, console output, and DOM snapshots. Emits standard OTLP over HTTP.

Ingest

OTEL Collector receives, batches, and inserts telemetry into ClickHouse. No intermediate queue or broker required.

Query

Three independent consumers read from the same tables. No data duplication, no sync lag. One source of truth.

Not just browser events — every EXOS API call is already traced.

EXOS Analytics captures OTEL traces from server-side APIs, not just browser events. Every EXOS API call is already instrumented with OpenTelemetry. EXOS Analytics turns that existing telemetry into product insight — no new client-side instrumentation required.

Server-side event tracking is easier to maintain, more reliable, and captures what matters most: the actual business logic executing behind the UI. Browser events show what users clicked. Server traces show what actually happened.

Component Stack

Every component in the stack, its role, and its licensing. No proprietary lock-in on the data path.

Component Role License Deployment
HyperDX Browser SDK Client-side auto-instrumentation. Captures clicks, navigation, network, console, and DOM for session replay. Emits OTLP. MIT NPM / CDN script tag
OpenTelemetry Collector Receives OTLP/HTTP on :4318. Batches and exports to ClickHouse via the clickhouseexporter. Apache 2.0 Docker container
ClickHouse Cloud Columnar OLAP database. Stores all traces, logs, and metrics. Handles both INSERT (write) and SELECT (query) workloads. Apache 2.0 (core) Managed service (GCP, us-central1)
HyperDX App Session replay UI, trace viewer, log explorer. Reads from ClickHouse via SQL. Apache 2.0 Docker container
Grafana Dashboard and alerting platform. Connects to ClickHouse as a data source for operational dashboards. AGPLv3 Docker container
Cloudflared Ingress tunnel. Routes external HTTPS traffic to internal Docker services without exposing ports. Proprietary Docker container
Vercel Serves the static portal (vanilla HTML) and the serverless API functions under /api/*, which query ClickHouse over HTTPS. Proprietary Vercel serverless
Vendor neutrality note: The entire data path (capture, ingest, store, query) uses open-source components with permissive licenses. Cloudflared handles ingress only and can be replaced with any reverse proxy or load balancer. ClickHouse Cloud can be swapped for a self-hosted ClickHouse instance with no schema or query changes.

Deployment Topology

Current production deployment. The portal and API run on Vercel; telemetry is stored in a managed ClickHouse Cloud instance on GCP (us-central1); telemetry ingest runs through a Cloudflare-tunneled OTEL collector on a self-hosted VM.

flowchart TB
    subgraph Internet
        Browser["End User Browser"]
        CloudFlare["Cloudflare Edge\n(tunnel ingress)"]
    end

    subgraph Ingest["OTEL Collector VM (self-hosted)"]
        CFD["cloudflared"]
        OTEL["otel-collector\n:4318 OTLP/HTTP"]
        CFD --> OTEL
    end

    subgraph Serve["Vercel"]
        Portal["Static Portal\n(vanilla HTML)"]
        API["Serverless API\napi/[action].js"]
    end

    subgraph CHCloud["ClickHouse Cloud — GCP us-central1"]
        CH[("ClickHouse\nSOC 2 Type II")]
    end

    Browser -->|HTTPS| CloudFlare
    CloudFlare -->|tunnel| CFD
    OTEL -->|"INSERT (batch)"| CH
    Browser -->|HTTPS| Portal
    Portal --> API
    API -->|"HTTPS :8443 (TLS)"| CH

    style Browser fill:#64748b,color:#fff,stroke:#94a3b8
    style CloudFlare fill:#f97316,color:#fff,stroke:#fb923c
    style OTEL fill:#06b6d4,color:#fff,stroke:#22d3ee
    style CFD fill:#64748b,color:#fff,stroke:#94a3b8
    style Portal fill:#0ea5e9,color:#fff,stroke:#38bdf8
    style API fill:#0ea5e9,color:#fff,stroke:#38bdf8
    style CH fill:#ca8a04,color:#fff,stroke:#facc15

Serving Tier (Vercel)

Platform Vercel
Portal Static vanilla HTML (no build step)
API Serverless functions via api/[action].js
Max duration 30 s per request
Ingest Cloudflare-tunneled OTEL collector (self-hosted VM)

Storage

Service ClickHouse Cloud (managed)
Cloud GCP (us-central1)
Connection HTTPS, port 8443 (TLS) from Vercel functions
Compliance SOC 2 Type II
Backup ClickHouse Cloud automated

Security & Compliance

Key security properties of the deployment. No third-party analytics vendors receive user data.

Data Residency

  • ✓ Telemetry is stored in your own ClickHouse Cloud instance — not sold or shared with third-party analytics vendors. (Ask AI sends only the typed question, never raw telemetry, to an Azure-hosted Claude endpoint.)
  • ✓ ClickHouse Cloud instance runs on GCP infrastructure in us-central1.
  • ✓ Cloudflare tunnel handles ingress routing only — telemetry payloads are not stored or inspected at the edge.

PII Handling

  • ✓ HyperDX SDK supports DOM masking: maskAllText and maskAllInputs flags redact sensitive content before it leaves the browser.
  • ✓ Selective masking available via CSS selectors for targeted redaction of PII fields.
  • ✓ Network request bodies can be excluded from capture via SDK configuration.

Certifications & Standards

  • ✓ ClickHouse Cloud: SOC 2 Type II certified. Managed encryption at rest and in transit.
  • ✓ OpenTelemetry: CNCF graduated project. Vendor-neutral telemetry standard backed by Google, Microsoft, and others.
  • ✓ No vendor lock-in: OTLP is a wire protocol. The collector can export to Jaeger, Elasticsearch, Datadog, or any OTLP-compatible backend without SDK changes.

Network Security

  • ✓ No public ports exposed on the OTEL collector VM. All telemetry ingress routed through the Cloudflare tunnel.
  • ✓ ClickHouse Cloud connection over TLS — the Vercel API queries over HTTPS (port 8443).
  • ✓ OTLP ingestion endpoint receives data over HTTPS only.

Access & Authentication

  • ✓Sign-in via Firebase Authentication with Microsoft and Google OAuth. Identity is attached to subsequent telemetry (user.email, user.uid).
  • ✓A read-only demo mode (?skipauth=true) bypasses only the front-end sign-in overlay for evaluation and test automation — it grants no additional data privilege.
  • ✓Every query runs as exos_readonly — SELECT on the default + exos databases only, with engine-level readonly=1 blocking remote()/url()/s3() escapes.
  • ✓NL-to-SQL and raw SQL additionally allowlist { default, exos } and reject cross-database references. Other data on the instance is unreachable from any portal surface.

Full detail: Docs → Access & Auth.

Data Schema

Primary table structure in ClickHouse. These tables follow the OpenTelemetry data model and are queryable by all three consumer applications.

otel_traces

Primary event table. One row per span. Powers session replay, trace analysis, and dashboard aggregations.

Column Type Description
Timestamp DateTime64(9) Nanosecond-precision event timestamp
TraceId String W3C trace context identifier. Links all spans in a distributed trace.
SpanId String Unique span identifier within the trace
ParentSpanId String Parent span for causal ordering
SpanName LowCardinality(String) Operation name (e.g., HTTP GET, click, routeChange)
ServiceName LowCardinality(String) Logical service identifier (e.g., clickstack-portal)
SpanAttributes Map(String, String) Span-level key-value metadata. Includes URL, status code, user agent, DOM element selectors.
ResourceAttributes Map(String, String) Resource-level metadata. Includes SDK version, browser info, OS.
Duration Int64 Span duration in nanoseconds
StatusCode LowCardinality(String) OTEL status: Unset, Ok, Error

otel_logs

Console captures and structured log events. Correlated to traces via TraceId.

Key columns: Timestamp, TraceId, SeverityText, Body, LogAttributes

otel_metrics

Web Vitals and custom metrics. LCP, FID, CLS, TTFB captured automatically by the SDK.

Key columns: Timestamp, MetricName, Value, Attributes

Query example: SELECT SpanName, count() AS events, avg(Duration)/1e6 AS avg_ms FROM otel_traces WHERE ServiceName = 'clickstack-portal' AND Timestamp > now() - INTERVAL 1 HOUR GROUP BY SpanName ORDER BY events DESC LIMIT 20
Properties. Data is segmented by ServiceName into distinct products — EXOS Operations (exos-*), the Analytics Portal (clickstack-portal), RadLo (radlo-web), and Docs (beppe-dotfiles-docs). The conceptual data model — the OpenTelemetry trace/span/attribute model, the core default and exos tables, and the property concept — is documented in Docs → Concepts & Data Model.

API Endpoints

The portal backend exposes these JSON API endpoints. All endpoints query ClickHouse in real time — no caching layer, no pre-aggregation.

Endpoint Description Parameters
GET /api/stats KPI snapshot: session count, event volume, and error rate. workspace, lob, operator, task, hours
GET /api/events Raw trace/log feed with search and pagination. table, hours, workspace, search, status, limit, offset, cursor
GET /api/funnels Funnel analysis with windowFunnel(). Returns step-by-step conversion rates for a defined sequence of events. steps, hours, workspace, list
GET /api/retention Cohort retention matrix. Groups users by first action date and measures return rates over subsequent periods. first_action, return_action, granularity, hours, workspace, list
GET /api/paths User journey analysis. Discovers the most common sequences of actions users take through the application. hours, workspace, depth
GET /api/errors Grouped error analysis. Returns error messages ranked by frequency with affected services and time distribution. hours, workspace, group
GET /api/sessions Session list; pass id for one enriched session's spans (replay drill-down). id, hours, limit, lob, operator, task, span
GET /api/operators Operator leaderboard; pass id for a single-operator profile. id, hours, lob, role
GET /api/ocpm Object-centric process mining: interactions, variants, lifecycle, actor-affinity. hours
GET /api/insights · /api/patterns Narrative process-mining insights and rolling-z-score anomaly detection. hours, min_hours, threshold
GET /api/workspaces · /api/lobs · /api/filters Discovery: available properties, lines of business, and cascading filter options. lob, operator
POST /api/query Natural-language → SQL (Claude on Azure), validated and executed read-only. { question } or { messages[] }
POST /api/sql Execute a raw read-only SELECT/WITH. DDL/DML and cross-database refs rejected. { sql, database? }
GET /api/schema Column catalog for the queryable default + exos tables. —
Note: Most analytics endpoints accept an optional workspace parameter to scope results to a single property (a ServiceName); omit it to query the default EXOS + Portal scope. All endpoints query ClickHouse in real time and run as the least-privilege exos_readonly user. Full per-endpoint reference: Docs → API & Query Reference.

Capacity & Performance

ClickHouse is a columnar OLAP engine optimized for analytical queries over large datasets. These are documented ClickHouse characteristics, not theoretical claims.

109+

Rows

ClickHouse handles billions of rows per table. Horizontal scaling via ClickHouse Cloud adds shards as volume grows.

<100ms

Query Latency

Columnar storage + vectorized execution. Sub-100ms on typical analytical queries scanning millions of rows.

10-20x

Compression

LZ4 + delta encoding on columnar data. Telemetry data (highly repetitive) compresses exceptionally well.

Why ClickHouse for Observability

✓ Columnar storage — Queries that scan a single column (e.g., count events by SpanName) skip all other columns entirely.
✓ No pre-aggregation needed — Fast enough to query raw events directly. No materialized views required for common access patterns.
✓ Standard SQL — No proprietary query language. Any SQL-capable tool (Grafana, DBeaver, CLI) can connect and query.
✓ TTL policies — Automatic data retention. Set per-table TTLs (e.g., 90 days for traces, 1 year for aggregated metrics) without manual cleanup.
✓ Real-time ingestion — Handles high-throughput INSERT batches from the OTEL Collector without write amplification issues.
✓ Cost efficiency — Compression ratios mean storage costs are a fraction of row-oriented alternatives at the same data volume.