Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .devcontainer/devcontainer-lock.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"features": {
"ghcr.io/devcontainers-extra/features/go-task:1": {
"version": "1.0.6",
"resolved": "ghcr.io/devcontainers-extra/features/go-task@sha256:4d1db153919976cadd3209ca05d655a761a01707767716994dad677b4538dc1b",
"integrity": "sha256:4d1db153919976cadd3209ca05d655a761a01707767716994dad677b4538dc1b"
},
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
"version": "1.10.0",
"resolved": "ghcr.io/devcontainers/features/docker-outside-of-docker@sha256:c2c2cf829505ead8e4892c88c31b6594ae94a2bbb209e16e1fac456c1a3a624e",
"integrity": "sha256:c2c2cf829505ead8e4892c88c31b6594ae94a2bbb209e16e1fac456c1a3a624e"
},
"ghcr.io/devcontainers/features/go:1": {
"version": "1.3.4",
"resolved": "ghcr.io/devcontainers/features/go@sha256:d85e921f91b41340055bb12b325d9d551170ed04b3b832e33530bf42f167c032",
"integrity": "sha256:d85e921f91b41340055bb12b325d9d551170ed04b3b832e33530bf42f167c032"
},
"ghcr.io/rails/devcontainer/features/bun:1.0.2": {
"version": "1.0.2",
"resolved": "ghcr.io/rails/devcontainer/features/bun@sha256:08057c197a8cde49b08749681607bf0d69aed79e280225cf43ca5d1782028789",
"integrity": "sha256:08057c197a8cde49b08749681607bf0d69aed79e280225cf43ca5d1782028789"
}
}
}
4 changes: 3 additions & 1 deletion .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@
"service": "app",
"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
"features": {
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {},
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
"moby": false
},
"ghcr.io/rails/devcontainer/features/bun:1.0.2": {},
"ghcr.io/devcontainers-extra/features/go-task:1": {},
"ghcr.io/devcontainers/features/go:1": {
Expand Down
145 changes: 107 additions & 38 deletions ANALYTICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,51 @@

Implementation notes for Lovely Eye analytics.

## Public Collect Contract

`POST /api/collect?site_key=<public_key>` is the only collect shape. `site_key` in the JSON body is not part of the contract.

The tracker sends JSON with `Content-Type: text/plain;charset=UTF-8` so `navigator.sendBeacon` can queue small analytics payloads without a custom request setup.

Page view:

```json
{ "path": "/pricing" }
```

Exit ping:

```json
{ "path": "/pricing", "exit": true }
```

Initial attribution, when present, is sent only on the first page view:

```json
{ "path": "/pricing", "referrer": "https://google.com", "utm_source": "google" }
```

Custom event:

```json
{ "name": "checkout_failed", "path": "/checkout", "properties": "{\"code\":\"PAYMENT_DECLINED\"}" }
```

The client does not send `duration`, `screen_width`, `last_alive`, session IDs, client IDs, or page-state decisions. All client data is treated as untrusted hints.

## Tracker Lifecycle

- Normal page views send only the current path, plus first-touch attribution if present.
- SPA navigation hooks send a new page view when the path changes.
- Exit pings use `visibilitychange` when the document becomes hidden.
- `pagehide` is kept only as a fallback.
- `beforeunload` is intentionally not used.

This follows browser guidance for small analytics payloads:
- [MDN `sendBeacon`](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon): intended for small analytics/diagnostic POSTs and avoids slowing navigation.
- [MDN `visibilitychange`](https://developer.mozilla.org/en-US/docs/Web/API/Document/visibilitychange_event): the hidden transition is the last reliably observable lifecycle point for many pages.
- [W3C Beacon](https://www.w3.org/TR/beacon/): defines asynchronous beacon delivery for analytics-style data.

## Visitor Identification

Server-generated visitor ID computed from minimized request signals:
Expand All @@ -16,6 +61,56 @@ Server-generated visitor ID computed from minimized request signals:
- Country is not part of the visitor ID
- The server secret helps reduce the impact of database-only leaks by making visitor IDs harder to recompute outside the app

## IP Address Handling

Client IP is resolved by `server/pkg/clientip`:
- `RemoteAddr` is authoritative unless it belongs to a configured trusted proxy CIDR.
- `X-Forwarded-For` and `X-Real-IP` are ignored from untrusted remotes.
- For trusted proxy chains, Lovely Eye scans `X-Forwarded-For` from right to left and selects the last non-trusted hop as the client.
- If every forwarded hop is trusted, the leftmost valid forwarded IP is used.
- IPs are truncated before hashing: IPv4 `/24`, IPv6 `/64`.
- IPs are used only for visitor identity, block checks, rate limiting, and optional country lookup. They are not stored.

The default `TRUSTED_PROXY_CIDRS` covers loopback, RFC1918 private IPv4 ranges, and IPv6 unique-local addresses for common Docker, Nginx, Traefik, and Kubernetes private-network deployments. Public CDN or edge proxy ranges must be configured explicitly. The hop-selection behavior matches the intent of Nginx `real_ip_recursive`: [Nginx realip module](https://nginx.org/en/docs/http/ngx_http_realip_module.html).

## Session Mechanics

Sessions are computed server-side:
- 30-minute inactivity timeout.
- Server receive time is used for event time, session entry time, exit time, and duration.
- A normal page view creates or extends the active session.
- A same-path exit ping updates `exit_time`, `exit_path`, and computed duration only.
- Same-path exit pings may close a single-page session for up to `ANALYTICS_MAX_SINGLE_PAGE_DURATION`, defaulting to `4h`; repeated exit pings cannot push that single-page duration past the cap.
- A different-path exit ping counts that path as a page view only while the session is still inside the normal 30-minute active window, then updates session exit fields.
- An exit ping without an active session is a no-op.
- A 10-second dedupe window suppresses repeated same-path page views before counters change.
- Bounce rate uses sessions with one page view.
- Average session duration uses positive server-computed durations, including single-page sessions closed by exit pings.

## Query Accuracy

- Top pages and active pages count distinct analytics clients, not sessions.
- Page-view time series are bucketed by event time, not session start time.
- Visitor and session time series are bucketed by session entry time.
- Dashboard overview errors are propagated instead of returning partial zero values.

## Limits And Hardening

Limits exist because analytics endpoints are public by design:
- `ANALYTICS_MAX_BODY_BYTES` defaults to `16384`; tracker payloads should be far smaller.
- `ANALYTICS_MAX_PROPERTIES_BYTES` defaults to `8192`; custom event properties are allowlisted and capped.
- `ANALYTICS_MAX_SINGLE_PAGE_DURATION` defaults to `4h`; this bounds long-lived open tabs while still allowing real long reads.
- `ANALYTICS_RATE_LIMIT_ENABLED` defaults to `true`.
- `ANALYTICS_RATE_LIMIT_PER_MINUTE` defaults to `120`; collect traffic is limited by client IP before site lookup and by site key plus client IP after validation.
- `ANALYTICS_RATE_LIMIT_BURST` defaults to `240`; it uses the same keying as the refill limit.
- `GRAPHQL_MAX_BODY_BYTES` defaults to `1048576`.
- `DASHBOARD_MAX_DAILY_RANGE_DAYS` defaults to `730`.
- `DASHBOARD_MAX_HOURLY_RANGE_DAYS` defaults to `31`.
- `DASHBOARD_MAX_FILTER_VALUES` defaults to `100`.
- `DASHBOARD_MAX_FILTER_STRING_LENGTH` defaults to `2048`.

These controls follow OWASP API guidance to constrain resource consumption and validate request sizes at the boundary: [OWASP REST Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/REST_Security_Cheat_Sheet.html).

## Bot Filtering

Filters non-human traffic:
Expand All @@ -25,49 +120,23 @@ Filters non-human traffic:
- Scrapers: curl, wget, python-requests
- Headless browsers: Puppeteer, Playwright

## Page View Deduplication

Prevents duplicate counting:
- 10-second deduplication window per visitor per page
- Filters double-clicks, script reloads, same-path SPA updates within 10s
- Duplicate hits are ignored before page-view counters or session exit metrics change
- Ensures accurate page view metrics

## Query Parameters

- By default, query parameters are not included in tracked page paths
- Use `data-include-query="true"` on the tracker script to include full query strings

## IP Address Handling

Extracts real client IP from proxied requests:
- Parses X-Forwarded-For header (first IP)
- Falls back to X-Real-IP header
- Strips port from RemoteAddr
- Truncates IP before hashing: IPv4 `/24`, IPv6 `/64`
- IPs used only for visitor identity and optional geolocation, never stored

## Session Management

Tracks browsing sessions:
- 30-minute inactivity timeout
- Records entry page, exit page, duration
- Calculates bounce rate
- Captures device, browser, OS, screen size
- Optional country detection via GeoIP
- By default, query parameters are not included in tracked page paths.
- Use `data-include-query="true"` on the tracker script to include full query strings.

## Privacy

- No client-side cookies or persistent identifiers
- Visitor IDs use UTC-day-skipped rotation
- Visitor IDs are derived server-side from minimized signals
- Site-scoped keying prevents reuse across sites
- Keyed visitor IDs reduce the value of database-only leaks
- IP addresses never stored in database
- Country-level geolocation only (no city data)
- No client-side cookies or persistent identifiers.
- Visitor IDs use UTC-day-skipped rotation.
- Visitor IDs are derived server-side from minimized signals.
- Site-scoped keying prevents reuse across sites.
- Keyed visitor IDs reduce the value of database-only leaks.
- IP addresses are never stored in the database.
- Country-level geolocation only, no city data.

## Event Allowlist

- Custom events are recorded only if the event name is allowlisted for the site
- Event properties are filtered to the allowed keys and types
- Required fields must be present for the event to be stored
- Custom events are recorded only if the event name is allowlisted for the site.
- Event properties are filtered to the allowed keys and types.
- Required fields must be present for the event to be stored.
69 changes: 66 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Self-hosted web analytics with a Go backend and a React dashboard. Lovely Eye tr
- Cookieless analytics with an identifier computed from minimized request data and keyed with a server-side secret
- SQLite by default, PostgreSQL when needed
- Bot filtering and page-view deduplication
- Server-side session timing; client timing fields are ignored
- Allowlisted custom events
- Optional country tracking
- Dashboard served as static assets by the Go server
Expand Down Expand Up @@ -116,12 +117,16 @@ SQLite is the default database. If `JWT_SECRET` is unset, Lovely Eye generates o
## Privacy And Tracking

- Lovely Eye does not use analytics cookies or local storage by default.
- The tracker sends a minimal payload. A page view sends `path`; an exit ping sends `path` plus `exit: true`.
- Timing is computed from server receive time. The client does not send `duration`, `screen_width`, or session state.
- Single-page exit duration is bounded by `ANALYTICS_MAX_SINGLE_PAGE_DURATION`, which defaults to `4h`; repeated exit pings cannot extend it past that cap.
- The analytics visitor identifier is computed from site ID, truncated IP prefix, browser family, and device class, and keyed with a server-side secret.
- The analytics visitor identifier is unique per site.
- The server computes hashes for `today` and `yesterday`.
- A visitor who returns at least once per UTC day keeps the same analytics client row.
- A new analytics client row is created only after the visitor skips a full UTC day between visits.
- Sessions are separate from the analytics visitor identifier and expire after 30 minutes of inactivity.
- Exit pings update the active session when they match the current path. If an exit ping names a different path while the session is still inside the 30-minute active window, the server counts that path as a page view before closing the session path; stale different-path exits are ignored.
- Country tracking is optional and is not part of the analytics visitor identifier.
- The dedicated `ANALYTICS_IDENTITY_SECRET` helps reduce the impact of database-only leaks because stored analytics rows do not contain enough information to recompute the identifier on their own.

Expand All @@ -133,6 +138,35 @@ SQLite is the default database. If `JWT_SECRET` is unset, Lovely Eye generates o
4. Copy the generated tracking code.
5. Add it to the site you want to track.

## Tracker API

The collect endpoint requires the public key in the query string:

```http
POST /api/collect?site_key=<public_key>
Content-Type: text/plain;charset=UTF-8
```

Page view:

```json
{ "path": "/pricing" }
```

Exit ping:

```json
{ "path": "/pricing", "exit": true }
```

Initial attribution, when present, is sent only on the first page view:

```json
{ "path": "/pricing", "referrer": "https://google.com", "utm_source": "google" }
```

The tracker uses `visibilitychange` with `sendBeacon`, with `pagehide` as a fallback. This follows the current MDN and W3C Beacon guidance for small analytics payloads that should not block navigation: [MDN sendBeacon](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon), [MDN visibilitychange](https://developer.mozilla.org/en-US/docs/Web/API/Document/visibilitychange_event), and [W3C Beacon](https://www.w3.org/TR/beacon/).

## Common Configuration

| Variable | Default | Meaning |
Expand All @@ -146,14 +180,29 @@ SQLite is the default database. If `JWT_SECRET` is unset, Lovely Eye generates o
| `INITIAL_ADMIN_USERNAME` | empty | Initial admin username. Requires `INITIAL_ADMIN_PASSWORD`. |
| `INITIAL_ADMIN_PASSWORD` | empty | Initial admin password. Requires `INITIAL_ADMIN_USERNAME`. |
| `GEOIP_MAXMIND_LICENSE_KEY` | empty | Optional MaxMind license key for country tracking |
| `ANALYTICS_MAX_BODY_BYTES` | `16384` | Maximum collect request body size. Small because tracker payloads are tiny. |
| `ANALYTICS_MAX_PROPERTIES_BYTES` | `8192` | Maximum custom-event `properties` JSON string size. |
| `ANALYTICS_MAX_SINGLE_PAGE_DURATION` | `4h` | Maximum same-path single-page duration accepted from an exit ping. |
| `ANALYTICS_RATE_LIMIT_ENABLED` | `true` | Enables per-process collect rate limiting. |
| `ANALYTICS_RATE_LIMIT_PER_MINUTE` | `120` | Refill rate for client IP admission and validated site key plus client IP admission. |
| `ANALYTICS_RATE_LIMIT_BURST` | `240` | Short burst allowance for the same collect admission keys. |
| `TRUSTED_PROXY_CIDRS` | private, loopback, and unique-local ranges | CIDRs allowed to supply `X-Forwarded-For` / `X-Real-IP`. Public CDN ranges must be configured explicitly. |
| `GRAPHQL_MAX_BODY_BYTES` | `1048576` | Maximum GraphQL request body size. |
| `DASHBOARD_MAX_DAILY_RANGE_DAYS` | `730` | Maximum daily dashboard date range. |
| `DASHBOARD_MAX_HOURLY_RANGE_DAYS` | `31` | Maximum hourly dashboard date range. |
| `DASHBOARD_MAX_FILTER_VALUES` | `100` | Maximum values per dashboard filter list. |
| `DASHBOARD_MAX_FILTER_STRING_LENGTH` | `2048` | Maximum byte length for each dashboard filter string. |

## Custom Events

```html
<script>
window.lovelyEye?.track("checkout_failed", {
code: "PAYMENT_DECLINED",
step: "confirm",
window.lovelyEye?.track({
name: "checkout_failed",
properties: {
code: "PAYMENT_DECLINED",
step: "confirm",
},
});
</script>
```
Expand Down Expand Up @@ -199,6 +248,20 @@ services:
- INITIAL_ADMIN_USERNAME=
- INITIAL_ADMIN_PASSWORD=
- ANALYTICS_IDENTITY_SECRET=replace-with-a-second-32-plus-character-secret
- ANALYTICS_MAX_BODY_BYTES=16384
- ANALYTICS_MAX_PROPERTIES_BYTES=8192
- ANALYTICS_MAX_SINGLE_PAGE_DURATION=4h
- ANALYTICS_RATE_LIMIT_ENABLED=true
- ANALYTICS_RATE_LIMIT_PER_MINUTE=120
- ANALYTICS_RATE_LIMIT_BURST=240
# Trust loopback and private network proxies by default.
# Add CDN/public reverse-proxy ranges explicitly.
- TRUSTED_PROXY_CIDRS=127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,fc00::/7
- GRAPHQL_MAX_BODY_BYTES=1048576
- DASHBOARD_MAX_DAILY_RANGE_DAYS=730
- DASHBOARD_MAX_HOURLY_RANGE_DAYS=31
- DASHBOARD_MAX_FILTER_VALUES=100
- DASHBOARD_MAX_FILTER_STRING_LENGTH=2048
- GEOIP_DB_PATH=/data/GeoLite2-Country.mmdb
- GEOIP_DOWNLOAD_URL=https://download.db-ip.com/free/dbip-country-lite.mmdb.gz
- GEOIP_MAXMIND_LICENSE_KEY=
Expand Down
4 changes: 2 additions & 2 deletions biome.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"$schema": "https://biomejs.dev/schemas/2.4.6/schema.json",
"$schema": "https://biomejs.dev/schemas/2.5.0/schema.json",
"vcs": {
"enabled": true,
"clientKind": "git",
Expand Down Expand Up @@ -60,7 +60,7 @@
"linter": {
"enabled": true,
"rules": {
"recommended": true,
"preset": "recommended",
"complexity": {
"noArguments": "error",
"noExcessiveCognitiveComplexity": {
Expand Down
Loading
Loading