# Observatory Agent Guide

Observatory is a multi-user cost and operational-metrics app. Each
signed-in user has an isolated set of provider connectors. The dashboard is
read-only; a user's agent configures connectors through the authenticated API.

## API

- `GET /api/connectors` — connector catalog and the current user's connection status.
- `PUT /api/connectors/{provider}` — create or replace one connector.
- `DELETE /api/connectors/{provider}` — disconnect one provider.
- `GET /api/series?days=30` — authenticated canonical provider envelopes. Cost windows use UTC calendar dates, not the last N reported rows: `period_start` is inclusive and `period_end` is today (any reported partial day is retained). Sparse and missing observations do not extend the window; missing dates are not confirmed zero spend. The AWS envelope also includes regional EC2 inventory refreshed independently of billing snapshots (at most 60 seconds cached). `?refresh=1` checks EC2 immediately without rerunning Cost Explorer. `inventory_generated_at` records the AWS check time; failed checks return `instances_available: false` and an explicit error instead of old running states. Each instance includes `compute_estimate` with monthly USD (730 running hours), hourly USD, and an explicit reason when unavailable. `instance_pricing` reports the AWS public Linux shared-tenancy On-Demand price publication date and source. These compute-only comparisons exclude storage, snapshots, public IPv4, data transfer, CPU credits, taxes, and discounts; stopped instances show zero compute while other charges may continue.
- `GET /api/series/{provider}?days=30` — query one provider independently.
- `GET /api/projection?days=30` — read the scheduled AI forecast with fresh owner-scoped BrightWrapper recorded costs overlaid as a floor. This read never generates an AI request; the recorded-cost fetch is bounded to ten seconds and cached for 60 seconds.
- `POST /api/metrics/query` — run a bounded AWS CloudWatch metric query.
- `GET /api/fleet-health` — every running EC2 instance in the connected account's region with 24 hours of 5-minute CloudWatch history in one batched read: CPU average/peak, CPU credit and surplus balances, charged surplus credits (`cpu_charged`, AWS `CPUSurplusCreditsCharged`, `Sum`), network out, EBS read ops and I/O burst balance, plus memory and root-disk use where the CloudWatch agent publishes them (`null` means no agent). The table shows the CPU mean over the window, current borrowed credits, and the sum of charged credits over the window. Borrowed surplus is repayable and does not prove a charge; missing charge datapoints are unavailable, not zero. Zero earned credits alone does not prove throttling. `GET /api/server-health?resource_id=<instance>` opens the detailed chart for any instance in that account.
- `GET /api/me` — authenticated viewer identity.
- `GET /api/health` and `GET /api/version` — public operational checks.

AWS billing is imported at most once every six hours; Refresh checks live EC2 inventory without buying another billing query. AWS `daily_service_costs` and `service_breakdown` cover the same UTC dates as `daily_costs` and the total, including revised historical charges. `billing_observed_at` identifies the last successful billing read.

Connector configuration is encrypted at rest, scoped to the authenticated user,
and never returned by the API. An agent may retrieve credentials from the user's
preferred secrets manager, then send them directly over HTTPS; Observatory does
not need access to that secrets manager. `PUT` is idempotent and replaces the
previous credential set.

Authenticated calls accept either the user's AuthReturn JWT or that user's
app-specific AuthReturn API key in `Authorization: Bearer …`. Agents should use
an API key for unattended access; keys remain user-scoped and revocable.

## Connect a provider

1. Ask the user which provider to connect and which secrets-manager entry holds
   its dedicated billing credential. Read the secret directly from that manager;
   do not ask the user to paste it into chat.
2. Authenticate as that same Observatory user. For an interactive, bounded setup,
   use the user's Observatory AuthReturn JWT. For unattended refreshes, use that
   user's app-specific, revocable Observatory API key. Never use another user's
   bearer credential.
3. Call `GET /api/connectors` and select one of the provider IDs below. Send the
   exact required fields in a `PUT /api/connectors/{provider}` request over HTTPS.
4. Confirm that the response reports `connected: true`, then call
   `GET /api/series/{provider}?days=30`. Report provider permission errors as-is;
   do not broaden the credential automatically.

For a bounded setup, obtain the JWT from the canonical app-scoped login endpoint.
Read the email and password from the user's secrets manager and keep the returned
token out of logs:

```http
POST https://authreturn.com/api/apps/observatory/login
Content-Type: application/json

{"email":"<Observatory account email>","password":"<Observatory password>"}
```

Use the response's `token` only as `Authorization: Bearer <token>` on Observatory
requests. Authentication must terminate with a visible error within 20 seconds;
do not retry invalid credentials. For unattended use, store a user-scoped
Observatory API key in the user's secrets manager and revoke it when the agent
no longer needs access.

Example request:

```http
PUT /api/connectors/openai
Authorization: Bearer <user JWT or app-specific API key>
Content-Type: application/json

{"config":{"admin_key":"<OpenAI organization admin key>"}}
```

The authenticated connector endpoints are on
`https://observatory.aisloppy.com`. Apply an explicit timeout to every request;
30 seconds is suitable for connector configuration and 45 seconds for the first
provider query. The API never returns the submitted credential.

Supported provider configuration:

| Provider | Required fields | Optional fields |
|---|---|---|
| `aws` | `access_key_id`, `secret_access_key` | `session_token`, `region` |
| `azure` | `tenant_id`, `client_id`, `client_secret`, `subscription_id` | — |
| `gcp` | `project_id`, `dataset_id`, `table_id`, `service_account` object | — |
| `openai` | `admin_key` | — |
| `anthropic` | `admin_key` | — |
| `scrapingbee` | `api_key` | — |

## Connect AWS costs and CloudWatch

The existing `aws` connector powers both Cost Explorer and CloudWatch. Use a
dedicated IAM principal with this minimum read-only policy; scope resources
further when the account's metric naming allows it:

```json
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ce:GetCostAndUsage","cloudwatch:GetMetricStatistics","cloudwatch:GetMetricData","cloudwatch:ListMetrics","ec2:DescribeInstances"],"Resource":"*"}]}
```

The metrics endpoint accepts JSON containing `namespace`, `metric_name`, an
optional `dimensions` object, `statistic`, `period`, and `hours`. Queries are
limited to 30 days, 1,440 points, ten dimensions, known CloudWatch statistics,
and a 20-second overall deadline. Successful identical queries are cached for
60 seconds. Example:

```http
POST /api/metrics/query
Authorization: Bearer <user JWT or app-specific API key>
Content-Type: application/json

{"namespace":"AWS/EC2","metric_name":"CPUUtilization","dimensions":{"InstanceId":"i-0123456789abcdef0"},"statistic":"Average","period":300,"hours":24}
```

CloudWatch errors are visible but connector credentials are redacted. The API
returns timestamp/value points and never returns the AWS credential.

Provider-specific payload shapes:

```json
{"config":{"access_key_id":"<AWS access key ID>","secret_access_key":"<AWS secret access key>","region":"us-east-1"}}
{"config":{"tenant_id":"<Entra tenant GUID>","client_id":"<app GUID>","client_secret":"<client secret>","subscription_id":"<subscription GUID>"}}
{"config":{"project_id":"<GCP project>","dataset_id":"<billing export dataset>","table_id":"<billing export table>","service_account":{"type":"service_account","project_id":"<GCP project>","private_key_id":"<key ID>","private_key":"<private key>","client_email":"<service account email>","client_id":"<client ID>","token_uri":"https://oauth2.googleapis.com/token"}}}
{"config":{"admin_key":"<OpenAI organization admin key>"}}
{"config":{"admin_key":"<Anthropic organization admin key>"}}
{"config":{"api_key":"<ScrapingBee API key>"}}
```

These examples are ordered as AWS, Azure, GCP, OpenAI, Anthropic, and ScrapingBee.
The Azure credential is a service principal with only the Cost Management Reader role on the subscription.
Send only the payload for the selected provider. Optional AWS `session_token`
credentials expire; replace the connector before expiry or use a dedicated
long-lived read-only IAM principal.

Use read-only billing or usage credentials with the narrowest provider permissions
available. Do not put credentials in query strings, logs, chat messages, or source
control. After configuration, query `/api/series/{provider}` to verify access; a
failed provider returns a visible error without affecting other connectors.

## Security model

- Public signup is enabled. Every connector, provider cache entry, and usage-history
  path is scoped by the verified AuthReturn user ID.
- Connector secrets terminate at Observatory: the service must decrypt them in
  memory to call providers. This is encrypted-at-rest isolation, not a zero-knowledge
  vault. Host administrators retain infrastructure-level access.
- Credentials are never returned after configuration. Sensitive values are redacted
  from upstream errors, protected API responses are `no-store`, and rendered provider
  content uses text nodes rather than HTML injection.
- Browser sessions require an AuthReturn Cognito ID token with verified issuer,
  audience, signature, expiry, and token type. Automation keys are verified against
  the `observatory` AuthReturn app and successful verification is cached for at most
  60 seconds.
- Use a dedicated read-only billing identity per provider. Do not supply general
  cloud-administrator, resource-write, or payment-method credentials.

Minimum credential guidance:

| Provider | Recommended boundary | Residual power |
|---|---|---|
| AWS | A dedicated IAM principal allowing `ce:GetCostAndUsage`, `cloudwatch:GetMetricStatistics`, and `cloudwatch:ListMetrics` | Do not reuse an infrastructure automation key. |
| GCP | BigQuery Job User plus Data Viewer limited to the billing-export dataset | The service-account key remains a reusable bearer credential. |
| OpenAI | A dedicated organization admin key used only for usage and costs | OpenAI's usage/cost endpoints require organization-level admin authority. |
| Anthropic | A dedicated organization admin key | The usage endpoint requires admin authority. |
| ScrapingBee | A dedicated account key | ScrapingBee keys can make paid scraping requests; no read-only usage key is assumed. |

Observatory currently accepts encrypted inline credentials; it does not yet
resolve references from an external user-controlled vault or use provider OAuth.

## Connector adoption analytics

Observatory tracks its own usage under the preserved app slug `observatory`. The backend emits
these privacy-safe events:

- `connection_attempted`: an authenticated connector `PUT` reached the backend.
  This is the primary adoption KPI.
- `connection_succeeded`: connector configuration passed validation and was
  encrypted and stored. It does not claim that the upstream provider accepted it.
- `connection_failed`: the request ended in one of `invalid_request`,
  `unknown_provider`, `invalid_config`, or `storage_error`.
- `guide_opened`: a signed-in user followed a disconnected card's setup link.

Event paths contain only the event, a catalog provider ID, and the controlled
failure category. Attempt sessions use an app-secret HMAC of the AuthReturn user
ID so usage analytics can count unique attempting tenants without receiving the raw
ID or email. Connector configs, API keys, secrets, raw errors, request bodies,
and email addresses are never analytics fields.

View the `observatory` panel at
`https://gridglance.com/`. Attempt totals and provider
breakdown are the page counts under `/events/connection_attempted/{provider}`;
attempt-only session analytics provide the unique-tenant count. No separate
Observatory analytics dashboard is maintained.

The cost-series response is schema version 2. AWS, GCP, OpenAI, Anthropic, and
ScrapingBee return independent availability envelopes; missing or failed
providers are never silently replaced with zeroes. Windows are bounded to
2–366 days. The dashboard includes a stacked daily cost graph, provider totals,
token usage, service/model breakdowns, and ScrapingBee credit headroom.

The final chart date may use a BrightWrapper projection because provider billing
buckets publish at different times. Projection payloads are labeled
`kind: ai_projection`, retain each provider's `observed_usd`, and never change
provider totals or cached source observations. Recent non-read-only CloudTrail
events are weak context when the connected AWS identity permits lookup; an
unavailable CloudTrail query is explicit and does not block history-based estimates.

Base URL: `https://observatory.aisloppy.com`

## Guide routes

`/welcome` is the minimal public landing page, without app navigation. Signed-out
visitors to `/` go there; signing in returns to the existing costs dashboard or
the original same-origin deep link. Signed-in visitors can open the app directly.

`GET /agent-guide.md` is this machine-readable contract; `/agent-guide` is its
rendered page. All browser pages are public shells; their account data remains
authenticated. The supported navigation is Costs & infrastructure, Analytics,
Storage, Computer, Usage, Profiles, and Agent guide.

## Reviewed peak investigations

`GET /api/server-health/investigation` returns `{available, data}` for the authenticated user’s currently monitored AWS resource. A report is a dated snapshot with separate CPU (%) and memory (GiB) series, overview and detail observations, event annotations, evidence excerpts, and explicit attribution limits. Missing reports return `available: false`. These historical snapshots are available through the API only; the dashboard shows current fleet health. Reviewed reports are stored in Observatory’s tenant-scoped `peak_investigations` table; they are not inferred automatically from timing correlations.


## Structured infrastructure embeds

`GET /api/embeds/infrastructure` returns a public HTML renderer, never tenant data.
Embed this exact URL in an iframe with `sandbox="allow-scripts"` and
`referrerpolicy="no-referrer"`. Only HTTPS FairyStack origins may frame it.
The response's CSP permits only hashed renderer scripts/styles and prohibits
network requests, forms, other frames, and same-origin privileges. It accepts no
HTML, script, credential, or caller-supplied URL.

On iframe load, transfer one MessageChannel port using
`postMessage({type: 'observatory:connect'}, '*', [port2])`. The wildcard is needed
for the opaque sandbox origin; transfer only the port, not evidence or tokens.
The renderer checks the sending window and FairyStack origin, binds once, and
returns `{type: 'ready'}` on the private channel. Send the report on port1 only
after readiness. It replies `{type: 'rendered'}` or
`{type: 'failed', error: '...'}`. The host must terminate a missing readiness or
render acknowledgement within 12 seconds and close the channel on removal.

Reports have `schema_version: 1`, `source_id`, `label`, `status` (`observing` or
`action_required`), `checked_at` (Unix seconds), `samples` (at most 1440), and
`annotations` (at most 82). Each ordered sample has `timestamp`, `load` (5-minute
load average), `cpus` (positive integer), `memory` and `disk` (used percentages).
Unavailable metrics are explicitly null. An annotation has `timestamp` and a
plain-text `label` (at most 300 characters). Invalid reports fail visibly.
Load is graphed per CPU, separately from capacity percentages. Missing samples
and collection gaps over three minutes break the line. Zoom, selected evidence,
focus, and unchanged nodes survive report updates.

FairyStack owns pair-health collection, bounded history, and incident delivery in
this first integration; Observatory owns the renderer. The embed does not store
reports or connect an Observatory user's account. Monitoring migration requires
a separate authenticated connector; this endpoint does not grant that access.


### Warning history and coverage

The default embed window includes both recorded warnings and sampled measurements.
`alerts` is an optional array (at most 200) of archived mailbox observations with
`id`, Unix-second `timestamp`, `kind: recorded_alert`, plain-text `label` (300
characters), `detail` (4000 characters), and nullable `load` / `cpus` values.
These measurements are isolated points, never interpolated into sampled history.
`unattributed_alerts` has the same shape but is explicitly labeled source unknown
and never plotted as a source's load measurement. An annotation may also preserve
its recorded numeric `load` and positive integer `cpus` after samples expire.
The time axis includes events outside the sampled interval; coverage is stated
above the chart. Warnings are selectable above the plot, with original text in a
shared disclosure. Explicit recent windows may exclude earlier events.

The renderer sends `{type: resize, height: <pixels>}` over the private port when
its content changes. Hosts may adjust the frame height within a bounded range
(350–850 pixels); this grants no navigation or parent DOM access.


## Historical server-health links

`GET /api/server-health?start=<ISO UTC>&end=<ISO UTC>&resource_id=<EC2 ID>`
uses the authenticated user's AWS connector and configured monitored server.
Both bounds are required together. A mismatched resource returns 400; missing
configuration returns 409. Queries have a 25-second overall deadline (504 on
expiry), and per-series missing data or upstream failures remain explicit.
The default remains the latest 24 hours. Historical bounds also work on
`POST /api/metrics/query` as `start` and `end` instead of `hours`; they are
included in cache identity and limited to 30 days and 1,440 points.

An existing server-health chart can open directly on an incident with
`/?health_start=<ISO UTC>&health_end=<ISO UTC>&resource=<EC2 ID>&warnings=<JSON>`.
`warnings` is a URL-encoded array of up to 20 `[Unix seconds, load, CPU count]`
records. These are caller-provided event annotations, not CloudWatch metric
values. The historical view shows the authenticated CloudWatch CPU, RAM, and
disk percentages, with warning times marked separately at the top of the chart.
It verifies server identity and omits unrelated cost panels in this view.


### Authenticated history in infrastructure embeds

A trusted FairyStack parent may set `history_enabled: true` on an infrastructure
report. The renderer requests `{type: 'history-request', key: <warning key>}` on
the existing private port. The parent authenticates its own inbox-history API,
which derives the episode window and EC2 identity from the owned incident.
Reply `{type:'history',key,status:'completed',data:<server-health data>}` or
`{type:'history',key,status:'failed',error:<plain text>}`. Historical data must
include explicit ISO `start`/`end`, a period, and CPU/memory/disk series. The
renderer validates percentages, ordered timestamps and incident coverage.

No credential or new network permission is added to the embed. The parent owns
a 35-second lookup deadline; the renderer has a 37-second missing-reply deadline.
Failures keep the original warning Details accessible. Selecting an older episode
requests its own window; unchanged evidence preserves plot nodes and selection.
The same percentage plot renderer is shared with Observatory's authenticated
historical view. Without `history_enabled`, recorded-load rendering remains
available for older parent versions during coordinated rollout.

OpenAI exposes `usage_today` (UTC date, availability, input/output tokens and
model requests), `observed_at`, and `latest_billing_date`. Current-day usage is
shown independently of billed dollars: absent billing is pending, never proof of
zero spend. Usage counts cover the organization's completions usage endpoint;
they are not credit charges. The legacy static token-price estimates must not be
used to reconcile credit balances. Source contract:
https://platform.openai.com/docs/api-reference/usage

## OpenAI usage freshness

OpenAI's completions usage API supports minute buckets; bucket width alone does
not guarantee ingestion freshness. Its Costs API currently uses daily buckets.
For an expensive running job, BrightWrapper's dashboard shows recorded
per-request costs sooner (Observatory does not duplicate it); final charges for a request may remain unknown until it finishes.
Sources: https://developers.openai.com/api/reference/resources/admin/subresources/organization/subresources/usage/methods/completions
and https://brightwrapper.com/agent-guide.md

## Prepaid provider credit balances

OpenAI and Anthropic cards show prepaid API credit balances separately from
spend, subscription allowances, and BrightWrapper credits. Their connected
public admin APIs do not currently provide a supported remaining-balance read.
Without a confirmed observation, the cards explain that the connected API does
not expose a prepaid credit balance and link directly to the provider's billing
page. “Connected” describes the cost feed, not live credit-balance access.
Absence is never rendered as zero. Refreshing costs cannot retrieve a balance.

An authenticated agent that has actually read the matching provider account's
billing page can record that observation with
`PUT /api/credit-balances/openai` or `/api/credit-balances/anthropic`:
`{"balance_usd":123.45,"observed_at":"2026-09-09T05:00:00Z","account_label":"Organization name"}`.
This is an example payload, not an account balance. Do not infer this value from
usage costs, assume grants or top-ups, or supply an unverified number.

`GET` on the same route returns the observation or explicit unavailable state.
`DELETE` removes the observation. Writes require an existing provider connector,
finite USD amount (negative balances are supported), a timestamp with timezone,
and an account label. Older observations cannot replace newer ones. Records
are user-scoped and bound to the current connector revision; replacing or
removing that connector invalidates the old account's balance. The card always
labels a stored value “last confirmed”, and marks it stale after 15 minutes.
Provider-series responses include `credit_balance` independently of cost caches.

Sources: https://help.openai.com/en/articles/8264644 and
https://support.claude.com/en/articles/8977456-how-do-i-pay-for-my-claude-api-usage

## Disk usage treemap

`/storage` is the authenticated disk monitoring page. Each owner may enroll up to
128 EC2 servers or personal devices without changing the existing CloudWatch health preset or billing
connector. The server links select a fixed instance; unknown or foreign IDs fail
rather than displaying a different server. Existing single-server scans migrate
atomically and remain the default view. Selecting a directory drills down; file
names and directory sizes are private account evidence. There are no deletion controls.

### Enrollment and collector credentials

Owner authentication is the normal Observatory JWT or app-specific AuthReturn key.
Never copy that account credential to a collector.

- `GET /api/storage-resources`: `{resources:[{resource_id,label,provider,updated_at,collector_enabled?}]}`.
  The existing health preset is included even before explicit storage enrollment.
- `POST /api/storage-resources`: `{resource_id:"i-…",label:"Finance app box"}`;
  idempotently enrolls or renames one resource, returning `{resource}`. Personal devices use `device-<24 lowercase hexadecimal FairyStack device ID>` and return `provider:"device"`; existing EC2 resources return `provider:"aws"`. EC2 IDs must
  have 8 or 17 lowercase hexadecimal digits. Labels have 1–80 characters.
- `PUT /api/storage-resources/<resource_id>/collector`: returns `{resource_id,
  revoked:false,ingest_token}`. This **rotates** the server's upload-only key;
  store the response directly in SOPS. The plaintext is shown only once. Lost
  responses require an explicit rotation; this endpoint must not be auto-retried.
- `DELETE /api/storage-resources/<resource_id>/collector`: idempotently revokes
  that server's key immediately. Retained scans remain readable. Missing or
  foreign resources return 404.
- `DELETE /api/storage-resources/<resource_id>`: removes a retired server with its
  collector key, scans, disk samples and vitals history. Missing or foreign
  resources return 404.
- `PUT /api/server-storage/ingest`: accepts only a dedicated `obs_storage_…`
  bearer key. It can upload only that key's owner and storage resource; it may GET `/api/storage-client` for its client updates, but cannot read
  snapshots, enroll/rename servers, rotate credentials, access billing, or execute
  commands. Wrong resource returns 403, invalid/revoked key 401, invalid schema
  422. Keys are stored only as SHA256 hashes. Responses and owner APIs are no-store.
- `GET /api/server-storage?resource_id=<EC2 ID>`: owner-scoped `{available,
  resource,resources,snapshot?,stale?}`. Omit the parameter for the existing health
  preset, or first explicitly enrolled server. Missing scans return
  `available:false`; unconfigured/foreign IDs return 409. Legacy owner-authenticated
  `PUT /api/server-storage` remains supported for existing installations.

Each upload has schema 1, `resource_id`, 32-lowercase-hex `run_id`, epoch-second
`started_at`, `deadline_at` (at most 960 seconds later), and status
`running|completed|partial|timed_out|failed`. Terminal results include
`completed_at`; directory results include `total_bytes`, `used_bytes`,
`available_bytes`, `reserved_bytes`, `scanned_bytes`, `entries`, up to 50,000
`{path,bytes,kind,groups?,leaf?}` nodes including `/`, `granularity_bytes` (1 GiB),
at most 20,000 small-entry group byte counts, and up to 20 `{path,error}` entries.
`kind` is `directory` or `file`. Payloads are at most 16 MiB. Uploads return
`{stored:true}`; obsolete/conflicting runs return `{stored:false}`. Exact terminal
retries are idempotent. Older scans never replace newer scans. A running scan
past its deadline is persisted as `timed_out`; a late worker cannot resurrect it.
Previous directory evidence stays available and visibly labeled while a scan runs
or fails. No received scan and a scan older than two hours remain explicit.

### Install on an explicitly authorized server

The existing `scripts/collect-storage.py` scans allocated blocks on the root
filesystem, without following symlinks or crossing devices. Hard-linked files
count once. It sends directory names, large-file names and sizes, **never file
contents**. Names may still be sensitive: enrollment requires the owner's consent.
Directories and individual files of at least 1 GiB are retained; immediate smaller
directories are named leaves, and smaller files/metadata are grouped. Missing
bytes can include filesystem metadata, open deleted files and scan gaps.

Persist `{resource_id,ingest_token}` in host-specific SOPS
`hosts/<host>/observatory-storage.json`, then run the canonical secrets sync.
Run `sudo python3 scripts/install-storage-collector.py --resource <EC2 ID>` from a
verified copy of the committed scripts. The installer verifies local IMDSv2
identity and the matching scoped secret, provisions a root-owned 0600 consumer,
and enables the hourly scan and minute reading timers. It refuses to replace an active collector. Record
the deployed commit alongside the installation receipt. No inbound listener,
SSH credential, AWS account key or general Observatory key is installed.

The collector runs with a read-only filesystem, one read-only secret mount,
only `CAP_DAC_READ_SEARCH`, low I/O priority, 25% CPU quota and a 512-MiB memory
limit. Root-level metadata access is needed for complete sizes; the script is
root-owned. HTTPS uploads refuse redirects. Idempotent uploads retry connection failures and HTTP 502/503/504 up to three times within a 20-second total deadline (at most 15 seconds per call); authentication, validation, and other failures are terminal. The scan
has a 900-second deadline, the process a 960-second alarm, and systemd a 990-second
limit. Scan errors become partial/failed results; lost workers time out in the
server view. The timer deduplicates scans through one systemd unit. Browser reads
terminate in 15 seconds, pause observation while hidden and stop at terminal state.

### Computer view

`/computer` frames the Computer diagram (computer.jessald.fairystack.com `?embed=1`)
for any instance in the connected AWS account. `GET /api/computer` (authenticated,
read-only, 15 s cache, 25 s deadline) returns `{servers:[{instance_id,name,state,
instance_type,collector}],machines:[…]}`. Each machine carries CloudWatch
`cpu_percent` (latest 5-minute average) and `cpu_count` from the instance type for
every server; memory, swap activity, load, filesystems, disk I/O, network, ports,
top processes and systemd services come only from that server's own collector and
are absent otherwise, which the diagram shows as not reported. Readings older than
five minutes are labelled stale. The enrolled collector pushes one ~1 s sample every
minute with `collect-storage.py --reading` (`observatory-reading.timer`) to
`PUT /api/server-vitals` `{schema_version:1,resource_id,reading}`, using the same
scoped `obs_storage_…` key (or the account's agent key for the primary resource);
only the latest reading per server is kept. Both installers install that timer.

`GET /api/storage-health` returns owner-scoped UTC daily counts for the last 30
days: attempts, terminal, failed, timed_out, partial, completed, and max_seconds.
Compact run receipts are retained for 90 days; recovery never erases an earlier
failure. Existing current/previous snapshots seed the history once, without
inventing older attempts. Reading health also terminates expired workers through
the snapshot timeout owner.
Collector upload keys cannot read this endpoint.

### Mac devices through FairyStack

Use the owner's existing FairyStack `/api/companions` connection to install
`scripts/install-storage-mac.py`; no new desktop app, inbound port, administrator
access, or account credential is needed. Enroll `device-<paired device ID>` using
the owner API, then issue its dedicated collector key. Pairing itself never enrolls
a laptop: the owner must authorize storage telemetry (directory names are private).

The installer receives JSON on stdin with `resource_id`, `origin`, `private_key`,
`encrypted_token` and the committed `client_source` (`scripts/storage-client.py`).
Generate an ephemeral RSA-2048 private key on the Mac in a mode-0700 setup directory;
return only its public key. Encrypt the scoped token using RSA OAEP SHA-256 on the
agent host. Only ciphertext may enter FairyStack command receipts. The installer
decrypts on the Mac, saves the key in login Keychain under
`com.fairystack.observatory-storage.<resource_id>`, verifies read-back, and removes
the ephemeral private key after successful installation. Keychain writes use stdin,
not process arguments. No token is persisted in a plist, config, script or log.

The user's LaunchAgent runs on login and hourly while awake. Its mode-0700 working
directory is `Library/Application Support/FairyStack/Observatory/<resource_id>`;
config contains no credential. It scans `/System/Volumes/Data` under the user's
normal rights without elevating privileges. Install Homebrew GNU coreutils with
`brew install coreutils` before installing the Mac collector. It uses `gdu` from
`/opt/homebrew/bin` or `/usr/local/bin` with NUL-terminated records, preserving
newlines, tabs and other filename characters without ambiguity. Native traversal
avoids millions of Python stat calls; hardlinks still count once. Empty directories
can be grouped with metadata. A missing dependency is an explicit failed scan.
macOS privacy denials yield partial
scans; the UI never presents them as full-disk coverage. Sleeping/offline laptops
retain the last observation and show stale after two hours. No file contents are
read or uploaded. This client does not delete files or change cloud sync settings.

For protected app data, the Mac owner must enable the collector's Python executable
in System Settings → Privacy & Security → Full Disk Access. macOS does not provide
a command to grant that permission. Resolve the installed Python executable before
adding it; an interpreter upgrade may change its path. Do not edit the privacy database
or disable platform protections. A scan after the grant verifies actual coverage.

`GET /api/storage-client` accepts only a live storage collector bearer and returns
`{schema_version:1,version,files:{"collect-storage.py":{source,sha256},
"storage-client.py":{source,sha256}}}`. It exposes only these client files. Each
hourly run verifies HTTPS (redirects refused), payload shape, SHA256 and Python
syntax before atomic file replacement. The new scanner runs immediately; updated
launcher code runs on the next hourly invocation. No update failure silently falls
back to old code: it records a failed observation when upload is available. Scans
include `scan_root`, `collector_version` and `collector_revision` (scanner SHA256),
visible in the page's Collector details. No perpetual process must restart to update.

One file lock prevents overlapping scans; the installer refuses replacement of an
active process. The client has a 990-second overall deadline, scans have the existing
900-second deadline, and Keychain operations have 15-second timeouts. Recurring client updates and uploads share the bounded 20-second transient-retry policy; initial installation has a 15-second download timeout. Durable
local `status.json` records the current step and terminal state. A lost collector
uses the existing server-side 960-second timeout. Never-resolving uploads cannot
leave the page indefinitely running. Revoke with the normal collector DELETE API;
uninstall by booting out the matching user LaunchAgent, deleting its plist and
Keychain item. Retained Observatory observations remain readable.

### Resource links

- `https://observatory.aisloppy.com/storage` — current account's default server.
- `https://observatory.aisloppy.com/storage?resource=i-05938f43701e1eef7` — Finance
  app box, visible only to the account that enrolled it.
- `/storage?resource=<EC2 ID>` — IDs returned by `/api/storage-resources`; ordinary
  owner authentication is required. These links select current evidence, not an
  immutable snapshot; the scan timestamp/run ID identifies the observation.

Fleet rollout should reuse this collector and enrollment API with a separate
owner/resource key for every host. This release does not automatically enroll
other customer instances or centralize their filenames under an administrator.
Existing FairyStack capacity alerts remain owned by FairyStack.

## Python release dependencies

Releases install `requirements.lock` with FairyStack’s protected uv wheel cache.
The lock records the complete installed package versions and SHA256 hashes.
Development `.venv` edits do not enter releases. Dependency updates must regenerate
the lock and pass the release import check before activation. Release environments
are read-only; identical wheels share storage across retained releases.

## Unified analytics (v5)

Observatory combines costs/infrastructure, usage, and telemetry. Account data is
isolated by verified Observatory identity. Usage collection and history are native to Observatory. Telemetry profiling history
can still be connected through `PUT /api/connectors/telemetry`, with
`{"config":{"api_key":"ar_…"}}`. Credentials are encrypted per account. Source reads have an
8-second overall deadline and report errors rather than fabricated zero totals.

### Native collection

- `POST /api/analytics/projects`: authenticated owner, body
  `{"name":"my-service","origins":["https://my-service.example"]}`. Returns
  `{id,name,origins,ingest_token}` with a **write-only key shown once**. Duplicate
  names return 409. Keep keys in SOPS; never expose them in browser code.
- `GET /api/analytics/projects`: owned and explicitly shared projects as `{projects:[{id,name,label,origins,access}]}`; read access does not grant ingestion or management authority.
- `POST /api/analytics/otlp/v1/{traces|logs|metrics}`: Bearer project ingestion key,
  OTLP/HTTP JSON or binary protobuf, optional gzip, max 2 MiB decoded / 2048 records.
  Traces retain measured start/end, parent identity, attributes and status; logs
  retain structured bodies; metrics retain type, unit, temporality and data point.
  Batches commit atomically. Exact retries are idempotent; changed immutable span
  identity is 409. Invalid encoding/schema is 422; unavailable storage is 503.
- `POST /api/analytics/browser/{project}`: compatible Faro envelope, max 64 KiB;
  only exact enrolled HTTPS Origin values accepted, 120 batches/IP/minute. This
  public endpoint stores sanitized browser signals and cannot read or write
  server signals. Exception messages and arbitrary event attributes are omitted.
- `GET /api/analytics/observations?project={id}&kind=span&hours=24&limit=200`:
  owner or explicitly authorized viewer; kinds `release`, `preparation`, `span`, `log`, `metric`, `event`, `load`, `failure`; hours 1–720;
  limit 1–1000. Optional `trace={32 lowercase hex}` reads all retained spans for
  the trace independent of the time filter. When `truncated` is true, pass the
  returned `next_cursor` as `cursor` with the same query filters to continue a
  stable time window. Returns records, truncated, next_cursor, window_end, limit,
  retention_days. Retention is 30 days. General traffic is capped at 100,000 records/project; release and preparation traces have a separate 100,000-record budget. Their oldest whole traces are evicted together, so noisy logs and metrics cannot displace release history.
- `GET /api/analytics/sources/telemetry` reads the current account's connected profiling source through its public API. It defaults to the latest 100 profiles; `limit=50|100|250|1000` and exact `project`, `kind`, or `id` filters are sent upstream before reading. `requested_limit` states the selected bound. This compatibility API retains older external evidence for agents; it is no longer called by the Profiles page. The former Grid Glance source route redirects to native `/api/usage/overview`.

### Request profiles

Profiles uses the existing native observation/trace reader at
`/analytics?kind=span`. It reads the selected authorized project's newest 200
spans in the chosen window; truncation stays explicit. Select a chart sample or
row to open the full trace (up to 1,000 retained spans) with measured start offsets
and durations. This includes instrumented HTTP endpoints and dependencies, not
automatic sampling of uninstrumented code. `/profiles` redirects here. It has no
runtime dependency on the older Telemetry collector or its connector credential.

### Resource links

| Browser route | Identity / authorization |
|---|---|
| `/analytics?project={id}&kind=release` | Project ID from projects API; owner login required for records |
| `/analytics?project={id}&kind=span&trace={trace_id}` | Immutable recorded trace; 32-character hexadecimal trace ID; owner-only |
| `/profiles` → `/analytics?kind=span` | Current account-authorized request and dependency traces |
| `/` | Existing costs and infrastructure analytics |

Example trace URL: `https://observatory.aisloppy.com/analytics?project=PROJECT_ID&kind=span&trace=TRACE_ID`.
IDs in this template are placeholders, not existing evidence. Pages are public
shells; all account evidence requires authentication. Missing/expired evidence
renders an empty state. Views refresh on demand, preserve local filters and
selected details, and terminate failed reads within 12 seconds.

Histograms are shown with their recorded sum/count and type in the detail view;
they are not silently treated as individual duration samples. This is a bounded
collector and evidence browser, not a PromQL or LogQL query engine.

Owner-only `POST /api/analytics/projects/{id}/quarantine` accepts
`{"trace_ids":["32-character-hex-id"],"reason":"Proven test contamination"}`.
It excludes those exact traces and their logs from every ordinary read, including
trace links and later replays. Evidence remains retained under normal retention;
the quarantine keeps its reason and timestamp. This is an agent maintenance API,
not a revision-pattern filter: never quarantine a real release by guessed naming.

### Release stage chart

`/analytics?project={id}&kind=release` presents a single stacked column chart,
chronological by completion, one column per attempt (repeated revisions are not
collapsed). It restores the release categories: backend tests, frontend tests,
preparation, signing/delivery, customer activation, marketing activation, queue,
and unattributed time. Colors stay fixed across refresh and queue visibility.
Values are minutes; the queue checkbox subtracts only recorded queue phases.
Columns with missing or inconsistent stage evidence are marked `?`, not inferred.
Hover/focus/tap uses Info Elements’ shared anchored-tooltip component, loaded with
an 8-second deadline. Selecting a column opens the existing measured trace.

Owner-only `GET /api/analytics/releases?project={id}&hours=720&limit=1000` returns the
ordinary release envelope plus each record's `phases:[{name,duration_ms,started_at}]`
from retained OTLP stage spans. Tenant boundaries and quarantine apply. Phase
spans are fetched together, not through one request per chart column. Releases default to 720 hours (30 days). When `truncated` is true, send the returned `next_cursor` with the same project/hours/limit to read the next page; cursors preserve the initial window end and are scoped to the query. The chart reads all pages within a 30-second deadline. Changing Window resets chart zoom; refresh within the same window preserves it. The URL retains the selected hours when opening or closing a trace.

Release speed cards compare the mean backend-test time and total release time of the latest 10 completed, measured releases against the preceding 10 in the loaded window. Total release always includes queues, independently of the chart toggle. Failed attempts and missing stage evidence are excluded from this comparison; fewer than 20 measured completions shows an explicit insufficient-history state.

Release durations default to hourly mean stacked bars, with flush edges and no borders. Average offers individual releases, 1 hour, 6 hours or 1 day. Buckets use fixed UTC boundaries, labeled in local time. Each stage uses the same denominator: releases with complete stage evidence, including failed attempts; missing evidence is excluded and counted in the sidebar. Include gaps defaults on: empty intervals remain blank rather than becoming zero-duration releases. Turning it off compresses only empty intervals; recorded attempts with missing evidence remain visible. Averages and their denominators are unchanged. Interval details include outcome counts and Show individual releases drills into the exact bucket. Latest 10 shows individual releases. Info Elements owns
both drag-range gestures (`chart-range.js`) and hover/focus/pinned details
(`chart-sidebar.js`, `<chart-detail-sidebar>`). The shared components load under
an 8-second deadline and fail visibly; they are not replaced with local copies.
Drag horizontally to zoom; Reset zoom and Latest 10 provide keyboard alternatives.
Refresh defers chart data changes during a drag and preserves the chosen range
and pinned sidebar. The sidebar's Open measured trace opens the existing trace
view. Missing evidence leaves its attempt unfilled and marked ?, rather than inventing zero-duration stages.

The release sidebar passes each category’s canonical color through Info Elements’
`metrics: {label: {value, color}}` contract; swatches match the connected bands and legend.

### Test-run evidence

Analytics storage uses SQLite WAL so readers retain a snapshot without blocking
writer commits. Ingestion applies the same retention limits with covering-index
counts and trims only excess rows; protected release traces are ranked only when
their dedicated cap is exceeded. Write-lock contention has a one-second bound and
returns HTTP 503 `Analytics database is busy; retry the request`; internal logs
retain the SQLite cause. Reads retain their four-second deadline.

`POST /api/analytics/test-reports` accepts a project write-only ingestion key and
an immutable envelope `{id,revision,trace_id,kind,report}`. `id` and `trace_id` are
32 lowercase hex characters, revision is 40; `kind` is publisher or background.
Publisher reports require their release trace. The report schema is version 1,
with gate/full suite, terminal status, updated_at, collected/recorded counts,
elapsed_ms, unaccounted_ms, optional cpu_ms, optional budget `{seconds,exhausted}`
(positive seconds and boolean exhausted), tests (50 slowest with setup/call/
teardown/duration and outcome), files (1,000 aggregates), optional results (10,000
complete per-test outcomes/durations). A completed budgeted run may have fewer
recorded than collected tests: these were not run, not passed or failed. The UI
shows the test-start budget and the unrun count; in-flight tests finish after
the budget, so elapsed time may exceed it. No extra pass circumvents the budget.
Max request 3 MiB. Unknown fields are
excluded. Retries are idempotent; conflicting identity returns 409. Retention is
30 days or 200 reports per project.

Owner-only `GET /api/analytics/test-reports?project={id}&revision={sha}&trace={trace}`
returns `{reports,retention_days}` for the exact publisher attempt and up to two
recent background runs for that revision. Missing reports are unavailable, never
implicitly passed. Quarantined release traces remain excluded.

Resource link: `/tests?project={project_id}&revision={40hex}&trace={32hex}` opens
the per-file and per-test drill-down. Same owner authorization as release traces;
public page shell does not grant access. The Backend tests sidebar metric links
here. Run choice, search, outcome and file filters are local drafts preserved
on refresh. Source reads terminate within 12 seconds.

Release charts default to individual releases, with Include gaps and Include queue wait unchecked. Optional hourly/daily intervals
show means across their attempts, which can differ from a selected release's
trace. Chart categories and measured trace stages share the same display labels;
combined categories retain their specific substep after the category name.

### Signing and delivery drill-down
`/analytics?project=<24hex>&kind=release&trace=<32hex>&stage=delivery` focuses the selected release's retained delivery intervals. Linked from the release sidebar. Owner authentication is required; immutable trace IDs select the attempt. Example: `/analytics?project=01badc780e971908930f965e&kind=release&trace=<release-trace-id>&stage=delivery`. Current publishers measure bundle creation/verification, checksum/signing, and upload/acknowledgement per allocation; older evidence has combined notification intervals. Missing time is explicitly unmeasured. Detail spans do not contribute additional time to top-level release stacks.

### Navigation and report discovery
Every full app page uses `/observatory-header.js`, the single `observatory-header` component. The Observatory brand links to `/`; navigation labels, order and active-page styling come from this shared component. Embedded infrastructure incident charts keep their compact embedded title.
`/analytics?panel=tests` and old Preparation listings open `/analytics`.
Individual `/tests?project=<24hex>&revision=<40hex>&trace=<32hex>` reports remain
available through release evidence. `GET /api/performance` returns administrators
the latest 100 cost-page request/server/render samples.

### Retained preparation evidence

Preparation has no standalone product page or navigation entry. Old Preparation
URLs open `/analytics`. Its existing ingestion and authenticated evidence API
retain operational records for agents; this UI cleanup does not delete history.

Project-key OTLP logs with body `kind: preparation` require operation, pair_id,
attempt_id and terminal outcome (completed/failed/timed_out/cancelled).
Optional duration_ms may be null for absent measurements. Companion spans carry
actual start/end times and parent identity. No private error text or credentials
are exported from preparation receipts.

Native reads use kind/time and kind/trace indexes so timing reads do not scan
unrelated runtime telemetry. SQLite query execution has a four-second deadline;
expiry returns 504 with a concrete retry message. The browser retains its
12-second overall read deadline, deduplicates concurrent initial loads, and
preserves edited filters on refresh.

Preparation records retain their immutable trace identity and distinct retries.


## Usage analytics

Usage analytics (tracker, visits, browser IDs, sessions, KPIs, device resources) moved to Grid Glance on 2026-10-07: https://gridglance.com/agent-guide.md. `https://observatory.aisloppy.com/api/usage/*` forwards there for trackers and integrations still using that URL, and the old `/usage`, `/usage-app`, `/usage-session` and `/resources` pages redirect to gridglance.com.

## Browser app loads and observation charts

`/analytics?project=<24hex>&kind=load` (View: Page loads) charts an enrolled
app's browser loads, capped at the 30-day navigation retention period and newest
200 samples. Enrollment's `view_url` is
that URL. Select a row for the immutable navigation's stage/resource timings;
`trace=<32hex>` preserves that selection.

`GET /api/analytics/observations` accepts `kind=load`, selecting only
`fairystack_load` browser events, and the same trace API supplies measured spans.
Each load record carries `slow: true` when it took over 10 s (`SLOW_LOAD_MS`); the
Page loads table marks it.

### Slow loads

`GET /api/analytics/slow-loads?hours=24&limit=100[&project=<24hex>]` (owner or
viewer; hours 1–720, limit 1–500) lists every readable app's loads over
`threshold_ms` (10,000), newest first, so nobody has to paste timings. Check it
before load-performance work. Each entry: `app`, `project`, `trace_id`,
`trace_url`, `time`, `duration_ms`, `status`, `name` (root step), `client`
(device and browser from the collector request's User-Agent, e.g.
`iPhone · Safari 26.5`; `Headless Chrome` is automation, not the operator),
`network_ms` (time with at least one request in flight), `no_request_ms` (the
rest: app/main-thread work or idle waiting), `transfer_bytes`, `resources`,
`mbps` (bytes over network-busy time, a lower bound on link speed),
`median_response_wait_ms` (request to first byte), `hidden_ms` (page in the
background), `browser_downlink_mbps`/`browser_rtt_ms` (Chromium estimates only).
`null` means the trace did not record it (profiler before 2.1.0). A slow
connection shows as high `network_ms` with low `mbps` and high wait; an app
defect shows as high `no_request_ms`, or as many megabytes or requests the step
did not need. Flags are evidence for an agent to read, not an automatic repair.
The browser uses the existing bounded analytics reader and ignores stale responses
when the app, window, or selected navigation changes.

The existing origin-restricted browser collector accepts `fairystack_load` events with the existing allowed status, kind (failed/current step), duration_ms and trace_id fields. Its related OTLP browser spans preserve only bounded numeric `load.*` fields: transfer/encoded bytes, resource count/omitted count, connection downlink, explicitly emulated Mbps (zero means no declared emulation), milestone/resource flags, response wait/download milliseconds; plus sanitized status, step and version. No URL queries, API identifiers, message content or credentials are recorded. Static asset names are retained; API requests and cross-origin resources use generic labels.

FairyStack's lightweight recorder starts near the beginning of HTML parsing and uses navigation's time origin. Terminal completion means the restored initial surface is rendered, window load fired and images belonging to that first conversation snapshot settled; it excludes future polls, later images and optional telemetry SDKs. A failed initial image is a failed load, not successful completion. Startup has a 180-second deadline from navigation, page exit records cancellation, and collection has a separate 15-second deadline. A reload is a new trace. Recorder delivery failure is exposed locally in `FairyStackLoadProfile.state()` and console; a lost/unreachable collector cannot promise a retained record. Browser/process termination before delivery can leave a gap. Human login time can be included; the authentication milestones identify it.

Waterfalls contain elapsed-from-navigation milestones and overlapping resource intervals: do not sum them. At most the 60 longest completed resource requests are retained; root fields show omissions. In-flight requests have no final resource duration. Zero transferred bytes may mean a cache hit or cross-origin timing restrictions. `load.downlink_mbps` is a browser estimate, not a measured speed test. Synthetic evidence must declare `load.emulated_mbps`; it is not a measurement of the operator's connection.

## Default app profiling

`PUT /api/analytics/sites/<exact-hostname>` enrolls one browser-only project for
an authenticated Observatory owner. Repeating it returns the same project;
other owners receive separate projects. Response: `{status:"completed",
project_id,origin,viewers,collector_url,script_url,view_url}`. No server ingestion key
is issued. Invalid hostnames return 422, missing authentication 401, busy storage
503. Use a ten-second request timeout.

The optional body `{"viewers":["operator@example.com"]}` (at most 10 emails)
replaces the project's read-only viewers; omitting it keeps them. A viewer signed
in with that verified email sees the project in `GET /api/analytics/projects`
(`access:"viewer"`, with a readable `label`) and reads its observations, releases
and test reports. Quarantine and enrollment remain owner-only; every other
account receives 404 `Telemetry project not found`. FairyStack grants the
installation owner so projects its agent enrolls are readable under the owner's
own login.

The `/app-profile.js` SDK has one owner: this file. FairyStack serves it at every
app's same-origin `/observatory-profile.js` from its ingress, through a short
nginx cache that keeps the last good copy when Observatory is unreachable, so
apps pick up a new version within minutes without a copy or a redeploy. Apps
keep only the include: `<script src="/observatory-profile.js" data-project="PROJECT_ID"
data-completion="manual"></script>` early in the head. The panel and each load
event (`profiler_version`) show the running version. It automatically appends
one collapsed, selectable timing tree. `ObservatoryProfile.begin(name,parentId)`
returns a span ID; `end(id,status,name)` closes it, optionally renaming it (for example with the received size). A fetch the app timed with `begin()` immediately before calling it is not listed again as a resource. `start(name)` begins a fresh measured navigation in the same collapsed tracker; `finish(status)` closes it. The tracker defaults on and offers an origin-scoped browser opt-out inside its disclosure. Opt-out survives reload and can be reversed with Enable and reload. Record/export pages omit profiling. `finish(status)` terminates the
initial load after data has rendered; statuses are completed, failed, cancelled
or timed_out. Without manual mode, window load completes the profile. Startup
has a 60-second deadline; collector delivery has a separate five-second deadline
and visible failure state. Reload creates a new trace; page exit cancels pending
startup. Resource intervals are captured automatically, named by path with numeric, hex and UUID segments masked as `:id`, plus transfer size. Since 2.1.0 the root span carries `load.network_ms`, `load.transfer_bytes`, `load.resources`, `load.omitted_resources`, `load.hidden_ms`, median `load.response_wait_ms` and, where the browser reports them, `load.downlink_mbps`/`load.rtt_ms`; Observatory's own uploads are excluded. `start()` clears the resource-timing buffer and the buffer grows to 8,000 entries, so long single-page sessions keep recording requests. The panel's Copy button copies the whole trace as compact text (page URL with credential-like query values redacted, user agent, network evidence, trace link, then every step as start/duration/name in start order) for pasting to an agent. Server-Timing durations appear locally as duration-only bars; their start offsets are unknown and are not exported as measured trace intervals.

Backend spans can be returned in the app's ordinary authenticated API response
and attached with `addSpans(rows,parentId,requestStart)`: rows contain id, parent,
name, start, duration (milliseconds from request start), status. Parent rows must
precede children. Only measured, non-sensitive names and intervals belong here.
No response bodies or URL queries are collected. This is elapsed-time tracing,
not a sampling CPU profiler; uninstrumented server internals remain unknown.
Browser evidence cannot independently authenticate server spans. Bars overlap
and must not be summed. At most 60 supplied spans and 60 resources are included.


Mac release handoffs are measured gaps between named pipeline stages, displayed
separately as “Release handoffs (internal steps unmeasured)”. Their boundaries do
not measure the internal distribution of packaging, dispatch, or waiting. Failed
update-check attempts remain Update verification spans. Release axis labels use
local 12-hour time with AM/PM.


### Anthropic reported costs

Anthropic chart values and `billed_cost` come from the organization's
`/v1/organizations/cost_report`, including input, cache, output and tool charges.
Amounts are converted from decimal cents to USD; all pages are imported within
a 90-second deadline. No static model-price fallback is used. The old
`estimated_cost` and incorrectly parsed token counts are retired. Existing
estimated history is replaced across all four windows by a full annual import.
Old estimates are not shown while that import is pending or has failed.

`cost_source`, `cost_note`, `observed_at` and `latest_billing_date` explain the
provenance. Days are UTC; a missing bucket is unreported, not confirmed zero.
Credit purchases, Claude subscriptions and Priority Tier charges are outside
this report. See [Anthropic's contract](https://platform.claude.com/docs/en/manage-claude/usage-cost-api).

## Session spending

Connect through `PUT /api/connectors/session_spend` with `config` containing `fairystack_sources_json` (JSON string of `{origin,api_key}` for Jessald and/or Finance; dedicated owner keys with only `costs:read`) and `brightwrapper_api_key`. Source keys are encrypted by the existing per-user connector store and never returned. Collection reads only network APIs and exports no prompts. The scheduler refreshes snapshots every 15 minutes with an 11-second source deadline; failed sources keep their last good snapshot, labeled stale/error. The snapshots feed the recorded-cost explorer.

Private session-spending sources may include `connect_address`, a Tailscale IPv4 address, alongside origin and api_key. This connects over the existing private network while retaining the origin hostname for SNI, certificate verification and Host; TLS verification is mandatory. It is useful where a release-only localhost DNS override otherwise captures the control hostname. No public ingress or release routing changes are required.


### Current-day cost floor

The default chart's UTC current-day point uses the maximum of provider-reported
costs, BrightWrapper recorded costs, and the cached full-day AI estimate.
BrightWrapper's `/api/usage/daily-costs` is read with the owner's existing
`session_spend` connector key, never a shared service account. Only OpenAI and
Anthropic recorded charges supplement their matching cost series; Gemini
charges are not folded into GCP infrastructure billing. BrightWrapper charges
include markup/rounding and overlap provider reports, so they are never added. Fractional cents from the recorded ledger are preserved until dollar amounts are formatted.
They exclude unpriced and unfinished requests; the tooltip identifies recorded
spend separately from the full-day estimate. Historical billing and reported
provider totals remain provider-reported.

Scheduled forecasts also receive today's recorded spend. A stale or missing AI
forecast cannot suppress new recorded spending: `/api/projection` reapplies the
floor on every read and returns `kind: recorded_costs` if no same-day AI forecast
is available. Recorded-cost failures keep today's last good evidence with a
visible stale/error label; no previous-day projection is applied to history.
Refresh reloads the projection as well as the provider snapshots.

### Prospective repair evidence

`GET /api/analytics/observations?project=<id>&kind=failure&hours=1&limit=100`
selects error logs, failed spans, HTTP 5xx and shared app-failure events. Responses
retain bounded `next_cursor` pagination (send `cursor` with the identical filters),
`truncated`, `window_end`, and `feeds` timestamps. Feed timestamps cover recently
received server telemetry or browser error-capture capability/error events; old
load-only browser SDK traffic is not evidence of error coverage. Records received
through current collectors have server-owned `ingestion: browser|server`; callers
cannot choose the ingestion authority. Browser Origin does not authenticate a
caller. Historical records without this marker remain unclassified.

The existing usage tracker loads `/app-errors.js` for pages with an enrolled
`observatory-profile.js` script. Normal page loads adopt current shared error
capture, including apps with older local timing SDKs. It sends fixed codes for
JavaScript exceptions, unexpected promise rejections and same-origin HTTP 5xx;
new timing SDK failed steps use the same reporter. No exception text, request URL,
query, response body or credentials are exported. Capture sends a capability
heartbeat and at most 20 events per page, one at a time with a five-second terminal
deadline. `ObservatoryErrors.state()` exposes completed/failed/timed_out delivery.
No enrolled script means no capture; absence of reports is unobserved, not healthy.


## Recorded app cost analysis

Observatory owns cross-app operational analysis: costs, latency, failures and links to execution evidence. BrightWrapper owns its charge ledger; FairyStack owns sessions and their history; apps own domain results. The shared renderer is presentation, not another execution or domain-data owner.

Cache fields on each cost aggregate: `cache_samples`, `cache_input_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`, `cache_creation_1h_input_tokens`, `cache_hit_requests`, `cache_savings_cents`. Reads / measured input gives token reuse. Savings compare the measured calls with uncached pricing at charge time, after write premiums and markup; negative means net extra cost. Missing historical/provider measurements remain unknown and are excluded, never treated as misses. The System health collector records today's measured cache reuse and net savings from the same API.

The Costs & infrastructure page begins with the shared recorded-cost explorer.
Deep link: `/?project=dcf-cell-by-cell&days=7&day=2026-09-27`.
Select an app, day or step to see the contributing calls, measured input/output
usage and exact session links. Missing session IDs stay in unattributed groups.
This is separate from provider invoices and subscription-equivalent estimates.

`GET /api/costs` accepts the existing owner-scoped AuthReturn JWT or Observatory
API key. Configure the existing `session_spend` connector; its BrightWrapper key
owns the cost source. Observatory reads BrightWrapper `/api/usage/cost-breakdown`; BrightWrapper owns aggregation. DCF reads that API directly and does not depend on this endpoint. Query: `days=1|3|7|14|30`, optional exact `project`, `day`
(UTC ISO date), `operation`, and optional `related_project` / `related_prefix`
for a separately reported `shared` table. Filters never change owner authority.
Schema 1 returns `totals`, `daily` (whole selected app/window), `steps`, `sessions`,
`projects` (owner's app directory), and `calls` (daily input groups when a step is
selected). Tokens retain sample counts; missing samples remain unknown. Related
charges are not added to the selected app. Session names/links come from retained
owner-authorized FairyStack snapshots; unknown or ambiguous origins stay unlinked.
No prompt/response bodies are returned and reading never makes a paid model call.

Source reads have a ten-second bound and owner/connector-version caches last five
minutes. Failed refreshes return the last successful snapshot with a visible
`error`; an unavailable first read returns 503. API consumers must preserve this
error and observation timestamp. No guessed prices or timing-based attribution.

Reusable browser artifact: `/cost-breakdown.js` and `/cost-breakdown.css`.
`ObservatoryCosts.mount(host, {request(filters, signal), project?, onData?, signal?})`
uses the host's authenticated API adapter, a 15-second overall deadline, aborts
superseded reads and returns `{destroy, reload}`. A fixed `project` hides the app
selector. Hosts may add operation labels and domain evidence in their adapter.
Vendor the published artifacts together. DCF uses the locally vendored renderer but reads BrightWrapper directly, removes the owner's unrelated app list and adds its own review evidence. This component does not require an Observatory data connection.

Unattributed operation rows are selectable. Their recorded input groups show the BrightWrapper dashboard call families (short prompt openings), with their charges and models. These previews explain older work without inventing operation or session IDs; complete-input fingerprint coverage remains separate. New image and speech calls retain their caller-supplied operation IDs.

Analytics readers open bounded read-only SQLite connections. Schema and indexes are prepared at serving-process startup; release stage queries use their own measured-phase index so unrelated spans do not enter the chart query.


## Shared FPS monitor

FairyStack’s same-origin `/.fairystack-analytics/tracker.js` automatically includes
the current `/app-fps.js` component. No per-app copy, database or extra collector
is needed. Existing apps adopt updates on an ordinary page load. The monitor
defaults on with a compact FPS readout and a collapsed one-minute sparkline.
Its disclosure offers an origin-scoped browser off/on switch, independent of
initial-load profiling. Turning it off cancels its frame loop and persists across
reload; the small “FPS off” control restores it. Hidden pages and record/export
pages do not sample. `<html data-observatory-fps="off">` is an explicit page/app
opt-out across every injected tracker. `data-fps="off"` skips an individual embed;
`data-fps="on"` enables an external tracker integration.

This measures foreground requestAnimationFrame callback delivery, not compositor,
video, or game-rendered frames. `ObservatoryFPS.state()` exposes the SDK version,
status, latest measured window and bounded local history. The graph labels FPS levels and time, and reports the minute’s minimum and largest gap. The window event
`observatory-fps-window` carries exact intervals, start/end, frames, milliseconds,
FPS, p95 and maximum gap; `observatory-fps-state` reports paused/disabled/running.
`ObservatoryFPS.mark("Play")` or `mark("Seek timeline")` records a bounded diagnostic action label without input text. The shared sampler observes long tasks and long animation frames where the browser supports them. A gap of at least 100 ms saves one correlated incident through the existing enrolled browser collector (at most once per ten seconds, one in flight, five-second deadline); unsupported attribution is explicitly recorded. Same-origin script paths have query/fragment removed; no DOM text, inputs, credentials or full page URLs are sent. Apps without a `script[data-project]` enrollment keep evidence local. Off disables sampling, observers and pending collection. Collector completion/failure is visible beside the dip, with a direct retained-trace link on success. These correlations identify recorded work around a dip; they do not prove causality or measure compositor frames.