# Architecture

> This document is not stable. It currently serves as the basis for refactoring
> and is subject to change.

This repository should be treated as a small ETL pipeline.

The current version is script-oriented: each script fetches data, interprets it,
and writes the final JSON files directly. That works for a first pass, but it
mixes source-specific logic with domain logic and presentation logic. It also
makes the whole system harder to extend. If Fitbit is replaced later, or if
nutrition starts coming from another source, too much of the pipeline would need
to change.

The better shape is a phased pipeline with clear boundaries:

1. source adapters 2. raw warehouse 3. canonical warehouse 4. projections

The key idea is that source-specific concerns should end early. Everything after
that should work with stable internal models rather than Fitbit or fddb quirks.

## Overview

The intended flow is:

```
source adapters → raw warehouse → canonical warehouse → projections
```

JSONL partitions are the durable storage format. SQLite is a working database
layered on top of that storage. It is useful for querying, joining, and
generating projections, but it should not be the source of truth.

That means the SQLite database can be deleted and rebuilt from JSONL partitions
at any time.

## Why This Shape

There are three problems with the current design.

First, it collapses the entire pipeline into a few scripts. Each script does
transport, transformation, storage, and projection work at the same time.

Second, it treats the presentation JSON as if it were the database. That is
backwards. `status.json` and the history JSON files should be outputs, not the
place where the data lives.

Third, it makes source replacement harder than it needs to be. If
Fitbit-specific assumptions are embedded in the data model and downstream logic,
then swapping to Garmin or Apple Health later becomes expensive.

The proposed architecture fixes that by making the internal model stable and the
adapters replaceable.

## Phase 1: Source Adapters

The first phase is a set of source adapters. These are thin modules that know
how to talk to a provider and nothing more.

A Fitbit adapter should know about:

- OAuth
- token refresh
- endpoint paths
- rate limits
- pagination
- raw response handling

An fddb adapter should know about:

- login
- session handling
- HTML fetching
- scraping mechanics
- request retries

These adapters should not know about concepts like dashboard status, daily
summaries, weight trends, or output JSON shapes.

They should return raw or near-raw source data in a consistent envelope that
includes provenance such as:

- source system
- endpoint or page name
- fetched at
- source record identifier if available
- source payload

This makes the adapters pluggable. If Fitbit is replaced later, the new provider
only has to satisfy the same adapter boundary.

## Phase 2: Raw Warehouse

The raw warehouse is where the fetched data is stored with minimal
interpretation.

Its purpose is simple: keep a durable, inspectable record of what each source
returned.

This layer should preserve source structure as much as possible. It is not yet
the place to decide what counts as a meal, a sleep stage, or an activity session
in the internal domain model. Those decisions belong in the next phase.

The raw warehouse is useful for a few reasons:

- it gives you replayability
- it makes debugging much easier
- it decouples source fetching from downstream transformation
- it lets you reprocess old data if the canonical model changes

This is also the right place for source provenance and ingestion metadata.

For storage, the raw warehouse should use JSONL, partitioned by date and source.
JSONL preserves the full structure of source payloads, diffs well in git, and
avoids the information loss that comes from flattening nested responses into CSV.

## Phase 3: Canonical Warehouse

The canonical warehouse is where source-specific data is translated into the
repository's internal model.

This is the most important layer in the system because it is what makes the rest
of the pipeline independent from the provider.

### Snapshot model

The canonical warehouse stores point-in-time observations. Each sync produces a
timestamped snapshot of the metrics available at that moment. For example, if
steps are 8000 at 15:00 and 8200 at 15:10, both observations are stored. Daily
totals, deltas, and trends are derived by projections, not baked into the
warehouse.

This preserves the intraday resolution of what actually happened over the course
of a day.

### Canonical entities

The canonical model defines these entities:

| Entity | Grain | Description |
|---|---|---|
| `heart-rate-sample` | per-minute | Individual heart rate reading |
| `step-snapshot` | per-sync | Cumulative step count at observation time |
| `calorie-snapshot` | per-sync | Cumulative calories burned at observation time |
| `active-minutes-snapshot` | per-sync | Cumulative active minutes at observation time |
| `activity-level-snapshot` | per-sync | Sedentary, light, fairly, very active minutes at observation time |
| `distance-snapshot` | per-sync | Distance, floors, elevation at observation time |
| `sleep-session` | per-session | One sleep period (bedtime, wake, duration, score) |
| `sleep-stage` | per-stage | Stage within a sleep session (deep, light, REM, awake) |
| `activity-session` | per-session | A logged exercise (name, calories, duration, steps) |
| `body-measurement` | per-observation | Weight, body fat, BMI, muscle mass, circumferences |
| `nutrition-day` | per-day | Daily calorie and macro totals, targets |
| `meal` | per-meal | Meal name, time, calories, macros |
| `meal-item` | per-item | Individual food item within a meal |

These are internal domain concepts. Downstream code should depend on these
concepts, not on Fitbit response shapes or fddb HTML structure.

Entity names use kebab-case in the filesystem and are named after domain
concepts, not vendor payloads.

### Cross-references

Entities are related through natural keys rather than synthetic IDs:

| Child | Parent | Natural key |
|---|---|---|
| `sleep-stage` | `sleep-session` | `(date, bedtime)` |
| `meal-item` | `meal` | `(date, meal_name)` |

`activity-session` and `sleep-session` are identified by `(date, start_time)`
where `start_time` is a full-precision timestamp.

Natural keys avoid the need for ID generation and keep JSONL rows
self-describing. They work because the underlying events are temporally
distinct — two sleep sessions cannot begin at the same moment, and two exercises
cannot start at the same timestamp.

### Provenance

This layer should carry provenance, because canonical rows should still be
traceable back to the source they came from. Every canonical record should
include:

- source system identifier
- fetch timestamp (when the pipeline observed the data)

### Observed and inferred data

Not all data will be available at true intraday resolution. Some will be
observed at a timestamp, some will be observed only at day or meal-bucket
precision, and some may later be inferred.

Those cases should be distinguished clearly.

Observed data means the source gave us the value directly at its own level of
precision. Inferred data means we derived a more detailed representation
locally.

Those two classes should not be merged casually. If a projection wants to use
inferred intraday data, that should be an explicit choice.

### Intraday first, but honest

Where a source offers intraday data, the canonical warehouse should store it at
that grain.

Where a source only offers daily or coarse meal-level data, the warehouse should
preserve that coarser truth rather than pretending to have more precision than
it does.

So the guiding rule is:

- store the finest reliable resolution the source actually gives us
- only add inferred finer-grained records in separate structures

## Phase 4: Projections

Projections are the consumer-facing outputs built from the canonical warehouse.

For this repository, the main projections are:

- `status.json`
- `history/vitals.json`
- `history/sleep.json`
- `history/nutrition.json`
- `history/measurements.json`
- `history/activity.json`

These files should not be written to directly by adapters or transformation
code. They should be generated from the warehouse.

That makes them disposable. If the projection logic changes, the JSON can be
rebuilt. If the site needs a different view later, it can be added without
changing the storage model.

Projections are decoupled from sync. Fetching and warehousing is one operation.
Generating projections is a separate operation. This means projections can be
rebuilt without re-fetching from sources, and projection bugs do not require a
full re-sync.

Each projection run produces a timestamped output directory. A symlink at
`build/projection/current` points to the latest run. Consumers follow the
symlink.

Derived values like weight trends and step streaks are projection concerns. The
warehouse stores only concrete observed data.

## Storage Model

There are two storage concerns in this design, and they should stay separate.

The first is durable storage. That is the warehouse on disk, partitioned into
JSONL files. This is what survives.

The second is query storage. That is SQLite, used locally to make extraction and
projection convenient.

SQLite is a tool here, not the database of record. All data flows through
SQLite, but it remains disposable — it can be deleted and rebuilt from JSONL
partitions at any time.

### Durable storage

The warehouse is partitioned by entity and by day:

```
data/
├── heart-rate-sample/
│   └── 2026-03-19.jsonl
├── step-snapshot/
│   └── 2026-03-19.jsonl
├── calorie-snapshot/
│   └── 2026-03-19.jsonl
├── active-minutes-snapshot/
│   └── 2026-03-19.jsonl
├── activity-level-snapshot/
│   └── 2026-03-19.jsonl
├── distance-snapshot/
│   └── 2026-03-19.jsonl
├── sleep-session/
│   └── 2026-03-19.jsonl
├── sleep-stage/
│   └── 2026-03-19.jsonl
├── activity-session/
│   └── 2026-03-19.jsonl
├── body-measurement/
│   └── 2026-03-19.jsonl
├── nutrition-day/
│   └── 2026-03-19.jsonl
├── meal/
│   └── 2026-03-19.jsonl
└── meal-item/
    └── 2026-03-19.jsonl
```

This layout keeps the data inspectable, git-diffable, and makes it possible to
rebuild the query layer selectively.

### Projections output

```
build/projection/<ISO 8601: YYYY-MM-DD-HHMMSS>/
├── status.json
└── history/
    ├── vitals.json
    ├── sleep.json
    ├── nutrition.json
    ├── measurements.json
    └── activity.json

build/projection/current → <latest timestamp directory>
```

### SQLite

SQLite should sit on top of the warehouse and expose:

- hydrated canonical tables
- views for daily rollups
- views for site projections

The implementation can choose whether SQLite is fully rebuilt on each run or
incrementally refreshed from touched partitions. That is an implementation
detail. Architecturally, the important thing is that SQLite remains disposable.

## Source Integration Requirements

This architecture is designed to be provider-pluggable. Fitbit and fddb are the
initial sources, but any future provider must satisfy these requirements to
integrate with the canonical model.

### Temporal requirements

- Sleep sessions must provide a start timestamp (bedtime) at minute precision
  or better. This is the natural key for cross-referencing sleep stages.
- Activity sessions must provide a start timestamp at full precision, not
  truncated to hours and minutes. This is the natural key for identifying
  sessions.
- Snapshot metrics (steps, calories, active minutes, distance, activity levels)
  must be attributable to an observation timestamp.
- Heart rate samples must carry a timestamp at minute precision or better.
- Nutrition and meal data must be attributable to a specific day.
- Meals within a day must have distinct identifiers (name, index, or label).

### Data requirements

- Body measurements must carry an observation date.
- Sleep stages must be associable with their parent session via the session's
  start timestamp.
- Meal items must be associable with their parent meal via the meal's day and
  identifier.

### Provenance requirements

- Every record must be attributable to a source system.
- Every record must carry a fetch timestamp indicating when the pipeline
  observed the data, distinct from when the source event occurred.

### Natural key constraint

If a source cannot provide timestamps precise enough to produce unique natural
keys for its entities, it cannot integrate without introducing synthetic
identifiers. That would be a signal to revisit the entity design for the
affected domain.

## Boundaries

The most important boundaries in the codebase should be these:

### Adapters

Provider-specific fetch logic only.

### Warehouse writers

Responsible for persisting canonical rows to durable JSONL storage.

### Canonical transformers

Responsible for mapping raw provider records into internal domain records.

### Repositories or query services

Responsible for reading canonical data from SQLite in a way that is convenient
for the rest of the application.

### Projection builders

Responsible for turning canonical data into output files for the site.

If these boundaries hold, the system stays modular. If they collapse, the code
will drift back into source-specific scripts that mutate final JSON files
directly.

## Pluggability

This architecture is intentionally designed to make providers replaceable.

If Fitbit becomes undesirable later, the rest of the pipeline should not care
much. The adapter changes, the warehouse gains a new source shape, and the
canonical transformer maps that provider into the same internal model.

That only works if the canonical model stays independent from provider
terminology. Internal tables should be named after domain concepts, not after
vendor payloads.

Good examples:

- `heart-rate-sample`
- `activity-session`
- `body-measurement`
- `meal`
- `meal-item`

Bad examples:

- `fitbit-intraday-heart-payload`
- `fddb-breakfast-html-row`

Provider-specific detail belongs in provenance fields, not in the stable
internal model.

## Suggested Initial Refactor

The implementation should probably happen in stages.

### Stage 1

Introduce source adapters that only fetch data and return raw payloads. Do not
change projections yet.

### Stage 2

Add the warehouse and store fetched source data there.

### Stage 3

Define the canonical warehouse schema and the transformations from raw to
canonical.

### Stage 4

Generate the existing JSON outputs from canonical data instead of writing them
directly.

### Stage 5

Remove the remaining direct JSON mutation logic from the current scripts.

This staged approach reduces risk. It also makes it easier to inspect the data
at each layer before the next abstraction is introduced.

## Open Design Questions

- How much historical intraday data should be kept for high-frequency metrics
  like heart rate.
- Whether inferred intraday records are needed at all in the first
  implementation.
- Whether SQLite hydration should be incremental or full rebuild.

Those are important choices, but they sit inside the architecture described
above. They do not change the overall shape of the pipeline.

## Summary

The repository should move toward a provider-pluggable ETL design.

Source adapters talk to Fitbit, fddb, or future providers. The warehouse stores
point-in-time observations as JSONL partitions. Projections generate the JSON
files the site needs.

JSONL partitions hold the durable data. SQLite makes the data queryable. The
canonical model, not the provider API, becomes the stable center of the system.
