# Development Guide

## Prerequisites

- Python >= 3.8
- Pipenv
- Git

## Setup

```sh
pipenv install
pip install -e .
```

## Credentials

| Variable | Purpose | Required by |
|----------|---------|-------------|
| `GITHUB_TOKEN` | GitHub PAT (`public_repo` scope) | `discover`, `mirror` |
| `BITBUCKET_TOKEN` | Workspace Access Token (Bearer) | `mirror` |
| `BITBUCKET_WORKSPACE` | Workspace slug | `mirror` |
| `BITBUCKET_PROJECT` | Project key (optional, for project-scoped access) | `mirror` |

`GITHUB_TOKEN` is technically optional for discovery but strongly
recommended to avoid GitHub API rate limits on unauthenticated requests.

## Two-Tier Cache

The meta crawler operates as a two-tier cache. The **remote tier** mirrors
upstream H5P repositories from GitHub to Bitbucket, providing a durable copy
independent of upstream availability. The **local tier** extracts
`library.json` metadata from those mirrors into a well-known directory
structure for build-time consumption.

### Data directory layout

```
data/
├── manifest.json                        ← discovered repos (discover)
├── mirror_index.json                    ← synced mirror registry (mirror)
├── index.json                           ← content type index (crawl)
├── mirrors/h5p/<repo>.git              ← bare mirror clones (mirror)
└── metadata/h5p/<repo>/library.json    ← extracted metadata (crawl)
```

### Bootstrapping

Run all three stages when no local data exists:

```sh
pipenv run make discover    # probe GitHub org for repos with library.json
pipenv run make mirror      # clone mirrors, push to Bitbucket
pipenv run make crawl       # extract library.json from mirrors
```

### Steady-state sync

When both tiers are already populated, re-fetch upstream changes and
re-extract only stale metadata:

```sh
pipenv run make mirror
pipenv run make crawl
```

The crawler compares `mirror_synced_at` against `downloaded_at` per entry
and only re-extracts when the mirror is newer.

### Re-discovery

When new content types appear upstream, re-run discovery first. Existing
manifest entries are preserved; new repos are appended:

```sh
pipenv run make discover
pipenv run make mirror
pipenv run make crawl
```

## Makefile Targets

All targets must be run via `pipenv run make <target>`.

| Target | Description | Required env vars |
|--------|-------------|-------------------|
| `discover` | Probe GitHub org for repos containing `library.json` | `GITHUB_TOKEN` |
| `mirror` | Clone/sync bare mirrors, push to Bitbucket | `GITHUB_TOKEN`, `BITBUCKET_TOKEN`, `BITBUCKET_WORKSPACE` |
| `crawl` | Extract `library.json` metadata from mirrors | (none) |
| `fmt` | Format source with autopep8 | (none) |
| `lint` | Run pylint and mypy | (none) |
| `check` | Run `fmt` then `lint` | (none) |
| `dist` | Build distribution package | (none) |
| `clean` | Remove build artifacts | (none) |

## Package Structure

```
src/byteb4rb1e/sphinxcontrib/h5p/
└── pkgutils/
    ├── util/
    │   ├── http.py          ← generic HTTP client with rate-limit retry
    │   ├── github.py        ← GitHub API wrapper (repo listing, file fetch)
    │   ├── git.py           ← bare repo primitives (clone, remote, push, read)
    │   └── bitbucket.py     ← Bitbucket Cloud REST API v2.0 wrapper
    └── content_type/
        ├── mirror.py        ← discover + sync (remote tier)
        └── meta_crawler.py  ← extract metadata (local tier)
```

### CLI entry points

```sh
# Discovery / mirror sync
python3 -m byteb4rb1e.sphinxcontrib.h5p.utils.h5porg.repository.mirror --discover
python3 -m byteb4rb1e.sphinxcontrib.h5p.utils.h5porg.repository.mirror \
    --bb-workspace WORKSPACE --bb-project PROJECT

# Metadata extraction
python3 -m byteb4rb1e.sphinxcontrib.h5p.utils.h5porg.repository.meta_crawler data/
```

Both commands accept `-v` / `-vv` for increased verbosity.

## Troubleshooting

**Empty `manifest.json`** — Discovery found no repos with `library.json`.
Verify `GITHUB_TOKEN` is set and the GitHub org (default `h5p`) is
accessible. Repos prefixed `h5p-editor` or `h5pmods` are skipped by
design.

**Empty `mirror_index.json`** — Mirror sync had nothing to process. Run
`discover` first to populate the manifest.

**Bitbucket 403** — The workspace access token lacks write permission, or
`BITBUCKET_WORKSPACE` does not match the token's scope. If using a
project-scoped access key, set `BITBUCKET_PROJECT` as well.

**Rate limiting (HTTP 429)** — The HTTP client retries automatically with
backoff (1 s, 2 s, 4 s). If retries are exhausted, wait a few minutes and
re-run. Setting `GITHUB_TOKEN` avoids the low unauthenticated rate limit.

**`GitError` during mirror sync** — A single repo failed to clone or push.
The remaining repos continue processing. Check network connectivity and
repository permissions for the failing repo.
