KB Source Resolution

Technical specification for the Knowledge Base (KB) source resolution algorithm in pair-cli.

Resolution Algorithm

Precedence Order

KB source is resolved in the following order (highest to lowest priority):

1. CLI flag --source <url|path>: Explicit user-provided source
2. Program-level flag --url <url|path>: Used as the source when the command names no --source; --source outranks it
3. Monorepo default: packages/knowledge-hub/dataset (when running from monorepo)
4. GitHub release auto-download: Latest release from configured repository

The monorepo default is the fallback for the default source only: a source you name explicitly always wins, --url <remote zip> included. In a development checkout that URL is downloaded and cached under its own slot rather than silently replaced by the local dataset, which is also what makes the --url path exercisable outside a released binary.

--url and --source name a source in exactly the same way: pair-cli install --url X and pair-cli install --source X resolve the same identity, download into the same cache slot and install the same bytes.

The only difference is scope, --url is declared on the program, so it applies to whichever subcommand runs (install, update, kb-info), while --source belongs to one command and therefore wins where both are given.

That equivalence is a rule, not a coincidence: a slot is keyed by source identity, so a flag that names a source without reaching resolution would fetch into a slot nothing reads and serve the official KB instead, silently a different KB than the one the user typed.

Decision Tree

                  KB Source Resolution
                         |
            --source (or --url) provided?
                /                  \
              YES                   NO
               |                     |
          --offline?            In monorepo?
          /        \            /          \
        YES        NO        YES          NO
         |          |          |            |
    Validate    Parse      Use monorepo   Download
    is local    source     dataset        from GitHub
    path        type
                  |
          ┌──────┼──────┐
         Git   Remote  Local
          |      URL    path
        Clone    |       |
        repo   Download  Use
        to     from URL  directly
        temp

Source Type Detection

Git URL Detection

function isGitUrl(source: string): boolean {
  const base = source.replace(/#.*$/, '')
  return (
    /^git:\/\//.test(base) ||
    /^git@[\w.-]+:/.test(base) ||
    /^ssh:\/\/git@/.test(base) ||
    (/^https?:\/\//.test(base) && base.endsWith('.git'))
  )
}

Checked before isRemoteUrl: HTTPS .git URLs would otherwise match remote URL detection.

URL Detection

function isUrl(source: string): boolean {
  return source.startsWith('http://') || source.startsWith('https://')
}

Local Path Detection

function isLocalPath(source: string): boolean {
  return (
    source.startsWith('/') || // Absolute Unix path
    source.startsWith('./') || // Relative path
    source.startsWith('../') || // Relative parent path
    /^[A-Za-z]:[/\\]/.test(source) // Windows absolute path
  )
}

Monorepo Detection

function isMonorepo(): boolean {
  const monorepoKbPath = path.resolve(process.cwd(), 'packages/knowledge-hub/dataset')
  return fs.existsSync(monorepoKbPath)
}

Option Validation

Constraint: --offline + --source Combination

Git Clone Process

When the source is a Git URL:

1. Parse ref: split url#ref into repo URL and optional branch/tag
2. Set auth env: if PAIR_GIT_TOKEN is set and the URL is https:// (not http://), set a child-process-scoped, origin-scoped http.<origin>.extraheader (Basic auth) via GIT_CONFIG_COUNT / GIT_CONFIG_KEY_n / GIT_CONFIG_VALUE_n, requires git ≥ 2.31; an older git silently ignores these and the clone proceeds unauthenticated. Never rewrites the URL and never appears on the git clone argv.

Also sets http.followRedirects=false: with the token attached, the clone refuses ANY HTTP redirect outright (fatal: … returned error: 3xx) rather than following it, since a redirect target is a host the caller never named and the token must not reach it. 3. Shallow clone: git clone --depth 1 [--branch ref] <url> <slot> into the git source's own cache slot (~/.pair/kb/external/git-{hash}/), replaced on each install 4. Cleanup: remove the clone's .git directory, leaving the dataset

Auth strategy:

• Default: git inherits system auth (SSH keys, credential manager, gh auth)
• Override: PAIR_GIT_TOKEN env var sets an origin-scoped http.extraheader for https:// sources, never http://, and never in the URL or the clone's argv
• Redirects refused while the token is attached: a git host that redirects a token-authenticated https:// clone (moved org, forced HTTPS) fails the clone instead of following the redirect. Point --source at the repository's current URL, or clone it yourself and install from the local path.

Resolution Implementation

interface KbSource {
  type: 'url' | 'local' | 'monorepo' | 'github-release'
  location: string
  requiresDownload: boolean
}
 
async function resolveKbSource(options: { source?: string; offline?: boolean }): Promise<KbSource> {
  // Precedence 1: Explicit --source flag
  if (options.source) {
    if (isUrl(options.source)) {
      if (options.offline) {
        throw new Error('Cannot use --offline with remote URL')
      }
      return { type: 'url', location: options.source, requiresDownload: true }
    } else if (isLocalPath(options.source)) {
      const resolvedPath = path.resolve(options.source)
      if (!fs.existsSync(resolvedPath)) {
        throw new Error(`KB source path does not exist: ${resolvedPath}`)
      }
      return { type: 'local', location: resolvedPath, requiresDownload: false }
    } else {
      throw new Error(`Invalid source format: ${options.source}`)
    }
  }
 
  // Precedence 2: Monorepo default
  if (isMonorepo()) {
    return {
      type: 'monorepo',
      location: path.resolve('packages/knowledge-hub/dataset'),
      requiresDownload: false,
    }
  }
 
  // Precedence 3: GitHub release auto-download
  if (options.offline) {
    throw new Error('Cannot auto-download KB in offline mode')
  }
  return {
    type: 'github-release',
    location: await getLatestReleaseUrl(),
    requiresDownload: true,
  }
}

Error Catalog

Download Process

When requiresDownload: true for URL sources:

1. HEAD request: Fetch URL metadata
2. Fetch checksum: Download .sha256 checksum file
3. Download KB: GET request with progress tracking, staged in a temporary file named after the source URL (kb-{version}-{hash}.zip under the system temp directory). An interrupted download leaves a .partial file that the next run of the same URL resumes; two different sources never share a staging file, so one source's bytes can never be resumed into another source's archive.
4. Validate SHA256: Verify integrity (if checksum available)
5. Extract to cache: Store in the source's own slot (see Cache Strategy below)
6. Unwrap a single root directory: an archive whose content sits under one top-level folder (acme-kb-1.0.0/.pair/…) is lifted to the slot root, so the dataset root is the slot itself whichever way the archive was packaged. Same normalization as a local ZIP.

Cache Strategy

Cache location: one slot per source, not per CLI version:

The slot is derived from the identity of the source (its resolved path or URL): the ZIP and git forms carry no name, only the content/url hash; the remote-URL form is the only one that keeps a human-readable label. Two different sources never share a slot and the same source always reuses its own.

Only the official KB writes the version-keyed slot: an external KB can never overwrite it or its manifest.json (foomakers/pair#395).

The root of every slot above is ~/.pair/kb unless PAIR_KB_CACHE_DIR overrides it (which must be an absolute path, slots are deleted recursively, so a relative value pointing at the current directory is rejected rather than honoured).

A directory source is never copied into the cache. --source <dir> validates the directory and installs from it in place, so it occupies no slot and there is nothing to invalidate: edit the directory and the next install picks the edit up. Only the archive forms (zip, remote zip, git clone) need a slot, because they have to be materialized somewhere before they can be read.

A slot whose manifest.json names a different KB than the one expected is treated as contaminated: it is discarded and re-fetched rather than served. The contaminated slot is set aside first, not deleted, so a failing re-fetch (offline, 5xx) leaves the previous content in place instead of no cache at all. A slot of the official KB with no manifest.json at all is treated as an aborted download and re-fetched too.

External slots declare no expected manifest name: there is nothing to compare against, so they are not inspected: every --source/--url install re-installs into its own slot regardless of what the slot already holds.

A fetched slot is never deleted before its replacement is in hand. Every source whose content comes over the network (the official download, --url, a git clone) sets its slot aside before it writes, and puts it back when the fetch throws. A local ZIP is the one exception, and deliberately: its source is a file you still have, so re-running the command is the recovery, and replacing the slot wholesale is what keeps a previous KB's files from lingering next to the new one.

Putting the slot back is itself a cleanup, and a cleanup never undoes or hides the work it follows: the set-aside copy is moved back first and the half-written one discarded afterwards (so a failing delete cannot cost you both), and a failure of that housekeeping is logged rather than raised: the error you see is the one that actually stopped the install ("Network error downloading KB…", a 404 with the manual-download URL), never an unrelated filesystem message from the recovery.

Should the recovery itself fail (a held handle on the directory, on Windows), it is retried the other way round first; if that fails too you get a warning naming the copy it kept and the path to move it back to, so your previous KB is never left recoverable-but-unnamed at <slot>.bak.

Scope of the self-heal: the official slot is keyed by CLI version, so detection only applies to the slot of the CLI version you are running. A slot polluted by an older CLI (before foomakers/pair#395) lives under that older version's directory and is never read again, abandoned, not discarded.

Disk: one slot per distinct source. Slots are plain directories with no hidden state, rm -rf ~/.pair/kb/external is always safe, and the next install re-populates what it needs. There is no automatic eviction (no LRU, no TTL): growth is proportional to the number of KBs you install, not to how often you install them. Use pair-cli kb-cache list to see what is there and pair-cli kb-cache prune (add --dry-run first) to reclaim what is dead. Four leftovers rm -rf ~/.pair/kb/external does not cover, all safe to delete by hand:

An external slot's own .discarded-* sits under external/ and is already covered by the rm -rf above; only the official slot's spelling needs stating.

What prune never deletes. The KB cache is shared by every project on the machine, so prune is deliberately timid: your running CLI version's official slot (the KB the next command reads), every external/ slot (one per source you chose: the CLI does not retain the URL or the archive, so deleting one is not undone by re-running an install), anything it cannot classify, and anything an install currently in flight owns.

That last one covers two shapes: a <slot>.tmp-<pid>-<n> stage whose process is still alive, and a <slot>.bak whose slot has not come back yet: that backup is the only copy of that KB. list marks both as kept, so a leftover-shaped entry that survives a prune is not a bug. With no CLI version resolvable, prune spares every official slot rather than guess.

Configuration

Environment Variables

Related

• CLI Commands Reference: Complete command documentation
• CLI Contracts Spec: Command contracts and validation
• CLI Help Examples: Practical usage examples