Copilot History Ingest — Conversation Mining

wiki-history-ingest

/wiki-history-ingest copilot

Before You Start

1. Read

   .env

   to get

   OBSIDIAN_VAULT_PATH

   and

   COPILOT_HISTORY_PATH

   (defaults to

   ~/.copilot/session-state

   ) and

   COPILOT_VSCODE_STORAGE_PATH

   (the VS Code

   workspaceStorage

   directory; platform-specific — ask the user if absent from

   .env

   )
2. Read

   .manifest.json

   at the vault root to check what's already been ingested
3. Read

   index.md

   at the vault root to know what the wiki already contains

Ingest Modes

Append Mode (default)

.manifest.json

• Sessions not in the manifest (new sessions)
• Sessions whose

  updated_at

  is newer than their

  ingested_at

  in the manifest

Full Mode

wiki-rebuild

GitHub Copilot Data Layout

Source 1:

~/.copilot/session-state/

(CLI sessions)

~/.copilot/session-state/
├── <session-uuid>/
│   ├── workspace.yaml           # Session metadata (id, cwd, summary_count, created_at, updated_at)
│   ├── vscode.metadata.json     # VS Code context (workspaceFolder, repositoryProperties, customTitle)
│   ├── events.jsonl             # Full event log — all turns, tool calls, reasoning
│   ├── session.db               # Per-session SQLite (todos/todo_deps only — skip for ingestion)
│   ├── index.md                 # Session summary written at session end
│   ├── checkpoints/             # Checkpoint JSON files (mid-session summaries)
│   │   └── <uuid>.json          # title, overview, history, work_done, technical_details,
│   │                            #   important_files, next_steps
│   ├── files/                   # Artifacts produced during session (plans, diagrams, etc.)
│   └── research/                # Research artifacts
└── ...

Source 2:

~/.copilot/session-store.db

(Global SQLite)

sessions       — id, cwd, repository, branch, summary, created_at, updated_at, host_type
turns          — session_id, turn_index, user_message, assistant_response, timestamp
checkpoints    — session_id, checkpoint_number, title, overview, history, work_done,
                 technical_details, important_files, next_steps, created_at
session_files  — session_id, file_path, tool_name, turn_index, first_seen_at
session_refs   — session_id, ref_type (commit/pr/issue), ref_value, turn_index, created_at
search_index   — FTS5 virtual table (content, session_id, source_type, source_id)

Source 3: VS Code Workspace Storage (

<workspaceStorage>/<hash>/GitHub.copilot-chat/

)

.env

<hash>/GitHub.copilot-chat/
├── transcripts/
│   └── <session-uuid>.jsonl     # Conversation transcripts (same JSONL format as events.jsonl)
├── memory-tool/
│   └── memories/
│       └── <base64-session-id>/ # Per-session saved artifacts (plan.md, etc.)
│           └── plan.md
└── codebase-external.sqlite     # Codebase index (skip — no conversation knowledge)

Key data sources ranked by value:

1. Checkpoints (

   session-store.db

   checkpoints

   table + per-session

   checkpoints/*.json

   ) — Pre-distilled summaries with

   overview

   ,

   work_done

   ,

   technical_details

   ,

   important_files

   ,

   next_steps

   . Gold.
2. Session summaries (

   session-store.db

   sessions.summary

   +

   index.md

   ) — One-paragraph synopsis per session.
3. Turns (

   session-store.db

   turns

   table +

   events.jsonl

   / transcript JSONL) — Full conversation. Rich but verbose.
4. Memory artifacts (

   memory-tool/memories/<id>/plan.md

   etc.) — Pre-written plans and structured notes the user saved explicitly. Worth importing verbatim (or lightly summarised).
5. File access patterns (

   session_files

   table +

   tool.execution_*

   events) — Which files the agent repeatedly touched — reveals high-value project files.
6. Session refs (

   session_refs

   table) — Commits, PRs, and issues linked to sessions.

7. vscode.metadata.json

   — Workspace folder path, branch,

   customTitle

   (user-set session label). Useful for grouping and naming.

Step 1: Survey and Compute Delta

.manifest.json

# --- Source 1: per-session directories ---
# Find all session directories (each has workspace.yaml)
ls ~/.copilot/session-state/

# For each session, read workspace.yaml for id/cwd/updated_at
# and vscode.metadata.json for customTitle / repositoryProperties

# --- Source 2: global database ---
# Query session-store.db with sqlite3 (or Python sqlite3)
SELECT s.id, s.cwd, s.repository, s.branch, s.summary, s.updated_at,
       COUNT(DISTINCT t.turn_index) AS turn_count,
       COUNT(DISTINCT c.id)         AS checkpoint_count
FROM sessions s
LEFT JOIN turns t ON t.session_id = s.id
LEFT JOIN checkpoints c ON c.session_id = s.id
GROUP BY s.id
ORDER BY s.updated_at DESC;

# --- Source 3: VS Code workspace storage ---
# For each <hash> directory under workspaceStorage, check for GitHub.copilot-chat/
# Find transcript files
ls <workspaceStorage>/<hash>/GitHub.copilot-chat/transcripts/

• New — not in manifest → needs ingesting
• Modified — in manifest but

  updated_at

  is newer → needs re-ingesting
• Unchanged — in manifest and not modified → skip in append mode

Step 2: Ingest Checkpoints and Summaries First

From

session-store.db

:

SELECT s.id, s.cwd, s.repository, s.branch, s.summary,
       c.checkpoint_number, c.title, c.overview, c.work_done,
       c.technical_details, c.important_files, c.next_steps,
       c.created_at
FROM checkpoints c
JOIN sessions s ON c.session_id = s.id
ORDER BY s.updated_at DESC, c.checkpoint_number ASC;

From per-session

checkpoints/*.json

:

title

overview

history

work_done

technical_details

important_files

next_steps

index.md

What to extract:

• overview

  → high-level description of what the session accomplished

• work_done

  → concrete tasks completed (good for skills / project pages)

• technical_details

  → implementation specifics (good for concepts pages)

• important_files

  → high-value files in the project (good for project pages)

• next_steps

  → open threads (good for linking to ongoing project work)

Step 3: Parse Session Turns

session-store.db

events.jsonl

From

session-store.db

:

SELECT turn_index, user_message, assistant_response, timestamp
FROM turns
WHERE session_id = '<uuid>'
ORDER BY turn_index ASC;

From

events.jsonl

/ transcript JSONL:

references/copilot-data-format.md

type

session.start

user.message

data.content

assistant.message

data.content

data.toolRequests

tool.execution_start

tool.execution_end

assistant.message

• data.content

  is the assistant's text response — extract this

• data.reasoningText

  is internal reasoning — skip (it's the unpacked

  reasoningOpaque

  field)

• data.toolRequests

  lists tool calls — skim tool names and arguments for file access patterns
• Skip

  type: "tool.execution_end"

  entirely

Step 3b: Process Memory Artifacts

memory-tool/memories/<base64-id>/

plan.md

import base64
session_id = base64.b64decode(dir_name).decode('utf-8')

skills/

concepts/

Step 3c: Extract File and Ref Patterns

session-store.db

-- Most-touched files per project
SELECT repository, file_path, COUNT(*) AS touch_count
FROM session_files
GROUP BY repository, file_path
ORDER BY touch_count DESC;

-- Linked commits/PRs/issues per session
SELECT session_id, ref_type, ref_value, turn_index
FROM session_refs
ORDER BY session_id, turn_index;

Step 4: Cluster by Topic

• Group extracted knowledge by topic across sessions
• A single session about "debugging auth + setting up CI" → two separate topics
• Three sessions across different days about "React performance" → one merged topic

• cwd

  /

  repository

  give you a natural first-level grouping;

  vscode.metadata.json

  's

  customTitle

  gives a human-readable session label

Step 5: Distill into Wiki Pages

cwd

repository

C:\Users\name\git\my-project   → my-project
/Users/name/code/another-app   → another-app

repository

owner/repo

session-store.db

cwd

Project-specific vs. global knowledge

projects/<name>/concepts/

projects/my-project/concepts/main-architecture.md

projects/<name>/skills/

projects/my-project/skills/api-rate-limiting.md

concepts/

concepts/react-server-components.md

skills/

skills/debugging-hydration-errors.md

entities/

entities/vercel-functions.md

synthesis/

synthesis/common-debugging-patterns.md

projects/<name>/<name>.md

_project.md

_project.md

_project

<name>.md

summary:

wiki-query

llm-wiki

• Checkpoints and index.md are pre-distilled by the system — treat checkpoint-derived claims as extracted (the system wrote them from observed actions).
• Memory artifacts are user-authored — treat as extracted.
• Conversation turn distillation is mostly inferred. You're synthesizing a coherent claim from many turns. Apply

  ^[inferred]

  liberally to synthesized patterns, generalizations across sessions, and "what the user really meant" interpretations.
• Use

  ^[ambiguous]

  when the user changed direction mid-session or when the session ended unresolved.
• Write a

  provenance:

  frontmatter block on every new/updated page summarizing the rough mix.

Step 6: Update Manifest, Journal, and Special Files

Update

.manifest.json

• ingested_at

  ,

  session_id

  ,

  updated_at

• source_type

  : one of

  "copilot_session"

  ,

  "copilot_checkpoint"

  ,

  "copilot_transcript"

  ,

  "copilot_memory_artifact"

• project

  : the decoded project name

• pages_created

  and

  pages_updated

  lists

projects

{
  "project-name": {
    "repository": "owner/repo",
    "cwd": "C:\\Users\\name\\git\\project-name",
    "vault_path": "projects/project-name",
    "last_ingested": "TIMESTAMP",
    "sessions_ingested": 5,
    "sessions_total": 8,
    "checkpoints_ingested": 12,
    "memory_artifacts_ingested": 3
  }
}

Create journal entry + update special files

index.md

log.md

- [TIMESTAMP] COPILOT_HISTORY_INGEST projects=N sessions=M checkpoints=C pages_updated=X pages_created=Y mode=append|full

hot.md

$OBSIDIAN_VAULT_PATH/hot.md

wiki-ingest

updated

Privacy

• Distill and synthesize — don't copy raw conversation text verbatim
• Skip anything that looks like secrets, API keys, passwords, tokens

• data.reasoningOpaque

  /

  data.reasoningText

  in assistant events is internal reasoning — skip entirely, never copy to wiki
• If you encounter personal/sensitive content, ask the user before including it
• The user's conversations may reference other people — be thoughtful about what goes in the wiki

Reference

references/copilot-data-format.md