docs

docs/architecture.md

@@ -0,0 +1,231 @@
+# Architecture
+
+## Modules
+
+### 1. Session
+Shared data schemas used across all other modules.
+
+- `ComponentCategory` enum: capacitor, resistor, inductor, diode, led, ic, connector, crystal, mosfet, sensor, other
+- `RowState` enum: pending → scouting → filtering → confirmed | flagged
+- `NormalizedParams`: package, value (SI float), value_str, unit, voltage_rating, current_rating, power_rating, tolerance, part_number, manufacturer
+- `BomRow`: row_id, designators, quantity, raw_value, footprint, category, normalized_params, state, applied_filters, facet_groups, confirmed_pick
+- `BomSession`: session_id, created_at, filename, preferred_suppliers, rows
+
+### 2. Ingestion
+Converts raw file bytes into a list of `BomRow` objects.
+
+**Parser** — detects format by extension, handles:
+- CSV: utf-8-sig decode, delimiter detection (`,` vs `;`), footer stripping, blank row skipping
+- XLSX: preamble skip via header-row detection (scans rows 0–14 for synonym matches), column-offset detection, phantom column stripping, footer/totals detection
+- XLS: same as XLSX via xlrd
+
+**Column Mapper** — maps raw column names to canonical fields:
+- Heuristic lookup against `column_synonyms.json` (exact normalized match)
+- Comment disambiguation: if BOM has both `Comment` and a richer value column, `Comment` → `part_number` (unless values look like passive component values)
+- AI fallback (Claude Haiku) when required `value` field is still unmapped
+- New AI-discovered synonyms persisted to `column_synonyms.json` to avoid future API calls
+- Unknown columns passed through with `_raw_` prefix
+
+**Normalizer** — per-row processing:
+- Skip empty / KiCad `~` rows
+- Parse and expand designators (`C1-5` → `[C1, C2, C3, C4, C5]`)
+- Parse quantity; qty=0 → flagged
+- DNP detection (value string, boolean columns, Mounting column)
+- Non-electronic detection (PCB, screw, nut, heatsink, etc.) → flagged
+- Calls classifier + param_extractor → emits `BomRow`
+
+### 3. Classification
+Determines component category and extracts normalized parameters.
+
+**Classifier** — `(designators, raw_value)` → `ComponentCategory`:
+- Designator prefix map: `C`→capacitor, `R/RV`→resistor, `L`→inductor, `D`→diode, `U/IC`→ic, `J/P/CON/USB`→connector, `Q/T`→mosfet, `X/Y`→crystal, `NTC/RT`→sensor
+- Special: D + LED keyword in value → led
+- Fallback (no designators): capacitance/resistance regex, alphanumeric part-number pattern → ic, else → other
+
+**Param Extractor** — `(raw_value, footprint, category)` → `NormalizedParams`:
+- Value parsing: capacitance (bare prefix, European notation), resistance (European 2k2/4k7, standard), inductance, frequency, IC/MOSFET (entire string → part_number)
+- Description fallback: scans full string for embedded values (`"CAP CER 0.1UF 100V X7R 0603"`)
+- Cross-category extractions: voltage_rating, current_rating, power_rating, tolerance
+- Package extraction cascade:
+  1. Exact match in `known_footprints` dict
+  2. Regex cascade from `package_patterns.json`
+  3. AI fallback (Claude Haiku) → result persisted to `known_footprints`
+
+### 4. Suppliers
+Adapter layer for electronic component marketplaces. All adapters implement the abstract `SupplierAdapter` base class, which supports both scraper mode (default, no credentials) and token mode (official API).
+
+```python
+class SupplierAdapter(ABC):
+    def __init__(self, api_token: str | None = None): ...
+    def search(self, category_id: int, params: NormalizedParams, page: int = 1) -> SearchResult: ...
+```
+
+#### 4a. LCSC Category Resolution
+Before searching, each `BomRow` must be mapped to a specific LCSC subcategory (name + integer ID). Generic keyword search is unreliable — LCSC expects category-scoped queries.
+
+**One-time setup** (`lcsc/fetch_categories.py`):
+- Scrapes ~10 LCSC category pages (Capacitors, Resistors, ICs, etc.) with 1.5s grace period between requests
+- Extracts subcategory name + ID from each page
+- Writes `lcsc/categories.json` — committed to repo, refreshed quarterly
+
+**Category JSON format:**
+```json
+[
+  {
+    "group": "Passives",
+    "category": "Capacitors",
+    "category_id": 11,
+    "subcategories": [
+      {"name": "Aluminum Electrolytic Capacitors", "id": 1140},
+      {"name": "Multilayer Ceramic Capacitors (MLCC)", "id": 1142},
+      {"name": "Film Capacitors", "id": 1144},
+      {"name": "Tantalum Capacitors", "id": 1150}
+    ]
+  }
+]
+```
+
+**LLM Category Resolver** (`lcsc/category_resolver.py`):
+- Cache key: `f"{internal_category}|{package}|{value_str}"` → check `lcsc/category_cache.json`
+- On miss: pre-filter category list to relevant subtree → build Claude Haiku prompt → parse JSON response
+- Prompt passes component info + subcategory name list; LLM returns one exact name
+- Persist result to `category_cache.json` (same file-backed learning pattern as `package_patterns.json`)
+
+Example: `category=capacitor, package=6.3x5.8, voltage=25V` → LLM picks `"Aluminum Electrolytic Capacitors"` → looked up → id `1140`
+
+**LCSC Search** (`lcsc/lcsc.py`):
+- Scraper mode (default): GET any LCSC category page to acquire CSRF cookie + session, then POST to `https://lcsc.com/api/products/search` with `catalog_id=<subcategory_id>` + parameter filters
+- Token mode: if `api_token` provided, use official LCSC API (`https://ips.lcsc.com/rest/wmsc2agent/`) with HMAC-signed requests
+
+### 5. Scouting Loop *(not yet built)*
+Drives each `BomRow` from `pending` to `confirmed` through an iterative user-guided process.
+
+```
+pending
+  → scouting   (LLM infers LCSC subcategory, user confirms or picks different)
+  → filtering  (search within confirmed subcategory + param filters, user browses results)
+  → confirmed  (user picks a specific part)
+  → flagged    (DNP / non-electronic / user skips)
+```
+
+Retry path: user rejects all results → back to scouting with modified parameters.
+
+### 6. API Layer *(not yet built)*
+FastAPI HTTP interface.
+
+- `POST /upload` — accept BOM file, run ingestion, return `BomSession`
+- `GET /session/{id}` — current session state
+- `GET /session/{id}/row/{row_id}/scout` — trigger LLM category inference, return `{category, alternatives[]}`
+- `POST /session/{id}/row/{row_id}/confirm-category` — user accepts/overrides category → state: filtering
+- `POST /session/{id}/row/{row_id}/search` — search within confirmed category + params, return page 1
+- `POST /session/{id}/row/{row_id}/confirm` — lock part pick, advance state to confirmed
+- `GET /session/{id}/export` — collate confirmed picks → CSV/XLSX download
+
+### 7. Frontend *(not yet built)*
+React SPA.
+
+- Upload page → triggers `POST /upload`
+- Row-by-row scouting view: category confirmation dropdown + results grid + confirm/reject buttons
+- Progress bar across all rows
+- Export button
+
+---
+
+## User Flow
+
+```
+User uploads BOM file
+  └─ Ingestion pipeline runs
+       ├─ Parser reads file → raw rows
+       ├─ Column Mapper normalizes column names
+       └─ Normalizer classifies + extracts params → BomSession
+
+For each pending BomRow:
+  └─ Scout
+       ├─ LLM infers LCSC subcategory from component info
+       │    (cache hit → instant; miss → Claude Haiku call → cached)
+       ├─ UI shows inferred subcategory + dropdown of all alternatives
+       ├─ User confirms or picks a different subcategory
+       ├─ Search within confirmed subcategory + param filters
+       ├─ User browses results, adjusts filters if needed
+       └─ User confirms pick  ──→  row state: confirmed
+                     └─ rejects  ──→  retry with different subcategory/filters
+
+All rows confirmed
+  └─ Export → order-ready BOM (CSV/XLSX)
+```
+
+---
+
+## System Flow
+
+```
+POST /upload
+  parse_bom(bytes, filename)
+    → map_columns(raw_rows)
+    → normalize(mapped_rows)
+    → BomSession stored in memory
+
+GET /session/{id}/row/{row_id}/scout
+  lcsc_category_resolver.resolve(row)          ← cache hit or Haiku call
+    → {subcategory: "Alum. Electrolytic...", id: 1140, alternatives: [...]}
+  row.state → scouting
+
+POST /session/{id}/row/{row_id}/confirm-category  {subcategory_id: 1140}
+  row.applied_filters["lcsc_category_id"] = 1140
+  row.state → filtering
+
+POST /session/{id}/row/{row_id}/search  {filters: {voltage_min: 25, ...}, page: 1}
+  lcsc.search(category_id=1140, params=row.normalized_params, filters=..., page=1)
+    → {count: N, results: [...]}
+
+POST /session/{id}/row/{row_id}/confirm  {pick: {...}}
+  row.confirmed_pick = pick
+  row.state → confirmed
+
+GET /session/{id}/export
+  [confirmed rows] → assemble BOM → return file
+```
+
+---
+
+## Physical File Map
+
+```
+instant-bom/
+├── bom_assistant/
+│   ├── session/
+│   │   └── models.py                      — Pydantic schemas
+│   │
+│   ├── ingestion/
+│   │   ├── parser.py                      — raw bytes → list[dict[str, str]]
+│   │   ├── column_mapper.py               — raw keys → canonical keys
+│   │   ├── column_synonyms.json           — file-backed synonym table (auto-updated by AI)
+│   │   └── normalizer.py                  — mapped rows → list[BomRow]
+│   │
+│   ├── classification/
+│   │   ├── classifier.py                  — designator prefix + value heuristics → ComponentCategory
+│   │   ├── param_extractor.py             — value string + footprint → NormalizedParams
+│   │   └── package_patterns.json          — file-backed regex + known_footprints (auto-updated by AI)
+│   │
+│   ├── suppliers/
+│   │   ├── base.py                        — abstract SupplierAdapter
+│   │   └── lcsc/
+│   │       ├── fetch_categories.py        — one-time CLI: scrape LCSC → categories.json
+│   │       ├── categories.json            — committed static file (refresh quarterly)
+│   │       ├── categories.py              — load + filter category tree
+│   │       ├── category_resolver.py       — LLM category inference + file-backed cache
+│   │       ├── category_cache.json        — auto-updated by resolver
+│   │       └── lcsc.py                    — search adapter (scraper + token mode)
+│   │
+│   └── api/                               (not yet built)
+│       └── main.py                        — FastAPI app + routes
+│
+├── frontend/                              (not yet built)
+│   └── src/
+│
+├── examples/
+│   └── bom/                               — 12 real BOM test files (CSV, XLSX, XLS)
+│
+└── requirements.txt
+```

docs/backlog.md

@@ -0,0 +1,66 @@
+# Backlog
+
+## Done
+
+### Ingestion Pipeline
+- [x] `session/models.py` — BomSession, BomRow, NormalizedParams, ComponentCategory, RowState
+- [x] `ingestion/parser.py` — CSV + XLSX + XLS, preamble skip, footer strip, delimiter detection
+- [x] `ingestion/column_mapper.py` — heuristic synonym lookup, AI fallback, synonym persistence
+- [x] `ingestion/column_synonyms.json` — file-backed synonym table, auto-updated on AI discovery
+- [x] `ingestion/normalizer.py` — designator parsing, range expansion, DNP detection, row state
+- [x] `classification/classifier.py` — designator prefix map + value heuristics → ComponentCategory
+- [x] `classification/param_extractor.py` — value parsing, description fallback, package extraction, AI fallback
+- [x] `classification/package_patterns.json` — file-backed regex patterns + known_footprints dict
+
+### Bug Fixes
+- [x] Double-space header normalization (`_normalize_key` collapses whitespace) — fixed `Ref Des  (Multi-4)` mismatch
+- [x] `Comment` → `part_number` disambiguation — passive value sampling prevents `pn='100nF'`
+- [x] Description fallback scanners — extracts values from long strings like `"CAP CER 0.1UF 100V X7R 0603"`
+- [x] `\b` → `(?<![A-Za-z])` lookbehind — fixes underscore-prefixed footprints (`Texas_HTSOP-8-1EP`)
+- [x] HTSOP overcapture fix — suffix pattern `(?:-[A-Z0-9]+)*` stops at `_digit`
+
+---
+
+## Todo
+
+### Ingestion
+- [ ] Strip `Elec_` prefix from package — store `"6.3x5.8"` not `"Elec_6.3x5.8"` (change capture group in `package_patterns.json`)
+- [ ] Persian-header BOM support — AI column mapping for non-Latin headers (currently returns 0 rows)
+
+### LCSC Category Resolution
+- [ ] `suppliers/lcsc/fetch_categories.py` — one-time CLI to scrape LCSC category pages → `categories.json` (1.5s grace period between requests)
+- [ ] `suppliers/lcsc/categories.json` — committed static category tree (refresh quarterly)
+- [ ] `suppliers/lcsc/categories.py` — `load_categories()`, `get_subcategories_for(internal_cat)` 
+- [ ] `suppliers/lcsc/category_resolver.py` — LLM (Claude Haiku) category inference + `category_cache.json` persistence
+- [ ] `suppliers/lcsc/category_cache.json` — file-backed cache (same pattern as `package_patterns.json`)
+
+### LCSC Search Adapter
+- [ ] `suppliers/base.py` — abstract `SupplierAdapter(api_token=None)` with `search(category_id, params, page)` interface
+- [ ] `suppliers/lcsc/lcsc.py` — scraper mode: CSRF session + POST to `lcsc.com/api/products/search`; token mode stub for official API
+
+### Scouting Loop
+- [ ] Scouting orchestrator — drive pending rows through scouting → filtering → confirmed
+- [ ] BomRow state machine transitions wired to category resolver + search adapter
+- [ ] Retry / re-scout flow when user rejects results or picks a different category
+
+### API
+- [ ] FastAPI app entry point (`api/main.py`)
+- [ ] `POST /upload` — ingest file, return BomSession
+- [ ] `GET /session/{id}` — return session state
+- [ ] `GET /session/{id}/row/{row_id}/scout` — LLM category inference, return `{subcategory, alternatives[]}`
+- [ ] `POST /session/{id}/row/{row_id}/confirm-category` — lock LCSC subcategory, advance state to filtering
+- [ ] `POST /session/{id}/row/{row_id}/search` — search within confirmed category + param filters
+- [ ] `POST /session/{id}/row/{row_id}/confirm` — lock part pick, advance state to confirmed
+- [ ] `GET /session/{id}/export` — collate confirmed picks → downloadable BOM
+
+### Future: Other Suppliers
+- [ ] DigiKey adapter
+- [ ] Mouser adapter
+- [ ] Nexar adapter
+
+### Frontend
+- [ ] React scaffold + build setup
+- [ ] Upload page
+- [ ] Row-by-row scouting UI (category dropdown, results grid, confirm/reject)
+- [ ] Session progress view (rows confirmed vs pending)
+- [ ] Export button