|
|
@@ -0,0 +1,231 @@
|
|
|
+# Architecture
|
|
|
+
|
|
|
+## Modules
|
|
|
+
|
|
|
+### 1. Session
|
|
|
+Shared data schemas used across all other modules.
|
|
|
+
|
|
|
+- `ComponentCategory` enum: capacitor, resistor, inductor, diode, led, ic, connector, crystal, mosfet, sensor, other
|
|
|
+- `RowState` enum: pending → scouting → filtering → confirmed | flagged
|
|
|
+- `NormalizedParams`: package, value (SI float), value_str, unit, voltage_rating, current_rating, power_rating, tolerance, part_number, manufacturer
|
|
|
+- `BomRow`: row_id, designators, quantity, raw_value, footprint, category, normalized_params, state, applied_filters, facet_groups, confirmed_pick
|
|
|
+- `BomSession`: session_id, created_at, filename, preferred_suppliers, rows
|
|
|
+
|
|
|
+### 2. Ingestion
|
|
|
+Converts raw file bytes into a list of `BomRow` objects.
|
|
|
+
|
|
|
+**Parser** — detects format by extension, handles:
|
|
|
+- CSV: utf-8-sig decode, delimiter detection (`,` vs `;`), footer stripping, blank row skipping
|
|
|
+- XLSX: preamble skip via header-row detection (scans rows 0–14 for synonym matches), column-offset detection, phantom column stripping, footer/totals detection
|
|
|
+- XLS: same as XLSX via xlrd
|
|
|
+
|
|
|
+**Column Mapper** — maps raw column names to canonical fields:
|
|
|
+- Heuristic lookup against `column_synonyms.json` (exact normalized match)
|
|
|
+- Comment disambiguation: if BOM has both `Comment` and a richer value column, `Comment` → `part_number` (unless values look like passive component values)
|
|
|
+- AI fallback (Claude Haiku) when required `value` field is still unmapped
|
|
|
+- New AI-discovered synonyms persisted to `column_synonyms.json` to avoid future API calls
|
|
|
+- Unknown columns passed through with `_raw_` prefix
|
|
|
+
|
|
|
+**Normalizer** — per-row processing:
|
|
|
+- Skip empty / KiCad `~` rows
|
|
|
+- Parse and expand designators (`C1-5` → `[C1, C2, C3, C4, C5]`)
|
|
|
+- Parse quantity; qty=0 → flagged
|
|
|
+- DNP detection (value string, boolean columns, Mounting column)
|
|
|
+- Non-electronic detection (PCB, screw, nut, heatsink, etc.) → flagged
|
|
|
+- Calls classifier + param_extractor → emits `BomRow`
|
|
|
+
|
|
|
+### 3. Classification
|
|
|
+Determines component category and extracts normalized parameters.
|
|
|
+
|
|
|
+**Classifier** — `(designators, raw_value)` → `ComponentCategory`:
|
|
|
+- Designator prefix map: `C`→capacitor, `R/RV`→resistor, `L`→inductor, `D`→diode, `U/IC`→ic, `J/P/CON/USB`→connector, `Q/T`→mosfet, `X/Y`→crystal, `NTC/RT`→sensor
|
|
|
+- Special: D + LED keyword in value → led
|
|
|
+- Fallback (no designators): capacitance/resistance regex, alphanumeric part-number pattern → ic, else → other
|
|
|
+
|
|
|
+**Param Extractor** — `(raw_value, footprint, category)` → `NormalizedParams`:
|
|
|
+- Value parsing: capacitance (bare prefix, European notation), resistance (European 2k2/4k7, standard), inductance, frequency, IC/MOSFET (entire string → part_number)
|
|
|
+- Description fallback: scans full string for embedded values (`"CAP CER 0.1UF 100V X7R 0603"`)
|
|
|
+- Cross-category extractions: voltage_rating, current_rating, power_rating, tolerance
|
|
|
+- Package extraction cascade:
|
|
|
+ 1. Exact match in `known_footprints` dict
|
|
|
+ 2. Regex cascade from `package_patterns.json`
|
|
|
+ 3. AI fallback (Claude Haiku) → result persisted to `known_footprints`
|
|
|
+
|
|
|
+### 4. Suppliers
|
|
|
+Adapter layer for electronic component marketplaces. All adapters implement the abstract `SupplierAdapter` base class, which supports both scraper mode (default, no credentials) and token mode (official API).
|
|
|
+
|
|
|
+```python
|
|
|
+class SupplierAdapter(ABC):
|
|
|
+ def __init__(self, api_token: str | None = None): ...
|
|
|
+ def search(self, category_id: int, params: NormalizedParams, page: int = 1) -> SearchResult: ...
|
|
|
+```
|
|
|
+
|
|
|
+#### 4a. LCSC Category Resolution
|
|
|
+Before searching, each `BomRow` must be mapped to a specific LCSC subcategory (name + integer ID). Generic keyword search is unreliable — LCSC expects category-scoped queries.
|
|
|
+
|
|
|
+**One-time setup** (`lcsc/fetch_categories.py`):
|
|
|
+- Scrapes ~10 LCSC category pages (Capacitors, Resistors, ICs, etc.) with 1.5s grace period between requests
|
|
|
+- Extracts subcategory name + ID from each page
|
|
|
+- Writes `lcsc/categories.json` — committed to repo, refreshed quarterly
|
|
|
+
|
|
|
+**Category JSON format:**
|
|
|
+```json
|
|
|
+[
|
|
|
+ {
|
|
|
+ "group": "Passives",
|
|
|
+ "category": "Capacitors",
|
|
|
+ "category_id": 11,
|
|
|
+ "subcategories": [
|
|
|
+ {"name": "Aluminum Electrolytic Capacitors", "id": 1140},
|
|
|
+ {"name": "Multilayer Ceramic Capacitors (MLCC)", "id": 1142},
|
|
|
+ {"name": "Film Capacitors", "id": 1144},
|
|
|
+ {"name": "Tantalum Capacitors", "id": 1150}
|
|
|
+ ]
|
|
|
+ }
|
|
|
+]
|
|
|
+```
|
|
|
+
|
|
|
+**LLM Category Resolver** (`lcsc/category_resolver.py`):
|
|
|
+- Cache key: `f"{internal_category}|{package}|{value_str}"` → check `lcsc/category_cache.json`
|
|
|
+- On miss: pre-filter category list to relevant subtree → build Claude Haiku prompt → parse JSON response
|
|
|
+- Prompt passes component info + subcategory name list; LLM returns one exact name
|
|
|
+- Persist result to `category_cache.json` (same file-backed learning pattern as `package_patterns.json`)
|
|
|
+
|
|
|
+Example: `category=capacitor, package=6.3x5.8, voltage=25V` → LLM picks `"Aluminum Electrolytic Capacitors"` → looked up → id `1140`
|
|
|
+
|
|
|
+**LCSC Search** (`lcsc/lcsc.py`):
|
|
|
+- Scraper mode (default): GET any LCSC category page to acquire CSRF cookie + session, then POST to `https://lcsc.com/api/products/search` with `catalog_id=<subcategory_id>` + parameter filters
|
|
|
+- Token mode: if `api_token` provided, use official LCSC API (`https://ips.lcsc.com/rest/wmsc2agent/`) with HMAC-signed requests
|
|
|
+
|
|
|
+### 5. Scouting Loop *(not yet built)*
|
|
|
+Drives each `BomRow` from `pending` to `confirmed` through an iterative user-guided process.
|
|
|
+
|
|
|
+```
|
|
|
+pending
|
|
|
+ → scouting (LLM infers LCSC subcategory, user confirms or picks different)
|
|
|
+ → filtering (search within confirmed subcategory + param filters, user browses results)
|
|
|
+ → confirmed (user picks a specific part)
|
|
|
+ → flagged (DNP / non-electronic / user skips)
|
|
|
+```
|
|
|
+
|
|
|
+Retry path: user rejects all results → back to scouting with modified parameters.
|
|
|
+
|
|
|
+### 6. API Layer *(not yet built)*
|
|
|
+FastAPI HTTP interface.
|
|
|
+
|
|
|
+- `POST /upload` — accept BOM file, run ingestion, return `BomSession`
|
|
|
+- `GET /session/{id}` — current session state
|
|
|
+- `GET /session/{id}/row/{row_id}/scout` — trigger LLM category inference, return `{category, alternatives[]}`
|
|
|
+- `POST /session/{id}/row/{row_id}/confirm-category` — user accepts/overrides category → state: filtering
|
|
|
+- `POST /session/{id}/row/{row_id}/search` — search within confirmed category + params, return page 1
|
|
|
+- `POST /session/{id}/row/{row_id}/confirm` — lock part pick, advance state to confirmed
|
|
|
+- `GET /session/{id}/export` — collate confirmed picks → CSV/XLSX download
|
|
|
+
|
|
|
+### 7. Frontend *(not yet built)*
|
|
|
+React SPA.
|
|
|
+
|
|
|
+- Upload page → triggers `POST /upload`
|
|
|
+- Row-by-row scouting view: category confirmation dropdown + results grid + confirm/reject buttons
|
|
|
+- Progress bar across all rows
|
|
|
+- Export button
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## User Flow
|
|
|
+
|
|
|
+```
|
|
|
+User uploads BOM file
|
|
|
+ └─ Ingestion pipeline runs
|
|
|
+ ├─ Parser reads file → raw rows
|
|
|
+ ├─ Column Mapper normalizes column names
|
|
|
+ └─ Normalizer classifies + extracts params → BomSession
|
|
|
+
|
|
|
+For each pending BomRow:
|
|
|
+ └─ Scout
|
|
|
+ ├─ LLM infers LCSC subcategory from component info
|
|
|
+ │ (cache hit → instant; miss → Claude Haiku call → cached)
|
|
|
+ ├─ UI shows inferred subcategory + dropdown of all alternatives
|
|
|
+ ├─ User confirms or picks a different subcategory
|
|
|
+ ├─ Search within confirmed subcategory + param filters
|
|
|
+ ├─ User browses results, adjusts filters if needed
|
|
|
+ └─ User confirms pick ──→ row state: confirmed
|
|
|
+ └─ rejects ──→ retry with different subcategory/filters
|
|
|
+
|
|
|
+All rows confirmed
|
|
|
+ └─ Export → order-ready BOM (CSV/XLSX)
|
|
|
+```
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## System Flow
|
|
|
+
|
|
|
+```
|
|
|
+POST /upload
|
|
|
+ parse_bom(bytes, filename)
|
|
|
+ → map_columns(raw_rows)
|
|
|
+ → normalize(mapped_rows)
|
|
|
+ → BomSession stored in memory
|
|
|
+
|
|
|
+GET /session/{id}/row/{row_id}/scout
|
|
|
+ lcsc_category_resolver.resolve(row) ← cache hit or Haiku call
|
|
|
+ → {subcategory: "Alum. Electrolytic...", id: 1140, alternatives: [...]}
|
|
|
+ row.state → scouting
|
|
|
+
|
|
|
+POST /session/{id}/row/{row_id}/confirm-category {subcategory_id: 1140}
|
|
|
+ row.applied_filters["lcsc_category_id"] = 1140
|
|
|
+ row.state → filtering
|
|
|
+
|
|
|
+POST /session/{id}/row/{row_id}/search {filters: {voltage_min: 25, ...}, page: 1}
|
|
|
+ lcsc.search(category_id=1140, params=row.normalized_params, filters=..., page=1)
|
|
|
+ → {count: N, results: [...]}
|
|
|
+
|
|
|
+POST /session/{id}/row/{row_id}/confirm {pick: {...}}
|
|
|
+ row.confirmed_pick = pick
|
|
|
+ row.state → confirmed
|
|
|
+
|
|
|
+GET /session/{id}/export
|
|
|
+ [confirmed rows] → assemble BOM → return file
|
|
|
+```
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## Physical File Map
|
|
|
+
|
|
|
+```
|
|
|
+instant-bom/
|
|
|
+├── bom_assistant/
|
|
|
+│ ├── session/
|
|
|
+│ │ └── models.py — Pydantic schemas
|
|
|
+│ │
|
|
|
+│ ├── ingestion/
|
|
|
+│ │ ├── parser.py — raw bytes → list[dict[str, str]]
|
|
|
+│ │ ├── column_mapper.py — raw keys → canonical keys
|
|
|
+│ │ ├── column_synonyms.json — file-backed synonym table (auto-updated by AI)
|
|
|
+│ │ └── normalizer.py — mapped rows → list[BomRow]
|
|
|
+│ │
|
|
|
+│ ├── classification/
|
|
|
+│ │ ├── classifier.py — designator prefix + value heuristics → ComponentCategory
|
|
|
+│ │ ├── param_extractor.py — value string + footprint → NormalizedParams
|
|
|
+│ │ └── package_patterns.json — file-backed regex + known_footprints (auto-updated by AI)
|
|
|
+│ │
|
|
|
+│ ├── suppliers/
|
|
|
+│ │ ├── base.py — abstract SupplierAdapter
|
|
|
+│ │ └── lcsc/
|
|
|
+│ │ ├── fetch_categories.py — one-time CLI: scrape LCSC → categories.json
|
|
|
+│ │ ├── categories.json — committed static file (refresh quarterly)
|
|
|
+│ │ ├── categories.py — load + filter category tree
|
|
|
+│ │ ├── category_resolver.py — LLM category inference + file-backed cache
|
|
|
+│ │ ├── category_cache.json — auto-updated by resolver
|
|
|
+│ │ └── lcsc.py — search adapter (scraper + token mode)
|
|
|
+│ │
|
|
|
+│ └── api/ (not yet built)
|
|
|
+│ └── main.py — FastAPI app + routes
|
|
|
+│
|
|
|
+├── frontend/ (not yet built)
|
|
|
+│ └── src/
|
|
|
+│
|
|
|
+├── examples/
|
|
|
+│ └── bom/ — 12 real BOM test files (CSV, XLSX, XLS)
|
|
|
+│
|
|
|
+└── requirements.txt
|
|
|
+```
|
n2749