# MiniProfiler Project Structure

## Directory Layout

```
MiniProfiler/
├── docs/                           # Documentation
│   ├── GETTING_STARTED.md         # Quick start guide
│   ├── PROTOCOL.md                # Communication protocol specification
│   └── PROJECT_STRUCTURE.md       # This file
│
├── host/                          # Host application (Python)
│   ├── miniprofiler/             # Main package
│   │   ├── __init__.py           # Package initialization
│   │   ├── analyzer.py           # Data analysis and visualization data generation
│   │   ├── cli.py                # Command-line interface
│   │   ├── protocol.py           # Binary protocol implementation
│   │   ├── serial_reader.py      # Serial communication
│   │   ├── symbolizer.py         # ELF/DWARF symbol resolution
│   │   └── web_server.py         # Flask web server with SocketIO
│   │
│   ├── web/                      # Web interface assets
│   │   ├── static/
│   │   │   ├── css/
│   │   │   │   └── style.css     # Stylesheet
│   │   │   └── js/
│   │   │       └── app.js        # JavaScript application logic
│   │   └── templates/
│   │       └── index.html        # Main HTML template
│   │
│   ├── tests/                    # Tests and utilities
│   │   ├── __init__.py
│   │   └── sample_data_generator.py  # Generate mock profiling data
│   │
│   ├── requirements.txt          # Python dependencies
│   ├── setup.py                  # Package setup
│   └── run.py                    # Quick start script
│
├── embedded/                     # Embedded module (Phase 2 - TODO)
│   ├── src/
│   ├── inc/
│   └── examples/
│
├── .gitignore                    # Git ignore rules
├── CLAUDE.md                     # Project overview for Claude
└── README.md                     # Main project README
```

## Module Descriptions

### Host Application (`host/miniprofiler/`)

#### `protocol.py`
**Purpose:** Binary protocol implementation for serial communication

**Key Components:**
- `ProfileRecord`: Data class for profiling records (14 bytes)
- `Metadata`: Device metadata (MCU clock, timer freq, etc.)
- `StatusInfo`: Device status information
- `CommandPacket`: Commands sent to device
- `ResponsePacket`: Responses from device
- CRC16 calculation and validation

**Used by:** `serial_reader.py`, `analyzer.py`, `sample_data_generator.py`

---

#### `serial_reader.py`
**Purpose:** Serial port communication and packet parsing

**Key Components:**
- `SerialReader`: Main class for serial I/O
  - Background thread for continuous reading
  - State machine for packet parsing
  - Callback-based event handling
  - Command sending (START, STOP, GET_STATUS, etc.)

**Callbacks:**
- `on_profile_data`: Profiling records received
- `on_metadata`: Device metadata received
- `on_status`: Status update received
- `on_error`: Error occurred

**Used by:** `web_server.py`

---

#### `symbolizer.py`
**Purpose:** Resolve function addresses to names using ELF/DWARF debug info

**Key Components:**
- `Symbolizer`: ELF file parser
  - Loads symbol table from `.elf` file
  - Parses DWARF debug info for file/line mappings
  - Address-to-name resolution
  - Handles function address ranges

**Dependencies:** `pyelftools`

**Used by:** `analyzer.py`, `web_server.py`

---

#### `analyzer.py`
**Purpose:** Analyze profiling data and generate visualization data structures

**Key Components:**
- `ProfileAnalyzer`: Main analysis engine
  - Build call tree from flat records
  - Compute statistics (call counts, durations)
  - Generate flame graph data (d3-flame-graph format)
  - Generate timeline data (Plotly format)
  - Generate statistics table data

**Data Structures:**
- `CallTreeNode`: Hierarchical call tree
- `FunctionStats`: Per-function statistics

**Used by:** `web_server.py`

---

#### `web_server.py`
**Purpose:** Flask web server with SocketIO for real-time updates

**Key Components:**
- `ProfilerWebServer`: Main server class
  - Flask HTTP routes (`/`, `/api/status`, `/api/flamegraph`, etc.)
  - SocketIO event handlers (connect, start_profiling, etc.)
  - Integrates `SerialReader`, `Symbolizer`, and `ProfileAnalyzer`
  - Real-time data streaming to web clients

**Routes:**
- `GET /`: Main web interface
- `GET /api/status`: Server status JSON
- `GET /api/flamegraph`: Flame graph data JSON
- `GET /api/timeline`: Timeline data JSON
- `GET /api/statistics`: Statistics table JSON

**SocketIO Events:**
- `connect_serial`: Connect to device
- `start_profiling`: Start profiling
- `stop_profiling`: Stop profiling
- `clear_data`: Clear all data
- Emits: `flamegraph_update`, `statistics_update`, etc.

**Used by:** `cli.py`

---

#### `cli.py`
**Purpose:** Command-line interface entry point

**Key Components:**
- Argument parsing (--host, --port, --debug, --verbose)
- Logging configuration
- Server initialization and startup

**Entry point:** `miniprofiler` command

---

### Web Interface (`host/web/`)

#### `templates/index.html`
**Purpose:** Main HTML page structure

**Features:**
- Connection controls (serial port, baud rate, ELF path)
- Profiling controls (start, stop, clear, reset)
- Status display
- Metadata panel
- Summary panel
- Three-tab interface (Flame Graph, Timeline, Statistics)

**Dependencies:**
- Socket.IO client
- D3.js
- d3-flame-graph
- Plotly.js

---

#### `static/css/style.css`
**Purpose:** Styling and layout

**Features:**
- Dark theme (VSCode-inspired)
- Responsive design
- Flexbox layouts
- Custom button styles
- Table styling
- Status indicators with animations

---

#### `static/js/app.js`
**Purpose:** Client-side application logic

**Key Functions:**
- `initializeSocket()`: Set up SocketIO connection
- `toggleConnection()`: Connect/disconnect from device
- `startProfiling()`, `stopProfiling()`: Control profiling
- `updateFlameGraph()`: Render flame graph with d3-flame-graph
- `updateTimeline()`: Render timeline with Plotly.js
- `updateStatistics()`: Update statistics table
- `showTab()`: Tab switching

**Event Handlers:**
- Socket events (connect, disconnect, data updates)
- Button clicks
- Window resize

---

### Tests (`host/tests/`)

#### `sample_data_generator.py`
**Purpose:** Generate realistic mock profiling data for testing

**Features:**
- Simulates typical embedded application (main, init, loop, sensors, etc.)
- Generates nested function calls with realistic timing
- Creates binary protocol packets
- Exports JSON files for visualization testing

**Outputs:**
- `sample_profile_data.bin`: Binary protocol data
- `sample_flamegraph.json`: Flame graph data
- `sample_statistics.json`: Statistics data
- `sample_timeline.json`: Timeline data

**Usage:**
```bash
cd host/tests
python sample_data_generator.py
```

---

## Data Flow

### Connection and Initialization

```
User                Web UI              Web Server          Serial Reader       Device
 │                    │                      │                    │               │
 │─── Open Browser ──►│                      │                    │               │
 │                    │                      │                    │               │
 │─── Enter Port ────►│                      │                    │               │
 │─── Click Connect ─►│─── connect_serial ──►│─── connect() ─────►│               │
 │                    │                      │                    │─── Open ─────►│
 │                    │                      │                    │               │
 │                    │                      │─── get_metadata() ►│─── CMD ──────►│
 │                    │                      │                    │◄── METADATA ──│
 │                    │◄── metadata ─────────│◄── on_metadata() ──│               │
 │◄── Display Info ───│                      │                    │               │
```

### Profiling Session

```
User                Web UI              Web Server          Analyzer            Device
 │                    │                      │                  │                 │
 │─── Start ─────────►│─── start_profiling ─►│─── start() ─────►│                 │
 │                    │                      │                  │─── CMD ────────►│
 │                    │                      │                  │                 │
 │                    │                      │                  │◄── DATA ────────│
 │                    │                      │◄── on_profile ───│                 │
 │                    │                      │                  │                 │
 │                    │                      │── add_records() ►│                 │
 │                    │                      │                  │─ Analyze        │
 │                    │                      │                  │─ Build Tree     │
 │                    │                      │                  │─ Compute Stats  │
 │                    │                      │◄── JSON ─────────│                 │
 │                    │◄─ flamegraph_update ─│                  │                 │
 │◄── Update Viz ─────│                      │                  │                 │
```

## Technology Stack

### Backend
- **Python 3.8+**: Main language
- **Flask 3.0+**: Web framework
- **Flask-SocketIO 5.3+**: Real-time WebSocket communication
- **pyserial 3.5+**: Serial port communication
- **pyelftools 0.29+**: ELF/DWARF parsing
- **crc 6.1+**: CRC16 calculation
- **eventlet**: Async I/O for SocketIO

### Frontend
- **HTML5/CSS3**: Structure and styling
- **JavaScript (ES6)**: Application logic
- **Socket.IO Client**: Real-time communication
- **D3.js v7**: Visualization library
- **d3-flame-graph 4.1**: Flame graph component
- **Plotly.js 2.27**: Timeline/chart visualization

### Development Tools
- **setuptools**: Package management
- **pip**: Dependency management
- **git**: Version control

## Configuration Files

### `requirements.txt`
Python package dependencies with minimum versions

### `setup.py`
Package metadata and installation configuration
- Entry point: `miniprofiler` CLI command
- Package data includes web assets

### `.gitignore`
Excludes:
- Python bytecode and caches
- Virtual environments
- IDE configs
- Build artifacts
- Generated test data

## Key Design Decisions

### Why Command-Response Protocol?
- Allows host to control profiling (start/stop)
- Can request status and metadata
- More flexible than auto-start mode
- Small overhead acceptable at 115200 baud

### Why Entry Time + Duration?
- Enables both flame graphs (aggregate) and timelines (chronological)
- Only 40% more data than duration-only
- Essential for debugging timing-sensitive embedded systems

### Why d3-flame-graph?
- Industry standard for flame graph visualization
- Interactive (zoom, search, tooltips)
- Customizable colors and layout
- Handles large datasets efficiently

### Why Separate Analyzer Module?
- Decouples data processing from I/O
- Easier to test in isolation
- Can swap visualization formats without changing protocol
- Allows offline analysis of captured data

## Extension Points

### Adding New Commands
1. Add to `Command` enum in `protocol.py`
2. Implement in `SerialReader.send_command()`
3. Add handler in `web_server.py` SocketIO events
4. Update embedded firmware to handle command

### Adding New Visualizations
1. Add route in `web_server.py` (e.g., `/api/callgraph`)
2. Implement data generation in `analyzer.py`
3. Add HTML tab in `index.html`
4. Add JavaScript rendering in `app.js`
5. Update CSS as needed

### Supporting More Microcontrollers
1. Ensure GCC toolchain supports `-finstrument-functions`
2. Implement timing mechanism (DWT, SysTick, or custom timer)
3. Port ring buffer and UART code to new MCU
4. Test and document

### Adding Compression
1. Update protocol version to 0x02
2. Implement compression in embedded module (e.g., delta encoding)
3. Add decompression in `protocol.py`
4. Update `ProfileDataPayload` parsing

## Future Enhancements

### Phase 2: Embedded Module
- [ ] STM32 HAL/LL implementation
- [ ] FreeRTOS integration
- [ ] Example projects for STM32F4/F7/H7
- [ ] CMake build system

### Phase 3: Advanced Features
- [ ] Statistical sampling mode
- [ ] ISR profiling
- [ ] Multi-core support (dual-core STM32H7)
- [ ] Task/thread tracking for RTOS
- [ ] Filtering and search

### Phase 4: Renode Integration
- [ ] Renode platform description
- [ ] Virtual UART setup
- [ ] CI/CD integration
- [ ] Automated regression tests

### Phase 5: Analysis Tools
- [ ] Differential profiling (compare two runs)
- [ ] Export to Chrome Trace Format
- [ ] Call graph visualization
- [ ] Performance regression detection
- [ ] Integration with debuggers (GDB)

## Performance Targets

### Embedded Overhead
- **Target**: <5% CPU overhead
- **Memory**: 2-10 KB RAM for buffers
- **Instrumentation**: 1-2 μs per function call

### Host Performance
- **Latency**: <100ms from device to visualization
- **Throughput**: Handle 500-1000 records/sec
- **Memory**: Scale to 100K+ records in browser

### Bandwidth
- **115200 baud**: ~780 records/sec
- **460800 baud**: ~3100 records/sec
- **921600 baud**: ~6200 records/sec

## Contributing

See individual module docstrings for implementation details.
Follow existing code style and structure when adding features.