atharva/MiniProfiler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
852957a7dec2698546c7e57506d4ea552f0fb47b
MiniProfiler/docs/PROJECT_STRUCTURE.md
Atharva Sawant 852957a7de Initialized MiniProfiler project 
- Contains the host code with a protocol implementation, data analyser and web-based visualiser
2025-11-27 20:34:41 +05:30

423 lines
14 KiB
Markdown
Raw Blame History

# MiniProfiler Project Structure
## Directory Layout
```
MiniProfiler/
├── docs/ # Documentation
│ ├── GETTING_STARTED.md # Quick start guide
│ ├── PROTOCOL.md # Communication protocol specification
│ └── PROJECT_STRUCTURE.md # This file
│
├── host/ # Host application (Python)
│ ├── miniprofiler/ # Main package
│ │ ├── __init__.py # Package initialization
│ │ ├── analyzer.py # Data analysis and visualization data generation
│ │ ├── cli.py # Command-line interface
│ │ ├── protocol.py # Binary protocol implementation
│ │ ├── serial_reader.py # Serial communication
│ │ ├── symbolizer.py # ELF/DWARF symbol resolution
│ │ └── web_server.py # Flask web server with SocketIO
│ │
│ ├── web/ # Web interface assets
│ │ ├── static/
│ │ │ ├── css/
│ │ │ │ └── style.css # Stylesheet
│ │ │ └── js/
│ │ │ └── app.js # JavaScript application logic
│ │ └── templates/
│ │ └── index.html # Main HTML template
│ │
│ ├── tests/ # Tests and utilities
│ │ ├── __init__.py
│ │ └── sample_data_generator.py # Generate mock profiling data
│ │
│ ├── requirements.txt # Python dependencies
│ ├── setup.py # Package setup
│ └── run.py # Quick start script
│
├── embedded/ # Embedded module (Phase 2 - TODO)
│ ├── src/
│ ├── inc/
│ └── examples/
│
├── .gitignore # Git ignore rules
├── CLAUDE.md # Project overview for Claude
└── README.md # Main project README
```
## Module Descriptions
### Host Application (`host/miniprofiler/`)
#### `protocol.py`
**Purpose:** Binary protocol implementation for serial communication
**Key Components:**
- `ProfileRecord`: Data class for profiling records (14 bytes)
- `Metadata`: Device metadata (MCU clock, timer freq, etc.)
- `StatusInfo`: Device status information
- `CommandPacket`: Commands sent to device
- `ResponsePacket`: Responses from device
- CRC16 calculation and validation
**Used by:** `serial_reader.py`, `analyzer.py`, `sample_data_generator.py`
---
#### `serial_reader.py`
**Purpose:** Serial port communication and packet parsing
**Key Components:**
- `SerialReader`: Main class for serial I/O
- Background thread for continuous reading
- State machine for packet parsing
- Callback-based event handling
- Command sending (START, STOP, GET_STATUS, etc.)
**Callbacks:**
- `on_profile_data`: Profiling records received
- `on_metadata`: Device metadata received
- `on_status`: Status update received
- `on_error`: Error occurred
**Used by:** `web_server.py`
---
#### `symbolizer.py`
**Purpose:** Resolve function addresses to names using ELF/DWARF debug info
**Key Components:**
- `Symbolizer`: ELF file parser
- Loads symbol table from `.elf` file
- Parses DWARF debug info for file/line mappings
- Address-to-name resolution
- Handles function address ranges
**Dependencies:** `pyelftools`
**Used by:** `analyzer.py`, `web_server.py`
---
#### `analyzer.py`
**Purpose:** Analyze profiling data and generate visualization data structures
**Key Components:**
- `ProfileAnalyzer`: Main analysis engine
- Build call tree from flat records
- Compute statistics (call counts, durations)
- Generate flame graph data (d3-flame-graph format)
- Generate timeline data (Plotly format)
- Generate statistics table data
**Data Structures:**
- `CallTreeNode`: Hierarchical call tree
- `FunctionStats`: Per-function statistics
**Used by:** `web_server.py`
---
#### `web_server.py`
**Purpose:** Flask web server with SocketIO for real-time updates
**Key Components:**
- `ProfilerWebServer`: Main server class
- Flask HTTP routes (`/`, `/api/status`, `/api/flamegraph`, etc.)
- SocketIO event handlers (connect, start_profiling, etc.)
- Integrates `SerialReader`, `Symbolizer`, and `ProfileAnalyzer`
- Real-time data streaming to web clients
**Routes:**
- `GET /`: Main web interface
- `GET /api/status`: Server status JSON
- `GET /api/flamegraph`: Flame graph data JSON
- `GET /api/timeline`: Timeline data JSON
- `GET /api/statistics`: Statistics table JSON
**SocketIO Events:**
- `connect_serial`: Connect to device
- `start_profiling`: Start profiling
- `stop_profiling`: Stop profiling
- `clear_data`: Clear all data
- Emits: `flamegraph_update`, `statistics_update`, etc.
**Used by:** `cli.py`
---
#### `cli.py`
**Purpose:** Command-line interface entry point
**Key Components:**
- Argument parsing (--host, --port, --debug, --verbose)
- Logging configuration
- Server initialization and startup
**Entry point:** `miniprofiler` command
---
### Web Interface (`host/web/`)
#### `templates/index.html`
**Purpose:** Main HTML page structure
**Features:**
- Connection controls (serial port, baud rate, ELF path)
- Profiling controls (start, stop, clear, reset)
- Status display
- Metadata panel
- Summary panel
- Three-tab interface (Flame Graph, Timeline, Statistics)
**Dependencies:**
- Socket.IO client
- D3.js
- d3-flame-graph
- Plotly.js
---
#### `static/css/style.css`
**Purpose:** Styling and layout
**Features:**
- Dark theme (VSCode-inspired)
- Responsive design
- Flexbox layouts
- Custom button styles
- Table styling
- Status indicators with animations
---
#### `static/js/app.js`
**Purpose:** Client-side application logic
**Key Functions:**
- `initializeSocket()`: Set up SocketIO connection
- `toggleConnection()`: Connect/disconnect from device
- `startProfiling()`, `stopProfiling()`: Control profiling
- `updateFlameGraph()`: Render flame graph with d3-flame-graph
- `updateTimeline()`: Render timeline with Plotly.js
- `updateStatistics()`: Update statistics table
- `showTab()`: Tab switching
**Event Handlers:**
- Socket events (connect, disconnect, data updates)
- Button clicks
- Window resize
---
### Tests (`host/tests/`)
#### `sample_data_generator.py`
**Purpose:** Generate realistic mock profiling data for testing
**Features:**
- Simulates typical embedded application (main, init, loop, sensors, etc.)
- Generates nested function calls with realistic timing
- Creates binary protocol packets
- Exports JSON files for visualization testing
**Outputs:**
- `sample_profile_data.bin`: Binary protocol data
- `sample_flamegraph.json`: Flame graph data
- `sample_statistics.json`: Statistics data
- `sample_timeline.json`: Timeline data
**Usage:**
```bash
cd host/tests
python sample_data_generator.py
```
---
## Data Flow
### Connection and Initialization
```
User Web UI Web Server Serial Reader Device
│ │ │ │ │
│─── Open Browser ──►│ │ │ │
│ │ │ │ │
│─── Enter Port ────►│ │ │ │
│─── Click Connect ─►│─── connect_serial ──►│─── connect() ─────►│ │
│ │ │ │─── Open ─────►│
│ │ │ │ │
│ │ │─── get_metadata() ►│─── CMD ──────►│
│ │ │ │◄── METADATA ──│
│ │◄── metadata ─────────│◄── on_metadata() ──│ │
│◄── Display Info ───│ │ │ │
```
### Profiling Session
```
User Web UI Web Server Analyzer Device
│ │ │ │ │
│─── Start ─────────►│─── start_profiling ─►│─── start() ─────►│ │
│ │ │ │─── CMD ────────►│
│ │ │ │ │
│ │ │ │◄── DATA ────────│
│ │ │◄── on_profile ───│ │
│ │ │ │ │
│ │ │── add_records() ►│ │
│ │ │ │─ Analyze │
│ │ │ │─ Build Tree │
│ │ │ │─ Compute Stats │
│ │ │◄── JSON ─────────│ │
│ │◄─ flamegraph_update ─│ │ │
│◄── Update Viz ─────│ │ │ │
```
## Technology Stack
### Backend
- **Python 3.8+**: Main language
- **Flask 3.0+**: Web framework
- **Flask-SocketIO 5.3+**: Real-time WebSocket communication
- **pyserial 3.5+**: Serial port communication
- **pyelftools 0.29+**: ELF/DWARF parsing
- **crc 6.1+**: CRC16 calculation
- **eventlet**: Async I/O for SocketIO
### Frontend
- **HTML5/CSS3**: Structure and styling
- **JavaScript (ES6)**: Application logic
- **Socket.IO Client**: Real-time communication
- **D3.js v7**: Visualization library
- **d3-flame-graph 4.1**: Flame graph component
- **Plotly.js 2.27**: Timeline/chart visualization
### Development Tools
- **setuptools**: Package management
- **pip**: Dependency management
- **git**: Version control
## Configuration Files
### `requirements.txt`
Python package dependencies with minimum versions
### `setup.py`
Package metadata and installation configuration
- Entry point: `miniprofiler` CLI command
- Package data includes web assets
### `.gitignore`
Excludes:
- Python bytecode and caches
- Virtual environments
- IDE configs
- Build artifacts
- Generated test data
## Key Design Decisions
### Why Command-Response Protocol?
- Allows host to control profiling (start/stop)
- Can request status and metadata
- More flexible than auto-start mode
- Small overhead acceptable at 115200 baud
### Why Entry Time + Duration?
- Enables both flame graphs (aggregate) and timelines (chronological)
- Only 40% more data than duration-only
- Essential for debugging timing-sensitive embedded systems
### Why d3-flame-graph?
- Industry standard for flame graph visualization
- Interactive (zoom, search, tooltips)
- Customizable colors and layout
- Handles large datasets efficiently
### Why Separate Analyzer Module?
- Decouples data processing from I/O
- Easier to test in isolation
- Can swap visualization formats without changing protocol
- Allows offline analysis of captured data
## Extension Points
### Adding New Commands
1. Add to `Command` enum in `protocol.py`
2. Implement in `SerialReader.send_command()`
3. Add handler in `web_server.py` SocketIO events
4. Update embedded firmware to handle command
### Adding New Visualizations
1. Add route in `web_server.py` (e.g., `/api/callgraph`)
2. Implement data generation in `analyzer.py`
3. Add HTML tab in `index.html`
4. Add JavaScript rendering in `app.js`
5. Update CSS as needed
### Supporting More Microcontrollers
1. Ensure GCC toolchain supports `-finstrument-functions`
2. Implement timing mechanism (DWT, SysTick, or custom timer)
3. Port ring buffer and UART code to new MCU
4. Test and document
### Adding Compression
1. Update protocol version to 0x02
2. Implement compression in embedded module (e.g., delta encoding)
3. Add decompression in `protocol.py`
4. Update `ProfileDataPayload` parsing
## Future Enhancements
### Phase 2: Embedded Module
- [ ] STM32 HAL/LL implementation
- [ ] FreeRTOS integration
- [ ] Example projects for STM32F4/F7/H7
- [ ] CMake build system
### Phase 3: Advanced Features
- [ ] Statistical sampling mode
- [ ] ISR profiling
- [ ] Multi-core support (dual-core STM32H7)
- [ ] Task/thread tracking for RTOS
- [ ] Filtering and search
### Phase 4: Renode Integration
- [ ] Renode platform description
- [ ] Virtual UART setup
- [ ] CI/CD integration
- [ ] Automated regression tests
### Phase 5: Analysis Tools
- [ ] Differential profiling (compare two runs)
- [ ] Export to Chrome Trace Format
- [ ] Call graph visualization
- [ ] Performance regression detection
- [ ] Integration with debuggers (GDB)
## Performance Targets
### Embedded Overhead
- **Target**: <5% CPU overhead
- **Memory**: 2-10 KB RAM for buffers
- **Instrumentation**: 1-2 μs per function call
### Host Performance
- **Latency**: <100ms from device to visualization
- **Throughput**: Handle 500-1000 records/sec
- **Memory**: Scale to 100K+ records in browser
### Bandwidth
- **115200 baud**: ~780 records/sec
- **460800 baud**: ~3100 records/sec
- **921600 baud**: ~6200 records/sec
## Contributing
See individual module docstrings for implementation details.
Follow existing code style and structure when adding features.
Reference in New Issue View Git Blame Copy Permalink