MiniProfiler/docs/PROJECT_STRUCTURE.md

MiniProfiler Project Structure

Directory Layout

MiniProfiler/
├── docs/                           # Documentation
│   ├── GETTING_STARTED.md         # Quick start guide
│   ├── PROTOCOL.md                # Communication protocol specification
│   └── PROJECT_STRUCTURE.md       # This file
│
├── host/                          # Host application (Python)
│   ├── miniprofiler/             # Main package
│   │   ├── __init__.py           # Package initialization
│   │   ├── analyzer.py           # Data analysis and visualization data generation
│   │   ├── cli.py                # Command-line interface
│   │   ├── protocol.py           # Binary protocol implementation
│   │   ├── serial_reader.py      # Serial communication
│   │   ├── symbolizer.py         # ELF/DWARF symbol resolution
│   │   └── web_server.py         # Flask web server with SocketIO
│   │
│   ├── web/                      # Web interface assets
│   │   ├── static/
│   │   │   ├── css/
│   │   │   │   └── style.css     # Stylesheet
│   │   │   └── js/
│   │   │       └── app.js        # JavaScript application logic
│   │   └── templates/
│   │       └── index.html        # Main HTML template
│   │
│   ├── tests/                    # Tests and utilities
│   │   ├── __init__.py
│   │   └── sample_data_generator.py  # Generate mock profiling data
│   │
│   ├── requirements.txt          # Python dependencies
│   └── run.py                    # Quick start script
│
├── embedded/                     # Embedded module (Phase 2 - TODO)
│   ├── src/
│   ├── inc/
│   └── examples/
│
├── .gitignore                    # Git ignore rules
├── CLAUDE.md                     # Project overview for Claude
└── README.md                     # Main project README

Module Descriptions

Host Application (host/miniprofiler/)

protocol.py

Purpose: Binary protocol implementation for serial communication

Key Components:

• ProfileRecord: Data class for profiling records (14 bytes)
• Metadata: Device metadata (MCU clock, timer freq, etc.)
• StatusInfo: Device status information
• CommandPacket: Commands sent to device
• ResponsePacket: Responses from device
• CRC16 calculation and validation

Used by: serial_reader.py, analyzer.py, sample_data_generator.py

---

serial_reader.py

Purpose: Serial port communication and packet parsing

Key Components:

• SerialReader: Main class for serial I/O
  • Background thread for continuous reading
  • State machine for packet parsing
  • Callback-based event handling
  • Command sending (START, STOP, GET_STATUS, etc.)

Callbacks:

• on_profile_data: Profiling records received
• on_metadata: Device metadata received
• on_status: Status update received
• on_error: Error occurred

Used by: web_server.py

---

symbolizer.py

Purpose: Resolve function addresses to names using ELF/DWARF debug info

Key Components:

• Symbolizer: ELF file parser
  • Loads symbol table from .elf file
  • Parses DWARF debug info for file/line mappings
  • Address-to-name resolution
  • Handles function address ranges

Dependencies: pyelftools

Used by: analyzer.py, web_server.py

---

analyzer.py

Purpose: Analyze profiling data and generate visualization data structures

Key Components:

• ProfileAnalyzer: Main analysis engine
  • Build call tree from flat records
  • Compute statistics (call counts, durations)
  • Generate flame graph data (d3-flame-graph format)
  • Generate timeline data (Plotly format)
  • Generate statistics table data

Data Structures:

• CallTreeNode: Hierarchical call tree
• FunctionStats: Per-function statistics

Used by: web_server.py

---

web_server.py

Purpose: Flask web server with SocketIO for real-time updates

Key Components:

• ProfilerWebServer: Main server class
  • Flask HTTP routes (/, /api/status, /api/flamegraph, etc.)
  • SocketIO event handlers (connect, start_profiling, etc.)
  • Integrates SerialReader, Symbolizer, and ProfileAnalyzer
  • Real-time data streaming to web clients

Routes:

• GET /: Main web interface
• GET /api/status: Server status JSON
• GET /api/flamegraph: Flame graph data JSON
• GET /api/timeline: Timeline data JSON
• GET /api/statistics: Statistics table JSON

SocketIO Events:

• connect_serial: Connect to device
• start_profiling: Start profiling
• stop_profiling: Stop profiling
• clear_data: Clear all data
• Emits: flamegraph_update, statistics_update, etc.

Used by: cli.py

---

cli.py

Purpose: Command-line interface entry point

Key Components:

• Argument parsing (--host, --port, --debug, --verbose)
• Logging configuration
• Server initialization and startup

Entry point: miniprofiler command

---

Web Interface (host/web/)

templates/index.html

Purpose: Main HTML page structure

Features:

• Connection controls (serial port, baud rate, ELF path)
• Profiling controls (start, stop, clear, reset)
• Status display
• Metadata panel
• Summary panel
• Three-tab interface (Flame Graph, Timeline, Statistics)

Dependencies:

• Socket.IO client
• D3.js
• d3-flame-graph
• Plotly.js

---

static/css/style.css

Purpose: Styling and layout

Features:

• Dark theme (VSCode-inspired)
• Responsive design
• Flexbox layouts
• Custom button styles
• Table styling
• Status indicators with animations

---

static/js/app.js

Purpose: Client-side application logic

Key Functions:

• initializeSocket(): Set up SocketIO connection
• toggleConnection(): Connect/disconnect from device
• startProfiling(), stopProfiling(): Control profiling
• updateFlameGraph(): Render flame graph with d3-flame-graph
• updateTimeline(): Render timeline with Plotly.js
• updateStatistics(): Update statistics table
• showTab(): Tab switching

Event Handlers:

• Socket events (connect, disconnect, data updates)
• Button clicks
• Window resize

---

Tests (host/tests/)

sample_data_generator.py

Purpose: Generate realistic mock profiling data for testing

Features:

• Simulates typical embedded application (main, init, loop, sensors, etc.)
• Generates nested function calls with realistic timing
• Creates binary protocol packets
• Exports JSON files for visualization testing

Outputs:

• sample_profile_data.bin: Binary protocol data
• sample_flamegraph.json: Flame graph data
• sample_statistics.json: Statistics data
• sample_timeline.json: Timeline data

Usage:

cd host/tests
python sample_data_generator.py

---

Data Flow

Connection and Initialization

User                Web UI              Web Server          Serial Reader       Device
 │                    │                      │                    │               │
 │─── Open Browser ──►│                      │                    │               │
 │                    │                      │                    │               │
 │─── Enter Port ────►│                      │                    │               │
 │─── Click Connect ─►│─── connect_serial ──►│─── connect() ─────►│               │
 │                    │                      │                    │─── Open ─────►│
 │                    │                      │                    │               │
 │                    │                      │─── get_metadata() ►│─── CMD ──────►│
 │                    │                      │                    │◄── METADATA ──│
 │                    │◄── metadata ─────────│◄── on_metadata() ──│               │
 │◄── Display Info ───│                      │                    │               │

Profiling Session

User                Web UI              Web Server          Analyzer            Device
 │                    │                      │                  │                 │
 │─── Start ─────────►│─── start_profiling ─►│─── start() ─────►│                 │
 │                    │                      │                  │─── CMD ────────►│
 │                    │                      │                  │                 │
 │                    │                      │                  │◄── DATA ────────│
 │                    │                      │◄── on_profile ───│                 │
 │                    │                      │                  │                 │
 │                    │                      │── add_records() ►│                 │
 │                    │                      │                  │─ Analyze        │
 │                    │                      │                  │─ Build Tree     │
 │                    │                      │                  │─ Compute Stats  │
 │                    │                      │◄── JSON ─────────│                 │
 │                    │◄─ flamegraph_update ─│                  │                 │
 │◄── Update Viz ─────│                      │                  │                 │

Technology Stack

Backend

• Python 3.8+: Main language
• Flask 3.0+: Web framework
• Flask-SocketIO 5.3+: Real-time WebSocket communication
• pyserial 3.5+: Serial port communication
• pyelftools 0.29+: ELF/DWARF parsing
• crc 6.1+: CRC16 calculation
• eventlet: Async I/O for SocketIO

Frontend

• HTML5/CSS3: Structure and styling
• JavaScript (ES6): Application logic
• Socket.IO Client: Real-time communication
• D3.js v7: Visualization library
• d3-flame-graph 4.1: Flame graph component
• Plotly.js 2.27: Timeline/chart visualization

Development Tools

• setuptools: Package management
• pip: Dependency management
• git: Version control

Configuration Files

requirements.txt

Python package dependencies with minimum versions

.gitignore

Excludes:

• Python bytecode and caches
• Virtual environments
• IDE configs
• Build artifacts
• Generated test data

Key Design Decisions

Why Command-Response Protocol?

• Allows host to control profiling (start/stop)
• Can request status and metadata
• More flexible than auto-start mode
• Small overhead acceptable at 115200 baud

Why Entry Time + Duration?

• Enables both flame graphs (aggregate) and timelines (chronological)
• Only 40% more data than duration-only
• Essential for debugging timing-sensitive embedded systems

Why d3-flame-graph?

• Industry standard for flame graph visualization
• Interactive (zoom, search, tooltips)
• Customizable colors and layout
• Handles large datasets efficiently

Why Separate Analyzer Module?

• Decouples data processing from I/O
• Easier to test in isolation
• Can swap visualization formats without changing protocol
• Allows offline analysis of captured data

Extension Points

Adding New Commands

1. Add to Command enum in protocol.py
2. Implement in SerialReader.send_command()
3. Add handler in web_server.py SocketIO events
4. Update embedded firmware to handle command

Adding New Visualizations

1. Add route in web_server.py (e.g., /api/callgraph)
2. Implement data generation in analyzer.py
3. Add HTML tab in index.html
4. Add JavaScript rendering in app.js
5. Update CSS as needed

Supporting More Microcontrollers

1. Ensure GCC toolchain supports -finstrument-functions
2. Implement timing mechanism (DWT, SysTick, or custom timer)
3. Port ring buffer and UART code to new MCU
4. Test and document

Adding Compression

1. Update protocol version to 0x02
2. Implement compression in embedded module (e.g., delta encoding)
3. Add decompression in protocol.py
4. Update ProfileDataPayload parsing

Future Enhancements

Phase 2: Embedded Module

• STM32 HAL/LL implementation
• FreeRTOS integration
• Example projects for STM32F4/F7/H7
• CMake build system

Phase 3: Advanced Features

• Statistical sampling mode
• ISR profiling
• Multi-core support (dual-core STM32H7)
• Task/thread tracking for RTOS
• Filtering and search

Phase 4: Renode Integration

• Renode platform description
• Virtual UART setup
• CI/CD integration
• Automated regression tests

Phase 5: Analysis Tools

• Differential profiling (compare two runs)
• Export to Chrome Trace Format
• Call graph visualization
• Performance regression detection
• Integration with debuggers (GDB)

Performance Targets

Embedded Overhead

• Target: <5% CPU overhead
• Memory: 2-10 KB RAM for buffers
• Instrumentation: 1-2 μs per function call

Host Performance

• Latency: <100ms from device to visualization
• Throughput: Handle 500-1000 records/sec
• Memory: Scale to 100K+ records in browser

Bandwidth

• 115200 baud: ~780 records/sec
• 460800 baud: ~3100 records/sec
• 921600 baud: ~6200 records/sec

Contributing

See individual module docstrings for implementation details. Follow existing code style and structure when adding features.