The Problem
Retail traders rely on expensive third-party scanners or manual watching. I wanted to build my own scanner infrastructure that could ingest real-time market data, run multiple scanning strategies in parallel, and deliver signals to dashboards in under 100ms.
Development Timeline
Built over 2 months (November 2025 – January 2026) with 134 commits. The project evolved rapidly:
Phase 1: Foundation (Nov 20-25)
- Initial commit: 81 files, 6,406 lines of code
- Core architecture: Flask dashboard, Go ingestion service, Kafka messaging
- First scanners: Breakout, Momentum, Volume
- Ansible deployment automation for LXC containers
- Audio notifications + signal highlighting
Phase 2: Scanner Expansion (Nov 27-30)
- RunningUp Scanner with RVOL/GAP% fields
- Replay System – timestamp-based scanner for backtesting
- RVOL, Burst, Sustained volume scanners
- VWAP-relative momentum with confirmed/unconfirmed signals
- Trade-triggered bar completion for replay mode
Phase 3: Data Persistence (Dec 1-10)
- SQL signal persistence for unlimited dashboard history
- Exponential backoff for Alpaca API retries
- Scanner Admin API (3-phase rollout)
- Heartbeat monitoring for service health alerts
- Multiprocess scanner support
- GitHub Actions CI/CD pipeline
Phase 4: User Experience (Dec 12-21)
- User authentication with TOTP 2FA
- Insider trades panel
- Worker metrics: CPU/mem/rate/lag + hover popup
- Per-scanner leaderboard panels
- Multi-pane charts + IndexedDB signal storage
Phase 5: Advanced Features (Dec 24-31)
- Earnings calendar system with Airflow DAGs
- Benzinga news migration + announcement system
- Stock split handling + enrichment sync
- Sidebar navigation + streamlined header
Phase 6: UI Overhaul (Jan 2026)
- Major UI overhaul with BSP layout
- Replay migrated from SQL to Parquet/DuckDB
- Real-time news via Alpaca WebSocket
- InPlay Scanner – high conviction setups
- Chat with @mention notifications
- Paper Trading Panel with daily P&L tracking
Architecture
+--------------------------------------------------------------+
| Alpaca WebSocket |
| (Real-time quotes) |
+---------------------------+----------------------------------+
|
+---------------------------v----------------------------------+
| Go Ingestion Service |
| (Normalizes, batches, publishes) |
+---------------------------+----------------------------------+
|
+---------------------------v----------------------------------+
| Apache Kafka |
| +---------+ +---------+ +---------+ +-----------------+ |
| | quotes | | trades | | signals | | scanner-events | |
| +---------+ +---------+ +---------+ +-----------------+ |
+---------------------------+----------------------------------+
|
+----------------------+----------------------+
| | |
+----v----+ +----v----+ +----v----+ +----v----+
|Momentum | | Volume | |Breakout | | RVOL | ... (8+ scanners)
| Scanner | | Scanner | | Scanner | | Scanner |
+----+----+ +----+----+ +----+----+ +----+----+
| | | |
+------------+------------+------------+
|
+---------------------------v----------------------------------+
| Redis |
| (Signal buffer for dashboards) |
+---------------------------+----------------------------------+
|
+---------------------------v----------------------------------+
| Flask + React Dashboard |
| (GridStack layout, real-time updates) |
+--------------------------------------------------------------+
Scanner Types
| Scanner | Added | Strategy |
|---|---|---|
| Breakout | Nov 20 | Breaking above resistance with volume |
| Momentum | Nov 20 | Price moving with volume confirmation |
| Volume | Nov 20 | Unusual volume vs 20-day average |
| RunningUp | Nov 27 | Multi-leg upward moves |
| RVOL | Nov 29 | Relative volume spikes (3x+ normal) |
| Burst | Nov 29 | Sudden price acceleration |
| Sustained | Nov 29 | Steady directional moves |
| Earnings | Dec 27 | Pre/post earnings movers |
| InPlay | Jan 4 | High conviction setups |
Challenges Overcome
| Issue | Solution |
|---|---|
| Stale DB connections | Timeouts + connection recycling |
| RVOL calculation errors | Fixed date range + SIP feed |
| Dashboard memory issues | Delta sync + chart lazy loading |
| Slow replay startup | Migrated to Parquet/DuckDB |
| Scaling | Multiprocess scanners + infinite Kafka retry |
Infrastructure
All scanners run in Proxmox LXC containers. Production cluster on VLAN 501 (10.31.11.x) with 15+ containers:
- Kafka – Central message broker
- Redis – Dashboard signal buffer
- SQL Server – Signal persistence
- Prometheus + Grafana – Monitoring
- Ansible – Deployment automation
Development Velocity
Peak activity: January 3, 2026 (15 commits). Consistent 3-7 commits per day during active development. Total: 134 commits over 2 months.