Multi-Kline Management System for High-Scale Trading Automation

Overview

The Multi-Kline Management system enables MagicTradeBot to efficiently load, cache, and maintain multiple timeframe candlestick (kline) data for hundreds of thousands of trading symbols simultaneously while respecting exchange API rate limits. This feature supports diverse trading strategies—from high-frequency scalping to long-term position trading—all operating concurrently with complete isolation.

Core Capabilities

Strategy-Based Interval Loading

The bot dynamically loads kline data based on the active trading strategy type. Each strategy has predefined optimal timeframes:

Strategy Intervals:

  • HFT (High-Frequency Trading): 1m – Ultra-fast execution on minute-level movements
  • Scalp: 15m – Short-term momentum plays on 15-minute charts
  • Day Trading: 15m, 30m, 1h, 2h, 4h – Intraday analysis across multiple timeframes
  • Swing Trading: 1h, 2h, 4h, 6h, 12h – Multi-hour patterns for position entries
  • Long-Term: 1d – Daily charts for macro trend analysis
  • Default: 5m, 15m, 1h – Balanced multi-timeframe approach

Each symbol can have one or more intervals loaded simultaneously based on its assigned strategy, enabling multi-timeframe confirmation while maintaining memory efficiency.

Intelligent Rate Limit Management

Adaptive Request Throttling

The system implements a sophisticated rate limiting algorithm that respects exchange API restrictions:

  • maxRequestsPerSecond: 10 – Exchange's hard limit per second
  • safeBuffer: 0.2 – Safety margin (uses 80% of max capacity)
  • Effective Rate: 8 requests/second (10 × 0.8)

Request Distribution Strategy

Chunked Symbol Processing:

  • chunk_size: 20 – Symbols fetched per update cycle
  • Spreads hundreds of thousands of symbols across time
  • Prevents API burst violations
Chunk Processing Time = chunk_size / effective_rate
Example: 20 symbols / 8 req/s = 2.5 seconds per chunk

Refresh Cycle:

  • kline_refresh_rate: 7 seconds between full update cycles
  • Ensures continuous data freshness without overwhelming APIs
  • Adaptive delay insertion based on response times

Multi-Exchange Support

  • Binance: 1200 requests/minute (20/sec)
  • OKX: 20 requests/2 seconds
  • Bybit: 120 requests/5 seconds

The bot auto-adjusts maxRequestsPerSecond based on detected exchange.

Memory-Efficient Kline Storage

Rolling Window Approach

To handle massive symbol counts without memory exhaustion:

  • Per-Symbol Kline Management:
    • max_kline_length: 100 – Maximum candles retained per interval
    • Circular buffer implementation (FIFO)
    • Old candles automatically purged as new ones arrive
Single Kline Object: ~200 bytes
Per Symbol (5 intervals × 100 candles): ~100 KB
100,000 symbols: ~10 GB total memory footprint

Lazy Loading & Caching

  • Initial load fetches full max_kline_length history
  • Subsequent updates only request latest candles
  • In-memory cache with LRU eviction for inactive symbols
  • Symbols inactive for 24h automatically cleared from memory

Concurrent Strategy Execution

Symbol-Strategy Mapping

BTCUSDT:
  strategy: "day"
  intervals: ["15m", "1h", "4h"]
  
ETHUSDT:
  strategy: "scalp"
  intervals: ["15m"]

XRPUSDT:
  strategy: "swing"
  intervals: ["1h", "4h", "12h"]

Isolated Signal Processing

  • signal_refresh_interval: 20 seconds (configurable per strategy)
  • Scalp → every 20 seconds
  • Long-term → every 1800 seconds (30 mins)

Each symbol computes its own indicators independently, with no cross-contamination between strategies. Worker threads run CPU-heavy tasks in parallel.

Scalability Architecture

Distributed Kline Fetching

Priority Queue System:

  1. High-priority symbols (active trades)
  2. Medium-priority (strong signals)
  3. Low-priority (monitoring only)
  4. Stale data (not updated in 2× refresh rate)

Batch Request Optimization:

  • Single API call can fetch multiple intervals for same symbol
  • Reduces total requests by ~40%
  • Uses exchange-specific batch endpoints

Fault Tolerance

  • Rate limit hit → Exponential backoff (2x, 4x, 8x)
  • Failed request → Up to 3 retries
  • Persistent failures → Symbol temporarily disabled
  • Connection loss → Queued until reconnect

Data validation includes:

  • Timestamp gap detection
  • Price anomaly detection
  • Automatic backfill
  • Stale data exclusion

Performance Optimization

Update Cycle Workflow

Per Refresh Cycle (every 7 seconds):

  1. Selection Phase (0.1s)
  2. Fetch Phase (2.5s)
  3. Processing Phase (1.5s)
  4. Idle Phase (2.9s)

Total Throughput:

  • 2.86 symbols/second
  • 10,285 symbols updated per hour
  • 100,000 symbols refreshed every ~9.7 hours

Strategy-Specific Optimizations

  • Scalp: faster refresh, smaller buffers
  • Long-Term: slow refresh, standard buffers

Configuration Best Practices

Small Portfolio (<1,000 symbols)

chunk_size: 50
kline_refresh_rate: 5
maxRequestsPerSecond: 15
safeBuffer: 0.3

Medium Portfolio (1,000–10,000 symbols)

chunk_size: 20
kline_refresh_rate: 7
maxRequestsPerSecond: 10
safeBuffer: 0.2

Large Portfolio (>100,000 symbols)

chunk_size: 10
kline_refresh_rate: 10
maxRequestsPerSecond: 8
safeBuffer: 0.15
max_kline_length: 50

Exchange-Specific Tuning

  • Binance: larger chunks, faster refresh
  • Strict exchanges: smaller chunks, slower refresh

Real-World Example

Scenario: Trading 50,000 symbols on Binance

chunk_size: 25
kline_refresh_rate: 8
maxRequestsPerSecond: 12
safeBuffer: 0.2
max_kline_length: 80

Performance:

  • Effective rate: 9.6 req/sec
  • Chunk processing: 2.6 seconds
  • Full refresh: Every ~4.4 hours
  • Memory usage: ~8 GB
  • API/day: ~275,000

Summary Analytics

The summary_interval: "1d" setting provides daily summary insights:

  • Total symbols monitored
  • API usage
  • Rate limit usage (%)
  • Data freshness
  • Signals per strategy
  • Memory usage

📎 Related Topics