Metrics Caching System

Overview

The metrics caching system prevents rate limiting and improves performance by:

1. Background Sync: Redis cron job runs every 30 minutes
2. Database Cache: Stores metrics in PostgreSQL tables
3. Instant Page Loads: Frontend reads from cached data (no live API calls)
4. Metrics History: 30-day snapshots for trend charts

Architecture

Before (Problematic)

User visits page → Frontend loops through resources → N live DigitalOcean API calls

Problems:

• 10 databases × 100 page loads = 1,000 DO API calls/day
• Rate limit: 5,000 requests/hour (easily exceeded)
• Sequential blocking calls = slow page loads (5+ seconds)

After (Optimized)

Redis Worker (every 30 min) → Batch fetch from DO API → Cache in PostgreSQL
User visits page → Frontend reads from database cache → Instant load (< 100ms)

Benefits:

• Only 48 sync jobs/day (one every 30 minutes)
• No rate limit concerns
• Instant page loads with no API calls
• Historical data for trend analysis

Database Tables

hosting_databases

Stores database metrics:

• cpu_count - Number of CPU cores
• memory_mb - Memory in MB
• disk_gb - Disk size in GB
• metrics_last_updated - Last sync timestamp

hosting_droplets

Stores droplet metrics:

• cpu_usage_percent - Average CPU usage (last hour)
• memory_usage_percent - Memory usage percentage
• disk_usage_percent - Disk usage percentage
• metrics_last_updated - Last sync timestamp

hosting_metrics_history

Time-series metrics for charts:

• resource_type - 'database', 'droplet', or 'app'
• resource_id - DigitalOcean resource ID
• cpu_value, memory_value, disk_value - Metric snapshots
• recorded_at - Timestamp for trend analysis

Running the Metrics Worker

DigitalOcean App Platform (Production)

The metrics worker is automatically deployed as a worker component in app-spec.yaml:

workers:
- name: metrics-sync
  run_command: node workers/metricsSync.js
  instance_size_slug: basic-xxs

To deploy:

cd rmm-psa-backend
git add -A && git commit -m "Update worker" && git push
# Worker deploys automatically with the app

To view logs:

# Get app ID
doctl apps list
 
# View worker logs
doctl apps logs <APP_ID> --type run --component metrics-sync --follow

Local Development

cd rmm-psa-backend
npm run worker:metrics

Environment Variables

BACKEND_URL=https://rmm-psa-backend.ondigitalocean.app
SYNC_INTERVAL=*/30 * * * *  # Every 30 minutes (default)

Manual Sync

You can trigger a manual sync via API:

curl -X POST https://rmm-psa-backend.ondigitalocean.app/api/hosting/sync

Response:

{
  "message": "DigitalOcean resources synced successfully",
  "syncStats": {
    "apps": 5,
    "databases": 3,
    "droplets": 2,
    "metrics": 5
  }
}

Frontend Changes

TabDatabases.jsx

Before:

for (const db of databases) {
    const metricsRes = await apiFetch(`/hosting/database/${db.do_database_id}/metrics`);
    // Live API call on EVERY page load! ❌
}

After:

const metricsData = {};
for (const db of databases) {
    metricsData[db.do_database_id] = {
        cpu_count: db.cpu_count,      // Read from cached field ✅
        memory_mb: db.memory_mb,
        disk_gb: db.disk_gb
    };
}

TabServers.jsx

Reads cpu_usage_percent from cached droplet records instead of live API calls.

DatabaseDetail.jsx

Displays cached metrics from database object on initial load.

Monitoring

Check Last Sync Time

SELECT tenant_id, last_sync_at 
FROM digitalocean_config 
ORDER BY last_sync_at DESC;

View Recent Metrics

SELECT database_name, cpu_count, memory_mb, disk_gb, metrics_last_updated
FROM hosting_databases
WHERE tenant_id = 1
ORDER BY metrics_last_updated DESC;

Metrics History (30-day trends)

SELECT 
    resource_type, 
    resource_id, 
    cpu_value, 
    memory_value, 
    recorded_at
FROM hosting_metrics_history
WHERE resource_id = 'abc123-database-id'
ORDER BY recorded_at DESC
LIMIT 30;

Troubleshooting

Worker not syncing?

1. Check worker logs: doctl apps logs <APP_ID> --component metrics-sync --follow
2. Verify worker is deployed in app-spec.yaml
3. Check BACKEND_URL and DO_API_TOKEN environment variables
4. Verify DigitalOcean API token is valid
5. Manually trigger sync to test: curl -X POST .../api/hosting/sync (requires tenant auth)

Metrics showing null?

1. Wait for first sync cycle (30 minutes) or check worker logs
2. Verify worker has successfully started and run initial sync
3. Check worker logs for API errors

Page loads still slow?

1. Verify frontend is NOT calling /database/:id/metrics endpoints
2. Check browser network tab for live API calls
3. Confirm metrics are in database: SELECT * FROM hosting_databases WHERE metrics_last_updated IS NOT NULL

Rate Limits

DigitalOcean API Limits:

• 5,000 requests per hour
• 250 requests per minute (burst)

Our Usage with Caching:

• ~48 sync cycles per day
• ~10 resource API calls per sync
• Total: ~480 API calls/day (well under limits)

Without Caching (Old System):

• 10 databases × 100 page loads = 1,000 calls/day
• 10 droplets × 50 page loads = 500 calls/day
• Risk of hitting limits during traffic spikes ⚠️

Next Steps

Future Enhancements

1. Metrics Charts: Add trend graphs using hosting_metrics_history data
2. Alert Thresholds: Notify when CPU > 80% or disk > 90%
3. Custom Intervals: Allow per-resource sync frequencies
4. Redis Queue: Use Bull/BullMQ for job management
5. Health Checks: Monitor worker uptime and sync success rate