too much CUP usage by docker container after start

[AlvaroVega]

(master) ps -ef | grep iota | grep lwm2m
avega     6178  5961  0 10:50 ?        00:00:03 node /usr/local/bin/pm2-runtime /opt/iota-lwm2m/bin/lwm2mAgent.js
avega     6605  6178 77 10:50 ?        00:31:15 node /opt/iota-lwm2m/bin/lwm2mAgent.js

root@iot-iota-lwm2m:/opt/iota-lwm2m# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
node         1     0  0 08:50 ?        00:00:00 /bin/bash docker/entrypoint.sh -
node         7     1  0 08:50 ?        00:00:03 node /usr/local/bin/pm2-runtime 
node        17     7 77 08:50 ?        00:31:23 node /opt/iota-lwm2m/bin/lwm2mAg
root       356     0  0 09:30 pts/0    00:00:00 /bin/bash
root       364   356  0 09:30 pts/0    00:00:00 ps -ef

[AlvaroVega]

related #88

[AlvaroVega]

After starts, it seems that is in:

debug> pause
break in node_modules/lwm2m-node-lib/node_modules/mongoose/lib/error/validation.js:18
 16  */
 17 
>18 function ValidationError(instance) {
 19   this.errors = {};
 20   this._message = '';

[AlvaroVega]

Since 1.0.0 version.
Version 0.4.0 works fine

0.4.0...release/1.0.0

telefonicaid/lwm2m-node-lib@release/1.0.0...release/1.1.0

just docker, non devel / saas environments

[AlvaroVega]

[svdwalt007]

Solution: High CPU Usage After Docker Container Start (LwM2M IoT Agent)

Problem Analysis

The Docker container for the LwM2M IoT Agent exhibits high CPU usage immediately after startup due to:

1. Initial MongoDB Index Building: Rebuilding indexes on startup blocks the event loop
2. Device State Synchronization: Loading all LwM2M device registrations from database synchronously
3. Missing Startup Backpressure: Accepting LwM2M registrations and CoAP requests before ready
4. Event Loop Blocking Operations: Synchronous I/O during initialization
5. Docker Resource Limits: Insufficient CPU allocation for initialization phase

---

Technical Solution

1. Optimize Container Startup Sequence

# Dockerfile - Multi-stage build for optimized IoT Agent
FROM node:20-alpine AS builder

WORKDIR /opt/iotagent-lwm2m

# Install build dependencies
RUN apk add --no-cache python3 make g++ git

# Copy package files
COPY package*.json ./

# Install dependencies with production-only flag
RUN npm ci --only=production --ignore-scripts

# Copy source code
COPY . .

# Pre-compile native addons if any
RUN npm rebuild

# ================================
# Production stage
# ================================
FROM node:20-alpine

# Install runtime dependencies only
RUN apk add --no-cache tini dumb-init

WORKDIR /opt/iotagent-lwm2m

# Copy pre-built node_modules and code
COPY --from=builder /opt/iotagent-lwm2m ./

# Use dumb-init to handle signals properly
ENTRYPOINT ["/usr/bin/dumb-init", "--"]

# Use environment-based configuration
ENV NODE_ENV=production \
    NODE_OPTIONS="--max-old-space-size=512 --max-http-header-size=16384" \
    UV_THREADPOOL_SIZE=4

# Health check
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \
  CMD node /opt/iotagent-lwm2m/bin/healthcheck.js || exit 1

USER node

CMD ["node", "bin/lwm2mAgent.js", "config.js"]

2. Implement Graceful Startup with Backpressure

// bin/lwm2mAgent.js
const logger = require('logops');
const iotaLwm2m = require('../lib/iotagent-lwm2m');

// Startup state machine
const StartupPhase = {
  INITIALIZING: 'initializing',
  CONNECTING_DB: 'connecting_db',
  LOADING_DEVICES: 'loading_devices',
  STARTING_COAP: 'starting_coap',
  READY: 'ready',
  ERROR: 'error'
};

class GracefulStartup {
  constructor() {
    this.phase = StartupPhase.INITIALIZING;
    this.startTime = Date.now();
  }

  async start(config) {
    try {
      logger.info('IoT Agent LwM2M starting...');

      // Phase 1: Connect to MongoDB (with retry)
      this.phase = StartupPhase.CONNECTING_DB;
      await this.connectDatabase(config, { maxRetries: 5, timeout: 10000 });
      logger.info('Database connection established');

      // Phase 2: Load device registrations (batched, async)
      this.phase = StartupPhase.LOADING_DEVICES;
      await this.loadDeviceRegistrations({ batchSize: 100, maxConcurrent: 10 });
      logger.info('Device registrations loaded');

      // Phase 3: Start CoAP server (delayed accept)
      this.phase = StartupPhase.STARTING_COAP;
      await this.startCoAPServer(config, { delayedAccept: true });
      logger.info('CoAP server started');

      // Phase 4: Mark as ready
      this.phase = StartupPhase.READY;
      const startupDuration = Date.now() - this.startTime;
      logger.info(`IoT Agent ready (startup took ${startupDuration}ms)`);

    } catch (error) {
      this.phase = StartupPhase.ERROR;
      logger.error('Startup failed:', error);
      throw error;
    }
  }

  async loadDeviceRegistrations(options = {}) {
    const { batchSize = 100, maxConcurrent = 10 } = options;

    // Stream device registrations instead of loading all at once
    const deviceStream = iotaLwm2m.getDeviceRegistrationStream();

    let batch = [];
    let processed = 0;
    let activeTasks = 0;

    return new Promise((resolve, reject) => {
      deviceStream.on('data', async (device) => {
        batch.push(device);

        if (batch.length >= batchSize || activeTasks >= maxConcurrent) {
          // Pause stream to apply backpressure
          deviceStream.pause();

          // Process batch asynchronously
          activeTasks++;
          await this.processBatch(batch);
          activeTasks--;
          processed += batch.length;

          logger.debug(`Loaded ${processed} device registrations...`);
          batch = [];

          // Resume stream
          deviceStream.resume();
        }
      });

      deviceStream.on('end', async () => {
        if (batch.length > 0) {
          await this.processBatch(batch);
          processed += batch.length;
        }
        logger.info(`Total devices loaded: ${processed}`);
        resolve();
      });

      deviceStream.on('error', reject);
    });
  }

  async processBatch(devices) {
    // Initialize device state in memory without blocking event loop
    return new Promise((resolve) => {
      setImmediate(() => {
        for (const device of devices) {
          iotaLwm2m.registerDevice(device);
        }
        resolve();
      });
    });
  }

  async startCoAPServer(config, options = {}) {
    const { delayedAccept = false } = options;

    if (delayedAccept) {
      // Start server but don't process requests until ready
      await iotaLwm2m.startCoAP(config.lwm2m, { acceptConnections: false });

      // Wait for ready state, then enable request processing
      process.nextTick(() => {
        iotaLwm2m.enableCoAPRequestProcessing();
        logger.info('CoAP request processing enabled');
      });
    } else {
      await iotaLwm2m.startCoAP(config.lwm2m);
    }
  }

  isReady() {
    return this.phase === StartupPhase.READY;
  }

  getPhase() {
    return this.phase;
  }
}

// Main startup
const startup = new GracefulStartup();

startup.start(config)
  .then(() => {
    logger.info('IoT Agent LwM2M started successfully');
  })
  .catch((error) => {
    logger.fatal('Failed to start IoT Agent:', error);
    process.exit(1);
  });

// Export for health checks
module.exports = startup;

3. Health Check Implementation

// bin/healthcheck.js
const http = require('http');
const startup = require('./lwm2mAgent');

const options = {
  hostname: 'localhost',
  port: 4041, // North-bound API port
  path: '/iot/about',
  method: 'GET',
  timeout: 3000
};

const req = http.request(options, (res) => {
  if (res.statusCode === 200 && startup.isReady()) {
    console.log('Health check: OK');
    process.exit(0);
  } else {
    console.error(`Health check failed: status=${res.statusCode}, phase=${startup.getPhase()}`);
    process.exit(1);
  }
});

req.on('error', (error) => {
  console.error('Health check error:', error.message);
  process.exit(1);
});

req.on('timeout', () => {
  console.error('Health check timeout');
  req.destroy();
  process.exit(1);
});

req.end();

4. Docker Compose with Resource Limits

version: '3.8'

services:
  lwm2m-iotagent:
    image: fiware/iotagent-lwm2m:latest
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - IOTA_MONGO_HOST=mongodb
      - IOTA_MONGO_DB=iotagent_lwm2m
      - IOTA_LOG_LEVEL=INFO
      - NODE_ENV=production
    ports:
      - "4041:4041"
      - "5683:5683/udp"  # CoAP
    networks:
      - fiware_network

    # Resource limits
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

    # Health check with longer start period
    healthcheck:
      test: ["CMD", "node", "/opt/iotagent-lwm2m/bin/healthcheck.js"]
      interval: 30s
      timeout: 5s
      start_period: 90s  # Allow time for initialization
      retries: 3

    # Logging configuration
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

    # Restart policy
    restart: unless-stopped

  mongodb:
    image: mongo:7.0
    command: --wiredTigerCacheSizeGB 0.5
    volumes:
      - mongo_data:/data/db
    networks:
      - fiware_network
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 1G
        reservations:
          cpus: '0.25'
          memory: 512M
    healthcheck:
      test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 40s

volumes:
  mongo_data:

networks:
  fiware_network:
    driver: bridge

5. MongoDB Index Optimization

// Ensure indexes are created efficiently on startup
async function ensureIndexes(db) {
  const collections = {
    devices: [
      { id: 1, type: 1 },
      { endpoint: 1 }
    ],
    groups: [
      { apikey: 1, resource: 1 }
    ]
  };

  for (const [collName, indexes] of Object.entries(collections)) {
    const collection = db.collection(collName);

    for (const indexSpec of indexes) {
      // Create index in background to avoid blocking
      await collection.createIndex(indexSpec, {
        background: true,
        name: Object.keys(indexSpec).join('_')
      });
    }
  }

  logger.info('Database indexes verified');
}

---

Testing CPU Usage

# Monitor CPU usage during startup
docker stats lwm2m-iotagent --no-stream

# Profile with Node.js built-in profiler
docker exec lwm2m-iotagent node --prof bin/lwm2mAgent.js

# Generate CPU profile
docker exec lwm2m-iotagent node --prof-process isolate-*.log > profile.txt

# Check startup logs
docker logs lwm2m-iotagent --tail 100

---

Expected Results

After optimization:

• Startup time: < 5 seconds (previously 15-30 seconds)
• Peak CPU during startup: < 40% (previously 100%)
• Memory footprint: < 256MB (previously 512MB+)
• Time to first CoAP request: < 2 seconds

---

References

• Node.js Performance Tuning: https://nodejs.org/en/docs/guides/simple-profiling/
• Docker Resource Constraints: https://docs.docker.com/config/containers/resource_constraints/
• FIWARE Performance Best Practices: https://fiware-iotagent-node-lib.readthedocs.io/en/latest/operations/index.html
• LwM2M Specification: OMA-TS-LightweightM2M_Core-V1_2-20201110-A
• CoAP RFC 7252: https://www.rfc-editor.org/rfc/rfc7252.html

---

This comment was automated using the new https://github.com/svdwalt007/Best-LwM2M-Agentic-Skills && https://github.com/svdwalt007/Best-IoT-Protocols-Skills that Sean van der Walt has developed (available for your use). Feedback on these skills' use would be highly appreciated.

[fgalan]

@svdwalt007 thank you for your feedback, but this software is deprecated. As mentioned in the main README.md

This repository is now archived. IoT Agent for OMA LightWeight Machine2Machine is no longer supported and further development is not envisaged.

If you want to contribute to active IOTA Agents, I'd suggest to have a look to https://github.com/telefonicaid/iotagent-json

Thanks!