Configure background scheduler in PM2 and optimize cron frequencies by Copilot · Pull Request #7 · enikqi/yapgrid

Copilot · 2025-11-04T22:10:21Z

The background scheduler exists but isn't configured to run automatically with PM2, causing cron jobs to fail. Additionally, the auto-processing job runs too frequently (every 30s), causing unnecessary CPU load.

Changes

PM2 Configuration - Added background-scheduler as second app in ecosystem.config.js with 500M memory limit and dedicated log files

Cron Frequency - Reduced auto-processing from every 30 seconds to every 2 minutes in background-scheduler.js

// Before: cron.schedule('*/30 * * * * *', async () => {
// After:  cron.schedule('*/2 * * * *', async () => {

Database Initialization - Added scripts/init-job-settings.js to upsert 7 job control settings (auto_processing_enabled, batch sizes, intervals, etc.)
Documentation - Added SCHEDULER_SETUP.md with deployment commands, troubleshooting guide, and configuration reference

Deployment

cd /home/ubuntu/apps/yapgrid/site
node scripts/init-job-settings.js
pm2 delete all && pm2 start ecosystem.config.js && pm2 save

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

checkpoint.prisma.io
- Triggering command: /usr/local/bin/node /home/REDACTED/work/yapgrid/yapgrid/site/node_modules/prisma/build/child {"product":"prisma","version":"6.17.0","cli_install_type":"local","information":"","local_timestamp":"2025-11-04T22:16:06Z","project_hash":"035950a5","cli_path":"/home/REDACTED/work/yapgrid/yapgrid/site/node_modules/.bin/prisma","cli_path_hash":"0fcc53eb","endpoint":"REDACTED","disable":false,"arch":"x64","os":"linux","node_version":"v20.19.5","ci":true,"ci_name":"GitHub Actions","command":"generate --postinstall \"UNABLE_TO_FIND_POSTINSTALL_TRIGGER__ENVAR_MISSING\"","schema_providers":["sqlite"],"schema_preview_features":[],"schema_generators_providers":["prisma-client-js"],"cache_file":"/home/REDACTED/.cache/checkpoint-nodejs/prisma-0fcc53eb","cache_duration":43200000,"remind_duration":172800000,"force":false,"timeout":5000,"unref":true,"child_path":"/home/REDACTED/work/yapgrid/yapgrid/site/node_modules/prisma/build/child","client_event_id":"","previous_client_event_id":"","check_if_update_available":true} (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

Fix Background Scheduler & Cron Jobs System

Problem Overview

The background scheduler exists but is not properly configured in PM2, causing jobs to not run automatically. Additionally, cron job frequencies need optimization and database settings need to be properly initialized.

Required Changes

1. Add Background Scheduler to PM2 Configuration

File: site/ecosystem.config.js

Currently only has yapgrid-nextjs. Need to add background-scheduler as a second app.

Change from:

module.exports = {
  apps: [
    {
      name: 'yapgrid-nextjs',
      script: 'node_modules/next/dist/bin/next',
      args: 'start -p 3002',
      cwd: '/home/ubuntu/apps/yapgrid/site',
      instances: 1,
      autorestart: true,
      watch: false,
      max_memory_restart: '1G',
      env: {
        NODE_ENV: 'production',
        PORT: 3002
      },
      error_file: '/home/ubuntu/apps/yapgrid/logs/web-error.log',
      out_file: '/home/ubuntu/apps/yapgrid/logs/web-out.log',
      time: true
    }
  ]
}

To:

module.exports = {
  apps: [
    {
      name: 'yapgrid-nextjs',
      script: 'node_modules/next/dist/bin/next',
      args: 'start -p 3002',
      cwd: '/home/ubuntu/apps/yapgrid/site',
      instances: 1,
      autorestart: true,
      watch: false,
      max_memory_restart: '1G',
      env: {
        NODE_ENV: 'production',
        PORT: 3002
      },
      error_file: '/home/ubuntu/apps/yapgrid/logs/web-error.log',
      out_file: '/home/ubuntu/apps/yapgrid/logs/web-out.log',
      time: true
    },
    {
      name: 'background-scheduler',
      script: 'background-scheduler.js',
      cwd: '/home/ubuntu/apps/yapgrid/site',
      instances: 1,
      autorestart: true,
      watch: false,
      max_memory_restart: '500M',
      env: {
        NODE_ENV: 'production',
        PORT: 3002
      },
      error_file: '/home/ubuntu/apps/yapgrid/logs/scheduler-error.log',
      out_file: '/home/ubuntu/apps/yapgrid/logs/scheduler-out.log',
      time: true
    }
  ]
}

2. Optimize Cron Job Frequency

File: site/background-scheduler.js

Line 9 - Change auto-processing frequency from every 30 seconds to every 2 minutes:

Change from:

cron.schedule('*/30 * * * * *', async () => {

To:

cron.schedule('*/2 * * * *', async () => {

Line 214 - Update console log to reflect new timing:

Change from:

console.log('  - Auto-processing: Every 30 seconds');

To:

console.log('  - Auto-processing: Every 2 minutes');

3. Add Database Settings Initialization Script

File: site/scripts/init-job-settings.js (CREATE NEW FILE)

Create a new script to initialize database settings for job control:

const { PrismaClient } = require('@prisma/client');
const prisma = new PrismaClient();

async function initializeJobSettings() {
  console.log('🔧 Initializing job settings...');
  
  try {
    // Auto-processing settings
    await prisma.setting.upsert({
      where: { key: 'auto_processing_enabled' },
      update: {},
      create: { key: 'auto_processing_enabled', value: 'true' }
    });
    
    await prisma.setting.upsert({
      where: { key: 'auto_processing_delay_seconds' },
      update: {},
      create: { key: 'auto_processing_delay_seconds', value: '10' }
    });
    
    await prisma.setting.upsert({
      where: { key: 'auto_processing_batch_size' },
      update: {},
      create: { key: 'auto_processing_batch_size', value: '10' }
    });
    
    // Auto-posting settings
    await prisma.setting.upsert({
      where: { key: 'auto_posting_enabled' },
      update: {},
      create: { key: 'auto_posting_enabled', value: 'true' }
    });
    
    await prisma.setting.upsert({
      where: { key: 'auto_posting_interval_minutes' },
      update: {},
      create: { key: 'auto_posting_interval_minutes', value: '30' }
    });
    
    await prisma.setting.upsert({
      where: { key: 'auto_posting_batch_size' },
      update: {},
      create: { key: 'auto_posting_batch_size', value: '5' }
    });
    
    // Auto-ingest settings
    await prisma.setting.upsert({
      where: { key: 'auto_ingest_enabled' },
      update: {},
      create: { key: 'auto_ingest_enabled', value: 'true' }
    });
    
    console.log('✅ Job settings initialized successfully!');
    
    // Display current settings
    const allSettings = await prisma.setting.findMany({
      where: {
        key: {
          in: [
            'auto_processing_enabled',
            'auto_processing_delay_seconds',
            'auto_processing_batch_size',
            'auto_posting_enabled',
            'auto_posting_interval_minutes',
            'auto_posting_batch_size',
            'auto_ingest_enabled'
          ]
        }
      }
    });
    
    console.log('\n📊 Current Settings:');
    allSettings.forEach(setting => {
      console.log(`  ${setting.key}: ${setting.value}`);
    });
    
  ...

</details>

*This pull request was created as a result of the following prompt from Copilot chat.*
> ## Fix Background Scheduler & Cron Jobs System
> 
> ### Problem Overview
> The background scheduler exists but is not properly configured in PM2, causing jobs to not run automatically. Additionally, cron job frequencies need optimization and database settings need to be properly initialized.
> 
> ---
> 
> ## Required Changes
> 
> ### 1. Add Background Scheduler to PM2 Configuration
> **File: `site/ecosystem.config.js`**
> 
> Currently only has `yapgrid-nextjs`. Need to add `background-scheduler` as a second app.
> 
> **Change from:**
> ```javascript
> module.exports = {
>   apps: [
>     {
>       name: 'yapgrid-nextjs',
>       script: 'node_modules/next/dist/bin/next',
>       args: 'start -p 3002',
>       cwd: '/home/ubuntu/apps/yapgrid/site',
>       instances: 1,
>       autorestart: true,
>       watch: false,
>       max_memory_restart: '1G',
>       env: {
>         NODE_ENV: 'production',
>         PORT: 3002
>       },
>       error_file: '/home/ubuntu/apps/yapgrid/logs/web-error.log',
>       out_file: '/home/ubuntu/apps/yapgrid/logs/web-out.log',
>       time: true
>     }
>   ]
> }
> ```
> 
> **To:**
> ```javascript
> module.exports = {
>   apps: [
>     {
>       name: 'yapgrid-nextjs',
>       script: 'node_modules/next/dist/bin/next',
>       args: 'start -p 3002',
>       cwd: '/home/ubuntu/apps/yapgrid/site',
>       instances: 1,
>       autorestart: true,
>       watch: false,
>       max_memory_restart: '1G',
>       env: {
>         NODE_ENV: 'production',
>         PORT: 3002
>       },
>       error_file: '/home/ubuntu/apps/yapgrid/logs/web-error.log',
>       out_file: '/home/ubuntu/apps/yapgrid/logs/web-out.log',
>       time: true
>     },
>     {
>       name: 'background-scheduler',
>       script: 'background-scheduler.js',
>       cwd: '/home/ubuntu/apps/yapgrid/site',
>       instances: 1,
>       autorestart: true,
>       watch: false,
>       max_memory_restart: '500M',
>       env: {
>         NODE_ENV: 'production',
>         PORT: 3002
>       },
>       error_file: '/home/ubuntu/apps/yapgrid/logs/scheduler-error.log',
>       out_file: '/home/ubuntu/apps/yapgrid/logs/scheduler-out.log',
>       time: true
>     }
>   ]
> }
> ```
> 
> ---
> 
> ### 2. Optimize Cron Job Frequency
> **File: `site/background-scheduler.js`**
> 
> **Line 9 - Change auto-processing frequency from every 30 seconds to every 2 minutes:**
> 
> Change from:
> ```javascript
> cron.schedule('*/30 * * * * *', async () => {
> ```
> 
> To:
> ```javascript
> cron.schedule('*/2 * * * *', async () => {
> ```
> 
> **Line 214 - Update console log to reflect new timing:**
> 
> Change from:
> ```javascript
> console.log('  - Auto-processing: Every 30 seconds');
> ```
> 
> To:
> ```javascript
> console.log('  - Auto-processing: Every 2 minutes');
> ```
> 
> ---
> 
> ### 3. Add Database Settings Initialization Script
> **File: `site/scripts/init-job-settings.js` (CREATE NEW FILE)**
> 
> Create a new script to initialize database settings for job control:
> 
> ```javascript
> const { PrismaClient } = require('@prisma/client');
> const prisma = new PrismaClient();
> 
> async function initializeJobSettings() {
>   console.log('🔧 Initializing job settings...');
>   
>   try {
>     // Auto-processing settings
>     await prisma.setting.upsert({
>       where: { key: 'auto_processing_enabled' },
>       update: {},
>       create: { key: 'auto_processing_enabled', value: 'true' }
>     });
>     
>     await prisma.setting.upsert({
>       where: { key: 'auto_processing_delay_seconds' },
>       update: {},
>       create: { key: 'auto_processing_delay_seconds', value: '10' }
>     });
>     
>     await prisma.setting.upsert({
>       where: { key: 'auto_processing_batch_size' },
>       update: {},
>       create: { key: 'auto_processing_batch_size', value: '10' }
>     });
>     
>     // Auto-posting settings
>     await prisma.setting.upsert({
>       where: { key: 'auto_posting_enabled' },
>       update: {},
>       create: { key: 'auto_posting_enabled', value: 'true' }
>     });
>     
>     await prisma.setting.upsert({
>       where: { key: 'auto_posting_interval_minutes' },
>       update: {},
>       create: { key: 'auto_posting_interval_minutes', value: '30' }
>     });
>     
>     await prisma.setting.upsert({
>       where: { key: 'auto_posting_batch_size' },
>       update: {},
>       create: { key: 'auto_posting_batch_size', value: '5' }
>     });
>     
>     // Auto-ingest settings
>     await prisma.setting.upsert({
>       where: { key: 'auto_ingest_enabled' },
>       update: {},
>       create: { key: 'auto_ingest_enabled', value: 'true' }
>     });
>     
>     console.log('✅ Job settings initialized successfully!');
>     
>     // Display current settings
>     const allSettings = await prisma.setting.findMany({
>       where: {
>         key: {
>           in: [
>             'auto_processing_enabled',
>             'auto_processing_delay_seconds',
>             'auto_processing_batch_size',
>             'auto_posting_enabled',
>             'auto_posting_interval_minutes',
>             'auto_posting_batch_size',
>             'auto_ingest_enabled'
>           ]
>         }
>       }
>     });
>     
>     console.log('\n📊 Current Settings:');
>     allSettings.forEach(setting => {
>       console.log(`  ${setting.key}: ${setting.value}`);
>     });
>     
>   } catch (error) {
>     console.error('❌ Error initializing settings:', error);
>   } finally {
>     await prisma.$disconnect();
>   }
> }
> 
> initializeJobSettings();
> ```
> 
> ---
> 
> ### 4. Add Deployment Instructions
> **File: `SCHEDULER_SETUP.md` (CREATE NEW FILE)**
> 
> Create deployment documentation:
> 
> ```markdown
> # Background Scheduler Setup
> 
> ## Overview
> The background scheduler runs three cron jobs:
> - **Auto-processing**: Processes NEW posts to READY status (every 2 minutes)
> - **Auto-publish**: Publishes READY posts to homepage (every 1 minute)
> - **Auto-fetch**: Fetches new posts from Reddit campaigns (every 30 minutes)
> 
> ## Initial Setup
> 
> ### 1. Initialize Database Settings
> Run this script once to create the required settings in the database:
> 
> \`\`\`bash
> cd /home/ubuntu/apps/yapgrid/site
> node scripts/init-job-settings.js
> \`\`\`
> 
> ### 2. Deploy with PM2
> The `ecosystem.config.js` now includes both the Next.js app and the background scheduler.
> 
> \`\`\`bash
> cd /home/ubuntu/apps/yapgrid/site
> pm2 delete all
> pm2 start ecosystem.config.js
> pm2 save
> \`\`\`
> 
> ### 3. Verify Jobs are Running
> Check PM2 status:
> 
> \`\`\`bash
> pm2 status
> \`\`\`
> 
> You should see:
> - yapgrid-nextjs (online)
> - background-scheduler (online)
> 
> Check logs:
> 
> \`\`\`bash
> pm2 logs background-scheduler
> \`\`\`
> 
> ## Job Control
> 
> All jobs are controlled via the admin panel at `https://yapgrid.com/admin/jobs`
> 
> ### Enable/Disable Jobs
> Jobs are enabled by default. To disable a job:
> 
> 1. Go to admin panel
> 2. Click "Stop" on the desired job
> 3. This updates the database setting (e.g., `auto_processing_enabled = 'false'`)
> 
> ### Manual Trigger
> You can manually trigger any job from the admin panel by clicking the "Start" button when the job is stopped, or by using the API endpoints.
> 
> ## Troubleshooting
> 
> ### Jobs Not Running
> Check if settings are enabled:
> 
> \`\`\`bash
> cd /home/ubuntu/apps/yapgrid/site
> node -e "const {PrismaClient}=require('@prisma/client');const p=new PrismaClient();p.setting.findMany({where:{key:{in:['auto_processing_enabled','auto_posting_enabled','auto_ingest_enabled']}}}).then(console.log).finally(()=>p.\$disconnect())"
> \`\`\`
> 
> ### Scheduler Not Starting
> Check logs:
> 
> \`\`\`bash
> pm2 logs background-scheduler --lines 100
> \`\`\`
> 
> Restart scheduler:
> 
> \`\`\`bash
> pm2 restart background-scheduler
> \`\`\`
> 
> ### High CPU Usage
> If auto-processing is causing high CPU usage, reduce the batch size:
> 
> \`\`\`sql
> UPDATE Setting SET value = '5' WHERE key = 'auto_processing_batch_size';
> \`\`\`
> 
> ## Configuration
> 
> Job settings are stored in the `Setting` table:
> 
> | Key | Default | Description |
> |-----|---------|-------------|
> | auto_processing_enabled | true | Enable/disable auto-processing |
> | auto_processing_delay_seconds | 10 | Delay between processing posts |
> | auto_processing_batch_size | 10 | Number of posts to process per run |
> | auto_posting_enabled | true | Enable/disable auto-publishing |
> | auto_posting_interval_minutes | 30 | Interval for publishing posts |
> | auto_posting_batch_size | 5 | Number of posts to publish per run |
> | auto_ingest_enabled | true | Enable/disable auto-fetching |
> ```
> 
> ---
> 
> ## Testing Checklist
> 
> After deployment:
> 
> 1. [ ] PM2 shows both apps running
> 2. [ ] Background scheduler logs show cron jobs executing
> 3. [ ] Admin panel shows correct job status
> 4. [ ] Can start/stop jobs from admin panel
> 5. [ ] NEW posts are processed to READY
> 6. [ ] READY posts are published to homepage
> 7. [ ] No high CPU usage
> 8. [ ] No memory leaks
> 
> ---
> 
> ## Expected Results
> 
> After these changes:
> 
> ✅ **Background scheduler runs automatically with PM2**
> - Starts on server boot
> - Auto-restarts on crash
> - Managed alongside Next.js app
> 
> ✅ **Optimized cron frequency**
> - Reduced from every 30s to every 2 minutes
> - Lower CPU usage
> - More stable performance
> 
> ✅ **Database settings initialized**
> - All jobs enabled by default
> - Can be controlled from admin panel
> - Settings persist across restarts
> 
> ✅ **Clear documentation**
> - Setup instructions
> - Troubleshooting guide
> - Configuration reference
> 
> ---
> 
> ## Deployment Steps for Server
> 
> After PR is merged, run these commands on the server:
> 
> \`\`\`bash
> # Pull latest changes
> cd ~/apps/yapgrid
> git pull origin main
> 
> # Initialize database settings
> cd site
> node scripts/init-job-settings.js
> 
> # Restart PM2 with new config
> pm2 delete all
> pm2 start ecosystem.config.js
> pm2 save
> 
> # Verify
> pm2 status
> pm2 logs background-scheduler --lines 20
> \`\`\`

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

…te init script and documentation Co-authored-by: enikqi <10384338+enikqi@users.noreply.github.com>

Initial plan

41f7fbd

Copilot AI assigned Copilot and enikqi Nov 4, 2025

Copilot started work on behalf of enikqi November 4, 2025 22:10 View session

Add background scheduler to PM2 config, optimize cron frequency, crea…

4621f33

…te init script and documentation Co-authored-by: enikqi <10384338+enikqi@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix background scheduler and cron jobs system~~ Configure background scheduler in PM2 and optimize cron frequencies Nov 4, 2025

Copilot AI requested a review from enikqi November 4, 2025 22:22

Copilot finished work on behalf of enikqi November 4, 2025 22:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure background scheduler in PM2 and optimize cron frequencies#7

Configure background scheduler in PM2 and optimize cron frequencies#7
Copilot wants to merge 2 commits into
mainfrom
copilot/fix-background-scheduler-cron-jobs

Copilot AI commented Nov 4, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Copilot AI commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Deployment

I tried to connect to the following addresses, but was blocked by firewall rules:

Fix Background Scheduler & Cron Jobs System

Problem Overview

Required Changes

1. Add Background Scheduler to PM2 Configuration

2. Optimize Cron Job Frequency

3. Add Database Settings Initialization Script

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Nov 4, 2025 •

edited

Loading