Schedule Agent Troubleshooting

Overview

The Schedule Agent feature allows users to schedule recurring AI workflow executions (GenSearch, Think Longer, Deep Research) and receive email reports on a defined cadence. The system consists of two components: the Scheduler, a cron job that runs every 15 minutes to enqueue due workflows; and the Processor, an always-on service with lane-based workers that claim and execute queued workflows, then send email notifications.

This document covers common failure scenarios for Schedule Agents and provides troubleshooting steps for resolution.

---

Failure Scenarios

1. Users Are Not Receiving Scheduled Reports

Triage:

Users report that their scheduled agent reports have stopped arriving. This can be caused by the processor pods being down, the scheduler not running, MongoDB connectivity issues, or email delivery failures.

Troubleshooting:

Verify that the scheduler and processor pods are running:

kubectl get pods -n applications | grep -E "ai-workflows-scheduler|ai-workflows-processor"

Check scheduler logs to confirm it is running every 15 minutes. Look for the startup log line and any errors:

kubectl logs -n applications deployment/ai-workflows-scheduler --tail=200 | grep -E "ScheduleCron|Error"

Expected healthy log entries:

[ScheduleCron] Starting schedule processing at <timestamp>
[ScheduleCron] Found N schedules due for execution

If you see Error processing schedules or no ScheduleCron log for more than 30 minutes, restart the scheduler:

kubectl rollout restart deployment/ai-workflows-scheduler -n applications

Check processor logs for workflow failures and email delivery errors:

kubectl logs -n applications deployment/ai-workflows-processor --tail=300 | grep -E "Failing workflow|Error sending email|Failed to build notification"

If the processor is down or crashing, check MongoDB connectivity by inspecting the pod logs for connection errors, then restart the processor:

kubectl rollout restart deployment/ai-workflows-processor -n applications

After restart, confirm the processor started successfully by checking for the startup capacity log:

kubectl logs -n applications deployment/ai-workflows-processor --tail=50 | grep "Processor max workers"

Expected output:

Processor max workers=16 (standard=7 tl=5 dr=4) defaults=(standard=7 tl=5 dr=4)

If pods are healthy but users still report missing emails, check whether email delivery is failing silently. Search processor logs for the affected user's ID:

kubectl logs -n applications deployment/ai-workflows-processor --tail=1000 | grep "<userId>"

Look for Email notification sent successfully or Error sending email / Failed to build notification payload lines for that user.

---

2. Scheduled Agent Reports Are Delayed

Triage:

Users receive their scheduled reports, but they arrive significantly later than the configured schedule time. This is typically caused by a large pending workflow queue, slow AI service responses, or worker slots being consumed by stuck workflows.

Troubleshooting:

Verify the scheduler is running on time by checking its logs:

kubectl logs -n applications deployment/ai-workflows-scheduler --tail=100 | grep "ScheduleCron"

The Starting schedule processing entry should appear every ~15 minutes. If it is taking longer than 60 seconds per run, MongoDB performance on the schedules collection may be degraded.

Check how many workflows are waiting in the queue by connecting to MongoDB and running:

// Total pending workflows
db.runnableScheduledAiWorkflows.countDocuments({status: 'PENDING'})

// Breakdown by execution mode
db.runnableScheduledAiWorkflows.aggregate([
  {$match: {status: 'PENDING'}},
  {$group: {_id: {mode: '$executionMode'}, count: {$sum: 1}}},
])

// Oldest pending workflow (how long has it been waiting?)
db.runnableScheduledAiWorkflows.find({status: 'PENDING'}).sort({createdAt: 1}).limit(1)

If the queue is large and growing, the processor does not have enough capacity. Increase per-pod worker concurrency by updating the following environment variables on the processor deployment.

To apply immediately (takes effect after the pod restarts):

kubectl set env deployment/ai-workflows-processor \
  STANDARD_MODE_CONCURRENCY_MAX_COUNT=<N> \
  THINK_LONGER_MODE_CONCURRENCY_MAX_COUNT=<N> \
  DEEP_RESEARCH_MODE_CONCURRENCY_MAX_COUNT=<N> \
  -n applications

The defaults are STANDARD=7, THINK_LONGER=5, DEEP_RESEARCH=4 (total 16 workers per pod).

Persistence

kubectl set env is overwritten on the next Helm upgrade. To make the change permanent, update the corresponding values in your Helm values.yaml for the ai-workflows-processor chart:

env:
  STANDARD_MODE_CONCURRENCY_MAX_COUNT: '<N>'
  THINK_LONGER_MODE_CONCURRENCY_MAX_COUNT: '<N>'
  DEEP_RESEARCH_MODE_CONCURRENCY_MAX_COUNT: '<N>'

Then apply with:

helm upgrade ai-workflows-processor <chart> -n applications -f values.yaml

To scale the processor to add more pods instead:

kubectl scale deployment/ai-workflows-processor -n applications --replicas=<N>

Total fleet capacity = (STANDARD + TL + DR) × number_of_pods.

Check whether worker slots are being held by stuck workflows (RUNNING for more than 20 minutes):

// Count stuck workflows
db.runnableScheduledAiWorkflows
  .find({
    status: 'RUNNING',
    startedAt: {$lt: new Date(Date.now() - 20 * 60 * 1000)},
  })
  .countDocuments()

Stuck workflows are reset to PENDING automatically by the next scheduler run (every 15 minutes). If many workflows are getting stuck consistently, check whether the AI service (GenSearch / Deep Research) is responding slowly or not completing within the 30-minute subscription timeout.

Check processor logs for subscription timeout messages:

kubectl logs -n applications deployment/ai-workflows-processor --tail=500 | grep -E "Subscription timed out|Subscription error|stream ended without final"

If the AI service is the bottleneck, refer to the GenSearch & Deep Research Troubleshooting runbook.

---

3. Individual Workflow Permanently Failed

Triage:

A specific user's scheduled agent stopped running and a record shows it has permanently failed. A workflow is permanently failed after 3 consecutive retry failures. The audit trail is preserved in the AiWorkflowExecutionHistory collection; the runnable record is deleted on final failure.

Troubleshooting:

Check recent failures in MongoDB for the affected user:

// Recent failures for a specific user
db.aiWorkflowExecutionHistory
  .find({userId: '<userId>', status: 'FAILED'})
  .sort({createdAt: -1})
  .limit(10)

// All failures in the last 2 hours
db.aiWorkflowExecutionHistory
  .find({
    status: 'FAILED',
    createdAt: {$gt: new Date(Date.now() - 2 * 60 * 60 * 1000)},
  })
  .sort({createdAt: -1})
  .limit(20)

Check the failure reason recorded in the history document. Common root causes:

• Subscription error / timeout — the AI service failed to return a result within 30 minutes across all 3 retries. Check AI service health using the GenSearch & Deep Research Troubleshooting runbook.
• Workflow not found — the workflow document was deleted while it was queued. Verify whether the user deleted the workflow.
• HTML export failure — the AI run succeeded but the email content could not be fetched; the execution is still recorded as failed. Check processor logs for Failed to get conversation message email html.

Check processor logs for the specific workflow ID:

kubectl logs -n applications deployment/ai-workflows-processor --tail=1000 | grep "<workflowId>"

Once the underlying issue is resolved (e.g. AI service restored), the user can manually re-run the schedule from the UI to trigger a new execution. The retry count resets on the next scheduled execution.

---

4. User Cannot Create a New Schedule

Triage:

A user receives an error when attempting to create a new scheduled agent. Each user has a maximum of 10 active schedules. Attempting to create an 11th schedule is rejected.

Troubleshooting:

Confirm how many active schedules the user currently has:

db.aiWorkflowSchedules.countDocuments({userId: '<userId>'})

If the count is 10, the user must delete an existing schedule before creating a new one. There is no admin override for this limit — it is enforced at the application layer.

If the count is below 10 and creation still fails, check for application errors in the main app logs:

kubectl logs -n applications deployment/graphql-ai-workflows --tail=200 | grep -E "Error|userId"

---

Validation Steps

After applying any resolution:

1. Verify that all affected pods are in Running state with no recent restarts:

   kubectl get pods -n applications | grep -E "ai-workflows"

2. Confirm the scheduler runs successfully on the next 15-minute interval by checking for [ScheduleCron] Starting schedule processing in the scheduler logs.

3. Confirm the processor is picking up workflows by checking for AskQuestion start event received in the processor logs.

4. For a user-specific issue, verify the affected user's schedule is active and that the next nextScheduleDateTime is set to a future time in MongoDB:

   db.aiWorkflowSchedules.find({userId: '<userId>'})

5. Wait for the next scheduled execution and confirm the user receives the email report.

6. If a service was restarted or scaled, monitor logs for healthy operation and confirm the pending queue is draining before closing the incident.