# Sync Queue

The Onboarded™ Sync Queue is a retry mechanism that ensures outbound sync operations complete successfully, even when Salesforce platform limits are reached or temporary failures occur.

## Sync Queue Overview

When outbound sync operations (sending data from Salesforce to Onboarded™) cannot be processed immediately due to queueable job limits or other constraints, the requests are stored in the `Onboarded_Sync_Queue__c` object. These queued requests are then processed asynchronously, with automatic retry on failure.

### Queue Activation Triggers

- Salesforce queueable job limits are reached (max 50 per transaction)
- Multiple sync operations triggered in rapid succession
- Flow-triggered syncs that exceed asynchronous limits
- Bulk operations that generate many outbound requests
- The Onboarded™ API rate limit is exceeded (HTTP 429). The platform accepts up to **3,000 records per minute**; requests beyond that are captured in the queue and retried automatically (see [API Rate Limits and Transient Failures](#api-rate-limits-and-transient-failures))


## Status Lifecycle

Each Sync Queue record progresses through a defined status lifecycle:

| Status | Description | Next Action |
|  --- | --- | --- |
| Pending | Record is waiting to be processed. This is the initial status when a sync request is queued. | System automatically picks up pending records for processing when queueable capacity is available. |
| Processing | Record is currently being processed by a queueable job. The system has claimed this record and is actively executing the sync operation. | If successful, the record is deleted. If failed, status changes to Failed. |
| Failed | The sync operation encountered an error. The `Error_Message__c` field contains details about what went wrong, and `Retry_Count__c` is incremented. | Automatically retried up to 3 times with backoff (15/30/60 minute intervals). After max retries, requires administrator review. |
| Do Not Retry | The sync operation failed with an unrecoverable error (such as HTTP 4xx client errors, invalid data, or decryption failures). Retrying will not fix the issue. | Requires administrator review. Must fix the underlying data issue and trigger a new sync operation. |


> **Success Behavior:** When a sync operation completes successfully, the corresponding Sync Queue record is automatically deleted. You should only see records in the queue that are pending, processing, or have failed.


## Sync Queue Fields

| Field | Description |
|  --- | --- |
| `Status__c` | Current processing status: Pending, Processing, or Failed |
| `Sync_Type__c` | Type of sync operation: Standard (record-based), Direct (payload-based), or PdfCapture (see [PDF/File Capture via the Sync Queue](#pdffile-capture-via-the-sync-queue)) |
| `Onboarded_Object_Type__c` | The Onboarded™ entity type (Employee, Employer, Client, etc.) |
| `SObject_Name__c` | The Salesforce object name for Standard sync operations |
| `Record_Ids__c` | Comma-separated list of Salesforce record IDs for Standard sync |
| `Operation__c` | The operation type: Create, Update, or Delete |
| `Payload__c` | Encrypted JSON payload for Direct sync operations (see Payload Encryption below) |
| `Error_Message__c` | Error details including message and stack trace when status is Failed |
| `Error_Logged__c` | Checkbox. Set once an Onboarded™ Error Log record has been written for this queue record (after retries are exhausted), so the same failure is not logged repeatedly on later passes. |
| `Retry_Count__c` | Number of times this record has been attempted. Useful for identifying persistent issues. |
| `Last_Attempt_Time__c` | Timestamp of the last processing attempt. Used by the automatic retry scheduler to implement backoff intervals. |


## Automatic Retry Mechanism

A scheduled job runs every **15 minutes** to process failed records automatically. The system uses a **backoff strategy** to avoid overwhelming the API during outages:

| Retry Attempt | Wait Time | Description |
|  --- | --- | --- |
| 1st retry | 15 minutes | First retry after initial failure |
| 2nd retry | 30 minutes | Second retry after first retry attempt |
| 3rd retry | 60 minutes | Final retry attempt |


> **Maximum Retries:** After 3 failed retry attempts, records remain in Failed status and require manual intervention. Records marked as Do Not Retry are never automatically retried.


### API Rate Limits and Transient Failures

The Onboarded™ platform enforces an ingestion limit of **3,000 records per minute**. When outbound sync traffic from Salesforce exceeds this threshold, the API responds with **HTTP 429 (Too Many Requests)**.

A 429 is treated as a **transient** failure — the data itself is valid; the platform is simply asking Salesforce to slow down. Rather than failing the record permanently, the application captures the affected outbound request in the Sync Queue and feeds it back through the [Automatic Retry Mechanism](#automatic-retry-mechanism) above. Other transient conditions — HTTP 5xx server errors and callout timeouts — are handled the same way.

Key behaviors:

- **No premature error logging.** A transient 429 does **not** create an Onboarded™ Error Log record on each attempt. An error is logged only if the record still fails after the retry attempts are exhausted; at that point the `Error_Logged__c` flag is set so the same record is not logged again on subsequent passes.
- **Self-throttling recovery.** The queue processor drains records in paced slices and chains itself between transactions, which naturally spreads retries back under the 3,000-per-minute ceiling instead of immediately re-hitting it.
- **Only failed records are retried.** When a batch partially succeeds, the successfully-sent records are dropped from the queue and only the failed record IDs are retried — minimizing repeat API calls and further rate-limit pressure.


> **Note:** A 429 is the one 4xx response that is **retried** rather than marked Do Not Retry. All other 4xx client errors (400/401/403/404) indicate a problem the retry cycle cannot fix and are handled as [Unrecoverable Errors](#unrecoverable-errors) below.


### Unrecoverable Errors

Certain errors are marked as **Do Not Retry** because retrying won't fix the underlying issue:

- **HTTP 4xx errors:** Bad Request (400), Unauthorized (401), Forbidden (403), Not Found (404). *(HTTP 429 rate-limit responses are the exception — they are transient and retried; see [API Rate Limits and Transient Failures](#api-rate-limits-and-transient-failures) above.)*
- **Data validation errors:** Invalid JSON payload, missing required fields
- **Encryption errors:** Decryption failures due to corrupted data


These records require administrator review to fix the underlying data issue before triggering a new sync.

> **Error Log created, but not reprocessable.** Every row that reaches Do Not Retry writes exactly one `Onboarded_Error_Log__c` entry (`Class_Name__c = 'OnboardedSyncQueueProcessorQueueable'`, `Impacted_Record__c` = the Sync Queue row's own Id) — the `Class_Name__c` value identifies it as originating from the queue's own repeated-failure handling, distinct from a live single-record outbound push. Because the **Reprocess** capability is scoped to that live outbound-push path (see [Error Handling](#error-handling)), this Error Log entry is for audit visibility only — no Reprocess button will appear for it. Resolve the underlying issue and retry from the Sync Queue record itself (see [Failed Record Retry](#failed-record-retry) above), not from the Error Log.


## Payload Encryption

When outbound sync operations store data in the Sync Queue, the `Payload__c` field is encrypted using AES-256 encryption to protect sensitive employee and employer information.

### Encryption Details

- **Algorithm:** AES-256 (FIPS 140-2 compliant via Salesforce Crypto class)
- **Key Management:** Encryption key is automatically generated during package installation and stored in the application's protected configuration — not visible or editable by any admin, including you
- **Randomization:** Each encryption operation uses industry-standard randomization to ensure identical data produces unique encrypted output
- **Automatic Encryption:** Payloads are encrypted automatically before insert via trigger


### Decrypted Payload Viewing

To view the original (decrypted) payload data for troubleshooting:

1. Assign the **Onboarded™ Encryption Access** permission set to your user
2. Navigate to the Sync Queue record
3. Click the **View Decrypted Payload** button
4. The decrypted JSON will be displayed in a modal dialog


> **Security Note:** Decrypted payloads may contain PII such as Social Security Numbers, dates of birth, and addresses. Only grant the Onboarded™ Encryption Access permission set to users with a legitimate need to troubleshoot sync failures.


## Failed Record Monitoring

Regularly monitor the Sync Queue for failed records to ensure data stays synchronized.

### Provided List Views

The package includes the following list views for monitoring:

- **All:** View all Sync Queue records
- **Failed (Pending Retry):** View records that will be automatically retried
- **Do Not Retry:** View records that require manual attention due to unrecoverable errors


### Custom List Views

You can also create custom list views:

1. Navigate to the **Onboarded™ Sync Queue** tab
2. Create a new list view with your desired filters
3. Add columns: Name, Status, Onboarded™ Object Type, Operation, Error Message, Retry Count, Last Attempt Time, Created Date
4. Sort by Created Date descending to see newest failures first


### Email Alerts

Create a workflow rule or Flow to send email notifications when Sync Queue records have Status = Failed:

1. Create a Flow that triggers on Sync Queue record update where Status changes to Failed
2. Send an email alert to your integration administrator
3. Include the `Error_Message__c` and record link in the notification


### Dashboard Component

Add a report chart to your admin dashboard showing:

- Count of Failed Sync Queue records
- Failed records grouped by Onboarded™ Object Type
- Trend of failures over time


## Failed Record Retry

> **Automatic Retries:** Most failed records will be retried automatically up to 3 times. You only need to intervene if records remain in Failed status after all retry attempts, or if records are marked as Do Not Retry.


1. **Identify the Failure Cause.** Review the `Error_Message__c` field to understand why the sync failed. Common causes include:
  - API authentication issues (token expired)
  - Network connectivity problems
  - Invalid data format in the payload
  - Onboarded™ API rate limiting
  - Missing required fields in the source record
2. **Resolve the Underlying Issue.** Address the root cause before retrying:
  - For auth issues: Re-authenticate in Onboarded™ Setup
  - For data issues: Correct the source record data
  - For API limits: Wait for the rate limit window to reset
3. **Change Status to Pending.** Once the issue is resolved, edit the failed Sync Queue record and change the `Status__c` field from **Failed** to **Pending**.
4. **Monitor for Success.** The system will automatically pick up the pending record and attempt to process it. If successful, the record will be deleted. If it fails again, review the updated error message.


> **Bulk Retry:** To retry multiple failed records at once, use Data Loader or the Salesforce Inspector browser extension to mass-update the `Status__c` field from Failed to Pending.


## Processing Behavior

### Record Processing

The Sync Queue processor operates as follows:

1. When a new Sync Queue record is inserted with Pending status, the trigger handler checks if queueable capacity is available
2. If capacity is available, a **bounded slice** of pending records is selected and their status is updated to Processing (the slice is deliberately small so the transaction stays well within Salesforce governor limits)
3. A queueable job is enqueued to process that slice
4. Each record is processed individually — failures don't affect other records in the batch — and record write-backs are applied in **bulk** (grouped per object) rather than one record at a time
5. Successful records are deleted; failed records are updated with error details
6. If more Pending records remain, the processor **chains itself** to continue with the next slice — so one drain works through the entire backlog across multiple transactions, keeping even a large burst safely within Salesforce governor limits


> **Recovering a stranded queue.** If a processor ever stops before finishing (or a backlog builds with no active job), the records simply wait as **Pending**/**Processing** — they are never lost. An admin can restart the drain at any time from **Onboarded™ Setup → Data Operation Settings → [Sync Queueable Operations](#sync-queueable-operations-on-demand-queue-recovery)**, which reclaims any orphaned rows and drains the whole queue in safe slices with a single click.


### Re-entrancy Prevention

The system prevents queue processing loops through these safeguards:

- A flag prevents new queueable jobs from being enqueued while the queue is already processing
- Records in Processing status are not picked up by subsequent trigger executions
- Inbound sync operations do not trigger outbound queue processing


### Standard vs Direct Sync Types

**Standard Sync:** Uses record IDs to query Salesforce data and transform it for the Onboarded™ API. The `Record_Ids__c` field contains the Salesforce IDs to sync.

**Direct Sync:** Uses a pre-built JSON payload (encrypted in `Payload__c`) that is sent directly to the Onboarded™ API. This is typically used by Flow invocable actions that construct the payload programmatically.