### Sync Queue


```markdown
# Sync Queue

The Onboarded™ Sync Queue is a retry mechanism that ensures outbound sync operations complete successfully, even when Salesforce platform limits are reached or temporary failures occur.

## Sync Queue Overview

When outbound sync operations (sending data from Salesforce to Onboarded™) cannot be processed immediately due to queueable job limits or other constraints, the requests are stored in the `Onboarded_Sync_Queue__c` object. These queued requests are then processed asynchronously, with automatic retry on failure.

### Queue Activation Triggers

- Salesforce queueable job limits are reached (max 50 per transaction)
- Multiple sync operations triggered in rapid succession
- Flow-triggered syncs that exceed asynchronous limits
- Bulk operations that generate many outbound requests

## Status Lifecycle

Each Sync Queue record progresses through a defined status lifecycle:

| Status | Description | Next Action |
|---|---|---|
| Pending | Record is waiting to be processed. This is the initial status when a sync request is queued. | System automatically picks up pending records for processing when queueable capacity is available. |
| Processing | Record is currently being processed by a queueable job. The system has claimed this record and is actively executing the sync operation. | If successful, the record is deleted. If failed, status changes to Failed. |
| Failed | The sync operation encountered an error. The `Error_Message__c` field contains details about what went wrong, and `Retry_Count__c` is incremented. | Automatically retried up to 3 times with backoff (15/30/60 minute intervals). After max retries, requires administrator review. |
| Do Not Retry | The sync operation failed with an unrecoverable error (such as HTTP 4xx client errors, invalid data, or decryption failures). Retrying will not fix the issue. | Requires administrator review. Must fix the underlying data issue and trigger a new sync operation. |

> **Success Behavior:** When a sync operation completes successfully, the corresponding Sync Queue record is automatically deleted. You should only see records in the queue that are pending, processing, or have failed.

## Sync Queue Fields

| Field | Description |
|---|---|
| `Status__c` | Current processing status: Pending, Processing, or Failed |
| `Sync_Type__c` | Type of sync operation: Standard (record-based) or Direct (payload-based) |
| `Onboarded_Object_Type__c` | The Onboarded™ entity type (Employee, Employer, Client, etc.) |
| `SObject_Name__c` | The Salesforce object name for Standard sync operations |
| `Record_Ids__c` | Comma-separated list of Salesforce record IDs for Standard sync |
| `Operation__c` | The operation type: Create, Update, or Delete |
| `Payload__c` | Encrypted JSON payload for Direct sync operations (see Payload Encryption below) |
| `Error_Message__c` | Error details including message and stack trace when status is Failed |
| `Retry_Count__c` | Number of times this record has been attempted. Useful for identifying persistent issues. |
| `Last_Attempt_Time__c` | Timestamp of the last processing attempt. Used by the automatic retry scheduler to implement backoff intervals. |

## Automatic Retry Mechanism

A scheduled job runs every **15 minutes** to process failed records automatically. The system uses a **backoff strategy** to avoid overwhelming the API during outages:

| Retry Attempt | Wait Time | Description |
|---|---|---|
| 1st retry | 15 minutes | First retry after initial failure |
| 2nd retry | 30 minutes | Second retry after first retry attempt |
| 3rd retry | 60 minutes | Final retry attempt |

> **Maximum Retries:** After 3 failed retry attempts, records remain in Failed status and require manual intervention. Records marked as Do Not Retry are never automatically retried.

### Unrecoverable Errors

Certain errors are marked as **Do Not Retry** because retrying won't fix the underlying issue:

- **HTTP 4xx errors:** Bad Request (400), Unauthorized (401), Forbidden (403), Not Found (404)
- **Data validation errors:** Invalid JSON payload, missing required fields
- **Encryption errors:** Decryption failures due to corrupted data

These records require administrator review to fix the underlying data issue before triggering a new sync.

## Payload Encryption

When outbound sync operations store data in the Sync Queue, the `Payload__c` field is encrypted using AES-256 encryption to protect sensitive employee and employer information.

### Encryption Details

- **Algorithm:** AES-256 (FIPS 140-2 compliant via Salesforce Crypto class)
- **Key Management:** Encryption key is automatically generated during package installation and stored securely in `OnboardedSettings__c` custom setting
- **Randomization:** Each encryption operation uses industry-standard randomization to ensure identical data produces unique encrypted output
- **Automatic Encryption:** Payloads are encrypted automatically before insert via trigger

### Decrypted Payload Viewing

To view the original (decrypted) payload data for troubleshooting:

1. Assign the **Onboarded™ Encryption Access** permission set to your user
2. Navigate to the Sync Queue record
3. Click the **View Decrypted Payload** button
4. The decrypted JSON will be displayed in a modal dialog

> **Security Note:** Decrypted payloads may contain PII such as Social Security Numbers, dates of birth, and addresses. Only grant the Onboarded™ Encryption Access permission set to users with a legitimate need to troubleshoot sync failures.

## Failed Record Monitoring

Regularly monitor the Sync Queue for failed records to ensure data stays synchronized.

### Provided List Views

The package includes the following list views for monitoring:

- **All:** View all Sync Queue records
- **Failed (Pending Retry):** View records that will be automatically retried
- **Do Not Retry:** View records that require manual attention due to unrecoverable errors

### Custom List Views

You can also create custom list views:

1. Navigate to the **Onboarded™ Sync Queue** tab
2. Create a new list view with your desired filters
3. Add columns: Name, Status, Onboarded™ Object Type, Operation, Error Message, Retry Count, Last Attempt Time, Created Date
4. Sort by Created Date descending to see newest failures first

### Email Alerts

Create a workflow rule or Flow to send email notifications when Sync Queue records have Status = Failed:

1. Create a Flow that triggers on Sync Queue record update where Status changes to Failed
2. Send an email alert to your integration administrator
3. Include the `Error_Message__c` and record link in the notification

### Dashboard Component

Add a report chart to your admin dashboard showing:

- Count of Failed Sync Queue records
- Failed records grouped by Onboarded™ Object Type
- Trend of failures over time

## Failed Record Retry

> **Automatic Retries:** Most failed records will be retried automatically up to 3 times. You only need to intervene if records remain in Failed status after all retry attempts, or if records are marked as Do Not Retry.

1. **Identify the Failure Cause.** Review the `Error_Message__c` field to understand why the sync failed. Common causes include:
    - API authentication issues (token expired)
    - Network connectivity problems
    - Invalid data format in the payload
    - Onboarded™ API rate limiting
    - Missing required fields in the source record
2. **Resolve the Underlying Issue.** Address the root cause before retrying:
    - For auth issues: Re-authenticate in Onboarded™ Setup
    - For data issues: Correct the source record data
    - For API limits: Wait for the rate limit window to reset
3. **Change Status to Pending.** Once the issue is resolved, edit the failed Sync Queue record and change the `Status__c` field from **Failed** to **Pending**.
4. **Monitor for Success.** The system will automatically pick up the pending record and attempt to process it. If successful, the record will be deleted. If it fails again, review the updated error message.

> **Bulk Retry:** To retry multiple failed records at once, use Data Loader or the Salesforce Inspector browser extension to mass-update the `Status__c` field from Failed to Pending.

## Processing Behavior

### Record Processing

The Sync Queue processor operates as follows:

1. When a new Sync Queue record is inserted with Pending status, the trigger handler checks if queueable capacity is available
2. If capacity is available, up to 99 pending records are selected and their status is updated to Processing
3. A queueable job is enqueued to process these records
4. Each record is processed individually — failures don't affect other records in the batch
5. Successful records are deleted; failed records are updated with error details

### Re-entrancy Prevention

The system prevents queue processing loops through these safeguards:

- A flag prevents new queueable jobs from being enqueued while the queue is already processing
- Records in Processing status are not picked up by subsequent trigger executions
- Inbound sync operations do not trigger outbound queue processing

### Standard vs Direct Sync Types

**Standard Sync:** Uses record IDs to query Salesforce data and transform it for the Onboarded™ API. The `Record_Ids__c` field contains the Salesforce IDs to sync.

**Direct Sync:** Uses a pre-built JSON payload (encrypted in `Payload__c`) that is sent directly to the Onboarded™ API. This is typically used by Flow invocable actions that construct the payload programmatically.
```