MuleSoft Batch Processing
Batch Processing handles large volumes of records efficiently. Instead of loading thousands of records into memory at once, MuleSoft's Batch Job splits the data into small chunks, processes each chunk in parallel, and reports the results. Use Batch Processing when you need to process CSV files with millions of rows, sync large datasets between systems, or run nightly data migration jobs.
When to Use Batch Processing
Use Batch Processing instead of a regular flow when:
- The dataset has more than a few hundred records
- Processing should continue even when individual records fail
- You need a success/failure report at the end
- Records can be processed independently of each other
Regular Flow vs Batch Job
Regular Flow (bad for large datasets): Load ALL 50,000 rows into memory Process row 1, row 2, row 3... row 50,000 If row 5,000 fails → entire flow fails All 50,000 rows must restart Batch Job (designed for large datasets): Load rows in chunks of 100 Process 10 chunks at a time (in parallel) If row 5,000 fails → only that row is marked failed Other 49,999 rows continue processing Final report: 49,999 succeeded, 1 failed
Batch Job Structure
A Batch Job has three phases that run in sequence:
- On Input Phase: Optional. Runs before record processing starts. Use it to fetch and prepare the full dataset.
- On Records Phase: The core phase. Runs once per record (or chunk). This is where you transform and send each record to the target system.
- On Complete Phase: Runs after all records finish. Use it to send a summary email, log statistics, or move the processed file to an archive folder.
Batch Job Phase Diagram
Trigger: [Scheduler: every day at 2 AM]
│
▼
┌─────────────────────────────────────────────┐
│ BATCH JOB: syncCustomersToCRM │
│ │
│ ┌─ ON INPUT ───────────────────────────┐ │
│ │ [Database: SELECT * FROM new_signups]│ │
│ │ → 12,500 customer records │ │
│ └──────────────────────────────────────┘ │
│ ↓ │
│ ┌─ ON RECORDS (runs per record) ────────┐ │
│ │ Chunk size: 200 records │ │
│ │ Max concurrency: 4 chunks at once │ │
│ │ │ │
│ │ Step 1: Transform to Salesforce format│ │
│ │ Step 2: Salesforce Connector Upsert │ │
│ └───────────────────────────────────────┘ │
│ ↓ │
│ ┌─ ON COMPLETE ──────────────────────────┐ │
│ │ [Logger: "Processed: #[batchJobResult │ │
│ │ .successfulRecords] records"]│ │
│ │ [Email: send summary report] │ │
│ └────────────────────────────────────────┘ │
└─────────────────────────────────────────────┘
Batch Steps
Inside the On Records phase, you add one or more Batch Steps. Each step runs on every record in sequence. Steps let you break complex record processing into organized stages.
Multiple Batch Steps
ON RECORDS PHASE:
Batch Step 1: "Validate Record"
[Choice: is email valid?]
YES → continue
NO → [Batch Record Variable: skip = true]
Batch Step 2: "Transform Record"
[Transform Message: JSON to Salesforce format]
[Set Variable: sfPayload = transformed data]
Batch Step 3: "Write to Salesforce"
[Salesforce: Upsert Contact]
(accepts filter: only process if skip != true)
Filtering Records in a Batch Step
Use the Accept Expression on a Batch Step to skip specific records. Records that do not pass the expression are counted as filtered, not failed.
Batch Step: "Write to Database" Accept Expression: vars.isValid == true Records where isValid = true → processed in this step Records where isValid = false → skipped, counted as "filtered"
Batch Job Results Object
In the On Complete phase, MuleSoft provides a batchJobResult variable with processing statistics.
batchJobResult Fields
batchJobResult.successfulRecords // number of records that succeeded batchJobResult.failedRecords // number of records that failed batchJobResult.totalRecords // total input records batchJobResult.elapsedTimeInMillis // total processing time batchJobResult.inputPhaseException // any error in the input phase ON COMPLETE Example: [Logger: message= "Batch complete. Total: #[batchJobResult.totalRecords], Success: #[batchJobResult.successfulRecords], Failed: #[batchJobResult.failedRecords], Time: #[batchJobResult.elapsedTimeInMillis / 1000]s" ]
Handling Failed Records
When a record fails in a Batch Step, MuleSoft moves it to the failed records bucket. You can add error handling inside a Batch Step to handle failures gracefully per record — log the bad record, write it to a dead-letter queue, or save it to a rejection table for manual review.
Per-Record Error Handling
Batch Step: "Write to Target System"
[Salesforce Upsert]
Error Handler:
On Error Continue: ANY
[Logger: "Record failed: #[payload.id] - #[error.description]"]
[Database: INSERT INTO failed_records (id, error, timestamp)
VALUES (#[payload.id], #[error.description], #[now()])]
(marks record as failed, continues to next record)
Configuring Chunk Size and Concurrency
Two settings control Batch Job performance:
- Chunk Size: How many records MuleSoft groups into one chunk for processing. Default is 100. Increase for simple transformations, decrease if each record uses lots of memory.
- Max Concurrency: How many chunks process in parallel. Default is 4. Increase if your target system supports high concurrency. Decrease if the target has rate limits.
Chunk Size and Concurrency Diagram
1,000 records, chunk size = 100 → 10 chunks Max Concurrency = 4: Time 0-2s: Chunks 1, 2, 3, 4 process simultaneously Time 2-4s: Chunks 5, 6, 7, 8 process simultaneously Time 4-6s: Chunks 9, 10 process Total: ~6 seconds Max Concurrency = 1 (sequential): Time 0-2s: Chunk 1 Time 2-4s: Chunk 2 ... Total: ~20 seconds
Typical Batch Job Use Cases
- Nightly database sync: Read all updated records from System A and upsert them into System B.
- CSV file import: Read a CSV with thousands of product updates and apply them to the database.
- Mass email campaign: Process a list of subscribers and send personalized emails in parallel.
- Data migration: Move data from a legacy system to a new platform in batches during a migration window.