MuleSoft Polling and Watermark
Polling lets your Mule application check a data source at regular intervals and process new records automatically. The Watermark feature ensures you only process records that are genuinely new since the last poll — not the same records over and over. Together, polling and watermark power common integration patterns like syncing new orders every five minutes or importing updated customer records nightly.
What Is Polling
Polling is the act of checking a source system repeatedly at a set interval. Your Mule application wakes up every N minutes, queries the source for new data, processes whatever it finds, and goes back to sleep. On the next wakeup, it checks again.
Polling Timeline
Time: 00:00 05:00 10:00 15:00 20:00
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
App: [poll] [poll] [poll] [poll] [poll]
finds finds finds finds finds
5 new 0 new 3 new 8 new 0 new
orders orders orders orders orders
Between polls: app is idle (no resources consumed)
The Scheduler Component
MuleSoft's Scheduler component triggers a flow at a fixed interval or on a cron expression. It is the standard way to build polling integrations.
Scheduler Configuration Options
Fixed Frequency (simple): Frequency: 5 Time Unit: MINUTES Start Delay: 0 SECONDS → runs every 5 minutes, immediately on startup Cron Expression (advanced): Expression: 0 0 2 * * ? → runs at 2:00 AM every day Expression: 0 0/15 9-17 * * MON-FRI → runs every 15 minutes, 9 AM to 5 PM, Monday to Friday Cron format: seconds minutes hours day-of-month month day-of-week
The Problem Without Watermark
Without a watermark, every poll reads all records from the source — even those already processed. This causes duplicates and wastes resources.
Without Watermark (Problem)
Database table: orders id=1, created=2024-01-15 09:00, status=new id=2, created=2024-01-15 09:03, status=new id=3, created=2024-01-15 09:07, status=new Poll at 09:05: Query: SELECT * FROM orders WHERE status='new' Gets: records 1, 2 → processes both → sends to CRM ✓ Poll at 09:10: Query: SELECT * FROM orders WHERE status='new' Gets: records 1, 2, 3 → sends 1 and 2 AGAIN (duplicates!) ✗ (status is still 'new' because we did not update it)
What Is a Watermark
A watermark is a saved marker that tracks the last successfully processed position in a dataset. On each poll, the query uses the watermark to request only records newer than the last known position. After processing, the watermark updates to the latest record's timestamp or ID.
With Watermark (Solution)
Initial watermark value: 2024-01-15 00:00:00 Poll at 09:05: Query: SELECT * FROM orders WHERE created > '2024-01-15 00:00:00' Gets: records 1 (09:00), 2 (09:03) → processes both ✓ Update watermark to: 2024-01-15 09:03:00 Poll at 09:10: Query: SELECT * FROM orders WHERE created > '2024-01-15 09:03:00' Gets: record 3 (09:07) only → processes once ✓ (no duplicates!) Update watermark to: 2024-01-15 09:07:00 Poll at 09:15: Query: SELECT * FROM orders WHERE created > '2024-01-15 09:07:00' Gets: nothing new → skips processing ✓
Implementing Watermark in MuleSoft
MuleSoft provides an Object Store to persist the watermark value between polls. The Object Store keeps the value even if the application restarts.
Watermark Flow Implementation
[Scheduler: every 5 minutes]
│
▼
[Object Store: Retrieve]
Key: "lastPollTimestamp"
Target Variable: lastPollTime
(if key not found, default = "1970-01-01T00:00:00")
│
▼
[Database: Select]
Query: SELECT * FROM orders
WHERE created_at > :lastPollTime
ORDER BY created_at ASC
Parameters: { lastPollTime: #[vars.lastPollTime] }
│
▼
[Choice: did we get any records?]
│
├── YES (payload is not empty):
│ [For Each: process each order]
│ [Salesforce: Upsert order record]
│ [End For Each]
│ [Set Variable: newWatermark = max created_at from results]
│ [Object Store: Store]
│ Key: "lastPollTimestamp"
│ Value: #[vars.newWatermark]
│
└── NO (no new records):
[Logger: "No new orders since " ++ vars.lastPollTime]
(watermark stays unchanged)
Watermark with Salesforce (Built-In Support)
The Salesforce Connector has built-in watermark support through the Query operation with watermarking enabled. You specify the field to use as the watermark (typically LastModifiedDate), and MuleSoft handles storing and updating the value automatically.
[Scheduler: every 10 minutes]
│
▼
[Salesforce: Query]
Query: SELECT Id, Name, Email FROM Contact
WHERE LastModifiedDate > :watermark
Watermark: enabled
Watermark Field: LastModifiedDate
Object Store Key: "sfContactWatermark"
│
▼
[For Each: process updated contacts]
[Database: UPSERT into local contacts table]
Watermark with File Polling
When polling a folder for new files, the File Connector supports watermarking by file creation or modification time. Only files newer than the last poll get picked up.
[File Listener: poll /input/orders/ every 1 minute] Watermark Mode: CREATION_TIME (or MODIFIED_TIME) Folder contents: orders_2024-01-15_08-00.csv (created 08:00) orders_2024-01-15_08-05.csv (created 08:05) orders_2024-01-15_08-10.csv (created 08:10) Poll at 08:06: Gets: orders_2024-01-15_08-00.csv, orders_2024-01-15_08-05.csv Watermark updates to 08:05 Poll at 08:11: Gets: orders_2024-01-15_08-10.csv (only the new one)
Handling Gaps and Overlaps
To avoid missing records created exactly at the watermark timestamp, subtract a small buffer from the watermark when querying. This overlapping window catches records that may have been committed to the database a few milliseconds after the recorded timestamp.
Instead of: WHERE created_at > '#[vars.lastPollTime]' Use a 5-second buffer: WHERE created_at > '#[vars.lastPollTime - |PT5S|]' Then deduplicate by ID using an Object Store or a unique constraint at the target system to handle the small overlap.
Best Practices for Polling
- Always use a watermark. Never poll without tracking what you already processed.
- Index the watermark column in the database. Without an index, every poll does a full table scan as the table grows.
- Set a reasonable poll interval. Polling every second wastes resources. Start with 5 minutes and adjust based on business needs.
- Use Object Store v2 on CloudHub for persistent watermarks that survive application restarts and redeployments.
- Log the watermark value at the start and end of each poll. This makes debugging much easier when records go missing.