Manage large messages in workflows by using chunking in Azure Logic Apps

Handling large messages in Azure Logic Apps using chunking is a great way to manage and process large data efficiently. Chunking breaks down large messages into smaller, manageable pieces, which are then processed individually.

Key Concepts

Chunking:
- Splits large messages into smaller, manageable chunks.
- Each chunk is processed individually.
Size Limitations:
- HTTP Action: 100 MB (for sending and receiving messages).
- Logic App Payload: 100 MB (for the entire workflow).
Chunking Workflow:
- Split the large message into chunks.
- Process each chunk individually.

Step-by-Step Guide

Step 1: Create a Logic App

Go to the Azure portal.
Click on Create a resource > Integration > Logic App.
Provide a name, subscription, resource group, and location for your Logic App.
Click Review + create and then Create.

Step 2: Add a Trigger

Open your Logic App in the designer.
Add a trigger to start the workflow (e.g., HTTP Request or Recurrence).

Step 3: Split the Large Message into Chunks

Add a Compose action to define the large message (e.g., a JSON array or XML data).
Add a For Each loop to iterate through the chunks:
- Use the Split function to divide the large message into smaller chunks.
- Example (for JSON array):
```
@chunk(outputs('Compose'),100)
```
  - This splits the array into chunks of 100 items each.

Step 4: Process Each Chunk

Inside the For Each loop, add actions to process each chunk:
- Use HTTP actions to send the chunk to an API.
- Use Data Operations to transform the chunk.

Here's how Logic App should look:

Example Workflow

Scenario

A large JSON array needs to be processed in chunks of 100 items each.
Each chunk is sent to an API for processing.

Step-by-Step Workflow

HTTP Request Trigger:
- Method: POST
- URI: (Generated URL)

Compose Action:

Input: The large JSON array. For example:

[
  { "id": 1, "name": "John Doe" },
  { "id": 2, "name": "Jane Smith" },
  ...
  { "id": 1000, "name": "Alice Johnson" }
]

For Each Loop:
- Iterate through the chunks:
```
@chunk(outputs('Compose'),100)
```
Process Each Chunk:
- Add an HTTP action to send the chunk to an API:
  - Method: POST
  - URI: https://api.example.com/process
  - Body: @items('For_each')

Testing the Workflow

Save and run the Logic App.
Send a large JSON array to the HTTP Request trigger.
The workflow will:
- Split the array into chunks of 100 items each.
- Process each chunk using the API.

Key Points

Chunking: Splits large messages into smaller chunks for processing.
For Each Loop: Iterates through the chunks.
HTTP Action: Sends each chunk to an API.

You can download the logic app template from the estudy247 GitHub repository – la-test-chunking-17

Previous lessons

Back to courses

Next lessons