Manage large messages in workflows by using chunking in Azure Logic Apps

Azure Logic Apps Manage large messages in workflows by using chunking in Azure Logic Apps

Handling large messages in Azure Logic Apps using chunking is a great way to manage and process large data efficiently. Chunking breaks down large messages into smaller, manageable pieces, which are then processed individually.

Key Concepts

Chunking:
- Splits large messages into smaller, manageable chunks.
- Each chunk is processed individually.
Size Limitations:
- HTTP Action: 100 MB (for sending and receiving messages).
- Logic App Payload: 100 MB (for the entire workflow).
Chunking Workflow:
- Split the large message into chunks.
- Process each chunk individually.

Step-by-Step Guide

Step 1: Create a Logic App

Go to the Azure portal.
Click on Create a resource > Integration > Logic App.
Provide a name, subscription, resource group, and location for your Logic App.
Click Review + create and then Create.

Step 2: Add a Trigger

Open your Logic App in the designer.
Add a trigger to start the workflow (e.g., HTTP Request or Recurrence).

Step 3: Split the Large Message into Chunks

Add a Compose action to define the large message (e.g., a JSON array or XML data).
Add a For Each loop to iterate through the chunks:
- Use the Split function to divide the large message into smaller chunks.
- Example (for JSON array):
```
@chunk(outputs('Compose'),100)
```
  - This splits the array into chunks of 100 items each.

Step 4: Process Each Chunk

Inside the For Each loop, add actions to process each chunk:
- Use HTTP actions to send the chunk to an API.
- Use Data Operations to transform the chunk.

Here's how Logic App should look:

Example Workflow

Scenario

A large JSON array needs to be processed in chunks of 100 items each.
Each chunk is sent to an API for processing.

Step-by-Step Workflow

HTTP Request Trigger:
- Method: POST
- URI: (Generated URL)

Compose Action:

Input: The large JSON array. For example:

[
  { "id": 1, "name": "John Doe" },
  { "id": 2, "name": "Jane Smith" },
  ...
  { "id": 1000, "name": "Alice Johnson" }
]

For Each Loop:
- Iterate through the chunks:
```
@chunk(outputs('Compose'),100)
```
Process Each Chunk:
- Add an HTTP action to send the chunk to an API:
  - Method: POST
  - URI: https://api.example.com/process
  - Body: @items('For_each')

Testing the Workflow

Save and run the Logic App.
Send a large JSON array to the HTTP Request trigger.
The workflow will:
- Split the array into chunks of 100 items each.
- Process each chunk using the API.

Key Points

Chunking: Splits large messages into smaller chunks for processing.
For Each Loop: Iterates through the chunks.
HTTP Action: Sends each chunk to an API.

You can download the logic app template from the estudy247 GitHub repository – la-test-chunking-17

Previous Lesson

Back to Course

Next Lesson

Azure Logic Apps