Elasticsearch Bulk API

Adding one document per request works for a handful of records. A catalog with two million products needs a faster route. Each single request pays a fixed cost for the network trip, the parsing, and the response. The Bulk API packs many operations into one request and pays that cost once.

One request per document (1,000 documents):

  Client ─→ ES   Client ─→ ES   Client ─→ ES   ...   1,000 round trips

Bulk request (1,000 documents):

  Client ═══════════════════════════════════→ ES    1 round trip

The Newline-Delimited Format

Bulk requests use NDJSON, which means one JSON object per line. Action lines describe what to do. Data lines carry the document. Every line ends with a newline character, including the last one. A missing final newline causes an error.

POST /_bulk
{ "index": { "_index": "products", "_id": "1" } }
{ "name": "Steel Bottle", "price": 18.5, "stock": 120 }
{ "index": { "_index": "products", "_id": "2" } }
{ "name": "Canvas Backpack", "price": 42.0, "stock": 35 }
{ "delete": { "_index": "products", "_id": "9" } }

Notice that a delete action needs only the action line. No data line follows it.

The Four Actions

Action	Behavior	Data Line Needed
index	Adds a document, or replaces one with the same ID	Yes
create	Adds a document, and fails if the ID already exists	Yes
update	Changes selected fields of an existing document	Yes, wrapped in "doc"
delete	Removes a document	No

POST /_bulk
{ "update": { "_index": "products", "_id": "1" } }
{ "doc": { "stock": 95 } }

Setting a Default Index

When every line targets the same index, place the name in the URL and drop it from the action lines. The requests become shorter and easier to generate.

POST /products/_bulk
{ "index": { "_id": "3" } }
{ "name": "Desk Lamp", "price": 26.0, "stock": 60 }
{ "index": { "_id": "4" } }
{ "name": "Notebook Set", "price": 9.0, "stock": 300 }

Sending a File with curl

Large loads usually come from a file. Save the NDJSON lines in a file named products.ndjson, and send it with the correct content type. The --data-binary flag keeps the newlines intact.

curl --cacert http_ca.crt -u elastic:YOUR_PASSWORD \
  -X POST "https://localhost:9200/_bulk" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @products.ndjson

Reading the Response

A bulk request succeeds as a whole even when single operations fail. The HTTP status stays 200, and the body carries a top-level "errors" flag. When the flag reads true, scan the "items" list for entries with an error message.

{
  "took": 30,
  "errors": true,                       ← at least one item failed
  "items": [
    { "index": { "_id": "1", "status": 201, "result": "created" } },
    { "create": { "_id": "2", "status": 409,
        "error": { "type": "version_conflict_engine_exception" } } }
  ]
}

Each item keeps the same position as the matching action in the request. Collect the failed items, fix the cause, and send only those again.

Choosing a Batch Size

Bigger batches reduce overhead, but huge ones strain memory. A practical starting point is 5 MB to 15 MB per request, or roughly 1,000 to 5,000 small documents. Test a few sizes against your own data and watch the indexing speed. Stop growing the batch once the speed stops improving.

Batch size test (example numbers):

  500 docs/request    → 18,000 docs per second
  2,000 docs/request  → 31,000 docs per second
  5,000 docs/request  → 33,000 docs per second   ← gains flatten
  20,000 docs/request → 29,000 docs per second   ← memory pressure

Speed Tips for Large Loads

Temporary settings change the cost of a big import. Turning off automatic refresh stops Elasticsearch from creating a searchable snapshot every second. Setting replicas to zero avoids copying every document twice while the load runs. Restore both settings when the import ends.

PUT /products/_settings
{ "index": { "refresh_interval": "-1", "number_of_replicas": 0 } }

(run the bulk import)

PUT /products/_settings
{ "index": { "refresh_interval": "1s", "number_of_replicas": 1 } }

Run several bulk requests in parallel from a few worker threads for even more speed. Watch for 429 status codes, which signal that the cluster needs a breather. Slow the senders down and retry those requests after a short pause.

Bulk Requests Inside Applications

Most real systems build bulk requests in code. A program reads rows from a database or a file, groups them into batches, and sends each batch. Official client libraries include helper functions that handle the grouping, the retries, and the error reports.

Source data (2,000,000 rows)
      │
      ▼
Split into batches of 2,000 rows  →  1,000 batches
      │
      ▼
Send batches in parallel (4 workers)
      │
      ▼
Collect failed rows  →  log them  →  retry them later

Log the IDs of failed items. A short list of failures is easy to fix, while a silent failure leaves holes in the search index that nobody notices for weeks. A final count check closes the loop: compare the number of source rows with the document count of the index.

GET /products/_count

Previous lesson

Back to course

Next lesson