OpenSearch not starting? Troubleshoot the TOO_MANY_REQUESTS/12 error like a pro!

If you’re reading this, chances are you’re frustrated and stuck with an OpenSearch cluster that refuses to start, throwing the infamous “TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block” error. Don’t worry, friend, you’re in the right place! This article will guide you through the troubleshooting process step-by-step, so you can get your OpenSearch cluster up and running in no time.

Understanding the Error

Before we dive into the fixes, let’s take a closer look at what’s causing the error. The “TOO_MANY_REQUESTS/12” error is usually triggered by high disk usage, which exceeds the flood-stage watermark. This watermark is a safety mechanism that prevents your cluster from consuming too many resources and becoming unresponsive.

[2023-03-01T14:30:00,123Z] INFO  [o.e.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk threshold [85%] exceeded, replica [my_index][0] will be made read-only
[2023-03-01T14:30:00,123Z] INFO  [o.e.c.r.a.DiskThresholdMonitor] [node-1] index [my_index] has read-only-allow-delete block, reason [disk usage exceeded flood-stage watermark]

Cause 1: High Disk Usage

One of the primary reasons for this error is high disk usage. This can be caused by:

• Indexing too much data: If you’re indexing a large amount of data without proper indexing strategies, it can lead to disk usage skyrocketing.
• Insufficient disk space: Running low on disk space can trigger the flood-stage watermark, causing your cluster to become read-only.
• Inefficient data retention policies: Not having a proper data retention policy in place can lead to old data accumulating, taking up valuable disk space.

Troubleshooting Steps

Now that we’ve identified the causes, let’s go through the troubleshooting steps to resolve the “TOO_MANY_REQUESTS/12” error:

Step 1: Check Disk Usage

Use the following command to check the current disk usage:

curl -XGET 'localhost:9200/_nodes/stats/fs'

This will give you an output like:

{
  "nodes": {
    "node-1": {
      "fs": {
        "total": {
          "total_in_bytes": 1073741824,
          "free_in_bytes": 536870912,
          "available_in_bytes": 536870912
        },
        "data": [
          {
            "path": "/var/lib/opensearch",
            "mount": "/var/lib/opensearch/",
            "total_in_bytes": 1073741824,
            "free_in_bytes": 536870912,
            "available_in_bytes": 536870912
          }
        ]
      }
    }
  }
}

Identify the node with the highest disk usage and proceed to the next step.

Step 2: Free Up Disk Space

Based on the disk usage output, you can take the following actions to free up disk space:

1. Delete unnecessary indices: Identify unused or redundant indices and delete them using the following command:

curl -XDELETE 'localhost:9200/my_unused_index'

2. Optimize your data retention policy: Implement a data retention policy that fits your use case. You can use the _delete_by_query API to remove old data:

curl -XPOST 'localhost:9200/my_index/_delete_by_query' -H 'Content-Type: application/json' -d '
{
  "query": {
    "range": {
      "my_date_field": {
        "lt": "now-30d"
      }
    }
  }
}
'

This will delete all documents in the my_index index that are older than 30 days.

Step 3: Adjust the Flood-Stage Watermark

If you’re not able to free up enough disk space, you can adjust the flood-stage watermark to give your cluster some breathing room:

curl -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.flood_stage": 90
  }
}
'

This sets the flood-stage watermark to 90%. Make sure to adjust this value according to your cluster’s specific needs.

Step 4: Restart OpenSearch

After making the necessary changes, restart your OpenSearch cluster to apply the changes:

sudo service opensearch restart

Preventing Future Occurrences

To avoid running into the “TOO_MANY_REQUESTS/12” error in the future, make sure to:

• Monitor disk usage regularly: Keep an eye on disk usage using the /_nodes/stats/fs API.
• Implement efficient indexing strategies: Use techniques like indexing in batches, using bulk APIs, and optimizing your indexing pipeline.
• Enforce data retention policies: Regularly clean up old data to prevent it from accumulating.
• Scale your cluster horizontally: Add more nodes to your cluster to distribute the load and prevent disk usage from becoming a bottleneck.

Conclusion

Troubleshooting the “TOO_MANY_REQUESTS/12” error can be a daunting task, but by following these steps, you should be able to identify and resolve the underlying issue. Remember to maintain a healthy disk usage, implement efficient indexing strategies, and enforce data retention policies to prevent future occurrences of this error.

OpenSearch is a powerful tool, and with the right troubleshooting techniques and best practices, you can ensure your cluster runs smoothly and efficiently. Happy troubleshooting!

Troubleshooting Step	Description
Step 1: Check Disk Usage	Use the /_nodes/stats/fs API to check disk usage.
Step 2: Free Up Disk Space	Delete unnecessary indices, optimize data retention policies, and remove old data.
Step 3: Adjust the Flood-Stage Watermark	Use the _cluster/settings API to adjust the flood-stage watermark.
Step 4: Restart OpenSearch	Restart the OpenSearch cluster to apply changes.

By following these steps, you’ll be well on your way to resolving the “TOO_MANY_REQUESTS/12” error and getting your OpenSearch cluster up and running smoothly.

Frequently Asked Question

Stuck with OpenSearch not starting due to “[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]” error? Worry not, we’ve got you covered! Here are some common questions and answers to get you back on track: