Skip to content

Add Query Resource Based Eviction#7488

Open
eeldaly wants to merge 6 commits intocortexproject:masterfrom
eeldaly:query-eviction
Open

Add Query Resource Based Eviction#7488
eeldaly wants to merge 6 commits intocortexproject:masterfrom
eeldaly:query-eviction

Conversation

@eeldaly
Copy link
Copy Markdown
Contributor

@eeldaly eeldaly commented May 7, 2026

What this PR does:
This PR builds on the current resource based throttling infrastructure (#6674) to allow for evicting currently running queries. This is currently only implemented on querier pods but can be extended to other pods similar to resourced based throttling.


Flags
All flags are prefixed with -querier.query-protection.eviction.

threshold.cpu-utilization: Max CPU utilization (0–1) before evicting the heaviest query (default 0)
threshold.heap-utilization: Max heap utilization (0–1) before evicting the heaviest query (default 0)
check-interval: How frequently the evictor checks resource utilization (default 1s)
cooldown-period: Number of check intervals to wait after an eviction before evicting again (default 3)
eviction-metric: Metric used to determine the heaviest query (fetched_samples, fetched_series, fetched_chunks, fetched_chunk_bytes) (default fetched_samples)
min-query-age: Minimum time a query must be running before it becomes eligible for eviction (default 10s)

The evictor will be disabled and will not check every check-interval if both cpu and heap utilization are disabled (set to 0).


How it works
This feature is completely disabled and the registry will not be created if cpu-utilization and heap-utilization are set to 0. If either of them is larger:

  • Picked up queries will be registered to a registry to track all current queries in a querier
  • The evictor will check for utilization every check-interval
  • Once a threshold is breached, all currently running queries who have been evaluted for longer than min-query-age will be evaluated from heaviest based on eviction-metric. The heaviest query will be evicted
  • We will wait check-interval before checking again if threshold is breached.

Why the current metrics?
The current metrics are not the best to detect the root cause for high heap, however, they are readily available and can be used as a proxy until work in Prometheus/Thanos is done to allow for better metrics. peak_samples is currently only available in query_stats after the query is completed and we have no way of tracking current heap usage by a query. Any new metrics can easily be added to this structure later.


Metrics
cortex_query_evictions_total{resource="cpu|heap", component="querier"}: Counter increments by one for every eviction that occurs. A single query may lead to multiple increments if it retries and ends up evicted again.


note: make doc added the config to store-gateway.md file even though this is not currently implemented on there as it is built on top of resource based throttling.


Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
eeldaly added 5 commits May 7, 2026 15:09
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant