Add Query Resource Based Eviction#7488
Open
eeldaly wants to merge 6 commits intocortexproject:masterfrom
Open
Conversation
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
This PR builds on the current resource based throttling infrastructure (#6674) to allow for evicting currently running queries. This is currently only implemented on querier pods but can be extended to other pods similar to resourced based throttling.
Flags
All flags are prefixed with
-querier.query-protection.eviction.threshold.cpu-utilization: Max CPU utilization (0–1) before evicting the heaviest query (default 0)threshold.heap-utilization: Max heap utilization (0–1) before evicting the heaviest query (default 0)check-interval: How frequently the evictor checks resource utilization (default 1s)cooldown-period: Number of check intervals to wait after an eviction before evicting again (default 3)eviction-metric: Metric used to determine the heaviest query (fetched_samples,fetched_series,fetched_chunks,fetched_chunk_bytes) (defaultfetched_samples)min-query-age: Minimum time a query must be running before it becomes eligible for eviction (default 10s)The evictor will be disabled and will not check every
check-intervalif both cpu and heap utilization are disabled (set to 0).How it works
This feature is completely disabled and the registry will not be created if
cpu-utilizationandheap-utilizationare set to 0. If either of them is larger:check-intervalmin-query-agewill be evaluated from heaviest based oneviction-metric. The heaviest query will be evictedcheck-intervalbefore checking again if threshold is breached.Why the current metrics?
The current metrics are not the best to detect the root cause for high heap, however, they are readily available and can be used as a proxy until work in Prometheus/Thanos is done to allow for better metrics.
peak_samplesis currently only available inquery_statsafter the query is completed and we have no way of tracking current heap usage by a query. Any new metrics can easily be added to this structure later.Metrics
cortex_query_evictions_total{resource="cpu|heap", component="querier"}: Counter increments by one for every eviction that occurs. A single query may lead to multiple increments if it retries and ends up evicted again.note: make doc added the config to
store-gateway.mdfile even though this is not currently implemented on there as it is built on top of resource based throttling.Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags