fix(ingester): prevent false series limit throttling during scale-up#7491
Open
yeya24 wants to merge 1 commit intocortexproject:masterfrom
Open
fix(ingester): prevent false series limit throttling during scale-up#7491yeya24 wants to merge 1 commit intocortexproject:masterfrom
yeya24 wants to merge 1 commit intocortexproject:masterfrom
Conversation
4c9f220 to
1e065f4
Compare
During ingester scale-up, the local series limit (derived from global_limit / num_ingesters * replication_factor) shrinks immediately when new ingesters join the ring. However, existing ingesters cannot shed their series until head compaction (~2h). This causes false throttling for tenants who are within their global limit. Add experimental -ingester.local-limit-cache-enabled flag (default false). When enabled, the limiter caches the previous local limit per tenant in convertGlobalToLocalLimit(). If the global limit has not changed and the new calculated limit is lower than the cached value, the cached (higher) limit is returned. The cache is only used when globalLimit == prev.globalLimit, ensuring: - Global limit decrease: immediately enforced (cache bypassed) - Global limit increase: recalculated fresh (cache bypassed) - Scale-up with same limit: cached (prevents false throttle) - Scale-down: new limit is higher, cache updates naturally ResetLocalLimitCache(userID) resets the cache for a specific tenant after their TSDB head compaction, when the ingester's series count reflects its true post-resharding ownership. Signed-off-by: Ben Ye <benye@amazon.com>
1e065f4 to
aea36ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
During ingester scale-up, the local series limit (derived from
global_limit / num_ingesters * replication_factor) shrinks immediately when new ingesters join the ring. However, existing ingesters cannot shed their series until head compaction (~2h). This causes false throttling for tenants who are within their global limit.Root Cause
When ingesters scale from N to M (e.g., 75 → 249), the local limit drops from
global / 75 * 3 = 108,000toglobal / 249 * 3 = 32,530. An ingester holding 50,000 series (well under the 2.7M global limit) is immediately throttled because50,000 > 32,530.The series cannot redistribute until head compaction flushes idle series, so the throttling persists for up to 2 hours.
Fix
Cache the previous local limit per tenant in
convertGlobalToLocalLimit(). If the global limit has not changed and the new calculated limit is lower than the cached value, return the cached (higher) limit.The cache is only used when
globalLimit == prev.globalLimit, ensuring:A
ResetLocalLimitCache()method is provided for callers to clear the cache after head compaction, when the ingester's series count reflects its true post-resharding ownership.How was this tested?