[refactor] Migrate persistent queue to consolidated metadata format #13126

malus2077 · 2025-06-02T05:00:42Z

This PR updates the persistence strategy to consolidate all relevant queue information (start, stop, in-progress, size) under a single key for each write operation.

Fixes [exporterhelper] Refactor persistent storage size backup #12890
Relates to Refactor: Write all queue state to a single key on each write for reliable backup #13047 (Sub-PR)

codecov · 2025-06-02T05:10:52Z

Codecov Report

Attention: Patch coverage is 75.36232% with 34 lines in your changes missing coverage. Please review.

Project coverage is 91.25%. Comparing base (1921e21) to head (1284e18).

Files with missing lines	Patch %	Lines
.../exporterhelper/internal/queue/persistent_queue.go	75.36%	22 Missing and 12 partials ⚠️

❌ Your patch check has failed because the patch coverage (75.36%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13126      +/-   ##
==========================================
- Coverage   91.27%   91.25%   -0.03%     
==========================================
  Files         514      514              
  Lines       28857    28943      +86     
==========================================
+ Hits        26340    26411      +71     
- Misses       1998     2001       +3     
- Partials      519      531      +12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bogdandrutu · 2025-06-02T17:24:36Z

Please use changes from #13125

bogdandrutu · 2025-06-03T13:09:10Z

Hey… something is wrong here

malus2077 · 2025-06-03T13:17:03Z

Hey… something is wrong here

It's done! Apologies, I'm still getting comfortable with git rebase operations.

exporter/exporterhelper/internal/queuebatch/persistent_queue.go

Embed QueueMetadata directly into persistentQueue, replacing the scattered metadata fields. 1. Add QueueMetadata as a member of persistentQueue. 2. Delete redundant fields (queueSize, readIdx, writeIdx, …). Relates to #13126

bogdandrutu · 2025-06-04T16:45:57Z

Please rebase with latest changes

…ry#13140) Embed QueueMetadata directly into persistentQueue, replacing the scattered metadata fields. 1. Add QueueMetadata as a member of persistentQueue. 2. Delete redundant fields (queueSize, readIdx, writeIdx, …). Relates to open-telemetry#13126

exporter/exporterhelper/internal/queuebatch/persistent_queue.go

bogdandrutu · 2025-06-05T16:17:02Z

exporter/exporterhelper/internal/queuebatch/persistent_queue.go

+		return err
+	}
+
+	if metadataSizerType != pq.set.sizerType {


Let's add support for this in a followup PR, for the moment in this case return error.

Not returning an error directly here for now. Please see my previous comment.

exporter/exporterhelper/internal/queuebatch/persistent_queue.go

malus2077 · 2025-06-06T22:43:48Z

Thanks for the review, @bogdandrutu! Changes are ready.

malus2077 · 2025-06-09T19:58:26Z

Thanks again for your patience and thorough feedback!

I’d really like to accelerate finalizing this PR, and I’m fully open to splitting it into smaller pieces if that helps us move faster. Please let me know how you’d prefer to proceed—I’m more than happy to accommodate your workflow to get this wrapped up efficiently.

.chloggen/malus2077_refactor-12890.yaml

dmitryax · 2025-06-13T03:54:49Z

exporter/exporterhelper/internal/queuebatch/persistent_queue.go

+	}
+
+	if metadataSizerType != pq.set.sizerType {
+		pq.logger.Warn("Sizer type mismatch in stored metadata")


Does it have to be a warning?

Probably this can be a warning, but I think we need to add some wording that the incoming requests are blocked until the queue is drained

Do we actually have to force draining when sizer changes? I imagine it'll be a significant delay to upgrade the collectors that accumulated a huge queue for any reason.

Can we maintain and store different metadata instances per each sizer type instead of having only one under qmv0?

When users change the sizerType, the current "force full drain" mechanism ensures consistency in queueSize and capacity calculations.

Implementing separate metadata instances per sizer type would create challenges:

Item Storage Overlap: Queue items use keys like "0", "1", "2". Different metadata instances would reference the same indices, requiring coordination.

Complex Capacity Management: Multiple metadata instances would complicate queue capacity handling.

While blocking during full drain causes delays for large queues, a non-blocking approach would significantly increase implementation complexity. We need to weigh the benefits against higher maintenance costs.

Do you think it would be better to work on enhancing the current force full drain?

Probably this can be a warning, but I think we need to add some wording that the incoming requests are blocked until the queue is drained

I would value your opinion on this. Please let me know if you see any room for improvement or have a different approach in mind. I'm happy to collaborate on a solution to ensure this issue #12890 is resolved properly.

@dmitryax

exporter/exporterhelper/internal/queuebatch/persistent_queue.go

…format.

…erations

…data

…ackup mechanism

…streamline metadata handling

…users

…tch check

malus2077 requested review from bogdandrutu, dmitryax and a team as code owners June 2, 2025 05:00

malus2077 mentioned this pull request Jun 2, 2025

Refactor: Write all queue state to a single key on each write for reliable backup #13047

Open

malus2077 requested review from mx-psi, dmathieu and andrzej-stencel as code owners June 3, 2025 06:54

malus2077 force-pushed the malus2077/refactor-12890 branch from 250d434 to 4aa8773 Compare June 3, 2025 13:13

bogdandrutu reviewed Jun 3, 2025

View reviewed changes

malus2077 mentioned this pull request Jun 4, 2025

[chore] Unify persistentQueue fields into QueueMetadata #13140

Merged

github-actions bot requested a review from bogdandrutu June 4, 2025 12:21

malus2077 requested a review from codeboten as a code owner June 5, 2025 00:47

malus2077 force-pushed the malus2077/refactor-12890 branch from 7d0eef9 to 1acbafc Compare June 5, 2025 01:59

malus2077 changed the title ~~refactor: migrate persistent queue to consolidated metadata format~~ [chore] Migrate persistent queue to consolidated metadata format Jun 5, 2025

malus2077 changed the title ~~[chore] Migrate persistent queue to consolidated metadata format~~ [refactor] Migrate persistent queue to consolidated metadata format Jun 5, 2025

bogdandrutu reviewed Jun 5, 2025

View reviewed changes

github-actions bot requested a review from bogdandrutu June 6, 2025 12:27

dmitryax reviewed Jun 13, 2025

View reviewed changes

malus2077 and others added 3 commits June 14, 2025 11:38

[Refactor] Migrate the persistent queue to the consolidated metadata …

022731c

…format.

[chore] Replace sizerTypeMismatch with atomic.Bool for thread-safe op…

d2655e7

…erations

[chore] Move persistentQueue state management entirely into QueueMeta…

09ae4e6

…data

malus2077 added 4 commits June 14, 2025 11:38

[chore] Add changelog entry for refactoring persistent storage size b…

9057c12

…ackup mechanism

[chore] Refactor persistentQueue to embed QueueMetadata directly and …

13feb01

…streamline metadata handling

[chore] Update changelog entry to indicate change is relevant to end …

c7dd30d

…users

[chore] update SizerType function calls and simplify sizer type misma…

1284e18

…tch check

malus2077 force-pushed the malus2077/refactor-12890 branch from f5bded2 to 1284e18 Compare June 14, 2025 04:03

[refactor] Migrate persistent queue to consolidated metadata format #13126

Are you sure you want to change the base?

[refactor] Migrate persistent queue to consolidated metadata format #13126

Uh oh!

Conversation

malus2077 commented Jun 2, 2025

Uh oh!

codecov bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bogdandrutu commented Jun 2, 2025

Uh oh!

bogdandrutu commented Jun 3, 2025

Uh oh!

malus2077 commented Jun 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bogdandrutu commented Jun 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bogdandrutu Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

malus2077 Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

malus2077 commented Jun 6, 2025

Uh oh!

malus2077 commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dmitryax Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

dmitryax Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmitryax Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

malus2077 Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

malus2077 Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jun 2, 2025 •

edited

Loading

malus2077 commented Jun 9, 2025 •

edited

Loading

dmitryax Jun 13, 2025 •

edited

Loading

dmitryax Jun 13, 2025 •

edited

Loading