Skip to content

refactor: optimize actual storage calculation to use PDP leaf counts#383

Open
ameeetgaikwad wants to merge 1 commit intofilecoin-project:masterfrom
ameeetgaikwad:fix/improve-calculate-actual-storage2
Open

refactor: optimize actual storage calculation to use PDP leaf counts#383
ameeetgaikwad wants to merge 1 commit intofilecoin-project:masterfrom
ameeetgaikwad:fix/improve-calculate-actual-storage2

Conversation

@ameeetgaikwad
Copy link
Copy Markdown

Closes #281

calculateActualStorage() was computing total stored bytes by creating a storage context for each dataset,
fetching every piece, and summing piece sizes. That path was doing much more work than needed for an aggregate storage total.

This change switches the calculation to use PDPVerifier.getDataSetLeafCount(dataSetId) and converts the result to bytes with PDP_LEAF_SIZE. This keeps the same public result shape and progress/abort behavior, but avoids the per-piece retrieval path.

@rvagg
Copy link
Copy Markdown
Member

rvagg commented Apr 30, 2026

@ameeetgaikwad this is close but not quite - see FilOzone/filecoin-services#451 where I discuss the FR32 expansion, so leaf count x 32 inflates the actual size by a little bit. The fix to that would be const dataSetBytes = (BigInt(leafCount) * BigInt(PDP_LEAF_SIZE) * 127n) / 128n

But, I think now the right approach might be to use the new getActivePieceCids iterator from Synapse and then getting the actual size out of the PieceCID itself, Synapse has utilities to decode a CID and give you the byte size of it.

So this is roughly correct, but looking back at #281 and my comments there, I think the third option I mentioned might be the right one. There'll be a newer Synapse release soon that has a better iterating getActivePieceCids type interface, I think the current one might work well enough but newer will be better, we just haven't triggered a release there yet.

Are you still able to take this on?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: calculate-actual-storage.ts can be improved.

2 participants