[archival] swtich state sync from archival store to checkpoints bucket #21955

phoenix-o · 2025-04-23T16:22:56Z

Description

Part 1 of deprecation of existing archival storage.
Notes:

switches StateSync from archival store to checkpoint bucket
removes the notion of archival from pruning policies
removes write-archival config from Sui node config

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

Protocol:
Nodes (Validators and Full nodes): Switches Sui archive mechanism for state sync from the existing format to checkpoint data ingestion bucket. The state-archive-read-config section of the full node config needs to be updated to include the ingestion-url field. For possible bucket options, see: https://docs.sui.io/guides/developer/advanced/custom-indexer#remote-reader.
gRPC:
JSON-RPC:
GraphQL:
CLI:
Rust SDK:

vercel · 2025-04-23T16:23:03Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 25, 2025 5:37pm

2 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview		Apr 25, 2025 5:37pm
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview		Apr 25, 2025 5:37pm

bmwill · 2025-04-23T18:03:56Z

Adding @aschran since he's more recently worked in this area than i have. I'm still planning on doing a review though :)

bmwill · 2025-04-23T18:00:17Z

crates/sui-data-ingestion-core/src/executor.rs

                    if let Some(limit) = upper_limit {
-                        if checkpoint.checkpoint_summary.sequence_number > limit {
+                        if sequence_number > limit && self.pool_senders.len() == 1 {
                            break;
                        }
                    }
+                }
+                Some(checkpoint) = checkpoint_recv.recv() => {


Is this a bug fix?

depends — the semantics have changed. Previously, it was "don't process checkpoints with sequence number > upper_limit", and now it's "ensure that all checkpoints up to upper_limit are processed". Both work for state sync, but I think the latter is better. From what I can tell, the parameter isn’t currently used anywhere in the codebase

bmwill · 2025-04-23T19:08:17Z

crates/sui-archival/src/lib.rs

-pub mod writer;
-
-#[cfg(test)]
-mod tests;


So the writer is being removed immediately? How does this impact the infra that we're running today for populating the archival? Or is this the archival that used to run inside the node and the one that runs as a part of ingestion framework still functions?

yeah, that's the old writer. Data ingestion still works after the change

bmwill · 2025-04-23T19:54:56Z

crates/sui-network/src/state_sync/mod.rs

+            'inner: {
+                let start = highest_synced
+                    .checked_add(1)
+                    .expect("Checkpoint seq num overflow");
+                let end = lowest_checkpoint_on_peers.unwrap();
+
+                let Some(ref archive_config) = archive_config else {
+                    warn!("Failed to find an archive reader to complete the state sync request");
+                    break 'inner;
+                };
+                let Some(ref ingestion_url) = archive_config.ingestion_url else {
+                    warn!("Archival ingestion url for state sync is not configured");
+                    break 'inner;
+                };
+                let reader_options = ReaderOptions {
+                    batch_size: archive_config.download_concurrency.into(),
+                    upper_limit: Some(end),
+                    ..Default::default()
+                };
+                let Ok((executor, _exit_sender)) = setup_single_workflow(
+                    StateSyncWorker(store.clone()),
+                    ingestion_url.clone(),
+                    start,
+                    1,
+                    Some(reader_options),
+                )
                .await
-            {
-                let txn_counter = Arc::new(AtomicU64::new(0));
-                let checkpoint_counter = Arc::new(AtomicU64::new(0));
-                if let Err(err) = archive_reader
-                    .read(
-                        store.clone(),
-                        checkpoint_range,
-                        txn_counter.clone(),
-                        checkpoint_counter.clone(),
-                        true,
-                    )
-                    .await
-                {
-                    warn!("State sync from archive failed with error: {:?}", err);
-                } else {
-                    info!("State sync from archive is complete. Checkpoints downloaded = {:?}, Txns downloaded = {:?}", checkpoint_counter.load(Ordering::Relaxed), txn_counter.load(Ordering::Relaxed));
+                else {
+                    break 'inner;
+                };


I'm having a little bit of trouble understanding the break 'inner control flow here. would this maybe be clearer if it was a separate function that could early return or something?

it just skips to the last statement in the loop(i.e. sleep), but I don't mind

bmwill

Thanks!

Lets also make sure to add some information into the release notes about this change.

cc @abhinavg6

abhinavg6 · 2025-04-25T04:04:29Z

@bmwill - By release notes, do you mean those would be available here - https://github.com/MystenLabs/sui/releases?

Also, considering this is part 1 of deprecation, is it worth documenting our overall intent now at https://docs.sui.io/guides/operator/archives and any other pertinent places? Or shall we wait until deprecation happens when we have the Bigtable interface for KV store?

bmwill · 2025-04-25T12:54:05Z

For release notes, checking a box and adding some info in the PR cover letter should suffice. As for documentation's our plans, yeah we should probably start doing that.

phoenix-o · 2025-04-25T13:24:20Z

I updated release note section. Feel free to edit it to better reflect the change.

Or shall we wait until deprecation happens when we have the Bigtable interface for KV store?

The change is not related to the KV store. The previous archive solution will be replaced by the existing ingestion bucket of checkpoints

MystenLabs#21955) ## Description Part 1 of deprecation of existing archival storage. Notes: * switches StateSync from archival store to checkpoint bucket * removes the notion of archival from pruning policies * removes write-archival config from Sui node config --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [x] Nodes (Validators and Full nodes): Switches Sui archive mechanism for state sync from the existing format to checkpoint data ingestion bucket. The `state-archive-read-config` section of the full node config needs to be updated to include the `ingestion-url` field. For possible bucket options, see: https://docs.sui.io/guides/developer/advanced/custom-indexer#remote-reader. - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env April 23, 2025 16:23 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs April 23, 2025 16:24 View deployment

phoenix-o force-pushed the archive_vol0 branch from 3ffc12a to f83de5d Compare April 23, 2025 16:33

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env April 23, 2025 16:34 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs April 23, 2025 16:35 View deployment

phoenix-o force-pushed the archive_vol0 branch from f83de5d to 7dadb30 Compare April 23, 2025 16:50

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env April 23, 2025 16:50 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs April 23, 2025 16:52 View deployment

phoenix-o marked this pull request as ready for review April 23, 2025 17:20

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env April 23, 2025 17:20 — with GitHub Actions Inactive

phoenix-o requested a review from bmwill April 23, 2025 17:21

bmwill requested a review from aschran April 23, 2025 18:03

bmwill reviewed Apr 23, 2025

View reviewed changes

phoenix-o force-pushed the archive_vol0 branch from 7dadb30 to 91d0301 Compare April 24, 2025 14:08

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env April 24, 2025 14:08 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs April 24, 2025 14:11 View deployment

bmwill approved these changes Apr 24, 2025

View reviewed changes

[archival] swtich state sync from archival store to checkpoints bucket

7567e1d

phoenix-o force-pushed the archive_vol0 branch from 91d0301 to 7567e1d Compare April 25, 2025 17:35

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env April 25, 2025 17:35 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs April 25, 2025 17:37 View deployment

phoenix-o merged commit 1dabb5c into MystenLabs:main Apr 28, 2025
47 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[archival] swtich state sync from archival store to checkpoints bucket #21955

[archival] swtich state sync from archival store to checkpoints bucket #21955

Uh oh!

phoenix-o commented Apr 23, 2025 •

edited by ebmifa

Loading

Uh oh!

vercel bot commented Apr 23, 2025 •

edited

Loading

Uh oh!

bmwill commented Apr 23, 2025 •

edited

Loading

Uh oh!

bmwill Apr 23, 2025

Uh oh!

phoenix-o Apr 23, 2025

Uh oh!

bmwill Apr 23, 2025

Uh oh!

phoenix-o Apr 23, 2025

Uh oh!

bmwill Apr 23, 2025

Uh oh!

phoenix-o Apr 23, 2025

Uh oh!

bmwill left a comment

Uh oh!

abhinavg6 commented Apr 25, 2025

Uh oh!

bmwill commented Apr 25, 2025

Uh oh!

phoenix-o commented Apr 25, 2025

Uh oh!

Uh oh!

Uh oh!

[archival] swtich state sync from archival store to checkpoints bucket #21955

[archival] swtich state sync from archival store to checkpoints bucket #21955

Uh oh!

Conversation

phoenix-o commented Apr 23, 2025 • edited by ebmifa Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release notes

Uh oh!

vercel bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bmwill commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bmwill Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

phoenix-o Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

bmwill Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

phoenix-o Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

bmwill Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

phoenix-o Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

bmwill left a comment

Choose a reason for hiding this comment

Uh oh!

abhinavg6 commented Apr 25, 2025

Uh oh!

bmwill commented Apr 25, 2025

Uh oh!

phoenix-o commented Apr 25, 2025

Uh oh!

Uh oh!

Uh oh!

phoenix-o commented Apr 23, 2025 •

edited by ebmifa

Loading

vercel bot commented Apr 23, 2025 •

edited

Loading

bmwill commented Apr 23, 2025 •

edited

Loading