Implement index update rework

[pietroalbini]

Implementation work needs to be done to add the things proposed in this docs.rs RFC.

[jyn514]

Some preliminary things can be done without switching completely:

• storing the current commit in the database is pretty easy
• we can implement the webhook endpoint in addition to the current index updates and run them both in parallel for a while to make sure they work
• synchronization needs to be implemented before fully switching over to webhooks, because there is no analogue of peek_changes for webhooks (there's no way to ensure durability in case of a crash)

[pietroalbini]

synchronization needs to be implemented before fully switching over to webhooks, because there is no analogue of peek_changes for webhooks (there's no way to ensure durability in case of a crash)

Yes there is, if we update the hash of the last visited commit in the same database transaction as adding crates to the queue we'd get the same level of consistency we have today.

[jyn514]

last visited commit

I don't think this makes sense in the context on web-hooks because we might not receive the hooks in the order the crates were released. If crates A and B release and we get the webhook for B before A, updating the commit to point to B would mean we could skip A.

[Nemo157]

The webhook is just a trigger "check what crates were released between last commit and now", it will just call the exact same update code as the time based trigger does.

[syphar]

The future container-based setup for docs.rs will have:

• web workers
• build servers
• one single registry watcher

which complicates the webhook setup, since the webhook is received by another physical server.

[syphar]

some notes [from this zulip thread](#t-crates-io > trigger docs.rs rebuilds from crates.io), cc @Mark-Simulacrum @Turbo87

if we want to drop the registry watcher from here we could go with two approaches:

to keep in mind

we're not only talking about new releases. All event types we handle are:

• new release
• version-delete
• crate-delete
• yank
• unyank

For the new/delete & and yank/unyank events order is important.

queue between crates.io & docs.rs

We would use a queue system that guarantees order (SQS FIFO?). crates.io would enqueue all changes when it gets them, in order. Docs.rs would then, one by one, handle these.

Thinking about scaling, the order is only important on a crate-level, between crates it doesn't matter. The single-queue design might be limited then.

simple "ping" for all events, docs.rs fetches sparse index.

• for all the events, crates.io would "ping" a web endpoint (for example), or we use a queue for these again.
• docs.rs would then fetch the crate file from the sparse index, and diff the local database with the index. That diff algorithm has to correctly determine all the events between the docs.rs database state, and the index state.

There might be race conditions, when we are getting a new ping while we are handling an old one, for the same crate.
Which is why we would probably use database-level locking (on the crate level) to tell crates.io to back off or wait.

[Mark-Simulacrum]

For the new/delete & and yank/unyank events order is important.

Is this actually true? I would have expected that ordering is not important if we use the 'update is available' design and re-fetch the index file, because we should always end up at a consistent state, assuming the index we read (a) has the information we need and (b) is already updated by the time we process the last event in a sequence for a particular crate.

That diff algorithm has to correctly determine all the events between the docs.rs database state, and the index state.

I wouldn't have expected you to need to compute the events, but rather 'just' update the database with the latest index state. e.g., if there was a series of yank/unyank/yank/unyank, you don't really care -- you just need to make sure the docs.rs database has the latest yank state.

Alternatively, for some of the states above, it may be reasonable to look at dropping that entirely from the docs.rs database and just always hitting the index (possibly cached for a short-ish ttl) to query the state, if it's purely UI affecting. For example I'm not sure if docs.rs really needs to persist yank state at all -- presumably you only need that information for the UI, in which case querying the index live should be reasonably OK? Similarly, deletes (of versions or full crates) just not showing the docs.rs page if the index indicates a crate/version in the index is missing -- that seems doable?

Then the only real event you're processing is "should I enqueue a build?" where the reconciliation between the index and docs.rs state I'd expect to be pretty straightforward.

[syphar]

For the new/delete & and yank/unyank events order is important.

Is this actually true? I would have expected that ordering is not important if we use the 'update is available' design and re-fetch the index file, because we should always end up at a consistent state, assuming the index we read (a) has the information we need and (b) is already updated by the time we process the last event in a sequence for a particular crate.

Yep, with your idea the order would not be important, when the information is transmitted in the event, it is.

That diff algorithm has to correctly determine all the events between the docs.rs database state, and the index state.

I wouldn't have expected you to need to compute the events, but rather 'just' update the database with the latest index state. e.g., if there was a series of yank/unyank/yank/unyank, you don't really care -- you just need to make sure the docs.rs database has the latest yank state.

This is true about the yank-state, it only gets interesting with version & crate deletes, every now and then with re-creates ( I had to manually fix database state a couple of times when authors recreated the crate with - instead of _ or something like that.

Alternatively, for some of the states above, it may be reasonable to look at dropping that entirely from the docs.rs database and just always hitting the index (possibly cached for a short-ish ttl) to query the state, if it's purely UI affecting. For example I'm not sure if docs.rs really needs to persist yank state at all -- presumably you only need that information for the UI, in which case querying the index live should be reasonably OK? Similarly, deletes (of versions or full crates) just not showing the docs.rs page if the index indicates a crate/version in the index is missing -- that seems doable?

Then the only real event you're processing is "should I enqueue a build?" where the reconciliation between the index and docs.rs state I'd expect to be pretty straightforward.

Now that's also a nice idea. Only caveats would be the cases where we need more information to render UI (like, do we have docs or not, binary/library crate), or the cases where the build is still queued or in-progress.