Parallel signing #1468
Open
Parallel signing #1468
+210
−98
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is (especially) useful with rekor 2 where the service only responds after log inclusion: We would prefer to get all signatures in the same inclusion batch. The change still affects both rekor v1 and rekor v2. This commit is a rebase of a bunch of Ramons commits. Signed-off-by: Ramon Petgrave <[email protected]> Signed-off-by: Jussi Kukkonen <[email protected]>
* Session thread safety is ambiguous: it may now be safe but situation is unclear * Use a single Session per create_entry() call: This may have a little more overhead but seems like the safest option * Note that RekorLog (v1) other methods may not be thread safe: I only reviewed the create_entry() flow Signed-off-by: Jussi Kukkonen <[email protected]>
It's a little unclear if Session is now thread safe or not: avoid reuse just in case Signed-off-by: Jussi Kukkonen <[email protected]>
* Separate the thread method so it's easier to see potential safety issues
* Using print() with the same file object is generally not thread safe: Avoid
it from the threaded method
The output remains effectively the same except:
* The b64 encoded signature is no longer printed to terminal
* Some print()s are now logger.info(): e.g.
Transparency log entry created at index: 5562
* Other print()s happen in a batch now, after he signing has finished
Signed-off-by: Jussi Kukkonen <[email protected]>
Signed-off-by: Jussi Kukkonen <[email protected]>
Signed-off-by: Jussi Kukkonen <[email protected]>
|
The included test only signs a few artifacts at a time but I have run this on more pathological cases: Runtime on current main branch
Runtime on parallel-sign branch
The time differences here are not that big but I think that's mostly because staging rekor 2 is currently configured with small and quick batches: With longer batching times the rekor2 difference will only grow |
Signed-off-by: Jussi Kukkonen <[email protected]>
woodruffw
reviewed
Jul 16, 2025
woodruffw
added
component:cli
Make every RekorLog have a Session of their own by default. This means RekorClient no longer needs to manage that. Signed-off-by: Jussi Kukkonen <[email protected]>
Colleen confirmed this for me: tessera checkpoint interval is currently configured at 2 seconds in staging, this might get bumped up to 10 seconds The effect of this with the 25 artifact rekor2 test (with the current 10 thread limit) is :
The timings for this branch depend on the max number of threads: current 10 threads means 25 artifacts splits to 3 checkpoints. The takeaways to me are:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This uses threads to sign multiple artifacts in parallel. The issue that is being solved here is that rekor v2 only responds to the entry request after the entry has been included in the log and the inclusion proof is available. When entry requests are done in parallel, they all get responses in the next "inclusion batch" (instead of one response per batch), speeding up multi-artifact signing by a lot. A practical example of the use case could be cpython releases where they sign ~10 artifacts.
There are multiple changes:
print()in threaded methodrequestswithurllib3#1040 we can likely get rid of this.Signer.sign_artifact()andSigner.sign_dsse()are now expected to be thread safe.