Skip to content

fix: prover node retries gathering needed txs - #11089

Merged
alexghr merged 5 commits into
masterfrom
ag/prover-node-retries
Jan 8, 2025
Merged

fix: prover node retries gathering needed txs#11089
alexghr merged 5 commits into
masterfrom
ag/prover-node-retries

Conversation

@alexghr

@alexghr alexghr commented Jan 7, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

This PR introduces retries when the prover node gathers the necessary txs for an epoch proof. This is necessary because on a fresh boot, the prover might not be connected to any peers so it will fail to acquire the necessary data.

Fix for #11063

TODO:

  • expose new config as env vars

alexghr
alexghr commented Jan 7, 2025
View reviewed changes
Comment thread yarn-project/prover-node/src/prover-node.test.ts
mock<L2Block>({
number: i + 20,
hash: () => new Fr(i),
body: mock<Body>({ txEffects: [mock<TxEffect>({ txHash: TxHash.random() } as TxEffect)] }),

@alexghr alexghr Jan 7, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This took longer than I want to admit to debug: mock makes the inner instances show up as undefined

PhilWindle
PhilWindle reviewed Jan 7, 2025
View reviewed changes

@PhilWindle PhilWindle left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me. Would appreciate @spalladino having a look also.

spalladino
spalladino approved these changes Jan 7, 2025
View reviewed changes

@spalladino spalladino left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Note the retry mechanism only makes sense with a p2p prover coordination, not with a direct connection to a node, but since the p2p is the one to be actually used for production I don't think we need to worry about the other scenario.

Also, I was thinking about having different timeouts for gathering txs for a quote vs for completing a job. But I don't really see a need for differentiating them if we set the timeout relatively high, and the cost for spending time collecting txs for a quote that we'll never manage to land is not too high.

@alexghr
alexghr merged commit 6f07132 into master Jan 8, 2025
@alexghr
alexghr deleted the ag/prover-node-retries branch January 8, 2025 11:12
@AztecBot AztecBot mentioned this pull request Jan 8, 2025
Merged
rahul-kothari pushed a commit that referenced this pull request Jan 8, 2025
@AztecBot
chore(master): Release 0.69.1 (#11028)
8105d37 
🤖 I have created a release *beep* *boop*
---


<details><summary>aztec-package: 0.69.1</summary>

##
[0.69.1](aztec-package-v0.69.0...aztec-package-v0.69.1)
(2025-01-08)


### Features

* Optionally handle rpc errors with 200 + err body
([#11083](#11083))
([b42756b](b42756b))


### Miscellaneous

* Representing `TxHash` as `Fr`
([#10954](#10954))
([84e67ac](84e67ac))
</details>

<details><summary>barretenberg.js: 0.69.1</summary>

##
[0.69.1](barretenberg.js-v0.69.0...barretenberg.js-v0.69.1)
(2025-01-08)


### Miscellaneous

* **barretenberg.js:** Synchronize aztec-packages versions
</details>

<details><summary>aztec-packages: 0.69.1</summary>

##
[0.69.1](aztec-packages-v0.69.0...aztec-packages-v0.69.1)
(2025-01-08)


### Features

* Acir formal proofs
([#10973](#10973))
([1cb7cd7](1cb7cd7))
* **blobs:** Blob sink
([#10079](#10079))
([94b6c86](94b6c86))
* Derive transcript structure between non-zk and zk flavors and between
Ultra and UltraKeccak
([#11086](#11086))
([48286c6](48286c6))
* Fix commitments and openings of masking polynomials used in zk
sumcheck
([#10773](#10773))
([fc48dcc](fc48dcc))
* Improve blob simulation speed
([#11075](#11075))
([fe845e2](fe845e2))
* Improve witness generation for cycle_group::batch_mul
([#9563](#9563))
([7da7f2b](7da7f2b))
* More efficient `compute_l2_to_l1_hash`
([#11036](#11036))
([60d43fd](60d43fd))
* Optionally handle rpc errors with 200 + err body
([#11083](#11083))
([b42756b](b42756b))
* Prover node checks txs availability before sending quote
([#10965](#10965))
([b9e7109](b9e7109)),
closes
[#10803](#10803)
* Slasher
([#10693](#10693))
([9dad251](9dad251))
* Use unconstrained helper in `append_tx_effects_for_blob`
([#11037](#11037))
([5355a5e](5355a5e))
* Validate block proposal txs iteratively
([#10921](#10921))
([c92129e](c92129e)),
closes
[#10869](#10869)


### Bug Fixes

* Add bytecode instances in reverse
([#11064](#11064))
([036496c](036496c))
* Can't use `self.field` in trait default implementations
([#11004](#11004))
([f31278f](f31278f))
* Check class registration nullifier in node before returning class
([#11074](#11074))
([649b590](649b590))
* **ci:** Update docs hash
([#11082](#11082))
([b0a8397](b0a8397))
* Optional check for architecture in bootstrap image-aztec
([#11085](#11085))
([fed44a5](fed44a5)),
closes
[#10957](#10957)
* Prover node retries gathering needed txs
([#11089](#11089))
([6f07132](6f07132))
* Reset pc to 0 for next enqueued call in avm witgen
([#11043](#11043))
([44e4816](44e4816))
* Update requests per call should be less than per tx
([#11072](#11072))
([da5e95f](da5e95f))
* Update schema naming
([#11038](#11038))
([547e556](547e556))


### Miscellaneous

* **avm:** Handle specific MSM errors
([#11068](#11068))
([a5097a9](a5097a9)),
closes
[#10854](#10854)
* **avm:** More column information in permutations
([#11070](#11070))
([8829f24](8829f24))
* Avoid getport race conditions when starting anvil
([#11077](#11077))
([b73f7f9](b73f7f9))
* Bump `noir-gates-diff`
([#11056](#11056))
([e076000](e076000))
* Bump `noir-gates-diff` commit
([#11042](#11042))
([c820a0e](c820a0e))
* Bump devnet prover agents
([#11046](#11046))
([55de1ce](55de1ce))
* Bump rc-1 prover agents
([#11033](#11033))
([fb58c16](fb58c16))
* **ci:** Fix CI to create baseline gate reports
([#11055](#11055))
([e2f6905](e2f6905))
* Clean up proof lengths and IPA
([#11020](#11020))
([800c834](800c834))
* Disable noir contracts tests until stabilized
([#11047](#11047))
([a76b52e](a76b52e))
* Fix customTags is not iterable in e2e-prover-full
([#11057](#11057))
([f35094f](f35094f))
* Fix invalid random log id
([#11076](#11076))
([b1b67b0](b1b67b0))
* Fix write_recursion_inputs flow in bootstrap
([#11080](#11080))
([cd5a615](cd5a615))
* Hide note_hashes log
([#11059](#11059))
([d9a14d2](d9a14d2))
* Let IndexedTreeLeafPreimage have LeafPreimage as a parent trait
([#10913](#10913))
([496a55a](496a55a))
* Load in the big dashboard during metrics install
([#11007](#11007))
([f6f2c12](f6f2c12))
* New test that you can register, deploy, and call a public function all
in one tx
([#11045](#11045))
([5e3183c](5e3183c))
* Pass fn signatures
([#10849](#10849))
([a2c4e98](a2c4e98))
* Patch jest to not use JSON serialization in message passing ci3
([#10964](#10964))
([d08f540](d08f540))
* Refactor tail public inputs
([#11031](#11031))
([4ed1530](4ed1530))
* Remove abi refs from publisher
([#10766](#10766))
([17d6802](17d6802))
* Remove some instances of `--silence-warnings`
([#11071](#11071))
([ecbd59e](ecbd59e))
* Renaming getIncomingNotes
([#10743](#10743))
([ffa7407](ffa7407))
* Replace relative paths to noir-protocol-circuits
([7194a7e](7194a7e))
* Replace relative paths to noir-protocol-circuits
([b00bd13](b00bd13))
* Replace relative paths to noir-protocol-circuits
([c4fcbc0](c4fcbc0))
* Replace relative paths to noir-protocol-circuits
([694343d](694343d))
* Representing `TxHash` as `Fr`
([#10954](#10954))
([84e67ac](84e67ac))
* Restore `prove_then_verify` test on `verify_rollup_honk_proof`
([#11018](#11018))
([79e289d](79e289d))
* Unify honk verifier contracts
([#11067](#11067))
([9968849](9968849))
* Update noir-bignum to v0.5.0
([#11066](#11066))
([bf10a5c](bf10a5c))
* Updated aztec-spartan.sh and the README
([#11088](#11088))
([56128a6](56128a6))
</details>

<details><summary>barretenberg: 0.69.1</summary>

##
[0.69.1](barretenberg-v0.69.0...barretenberg-v0.69.1)
(2025-01-08)


### Features

* Acir formal proofs
([#10973](#10973))
([1cb7cd7](1cb7cd7))
* Derive transcript structure between non-zk and zk flavors and between
Ultra and UltraKeccak
([#11086](#11086))
([48286c6](48286c6))
* Fix commitments and openings of masking polynomials used in zk
sumcheck
([#10773](#10773))
([fc48dcc](fc48dcc))
* Improve witness generation for cycle_group::batch_mul
([#9563](#9563))
([7da7f2b](7da7f2b))


### Bug Fixes

* Add bytecode instances in reverse
([#11064](#11064))
([036496c](036496c))
* Reset pc to 0 for next enqueued call in avm witgen
([#11043](#11043))
([44e4816](44e4816))
* Update requests per call should be less than per tx
([#11072](#11072))
([da5e95f](da5e95f))


### Miscellaneous

* **avm:** Handle specific MSM errors
([#11068](#11068))
([a5097a9](a5097a9)),
closes
[#10854](#10854)
* **avm:** More column information in permutations
([#11070](#11070))
([8829f24](8829f24))
* Clean up proof lengths and IPA
([#11020](#11020))
([800c834](800c834))
* Fix write_recursion_inputs flow in bootstrap
([#11080](#11080))
([cd5a615](cd5a615))
* Restore `prove_then_verify` test on `verify_rollup_honk_proof`
([#11018](#11018))
([79e289d](79e289d))
* Unify honk verifier contracts
([#11067](#11067))
([9968849](9968849))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
@alexghr alexghr linked an issue Jan 8, 2025 that may be closed by this pull request
prover-node should be ablet to recover after a crash by picking up txs from the p2p network #11063
Closed
AztecBot added a commit to AztecProtocol/barretenberg that referenced this pull request Jan 9, 2025
@AztecBot AztecBot
chore(master): Release 0.69.1 (#11028)
be1d5a3 
🤖 I have created a release *beep* *boop*
---


<details><summary>aztec-package: 0.69.1</summary>

##
[0.69.1](AztecProtocol/aztec-packages@aztec-package-v0.69.0...aztec-package-v0.69.1)
(2025-01-08)


### Features

* Optionally handle rpc errors with 200 + err body
([#11083](AztecProtocol/aztec-packages#11083))
([b42756b](AztecProtocol/aztec-packages@b42756b))


### Miscellaneous

* Representing `TxHash` as `Fr`
([#10954](AztecProtocol/aztec-packages#10954))
([84e67ac](AztecProtocol/aztec-packages@84e67ac))
</details>

<details><summary>barretenberg.js: 0.69.1</summary>

##
[0.69.1](AztecProtocol/aztec-packages@barretenberg.js-v0.69.0...barretenberg.js-v0.69.1)
(2025-01-08)


### Miscellaneous

* **barretenberg.js:** Synchronize aztec-packages versions
</details>

<details><summary>aztec-packages: 0.69.1</summary>

##
[0.69.1](AztecProtocol/aztec-packages@aztec-packages-v0.69.0...aztec-packages-v0.69.1)
(2025-01-08)


### Features

* Acir formal proofs
([#10973](AztecProtocol/aztec-packages#10973))
([1cb7cd7](AztecProtocol/aztec-packages@1cb7cd7))
* **blobs:** Blob sink
([#10079](AztecProtocol/aztec-packages#10079))
([94b6c86](AztecProtocol/aztec-packages@94b6c86))
* Derive transcript structure between non-zk and zk flavors and between
Ultra and UltraKeccak
([#11086](AztecProtocol/aztec-packages#11086))
([48286c6](AztecProtocol/aztec-packages@48286c6))
* Fix commitments and openings of masking polynomials used in zk
sumcheck
([#10773](AztecProtocol/aztec-packages#10773))
([fc48dcc](AztecProtocol/aztec-packages@fc48dcc))
* Improve blob simulation speed
([#11075](AztecProtocol/aztec-packages#11075))
([fe845e2](AztecProtocol/aztec-packages@fe845e2))
* Improve witness generation for cycle_group::batch_mul
([#9563](AztecProtocol/aztec-packages#9563))
([7da7f2b](AztecProtocol/aztec-packages@7da7f2b))
* More efficient `compute_l2_to_l1_hash`
([#11036](AztecProtocol/aztec-packages#11036))
([60d43fd](AztecProtocol/aztec-packages@60d43fd))
* Optionally handle rpc errors with 200 + err body
([#11083](AztecProtocol/aztec-packages#11083))
([b42756b](AztecProtocol/aztec-packages@b42756b))
* Prover node checks txs availability before sending quote
([#10965](AztecProtocol/aztec-packages#10965))
([b9e7109](AztecProtocol/aztec-packages@b9e7109)),
closes
[#10803](AztecProtocol/aztec-packages#10803)
* Slasher
([#10693](AztecProtocol/aztec-packages#10693))
([9dad251](AztecProtocol/aztec-packages@9dad251))
* Use unconstrained helper in `append_tx_effects_for_blob`
([#11037](AztecProtocol/aztec-packages#11037))
([5355a5e](AztecProtocol/aztec-packages@5355a5e))
* Validate block proposal txs iteratively
([#10921](AztecProtocol/aztec-packages#10921))
([c92129e](AztecProtocol/aztec-packages@c92129e)),
closes
[#10869](AztecProtocol/aztec-packages#10869)


### Bug Fixes

* Add bytecode instances in reverse
([#11064](AztecProtocol/aztec-packages#11064))
([036496c](AztecProtocol/aztec-packages@036496c))
* Can't use `self.field` in trait default implementations
([#11004](AztecProtocol/aztec-packages#11004))
([f31278f](AztecProtocol/aztec-packages@f31278f))
* Check class registration nullifier in node before returning class
([#11074](AztecProtocol/aztec-packages#11074))
([649b590](AztecProtocol/aztec-packages@649b590))
* **ci:** Update docs hash
([#11082](AztecProtocol/aztec-packages#11082))
([b0a8397](AztecProtocol/aztec-packages@b0a8397))
* Optional check for architecture in bootstrap image-aztec
([#11085](AztecProtocol/aztec-packages#11085))
([fed44a5](AztecProtocol/aztec-packages@fed44a5)),
closes
[#10957](AztecProtocol/aztec-packages#10957)
* Prover node retries gathering needed txs
([#11089](AztecProtocol/aztec-packages#11089))
([6f07132](AztecProtocol/aztec-packages@6f07132))
* Reset pc to 0 for next enqueued call in avm witgen
([#11043](AztecProtocol/aztec-packages#11043))
([44e4816](AztecProtocol/aztec-packages@44e4816))
* Update requests per call should be less than per tx
([#11072](AztecProtocol/aztec-packages#11072))
([da5e95f](AztecProtocol/aztec-packages@da5e95f))
* Update schema naming
([#11038](AztecProtocol/aztec-packages#11038))
([547e556](AztecProtocol/aztec-packages@547e556))


### Miscellaneous

* **avm:** Handle specific MSM errors
([#11068](AztecProtocol/aztec-packages#11068))
([a5097a9](AztecProtocol/aztec-packages@a5097a9)),
closes
[#10854](AztecProtocol/aztec-packages#10854)
* **avm:** More column information in permutations
([#11070](AztecProtocol/aztec-packages#11070))
([8829f24](AztecProtocol/aztec-packages@8829f24))
* Avoid getport race conditions when starting anvil
([#11077](AztecProtocol/aztec-packages#11077))
([b73f7f9](AztecProtocol/aztec-packages@b73f7f9))
* Bump `noir-gates-diff`
([#11056](AztecProtocol/aztec-packages#11056))
([e076000](AztecProtocol/aztec-packages@e076000))
* Bump `noir-gates-diff` commit
([#11042](AztecProtocol/aztec-packages#11042))
([c820a0e](AztecProtocol/aztec-packages@c820a0e))
* Bump devnet prover agents
([#11046](AztecProtocol/aztec-packages#11046))
([55de1ce](AztecProtocol/aztec-packages@55de1ce))
* Bump rc-1 prover agents
([#11033](AztecProtocol/aztec-packages#11033))
([fb58c16](AztecProtocol/aztec-packages@fb58c16))
* **ci:** Fix CI to create baseline gate reports
([#11055](AztecProtocol/aztec-packages#11055))
([e2f6905](AztecProtocol/aztec-packages@e2f6905))
* Clean up proof lengths and IPA
([#11020](AztecProtocol/aztec-packages#11020))
([800c834](AztecProtocol/aztec-packages@800c834))
* Disable noir contracts tests until stabilized
([#11047](AztecProtocol/aztec-packages#11047))
([a76b52e](AztecProtocol/aztec-packages@a76b52e))
* Fix customTags is not iterable in e2e-prover-full
([#11057](AztecProtocol/aztec-packages#11057))
([f35094f](AztecProtocol/aztec-packages@f35094f))
* Fix invalid random log id
([#11076](AztecProtocol/aztec-packages#11076))
([b1b67b0](AztecProtocol/aztec-packages@b1b67b0))
* Fix write_recursion_inputs flow in bootstrap
([#11080](AztecProtocol/aztec-packages#11080))
([cd5a615](AztecProtocol/aztec-packages@cd5a615))
* Hide note_hashes log
([#11059](AztecProtocol/aztec-packages#11059))
([d9a14d2](AztecProtocol/aztec-packages@d9a14d2))
* Let IndexedTreeLeafPreimage have LeafPreimage as a parent trait
([#10913](AztecProtocol/aztec-packages#10913))
([496a55a](AztecProtocol/aztec-packages@496a55a))
* Load in the big dashboard during metrics install
([#11007](AztecProtocol/aztec-packages#11007))
([f6f2c12](AztecProtocol/aztec-packages@f6f2c12))
* New test that you can register, deploy, and call a public function all
in one tx
([#11045](AztecProtocol/aztec-packages#11045))
([5e3183c](AztecProtocol/aztec-packages@5e3183c))
* Pass fn signatures
([#10849](AztecProtocol/aztec-packages#10849))
([a2c4e98](AztecProtocol/aztec-packages@a2c4e98))
* Patch jest to not use JSON serialization in message passing ci3
([#10964](AztecProtocol/aztec-packages#10964))
([d08f540](AztecProtocol/aztec-packages@d08f540))
* Refactor tail public inputs
([#11031](AztecProtocol/aztec-packages#11031))
([4ed1530](AztecProtocol/aztec-packages@4ed1530))
* Remove abi refs from publisher
([#10766](AztecProtocol/aztec-packages#10766))
([17d6802](AztecProtocol/aztec-packages@17d6802))
* Remove some instances of `--silence-warnings`
([#11071](AztecProtocol/aztec-packages#11071))
([ecbd59e](AztecProtocol/aztec-packages@ecbd59e))
* Renaming getIncomingNotes
([#10743](AztecProtocol/aztec-packages#10743))
([ffa7407](AztecProtocol/aztec-packages@ffa7407))
* Replace relative paths to noir-protocol-circuits
([7194a7e](AztecProtocol/aztec-packages@7194a7e))
* Replace relative paths to noir-protocol-circuits
([b00bd13](AztecProtocol/aztec-packages@b00bd13))
* Replace relative paths to noir-protocol-circuits
([c4fcbc0](AztecProtocol/aztec-packages@c4fcbc0))
* Replace relative paths to noir-protocol-circuits
([694343d](AztecProtocol/aztec-packages@694343d))
* Representing `TxHash` as `Fr`
([#10954](AztecProtocol/aztec-packages#10954))
([84e67ac](AztecProtocol/aztec-packages@84e67ac))
* Restore `prove_then_verify` test on `verify_rollup_honk_proof`
([#11018](AztecProtocol/aztec-packages#11018))
([79e289d](AztecProtocol/aztec-packages@79e289d))
* Unify honk verifier contracts
([#11067](AztecProtocol/aztec-packages#11067))
([9968849](AztecProtocol/aztec-packages@9968849))
* Update noir-bignum to v0.5.0
([#11066](AztecProtocol/aztec-packages#11066))
([bf10a5c](AztecProtocol/aztec-packages@bf10a5c))
* Updated aztec-spartan.sh and the README
([#11088](AztecProtocol/aztec-packages#11088))
([56128a6](AztecProtocol/aztec-packages@56128a6))
</details>

<details><summary>barretenberg: 0.69.1</summary>

##
[0.69.1](AztecProtocol/aztec-packages@barretenberg-v0.69.0...barretenberg-v0.69.1)
(2025-01-08)


### Features

* Acir formal proofs
([#10973](AztecProtocol/aztec-packages#10973))
([1cb7cd7](AztecProtocol/aztec-packages@1cb7cd7))
* Derive transcript structure between non-zk and zk flavors and between
Ultra and UltraKeccak
([#11086](AztecProtocol/aztec-packages#11086))
([48286c6](AztecProtocol/aztec-packages@48286c6))
* Fix commitments and openings of masking polynomials used in zk
sumcheck
([#10773](AztecProtocol/aztec-packages#10773))
([fc48dcc](AztecProtocol/aztec-packages@fc48dcc))
* Improve witness generation for cycle_group::batch_mul
([#9563](AztecProtocol/aztec-packages#9563))
([7da7f2b](AztecProtocol/aztec-packages@7da7f2b))


### Bug Fixes

* Add bytecode instances in reverse
([#11064](AztecProtocol/aztec-packages#11064))
([036496c](AztecProtocol/aztec-packages@036496c))
* Reset pc to 0 for next enqueued call in avm witgen
([#11043](AztecProtocol/aztec-packages#11043))
([44e4816](AztecProtocol/aztec-packages@44e4816))
* Update requests per call should be less than per tx
([#11072](AztecProtocol/aztec-packages#11072))
([da5e95f](AztecProtocol/aztec-packages@da5e95f))


### Miscellaneous

* **avm:** Handle specific MSM errors
([#11068](AztecProtocol/aztec-packages#11068))
([a5097a9](AztecProtocol/aztec-packages@a5097a9)),
closes
[#10854](AztecProtocol/aztec-packages#10854)
* **avm:** More column information in permutations
([#11070](AztecProtocol/aztec-packages#11070))
([8829f24](AztecProtocol/aztec-packages@8829f24))
* Clean up proof lengths and IPA
([#11020](AztecProtocol/aztec-packages#11020))
([800c834](AztecProtocol/aztec-packages@800c834))
* Fix write_recursion_inputs flow in bootstrap
([#11080](AztecProtocol/aztec-packages#11080))
([cd5a615](AztecProtocol/aztec-packages@cd5a615))
* Restore `prove_then_verify` test on `verify_rollup_honk_proof`
([#11018](AztecProtocol/aztec-packages#11018))
([79e289d](AztecProtocol/aztec-packages@79e289d))
* Unify honk verifier contracts
([#11067](AztecProtocol/aztec-packages#11067))
([9968849](AztecProtocol/aztec-packages@9968849))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@spalladino spalladino spalladino approved these changes
+1 more reviewer
@PhilWindle PhilWindle PhilWindle left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

prover-node should be ablet to recover after a crash by picking up txs from the p2p network

3 participants

@alexghr @spalladino @PhilWindle