Skip to content

feat: unify all acir recursion constraints based on RecursionConstraint and proof_type - #7993

Merged
ledwards2225 merged 24 commits into
masterfrom
lde/proof_type
Aug 15, 2024
Merged

feat: unify all acir recursion constraints based on RecursionConstraint and proof_type#7993
ledwards2225 merged 24 commits into
masterfrom
lde/proof_type

Conversation

@ledwards2225

@ledwards2225 ledwards2225 commented Aug 14, 2024

Copy link
Copy Markdown
Contributor

This PR moves us towards a model where the type of recursive verifier (plonk/honk, eventually IVC) can be specified from noir by setting a proof_type constant input to std::verify_proof(). The new mechanism has been integrated into the verify_honk_proof test program but not yet into the protocol circuits. (The method defaults to Plonk recursion so this is not a breaking change).

Other changes/updates:

  • All types of recursion are specified through RecursionConstraint (no more HonkRecursionConstraint)
  • Move handling of recursion_constraints and honk_recursion_constraints into individual methods process_plonk/honk_recursion_constraints() in acir_format for greater clarity. (More cleanup along these lines to come)
  • Move the gate count tracking functionality to a class GateCounter

@ledwards2225 ledwards2225 self-assigned this Aug 14, 2024
Comment thread barretenberg/cpp/src/barretenberg/dsl/acir_format/acir_to_constraint_buf.cpp Outdated
@AztecBot

ghost commented Aug 15, 2024

Copy link
Copy Markdown
Collaborator

Benchmark results

Metrics with a significant change:

  • avm_simulation_time_ms (Token:mint_public): 43.9 (-39%)
  • avm_simulation_time_ms (Token:assert_minter_and_mint): 67.8 (+69%)
  • avm_simulation_time_ms (Token:transfer_public): 21.4 (-48%)
Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Proof generation

Each column represents the number of threads used in proof generation.

Metric 1 threads 4 threads 16 threads 32 threads 64 threads
proof_construction_time_sha256_ms 5,777 1,561 (+1%) 708 (-1%) 762 (+2%) 765 (+1%)
proof_construction_time_sha256_30_ms 11,879 (+1%) 3,184 1,408 (-1%) 1,439 1,478
proof_construction_time_sha256_100_ms 45,359 12,067 5,443 (-1%) 5,394 (-1%) 5,357 (-1%)
proof_construction_time_poseidon_hash_ms 78.0 34.0 34.0 58.0 (-3%) 87.0 (-1%)
proof_construction_time_poseidon_hash_30_ms 1,536 422 202 232 (+1%) 268
proof_construction_time_poseidon_hash_100_ms 5,659 1,516 678 733 (-2%) 751 (+1%)

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric 4 txs 8 txs 16 txs
l1_rollup_calldata_size_in_bytes 4,324 7,844 14,852
l1_rollup_calldata_gas 49,708 92,458 177,644
l1_rollup_execution_gas 1,383,157 2,130,875 3,958,100
l2_block_processing_time_in_ms 243 (-7%) 444 (-1%) 808 (-3%)
l2_block_building_time_in_ms 9,378 (+1%) 18,185 36,405 (+1%)
l2_block_rollup_simulation_time_in_ms 9,377 (+1%) 18,184 36,405 (+1%)
l2_block_public_tx_process_time_in_ms 7,890 16,628 34,820 (+1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 8 txs.

Metric 3 blocks 5 blocks
node_history_sync_time_in_ms 2,966 (-1%) 3,771 (-3%)
node_database_size_in_bytes 12,623,952 16,728,144
pxe_database_size_in_bytes 16,254 26,813

Circuits stats

Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Circuit simulation_time_in_ms witness_generation_time_in_ms input_size_in_bytes output_size_in_bytes proving_time_in_ms
private-kernel-init 98.6 392 (+2%) 21,673 44,858 N/A
private-kernel-inner 165 (+1%) 687 (-2%) 72,309 45,005 N/A
private-kernel-reset-tiny 510 (+1%) 871 65,974 44,844 N/A
private-kernel-tail 209 (+1%) 161 (+1%) 50,728 52,256 N/A
base-parity 5.50 (-1%) N/A 160 96.0 N/A
root-parity 33.4 (+1%) N/A 69,084 96.0 N/A
base-rollup 2,889 (+1%) N/A 187,817 664 N/A
root-rollup 38.5 N/A 54,525 716 N/A
public-kernel-setup 97.4 N/A 103,760 71,222 N/A
public-kernel-app-logic 104 (+1%) N/A 103,599 71,222 N/A
public-kernel-tail 575 N/A 409,190 16,414 N/A
private-kernel-reset-small 541 (+11%) N/A 66,085 45,629 N/A
private-kernel-tail-to-public 817 (-4%) 633 (-1%) 492,960 1,697 N/A
public-kernel-teardown 91.1 N/A 104,005 71,222 N/A
merge-rollup 19.5 (+2%) N/A 35,742 664 N/A
undefined N/A N/A N/A N/A 67,217

Stats on running time collected for app circuits

Function input_size_in_bytes output_size_in_bytes witness_generation_time_in_ms
ContractClassRegisterer:register 1,344 11,731 346
ContractInstanceDeployer:deploy 1,408 11,731 18.2
MultiCallEntrypoint:entrypoint 1,920 11,731 425 (-1%)
FeeJuice:deploy 1,376 11,731 395 (+2%)
SchnorrAccount:constructor 1,312 11,731 105 (-1%)
SchnorrAccount:entrypoint 2,304 11,731 433 (-1%)
Token:privately_mint_private_note 1,280 11,731 139 (+3%)
FPC:fee_entrypoint_public 1,344 11,731 27.0 (+1%)
Token:transfer 1,312 11,731 281
Benchmarking:create_note 1,344 11,731 102 (-4%)
SchnorrAccount:verify_private_authwit 1,280 11,731 27.9 (+2%)
Token:unshield 1,376 11,731 563 (+1%)
FPC:fee_entrypoint_private 1,376 11,731 763 (+1%)

AVM Simulation

Time to simulate various public functions in the AVM.

Function time_ms bytecode_size_in_bytes
FeeJuice:_increase_public_balance 55.4 (-2%) 8,139
FeeJuice:set_portal 13.1 (-3%) 2,362
Token:constructor 87.5 (+4%) 31,107
FPC:constructor 55.5 (+1%) 22,380
FeeJuice:mint_public 38.4 (+1%) 6,150
Token:mint_public ⚠️ 43.9 (-39%) 11,720
Token:assert_minter_and_mint ⚠️ 67.8 (+69%) 8,028
AuthRegistry:set_authorized 45.4 (+3%) 4,537
FPC:prepare_fee 245 (-2%) 8,812
Token:transfer_public ⚠️ 21.4 (-48%) 47,374
FPC:pay_refund 63.3 (+7%) 12,114
Benchmarking:increment_balance 975 7,450
Token:_increase_public_balance 38.2 (-14%) 8,960
FPC:pay_refund_with_shielded_rebate 66.3 (-4%) 12,663

Public DB Access

Time to access various public DBs.

Function time_ms
get-nullifier-index 0.165 (-2%)

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric 1 leaves 16 leaves 64 leaves 128 leaves 256 leaves 512 leaves 1024 leaves
batch_insert_into_append_only_tree_16_depth_ms 2.18 3.92 (+1%) N/A N/A N/A N/A N/A
batch_insert_into_append_only_tree_16_depth_hash_count 16.8 31.7 N/A N/A N/A N/A N/A
batch_insert_into_append_only_tree_16_depth_hash_ms 0.112 0.111 (+1%) N/A N/A N/A N/A N/A
batch_insert_into_append_only_tree_32_depth_ms N/A N/A 11.6 (+1%) 18.2 (-5%) 32.3 (+1%) 60.2 (+2%) 113 (-1%)
batch_insert_into_append_only_tree_32_depth_hash_count N/A N/A 95.9 159 287 543 1,055
batch_insert_into_append_only_tree_32_depth_hash_ms N/A N/A 0.111 0.106 (-5%) 0.105 (+1%) 0.104 (+2%) 0.102 (+1%)
batch_insert_into_indexed_tree_20_depth_ms N/A N/A 14.5 (+1%) 25.9 (+1%) 43.6 85.0 (+3%) 161 (-4%)
batch_insert_into_indexed_tree_20_depth_hash_count N/A N/A 109 207 355 691 1,363
batch_insert_into_indexed_tree_20_depth_hash_ms N/A N/A 0.110 (+1%) 0.104 0.106 (+1%) 0.106 (+3%) 0.102 (-4%)
batch_insert_into_indexed_tree_40_depth_ms N/A N/A 16.6 (+1%) N/A N/A N/A N/A
batch_insert_into_indexed_tree_40_depth_hash_count N/A N/A 132 N/A N/A N/A N/A
batch_insert_into_indexed_tree_40_depth_hash_ms N/A N/A 0.106 (+1%) N/A N/A N/A N/A

Miscellaneous

Transaction sizes based on how many contract classes are registered in the tx.

Metric 0 registered classes 1 registered classes
tx_size_in_bytes 64,779 668,997

Transaction size based on fee payment method

| Metric | |
| - | |

* @brief Test recursive honk recursive verification
*
*/
TEST_F(AcirIntegrationTest, DISABLED_HonkRecursion)

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a convenient way to debug honk recursion without getting bb involved

@ledwards2225
ledwards2225 marked this pull request as ready for review August 15, 2024 19:27
std::vector<EcAdd> ec_add_constraints;
std::vector<RecursionConstraint> recursion_constraints;
std::vector<HonkRecursionConstraint> honk_recursion_constraints;
std::vector<RecursionConstraint> honk_recursion_constraints;

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually we can probably move to a model where there is only one vector of RecursionConstraints and they are handled accordingly based on proof_type but that's going to require a larger refactor and I've already strayed a bit from the original intent of this PR :)

ghost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

// final recursion output.
builder.set_recursive_proof(current_aggregation_object);
}
};

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume all of this was just moving around and not any actual changes?

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct

std::string test_name = "verify_honk_proof"; // arbitrary program with RAM gates
auto acir_program = get_program_data_from_test_file(
test_name,
/*honk_recursion=*/false); // WORKTODO: TODO(https://github.com/AztecProtocol/barretenberg/issues/1013):

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this false here?

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its false because with this new pattern, the honk_recursion flag is only used for the "this circuit will be recursively verified with honk" use case. The verifier is set to a honk verifier via the noir verify_proof_with_type call directly

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a comment to this effect


AggregationObjectIndices create_honk_recursion_constraints(Builder& builder,
const HonkRecursionConstraint& input,
const RecursionConstraint& input,

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should add an assert in this function that it's actually taking in a honk recursion constraint

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea. added

ghost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Some very minor nits that are non-blocking

) {
std::verify_proof(verification_key, proof, public_inputs, key_hash);
// Proof type input specified as 1 to indicate honk
std::verify_proof_with_type(verification_key, proof, public_inputs, key_hash, 1);

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: probably could make this a global like global HONK_IDENTIFIER, this is non-blocking though

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread noir/verify_honk_proof/src/main.nr
// the only means for setting the proof type. use of honk_recursion flag in this context can go away
// once all noir programs (e.g. protocol circuits) are updated to use the new pattern.
if (honk_recursion && proof_type_in != HONK_RECURSION) {
// info("WARNING: Recursion type is not being specified correctly via noir verify_proof()!");

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// info("WARNING: Recursion type is not being specified correctly via noir verify_proof()!");

?

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having prints here seemed to make the gate_counts tests on CI complain?

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was suggesting to delete this as I was surprised to see it left here with no comment

};
af.recursion_constraints.push_back(c);
af.original_opcode_indices.recursion_constraints.push_back(opcode_index);
info("Invalid PROOF_TYPE in RecursionConstraint!");

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: could probably print the valid proof types here vs. what was specified but this is a nit and can come later. In general, before moving to honk as the main public facing backend for noir we probably need to improve some of the error messages

ghost Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave this one for a follow on since I'll be touching this code a lot in coming weeks

Comment thread barretenberg/cpp/src/barretenberg/dsl/acir_format/recursion_constraint.hpp Outdated
Comment thread barretenberg/cpp/src/barretenberg/dsl/acir_format/acir_format.cpp Outdated
ledwards2225 and others added 5 commits August 15, 2024 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants