Skip to content

Add a step counter to RunLimits - #352

Open
saoirse-a wants to merge 3 commits into
eclipse-biscuit:mainfrom
saoirse-a:saoirse/step-counter
Open

saoirse-a wants to merge 3 commits into
eclipse-biscuit:mainfrom
saoirse-a:saoirse/step-counter

Conversation

@saoirse-a

Copy link
Copy Markdown
Contributor

This adds support for limiting the number of logical "steps" taken during biscuit authorization, as a way of limiting evaluation runtime without relying on wall clock time.

There are basically two ways a step is counted:

  1. When a candidate fact is found to support a predicate in a rule, this counts as a step. (To make it so the amount of work this takes is more consistent, Datalog evaluation has changed to store facts in a hash map keyed by the fact's name and arity; this stops us from iterating every fact in the world to find candidates).
  2. When an operation is executed against some operand(s), this counts as a step.

Steps are persisted through authorization so the same limit is used for rule evaluation and for all the checks and policies that run against the resolved world.

As a side change, we make the contents of the Datalog module mostly private to avoid leaking internal implementation details in the public API.

Comment thread biscuit-auth/src/format/schema.proto Resolved
@divarvel

divarvel commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

This generally LGTM except the new required field on snapshots. The two failing jobs are due to a formatting issue in the prost-generated file.

I’ll need to review the query parts carefully to make sure the remaining step counts is carried over after fact generation like max_time is.

We’ll also need to justify the default step count, since it will be silently added to future evaluation through default. In any case it will need a thorough explanation in the release notes, but weighing the new default against a couple use cases, including the samples suite, will be important (we can also merge as is, try it out in a few downstream projects and adjust defaults here before cutting a release)

@saoirse-a

Copy link
Copy Markdown
Contributor Author

We’ll also need to justify the default step count, since it will be silently added to future evaluation through default. In any case it will need a thorough explanation in the release notes, but weighing the new default against a couple use cases, including the samples suite, will be important (we can also merge as is, try it out in a few downstream projects and adjust defaults here before cutting a release)

Agreed, I have no basis for deciding what the "correct" default value would be and chose something arbitrary.

@divarvel

divarvel commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

I think snapshots should carry the elapsed steps, as they carry elapsed time. I can handle that in a follow-up PR as needed.

While experimenting on this idea, it appeared that the number of steps is not deterministic due to the use of hash-based data structures. Since there was care about deterministic step counting, I think that’s something we want to fix. We already use BTree-based data structures for deterministic ordering in the codebase. Note: this would make a given implementation deterministic, but making it consistent over implementations would be harder (that would require defining a well-defined order across implementations and making sure each implementation walks over trees in the same direction)

@divarvel

divarvel commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

I’ve run the samples suite with a patch that keeps track of elapsed steps, they took between 1 and 72 steps. The example from https://doc.biscuitsec.org/recipes/role-based-access-control.html runs in 60 steps.

I’ll run more tests on more realistic test cases, but I think 1000 is a comfortable default.

I don’t know how far we can go on deterministic step tracking across implementations. Fact generation is exhaustive, but checks and policies can early abort, so a failing candidate being at the front or at the back of a list can have a visible effect.

@divarvel

Copy link
Copy Markdown
Contributor

saoirse-a#2 explores what it could look like

@saoirse-a

saoirse-a commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

I don’t know how far we can go on deterministic step tracking across implementations. Fact generation is exhaustive, but checks and policies can early abort, so a failing candidate being at the front or at the back of a list can have a visible effect.

Agreed. The amount of work it takes is non-deterministic unless you define a specific order that evaluation is supposed to run in, which seems unrealistic to me.

We'll need to specify this feature in a way that a particular run in a particular implementation is permitted to take fewer steps so long is it actually does less work. This could mean either because the implementation has some optimization or because the order it happened to examine candidates in in this run took fewer steps.

This also means setting the step limit anywhere near where a legitimate token could hit it could mean non-deterministic failure, so it should be set pretty far above what real tokens look like.

All of this is also true of the time limit of course, though the time limit has the added non-determinism added by physical hardware, language runtime, etc.

If the highest any in the test corpus get is 72, 1000 doesn't seem like a bad default, some of those have more complex rule relations than I think would be common.

Instead of iterating every visible fact to find candidate matches when
querying, we index the FactSet by (name, arity). This makes finding
candidate matches O(1) instead of O(facts).

Move the step counter up so that we could a step for each candidate we
encounter.

This has the result that each step takes a roughly constant amount of
work, rather than taking an amount of work determined by the number of
facts there are. This makes it a more reliable limit on the amount of
work performed during authorization.

Also make most of the contents of datalog/mod.rs pub(crate) to avoid
internal implementation details being part of the public API contract.
@divarvel
divarvel force-pushed the saoirse/step-counter branch from 70a12ea to 3c21c94 Compare November 5, 2026 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants