Data tracks schema metadata

[ladvoc]

Closes BOT-318

[changeset-bot]

🦋 Changeset detected

Latest commit: 5a3a2dd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
github.com/livekit/protocol	Patch
@livekit/protocol	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[boks1971]

Probably hold this till we have some form of implementation?

Should AddDataTrack request fail if there is an existing schema id, but the definition bytes do not match? If so, need to add another error reason for this case.

Should schemas be per publisher? If that is the case, GetSchema should include participant.

If not per publisher, (i. e. per room), becomes trickier to coordinate in a distributed room, so would be good to avoid that.

[boks1971]

Should AddDataTrack request fail if there is an existing schema id, but the definition bytes do not match? If so, need to add another error reason for this case.

Also a good callout. The question is, is there a real use case where a schema would need to get redefined during the same session? @stephen-derosa, @alan-george-lk what do you think?

since a DataTrack is unique per participant, i would think that we should fail if the same schema id is used but the bytes dont match. This would likely be a an application side error since it means the same participant has two different message definitions for the same id

sounds good. Should add an error reason for this.

[boks1971]

If not per publisher, (i. e. per room), becomes trickier to coordinate in a distributed room, so would be good to avoid that.

I see, so, since this will live in an in-memory store and participants can be connected to different nodes, the schema will be stored on the node the publisher is connected to? Therefore, when a GetSchema request is made, the node the requesting participant is connected to may need to make an RPC to the other to get the schema definition. Just want to make sure I have a clear picture here.

Most probably, this will be propagated to other nodes a la DataTrackInfo. The challenge is more on AddDataTrack side. The schema has to be synced to all nodes so that when a publisher adds a schema it can be checked for collisions etc. That leads to races as cross-region sync could race with some other user publishing. So, per publisher makes that simpler as a participant is connected to one node. And GetSchema specifying publisher identity makes it cleaner to lookup the specific publisher.

Still have to think about migration cases, i. e. should clients add the schemas to SyncState?

[ladvoc]

@boks1971, we thought though the edge cases of having schemas being scoped to specific participants and the conclusion is this will work with some minor caveats. Question: should this work based on participant ID or identity for the purposes of the protocol?