Address REP-0157 peer reviews

[hidmic]

Follow-up to #22. This patch addresses most comments that were left standing. I believe we agree on the substance, we have time to discuss about the form. I bet some nuances will come up during implementation.

FYI @wjwwood @MiguelCompany

[hidmic]

@wjwwood whenever you have a minute.

[tfoote]

To confirm you'd like to update the posted draft with a merge or old this here for more review inline before merging?

[hidmic]

@tfoote I think it'd be good for @wjwwood @MiguelCompany and perhaps @jmachowinski @allenh1 to take a look before merging. Or merge so that it's easier to read? I don't mind either way.

[jmachowinski]

Hey, I found time to go through the REP and I got some thoughts on it.

From my point of view we had a shift in the IDL generation from the pure description of the type, to also generate the serialization format. E.g. rosidl_typesupport_fastrtps_cpp is a good example of this.
rosidl_buffer_backends is a step in a similar direction, and this REP feels like it actually aims to do the same thing.

My question would now be, if it makes more sense to try to standardize the interface here more than the taken approach.
What this REP introduces is a new way to serialize messages that are compatible. But it focuses hard on the serialization method used, and not on how an RWM can actually use this method, or over which interface it is exposed.

My thoughts are going more in the direction of specifying a new interface that RMWs can use to figure out, if the message that got passed down to them is already in shm, or if it is a flatbuffer that does not need serialization.
The idea goes into the direction, that one can specify multiple serialization methods, in respect to the properties of the endpoint.
E.g.
I got a message in shm, and the endpoint is on the same machine
-> Return a serialization function that generates short metadata message, that the
endpoint can use to pull the message from some shm manager

I got a message in shm, and the endpoint is on the network
-> Return a serialization function (existing)

The second point I am wondering about is, if we should generate multiple C / C++ types in respect to the used idl_generator.
E.g. allow something like directly using cap'n proto messages as long as we provide a serialization function to the 'default' representation in order to allow a (slow) backwards compatible path.

[hidmic]

What this REP introduces is a new way to serialize messages that are compatible. But it focuses hard on the serialization method used, and not on how an RWM can actually use this method, or over which interface it is exposed.

Yes and no. What we are proposing here is to decouple message structure and semantics from memory layout. A useful side effect of this is that if you align that memory layout with a memory-addressable serialization format you get zero copy on transfer. But that's a choice that an specific type support makes. Other type supports can still make other choices. Granted, the middleware doesn't have a direct say, but it does choose the type support.

The idea goes into the direction, that one can specify multiple serialization methods, in respect to the properties of the endpoint.

Hmm, isn't this possible already with a type support choice?

I got a message in shm, and the endpoint is on the same machine
-> Return a serialization function that generates short metadata message, that the
endpoint can use to pull the message from some shm manager

Oh, I see. You mean properties of the message itself. It gets hairy. You are taking some agency away from the middleware. We would need to define new middleware-level interfaces to do this. Similar to buffer backends.

If I follow your logic though, you are essentially questioning the rmw design. We have a monolith that can do pubsub and rpc in some implementation defined way. Instead, it could feature a smaller core that interfaces with external (reusable?) modules to decide how to transport data. You could think of buffer backends as the first of such modules.

I mean, it sounds sensible and it should be doable, but it's a big leap, and IMHO orthogonal to this REP.

The second point I am wondering about is, if we should generate multiple C / C++ types in respect to the used idl_generator.

Let me walk through the thought experiment. It's 2028. We have protobuf, rosidl, flatbuffer, and cap'n proto equivalent geometry_msgs/msg/Pose messages. Do they all derive from the same .msg file? Why would someone do protobuf like that? Do each stick to their own IDLs? What keeps them in line? Let's say someone likes protobuf and chooses protobuf. Is it interoperable with all the other ROS code out there that chose a different messaging system? Surely they are, otherwise we would have ROS tribes that can't talk to each other. But then, a protobuf pub has to talk to a flatbuffer sub. Would there be custom serdes code?

TBH I think that multi-messaging system support falls square in the bucket of cool features that we shouldn't build. It's complex to build and maintain, and it creates friction that the vanilla IDLs themselves do not have (custom serdes code, partial feature support), which people will complain about, unless we compromise the cohesiveness of the ROS ecosystem. For those that need integration with external messaging systems, REP-2007 already serves that purpose. We just need to implement it in more places than just rclcpp.

I know, I know, I don't live under a rock, I understand the convenience of protoc or pip install flatbuffers, but I don't think the solution is "now ROS publishes Protobuf too!". The solution is to make rosidl a sensible choice, which is not right now. REP-0157 already gives it a feature that I've not seen any other messaging system offer: you can go dynamic like Protobuf or you can go constrained like Flatbuffers just the same. Next step is to burn all the CMake + EmPy and have a normal, pip installable rosidlc compiler, with extensions.