Plug-in architecture for master keys

[hoangelos]

We have discovered Sops and like it. Some of our company uses PGP and others use an in-house KMS. It’s a proprietary KMS so there’s not much benefit of anyone except us. I’m sure if there were some sort of plugin architecture for the KMS it would benefit a wider audience.

[bookshelfdave]

https://github.com/hashicorp/go-plugin

[autrilla]

I am not against the idea, but I think it'd be pretty hard to come up with a generic API that all KMSs support. We already have a gRPC service for encryption/decryption, so we could theoretically use that... except we use a oneof for the key itself:

https://github.com/mozilla/sops/blob/master/keyservice/keyservice.proto#L3-L10

I suppose we could change that to https://developers.google.com/protocol-buffers/docs/proto3#any and have each "plugin" only handle the key types they know about.

[hoangelos]

Hardcoding the key types in that method is sub-optimal too, and not as flexible. If there's a way to do this just with the gRPC approach itself, we'd be game.

[autrilla]

Yeah, I agree. Having an opaque "KeyInformation" or similar would have been a better design than a oneof of all possible key types. But what's done is done, so we'd need to support the current protos too, because there will be people running outdated clients or servers.

The UX for this could get to okay-ish levels without any major changes to sops -- you could just run a separate keyservice (through systemd or whatever) that supports your custom KMS, and pass it as an argument to sops invocations.

[hoangelos]

@autrilla Can you provide any pointers to documents that would serve as a guide for this? Really, like what you all have done. This is much better than an encrypted tarball and a bunch of custom scripts that my current employer utilizes. Some teams have graduated to a custom approach that also does value encrypting, but it's very manual. So, I'm happy to have come across you guys.

[autrilla]

I'm not sure what kind of documentation you're looking for. Are you looking for help implementing the solution for this issue? Or documentation on how Mozilla uses sops internally?

[hoangelos]

The former. My team uses sops via PGP key, but we are prepping the way for another team that's going to need to use the internal KMS. So any documentation you can show or even examples on how to do a separate keyservice.

[autrilla]

Oh! The code for the current server is here:

https://github.com/mozilla/sops/blob/master/keyservice/server.go

And the server binary:

https://github.com/mozilla/sops/blob/master/cmd/sops/subcommand/keyservice/keyservice.go

You should be able to copy most of that into your own binary that handles whichever key type you want. You'll need to make some changes to the proto files and the server code to make it only handle key types it knows about. A quick way around it would be to define something like an ArbitraryKey type and add it to the oneof.

You can then run sops with --keyservice unix:///tmp/your_keyservice.sock (or wherever your keyservice listens on).