Repository navigation
F# Records not compatible with ML .NET #180
Description
Perhaps related to #92. Apparently ML .NET doesn't work with F# records! Is this perhaps to do with the fact that F# Records use a different naming convention to Classes for backing fields, and the ML .NET Library. If you use records, the library simply can't find the columns and complains of missing columns. If you port from records to classes with mutable properties, it doesn't fail with that error any longer.
However, mutable classes are pretty much non-idiomatic in F# - no one uses them for the sorts of data-bound workloads that you'll see with ML.
Instead of this:
type SentimentData =
{ [<Column(ordinal = "0")>] SentimentText : string
[<Column(ordinal = "1", name = "Label")>] Sentiment : float }
[<CLIMutable>]
type SentimentPrediction =
{ [<ColumnName "PredictedLabel">] Sentiment : bool }You'll have to use something like this monstrosity.
type SentimentData() =
[<Column(ordinal = "0"); DefaultValue>]
val mutable SentimentText : string
[<Column(ordinal = "1", name = "Label"); DefaultValue>]
val mutable Sentiment : float32
type SentimentPrediction() =
[<ColumnName "PredictedLabel"; DefaultValue>]
val mutable Sentiment : boolYou can't even use the member val shorthand syntax that F# provides for mutable get / set properties since the ColumnName attribute doesn't work with them. Plus, you lose all the standard features that Records bring such as lightweight syntax, easy creation, copy-and-update, immutability and structural equality.
I strongly recommend adding support for them by ensuring that whatever internal hydration logic that is currently coupled to classes supports records as well.
Activity
forki commentedon May 17, 2018
should this be part of unittests and samples? How can it bind to backing fields in the first place!?
isaacabraham commentedon May 17, 2018
The "backing fields" bit is an assumption on my part - normally
[<CLIMutable>]does the trick for libraries that want to use a default constructor and then set the properties using e.g. reflection, but it doesn't work here. So I assume that it's using some field-based reflection.Having said that, if we want a truly F#-friendly API then even
[<CLIMutable>]shouldn't be required.asthana86 commentedon May 22, 2018
@cartermp
isaacabraham commentedon May 23, 2018
@asthana86 This is a common issue when APIs are designed for use with C# and use e.g. reflection. I'm not sure if @cartermp will have any answers here - you probably need to look into the ML .NET codebase to see where and how it does the hydration.
terrajobst commentedon Jun 6, 2018
Assuming F# records are just producing get/set properties it’s not even just a problem for F# — public fields aren’t canonical C# code either. We gave the same feedback in the API review but it didn’t make the cut yet.
Presumably #254 would fix that then.
cartermp commentedon Jun 6, 2018
In practice, the trick for something like working with an ORM or equivalent piece of machinery that wants to do things to properties with reflection is ye olde
[<CLIMutable>]:Which effectively turns it into a property with
get/setso the underlying thingie can go to town on it.So I would expect #254 to fix this. That said, it would be nice if we didn't need to set that somehow.
isaacabraham commentedon Jun 7, 2018
@terrajobst They produce getter-only properties (with backing fields that are generated with a different convention to C# backing fields).
You can avoid the need for
[<CLIMutable>]completely, it's not that difficult - F# Records without the attribute always have a public constructor with an argument for each property, and this is (AFAIK) done in a deterministic order (although I can't remember what that order is!).terrajobst commentedon Jun 7, 2018
I see. Supporting immutable types might be harder in the engine. @glebuk would know for sure.
isaacabraham commentedon Jun 7, 2018
I would hope that the library doesn't modify data that you supply to it! :-)
isaacabraham commentedon Jun 7, 2018
To be honest something else we discussed was to allow the caller to supply data to the library rather than the library hydrating data itself. This is kind of a prerequisite for in memory data support anyway. At that point it should not matter whether it's a record or not.
But that's kind of a separate thing.
voronoipotato commentedon Jun 14, 2018
Even if it was "harder" I don't understand why you couldn't construct a mutable from the record, then construct a new record from the final result of the mutation.
record -> mutable ~> mutable -> record
Creating a kind of "sh#t sandwich" . No judgement intended it's just a euphemism for wrapping something you don't want, with two things you do.
isaacabraham commentedon Jun 14, 2018
@voronoipotato that would probably be even more work as it would still mean creating initial records with default values for all the fields and then modifying them later manually by creating new records. I think either use the CLIMutable approach, which is probably easier but sub-optimal, or create it based on the ctor.
dsyme commentedon Jul 30, 2018
#254 is definitely the underlying problem here.
@asthana86 Who is the expert on the schema mapping code? If we wanted to allow get/set properties (rather than just public fields) as the basis for schema mappings then who should we talk to? Thanks
dsyme commentedon Jul 30, 2018
@terrajobst re this:
I could go ahead and do the work to allow get/set properties but I'd need it carefully reviewed. Do you know who is the right person in the ML.NET team to talk to about this? Thanks
dsyme commentedon Jul 30, 2018
I searched for uses of "GetFields" and here's what I see:
Relevant:
Not relevant:
isaacabraham commentedon Jul 30, 2018
Any chance of getting ctor support as well to avoid the need for CLIMutable?
dsyme commentedon Jul 30, 2018
@isaacabraham I'm not sure. First things first.
dsyme commentedon Jul 31, 2018
@isaacabraham Some notes on immutable F# records:
For immutable records, a schema mapper or serialization framework really has no choice but to work with the "internal" representation of the record, i.e. the hidden, private fields. This is what binary serialization does, for example. This is because there are no setter properties on those immutable records, so there's really no other choice: the IDataView schema mapper works by setting fields one at a time on target objects and there is no way we could convert it to use the immutable object constructor for example.
However inputs to ML.NET currently uses field/property names in string form, e.g.
"SentimentText"here:So, if the ML.NET IDataView schema mapper is dealing with internal fields (e.g. private field
SentimentText@used by F#), then it would also have to know about the relationship between internal and external names, i.e.It's possible but F# emits no specific metadata about this relationship, so the rule would have to be F#-specific. It's a stretch but not impossible. However would require more subtle coding than I'm willing to try for now, so CLIMutable is a reasonable stepping point.
isaacabraham commentedon Jul 31, 2018
Indeed, CLIMutable is a great step forward. But constructor support seems like it's a no-go because of how schema mapping in ML .NET works internally i.e. setting fields one at a time? Which, as you say, leaves just some custom "F# field mapping" based on convention as the only way to remove CLIMutable.
It's definitely not the end of the world leaving it there, but it would be nice to not require it.
dsyme commentedon Aug 2, 2018
Fixed in #616
charlesroddie commentedon Feb 2, 2019
To me this also looks monstrous. Magic strings everywhere, even ints represented as strings, loss of compiler helpfulness with the reliance on annotations.
Can this be done in ML.Net in a way that is not reflection-based?
isaacabraham commentedon Feb 3, 2019
It's been a while since I looked at this but can you now actually access ML net without being coupled to the data access layer? If so, that would ideally remove all this stuff and we could just worry about that ourselves.
charlesroddie commentedon Feb 7, 2019
Two ways to do this!