IFormatProvider in TextLoader #126

Activity

1. veikkoeeva commented on May 11, 2018

   

   veikkoeeva

   on May 11, 2018

   Last edited by veikkoeeva

   Contributor

   More actions

   Cross-referencing since related: #74 and #109. And since this is a real problem, feeding thoughts: https://www.nuget.org/packages?q=CSV.
2. shauheen

   assigned

   codemzs

   on May 11, 2018
3. veikkoeeva commented on May 11, 2018

   

   veikkoeeva

   on May 11, 2018

   Last edited by veikkoeeva

   Contributor

   More actions

   @ddobric You can see it at

   machinelearning/src/Microsoft.ML/TextLoader.cs

   Line 16 in 3780923

   public class TextLoader<TInput> : ILearningPipelineLoader

   .

   I wonder if it were possible to do something like this

       /// <summary>
       /// A canonical interface for deserializer.
       /// </summary>
       public interface IDeserializer
       {
          /// <summary>
          /// Can this provider stream data.
          /// </summary>
          bool CanStream { get; }
   
          /// <summary>
          /// Deserializes the given data.
          /// </summary>
          /// <param name="data">The data to be serialized.</param>
          /// <param name="destination">The destination type.</param>
          /// <returns>The deserialized object.</returns>
          object Deserialize(object data, Type destination);
   
   /// <summary>
   /// Deserializes the given data from a stream.
   /// </summary>
   /// <param name="dataStream">The stream from which to serialize.</param>
   /// <param name="destination">The destination type.</param>
   /// <returns>The deserialized object.</returns>
   object Deserialize(Stream dataStream, Type destination);
   
   /// <summary>
   /// Deserializes the given data from a text stream.
   /// </summary>
   /// <param name="reader">The text stream from which to serialize.</param>
   /// <param name="destination">The destination type.</param>
   /// <returns>The deserialized object.</returns>
   object Deserialize(TextReader reader, Type destination);
   }
   
   /// <summary>
       /// A default JSON deserializer.
       /// </summary>
       [DebuggerDisplay("CanStream = {CanStream}")]
       public class DefaultJsonDeserializer: IDeserializer
       {
           public JsonSerializer Deserializer { get; }
           public JsonSerializerSettings Settings { get; }
           public bool CanStream { get; } = true;
   
           public DefaultJsonDeserializer(JsonSerializerSettings settings)
           {
               if(settings == null)
               {
                   throw new ArgumentNullException(nameof(settings));
               }
               Deserializer = JsonSerializer.Create(settings);
               Settings = settings;
           }
   
           public object Deserialize(Stream dataStream, Type destination)
           {
               using(var streamReader = new StreamReader(dataStream))
               {
                   using(var reader = new JsonTextReader(streamReader))
                   {
                       return Deserializer.Deserialize(reader, destination);
                   }
               }
           }
   
           public object Deserialize(object data, Type destination)
           {
               return JsonConvert.DeserializeObject((string)data, destination, Settings);
           }
   
           public object Deserialize(TextReader reader, Type destination)
           {
               using(var jsonReader = new JsonTextReader(reader))
               {
                   return Deserializer.Deserialize(jsonReader, destination);
               }
           }
       }
   }
   

   The real case would be more complicated and the interface could support, say IAsyncEnumerable<T>. The intention would be to make canonical interfaces for input (and output) and make concrete implementations in arbitrary ways, users could bring their own (de)serializers. This might be possible in the current system. I don't know what is schema, for instance, in MachineLearning.NET. :)
4. sfilipi commented on Nov 27, 2018

   

   sfilipi

   on Nov 27, 2018

   Member

   More actions

   @ddobric you can specify the column separator in the TextLoader, in ML.Net.
   The TextLoader's default separator is ' \t'.

   The culture does not play a factor.
5. rustd

   added

   area-Microsoft.CSharp

   on Dec 4, 2018
6. shauheen

   removed

   area-Microsoft.CSharp

   on Dec 5, 2018
7. codemzs commented on Jun 30, 2019

   

   codemzs

   on Jun 30, 2019

   Member

   More actions

   Please try with the new API in post 1.0 release and let me know if you still see this issue.
8. codemzs

   closed this as completedon Jun 30, 2019
9. ghost locked as resolved and limited conversation to collaborators on Mar 30, 2022