DataFrame (Microsoft.Data.Analysis) Tracking Issue

Summary

This issue tracks priorities and discussions around DataFrame improvements based on issues and feedback.

Microsoft.Data.Analysis Open Issue Query: https://github.com/dotnet/machinelearning/issues?q=is%3Aopen+is%3Aissue+label%3AMicrosoft.Data.Analysis

Work Items

Related Issues

Create DataFrames

Data Formatting

Data Sources

Combine DataFrames

Group DataFrames

Grouping
- Enable multi column group by for Dataframe #5724
- Improve support for Null values in DataFrame GroupBy #5691
Aggregate Functions
- Add Sort to GroupBy in DataFrame #5681

Summarize DataFrames

Descriptive statistics
PrimitiveDataFrameColumnComputations produce wrong result for Min/Max functions #5759

Handle Missing Data

Drop missing data
Impute / Fill missing data

DataTypes

Array / Vector / VBuffer

DateTime

Other

Misc

LINQ / EFCore?
DateTime operations

Can you please finish the dataframe? I realize it is not a high priority but this would be a great addition to .net even just for data-wrangling. Also, Is there a way to convert the dataframe to a C# List of a Custom Class?

I was wondering if there could be a LoadExcel Method for the dataframe for .xlsx or .xls files like Pandas has?

Can you please finish the dataframe? I realize it is not a high priority but this would be a great addition to .net even just for data-wrangling. Also, Is there a way to convert the dataframe to a C# List of a Custom Class?

Not built-in, but one way you might want to try doing it is by accessing the DF rows. Something like:

Thanks!

Can you please add first-class support for time series analysis? I see lots of outcry in the GH Issues.... First of all DateTime columns are unusable in VS code due to #5698

Hey, This is a great summary and a good list of features.

I am not sure why this doesn't seem to have higher priority. I can tell you that from an industry perspective, a proper dataframe solution - especially involving time series functionality - is a prerequisite for dotnet to be competetive in the analysis space. I can't leave python even if I wanted to.

Great list, hope to see this implemented!

Looks like neither DataFrame, nor IDataView, has an ability to convert types of vectors.
Developers can convert column of String type to Single type, but can't convert String[] to Single[].

I'm trying to use vectors of variable length as features. The data view with features.

Label  |  Feature1  |  Feature2
--------------------------------
1      |  5#15#25   |  25#75
2      |  10        |  100#65#0
3      |  115#90    |  5#80

Using pipeline estimator TokenizeIntoWords, I can split selected features into a vector of words.

var pipeline = Context
  .Transforms
  .Conversion
  .MapValueToKey("Output", "Label")
  .Append(Context.Transforms.Text.TokenizeIntoWords("Tokens", "Feature1", new[] { '#' }))

After this, how can I specify that Tokens is a vector of numbers, not strings? Also, how do I normalize this vector, considering that this is a row-based vector and standard ML transformers, like NormalizeMinMax, apply to column-based data?

The implementation is pretty bad, not thread-safe, not GC friendly, etc, but this is what would be nice to have available in DataFrame. Besides support for vector types, would be nice to add more flexible custom mapping - Converter that doesn't enforce to specify input and output type.

using Microsoft.ML;
using Microsoft.ML.Data;
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;

namespace DemoSpace
{
  public class View : IDataView
  {
    protected IDictionary<string, Func<object, object>> _cache = new Dictionary<string, Func<object, object>>();

    /// <summary>
    /// Accessors
    /// </summary>
    public virtual IList Items { get; set; }
    public virtual DataViewSchema Schema { get; set; }
    public virtual IDictionary<string, DataViewType> Columns { get; set; }
    public virtual Func<IEnumerator, DataViewSchema.Column, dynamic> Converter { get; set; }

    /// <summary>
    /// View implementation
    /// </summary>
    public virtual bool CanShuffle => false;
    public virtual long? GetRowCount() => Items.Count;
    public virtual DataViewRowCursor GetRowCursor(IEnumerable<DataViewSchema.Column> columns, Random seed = null) => new ViewCursor(this);
    public virtual DataViewRowCursor[] GetRowCursorSet(IEnumerable<DataViewSchema.Column> columns, int o, Random seed = null) => new[] { GetRowCursor(columns, seed) };

    /// <summary>
    /// Constructor
    /// </summary>
    /// <param name="columns"></param>
    /// <param name="items"></param>
    public View(IDictionary<string, DataViewType> columns, IList items)
    {
      var schemaBuilder = new DataViewSchema.Builder();

      foreach (var column in columns)
      {
        schemaBuilder.AddColumn(column.Key, column.Value);
      }

      Items = items;
      Columns = columns;
      Schema = schemaBuilder.ToSchema();
    }

    /// <summary>
    /// Converter
    /// </summary>
    /// <param name="enumerator"></param>
    /// <param name="column"></param>
    /// <returns></returns>
    public virtual dynamic ConverterImplementation(IEnumerator enumerator, DataViewSchema.Column column)
    {
      var name = column.Name;

      if (_cache.ContainsKey(name) is false)
      {
        _cache[name] = enumerator.Current.GetType().GetProperty(name).GetValue;
      }

      var value = _cache[name](enumerator.Current);

      if (value is string)
      {
        return $"{ value }".AsMemory();
      }

      if (value is IList)
      {
        var items = value as IList;
        var itemType = items.GetType().GetElementType();

        switch (true)
        {
          case true when Equals(itemType, typeof(int)): return new VBuffer<int>(items.Count, (items as IList<int>).ToArray());
          case true when Equals(itemType, typeof(float)): return new VBuffer<float>(items.Count, (items as IList<float>).ToArray());
          case true when Equals(itemType, typeof(double)): return new VBuffer<double>(items.Count, (items as IList<double>).ToArray() );
          case true when Equals(itemType, typeof(string)): return new VBuffer<ReadOnlyMemory<char>>(items.Count, (items as IList<string>).Select(o => $"{o}".AsMemory()).ToArray());
        }
      }

      return value;
    }

    /// <summary>
    /// Cursor
    /// </summary>
    public class ViewCursor : DataViewRowCursor
    {
      protected int _index = -1;
      protected View _view = null;
      protected IEnumerator _enumerator = null;

      /// <summary>
      /// Cursor implementation
      /// </summary>
      public override long Batch => 0;
      public override long Position => _index;
      public override DataViewSchema Schema => _view.Schema;
      public override bool IsColumnActive(DataViewSchema.Column column) => true;
      public override ValueGetter<DataViewRowId> GetIdGetter() => (ref DataViewRowId id) => id = new DataViewRowId();

      public ViewCursor(View view)
      {
        _view = view;
        _enumerator = _view.Items.GetEnumerator();
      }

      public override ValueGetter<TValue> GetGetter<TValue>(DataViewSchema.Column column)
      {
        return (ref TValue value) => value = _view.Converter(_enumerator, column);
      }

      protected override void Dispose(bool disposing) => base.Dispose(disposing);

      public override bool MoveNext()
      {
        if (_enumerator.MoveNext())
        {
          _index++;

          return true;
        }

        Dispose();

        return false;
      }
    }
  }
}

Usage example

var columns = new Dictionary<string, DataViewType>
{
  ["Id"] = NumberDataViewType.Int32,
  ["Name"] = TextDataViewType.Instance,
  ["Ints"] = new VectorDataViewType(NumberDataViewType.Int32),
  ["Doubles"] = new VectorDataViewType(NumberDataViewType.Double),
  ["Strings"] = new VectorDataViewType(TextDataViewType.Instance),
};

var items = new List<object>
{
  new { Id = 1, Name = "A", Doubles = new double[] { 10, 200 }, Ints = new int[] { 1, 2, 3 }, Strings = new string[] { "" } },
  new { Id = 2, Name = "B", Doubles = new double[] { 5, 20, 15 }, Ints = new int[] { 1 }, Strings = new string[] { "" } },
  new { Id = 3, Name = "C", Doubles = new double[] {}, Ints = new int[] { }, Strings = new string[] { "" } }
};

var view = new View(columns, items);

view.Converter = view.ConverterImplementation;  // Use predefined converter
view.Converter = (enumerator, column) =>        // Or define a custom one
{
  if (enumerator.Current is double)
  {
    return enumerator.Current.GetType().GetProperty(name).GetValue(enumerator.Current) as double;
  }

  return enumerator.Current;
}; 

var x = view.Preview();
var x1 = view.GetColumn<string>("Name");

@luisquintanilla

Hi @luisquintanilla , thank you for this overview. Would you consider adding my issue as well ?
#5652

Edit:
And also maybe this one
#5656

@integerman

Hi @integerman , there is an easier way to pull data out of a column.
As each column implements IEnumerable<T?> you can use

var dataX = ((SingleDataFrameColumn)df["Credit Amount"] ).ToArray();

However, there are several issues with this approach. First issue is related to using Single data type (not Double)) in your example. SingleDataFrameColumn stores float values, that you have to explicitly convert to doubles to pass them to ScottPlot. Second issue is that DataFrame allows to store Nulls, so actual type of dataX is float?[].
Actually, using Select( x => (double) x).ToArray() may fail with System.InvalidOperationException "Nullable object must have a value." in case of any Null value inside your column.

So my suggestion is:

Use DoubleDataFrameColumn instead of SingleDataFrameColumn
Write your own small extension method for IEnumerable<T?>

public static T[] ConvertToArray<T>(this IEnumerable<T?> enumerable)
    where T : struct
{
    return enumerable.Select(x => x ?? default).ToArray();
}

After that you will be able to simplify your code to:

var dataX = ((DoubleDataFrameColumn)df["Credit Amount"] ).ConvertToArray();
var dataY = ((DoubleDataFrameColumn)df["Age"] ).ConvertToArray();

var plt = new ScottPlot.Plot(300, 400);
plt.AddScatter(dataX, dataY);

I could not find a way to apply lambdas to DF or to results of GroupBy (like the Apply function in Pandas).
In particular, after a GroupBy the only way to apply lambdas to the groups and columns is to loop through the groups and add the new transformed rows into a new DF. Something like this:

        var dataFrameGroupBy = dataFrame.GroupBy<int>("Dimension1");
        
        var cols = new List<DataFrameColumn>
        {
            new Int32DataFrameColumn("Identifier"),
            new DoubleDataFrameColumn("Value1"),
            new DoubleDataFrameColumn("Value2"),
            new Int32DataFrameColumn("Dimension1"),
            new Int32DataFrameColumn("Dimension2")
        };
        
        var df = new DataFrame(cols);

        foreach (var group in dataFrameGroupBy.Groupings)
        {
            int id = 1;
            var value1= group.Max<DataFrameRow, double>(each => (double)each["Value1"]);
            var value2= group.Min<DataFrameRow, double>(each => (double)each["Value2"]);
            var key = group.Key;
            var dim2 = group.Max<DataFrameRow, int>(each => (int)each["Dimension2"]);

            df.Append(new KeyValuePair<string, object>[]{new("Identifier", id), new("Value1", value1), new("Value2", value2), new ("Dimension1", key), new ("Dimension2", dim2)}, inPlace: true);
        };

Is there a better was to achieve this?
Is there any plan to implement an Apply method?

@lucvalentino

Hi @lucvalentino. Dataframe columns have two methods that take a lambda and apply it to the column content: ApplyElementwise and Apply. However, as lambda function is generic, these method are defined on the level of the inheritors of the base DataFrameColumn class.

For example for PrimitiveDataFrameColumn<T>:

void ApplyElementwise(Func<T?, long, T?> func)
PrimitiveDataFrameColumn<TResult> Apply<TResult>(Func<T?, TResult?> func) where TResult : unmanaged

For ArrowStringDataFrameColumn:

Apply(Func<string, string> func)

For some reason StringDataFrameColumn and VBufferDataFrameColumn don't support such functionality.

ApplyElementwise applies lambba in place and Apply method returns new column as a result and allows result to have different underlying data type.

As regarding grouping, unfortunately it seems that current implementation of GroupBy class (result of the DataFrame.GroupBy method) doesn't allow to specify different aggregation functions to different columns. I think the simpliest way to achive expected result is to use code like this:

var dataFrameGroupBy = dataFrame.GroupBy<int>("Dimension1");
var max = dataFrameGroupBy.Max("Value1", "Dimension2");
var min = dataFrameGroupBy.Min("Value2");
var res = max.Merge<int>(min, "Identifier", "Identifier");

These implementation can be very CPU and memory consuming for large dataframe, so I absolutely agree with you that current GroupBy API requires redesign.

@asmirnov82

Thanks @asmirnov82 for your prompt reply and clarifications!

I saw those Apply and ApplyElementwise methods at column level, and also realized that they were not available for StringDataFrameColumn. But I thought there was a way to apply lambdas directly to DFs similarly to Pandas or PySpark.

It would be nice to make the Apply and ApplyElementwise available to all column types.

And yes, the GroupBy would need some redesign, with the current design the user gets somehow stuck after that operation.

But hey, thanks to all developers contributing to this project!

I know this project is basically abandoned but now that our robot friends can do most of the work is it worth setting them loose on the backlog and resurrecting this library?

@MgSam

@MgSam, this project isn't abandoned, meaningful contributions welcomed :-)

Triage: closing as outdated tracking issue.

Navigation Menu

Repository navigation

DataFrame (Microsoft.Data.Analysis) Tracking Issue #6144

Summary

Work Items

Related Issues

Create DataFrames

Data Formatting

Data Sources

Other

Reshape DataFrames

Filter / Sort DataFrames

Combine DataFrames

Group DataFrames

Summarize DataFrames

Handle Missing Data

DataTypes

Array / Vector / VBuffer

DateTime

Other

Misc

16 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DataFrame (Microsoft.Data.Analysis) Tracking Issue #6144

Description

Summary

Work Items

Related Issues

Create DataFrames

Data Formatting

Data Sources

Other

Reshape DataFrames

Filter / Sort DataFrames

Combine DataFrames

Group DataFrames

Summarize DataFrames

Handle Missing Data

DataTypes

Array / Vector / VBuffer

DateTime

Other

Misc

Activity

Bigwiggz commented on Mar 30, 2022

Bigwiggz commented on Mar 31, 2022

luisquintanilla commented on Mar 31, 2022

Bigwiggz commented on Apr 2, 2022

nodakai commented on Apr 8, 2022

MikaelUmaN commented on Jun 20, 2022

artemiusgreat commented on Jun 23, 2022

artemiusgreat commented on Jun 26, 2022

chriss2401 commented on Jul 29, 2022

16 remaining items

asmirnov82 commented on Nov 30, 2023

lucvalentino commented on Mar 25, 2024

asmirnov82 commented on Mar 25, 2024

lucvalentino commented on Mar 25, 2024

MgSam commented on Aug 12, 2026

rosebyte commented on Sep 30, 2026

rosebyte commented on Sep 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions