Repository navigation
API reference - Samples for Transforms #1209
Description
We need to add samples on how to use the new transformer, and estimators than reference those samples from the XML documentation so that in docs.microsoft.com users can copy/paste the sample and have a head-starts.
Mot of the tests that got added as part of the transformer work are a good start for creating a sample.
MLContext Catalogs
| Catalog | Total APIs | Samples Owner | Samples Status / ETA |
|---|---|---|---|
| MLContext.Transforms (root) | 19 | Senja | Remaining: 4 overrides for the normalizer multicolumn examples |
| MLContext.Transforms.Categorical | 2 | ZeeshanA | Done v1 |
| MLContext.Transforms.Conversion | 6 | Senja | DoneV1 |
| MLContext.Transforms.FeatureSelection | 4 | ZeeshanA | Done v1 |
| MLContext.Transforms.TimeSeries | 4 | Senja | Done V1 |
| MLContext.Transforms.Text | 29 | ZeeshanA | Done V1 |
| MLContext.Data | 10 | Senja | DoneV1 |
| MLContext.Model (root) | 4 | ZeeshanS | DoneV1 |
P0+P1 Public API (extension methods) per Catalog
| MLContext.Transforms (root) | Num Overloads | Documentation | Sample | API Owner |
|---|---|---|---|---|
| CopyColumns | 2 | Yes | 2 Can remove dependency on DatasetUtils. | Zeeshan |
| Concatenate | 1 | Yes, needs improvement. | 1 - Can remove dependency on DatasetUtils. | Zeeshan |
| DropColumns | 1 | Yes | 1 Can remove dependency on DatasetUtils. | Zeeshan |
| SelectColumns | 2 | Yes, needs improvement. | 2 - Can remove dependency on DatasetUtils. | Zeeshan |
| Normalize | 1 | Done. | 1 #3244 | Ivan |
| CustomMapping | 1 | Yes, needs improvement. | Done-v1 #3275 | Artidoro |
| IndicateMissingValues | 2 | Done-v1 #3275 | Artidoro | |
| ReplaceMissingValues | 2 | Done-v1 #3275 | Artidoro | |
| ConvertToGrayscale | 1 | Yes, needs fixes. Example not displaying. | 1 #3165 | Abhishek |
| LoadImages | 1 | Yes, needs fixes. Example not displaying. | 1 #3165 | Abhishek |
| ExtractPixels | 2 | Yes, needs fixes. Example not displaying. | 1 #3165 | Abhishek |
| ResizeImages | 2 | Yes. Example not displaying. | 1 #3165 | Abhishek |
| ConvertToImage | 2 | Yes. | 1 #3165 | Abhishek |
| IidChangePointEstimator | 1 | 1- Done | Senja | |
| IidSpikeEstimator | 1 | 1 - Done | Senja | |
| SsaChangePointEstimator | 1 | 1 - Done | Senja | |
| SsaSpikeEstimator | 1 | 1 - Done | Senja | |
| ApplyOnnxModel | 3 | DoneV1 | #3349 | Gani |
| DnnFeaturizeImage | 1 | Yes, needs improvement. | 1 - Done | Senja |
| NormalizeGlobalContrast | 1 | Done | 0 #3232 | Ivan |
| NormalizeLpNorm | 1 | Done. | 0 #3232 | Ivan |
| ApproximatedKernelMap | 1 | Done | 0 #3232 | Ivan |
| mlContext.Transforms. CalculateFeatureContribution | 1 | Yes, needs improvement | Rogan |
| MLContext.Transforms.Categorical | Num Overloads | Documentation | Sample | API Owner |
|---|---|---|---|---|
| OneHotEncoding | 2 | 2 #3179 | Abhishek | |
| OneHotHashEncoding | 2 | 2 #3179 | Abhishek | |
| MLContext.Transforms.Conversion | Num Overloads | Documentation | Sample | API Owner |
|---|---|---|---|---|
| Hash | 2 | can't find the API | Done | Senja |
| ConvertType | 2 | Yes, needs improvement. | Done | Senja |
| MapKeyToValue | 2 | Yes, needs improvement. | Done | Senja |
| MapKeyToVector | 2 | Yes, needs improvement. | Done | Senja |
| MapValueToKey | 2 | Yes. | Done | Senja |
| MapKeyToBinaryVector | 2 | Yes, needs improvement. | Done | Senja |
| MLContext.Transforms.FeatureSelection | Num Overloads | Documentation | Sample | API Owner |
|---|---|---|---|---|
| SelectFeaturesBasedOnMutualInformation | 2 | need a better example to show MI computation. something like this | 2 #3184 | Abhishek |
| SelectFeaturesBasedOnCount | 2 | 2 #3184 | Abhishek | |
| MLContext.Transforms.Text | Num Overloads | Documentation | Sample | API Owner |
|---|---|---|---|---|
| FeaturizeText | 2 | #3120 | Zeeshan | |
| TokenizeCharacters | 1 | #3123 | Zeeshan | |
| NormalizeText | 1 | #3133 | Zeeshan | |
| ExtractWordEmbeddings | 1 | #3142 | Zeeshan | |
| TokenizeWords | 1 | #3156 | Zeeshan | |
| ProduceNgrams | 3 | #3177 | Zeeshan | |
| RemoveDefaultStopWords | 2 | #3156 | Zeeshan | |
| RemoveStopWords | 2 | #3156 | Zeeshan | |
| ProduceWordBags | 3 | #3183 | Zeeshan | |
| ProduceHashedWordBags | 3 | #3183 | Zeeshan | |
| ProduceHashedNgrams | 3 | #3177 | Zeeshan | |
| LatentDirichletAllocation | 2 | #3191 | Zeeshan |
For the Data catalog, all API's documentations needs to be augmented with suggestions for when would one use this API.
| MLContext.Data | Num Overloads | Documentation | Sample | API Owner |
|---|---|---|---|---|
| LoadFromEnumerable | 1 | Done. | 1 - Done. | Senja |
| CreateEnumerable | 2 | Done. The second overload of this API is a P4 scenario. the use case for that API would be: users has a model which has slot names preserved for the features, and when they load the models, they also get the schema out of the loaded model and pass that schema, together with the TRow type they want to load the data to this API. This API will then populate the Annotations (former metadata) for the feature column. | 1 | Senja |
| BootstrapSample | 1 | Done. | 1 - Done. | Senja |
| Cache | 1 | Done. | 1 - Done. | Senja |
| FilterRowsByColumn | 1 | Done. | 1 - Done. | Senja |
| FilterRowsByKeyColumnFraction | 1 | Done. | 1 - Done. | Senja |
| FilterRowsByMissingValues | 1 | Done. | 1 - Done. | Senja |
| ShuffleRows | 1 | Done. | 1 - Done. | Senja |
| SkipRows | 1 | Done. | 1 - Done. | Senja |
| TakeRows | 1 | Done. | 1 - Done. | Senja |
| Other | Num Overloads | Documentation | Sample | API Owner |
|---|---|---|---|---|
| Permutation Feature Importance | 4 | Yes, but needs work | Yes, but needs work | Rogan |
Activity
[-]Documentation samples for the transforms of the new API[/-][+]Documentation samples for the componentsof the new API[/+][-]Documentation samples for the componentsof the new API[/-][+]Documentation samples for the components of the new API[/+][-]Documentation samples for the components of the new API[/-][+]Documentation and samples for the API reference site[/+]sfilipi commentedon Jan 30, 2019
jwood803 commentedon Feb 8, 2019
Would the logistic regression one be done with PR #2256? I wonder if others may be done, too. I can try to go through and see if they have XML doc examples.
sfilipi commentedon Feb 8, 2019
Hi @jwood803, yes you took care of LogisticRegression with #2256. Thanks!
This workitem is to complete everything: XML doc over the extensions, estimators etc.
If you have bandwidth, and are looking for something to do, you can contribute to the samples, and replicate the work you did on Logistic Regression for #2256 for the other binary trainers:
LightGBM,
FastTree,
AveragedPerceptron,
SDCA,
LinearSVM
SymSGD
basically every extension on the BinaryClassificationCatalog.BinaryClassificationTrainers.
this BinaryClassificationCatalog.BinaryClassificationTrainers catalog,cc @shmoradims @rogancarr FYI.
If you claim those, feel free to update the table with your username.
[-]Documentation and samples for the API reference site[/-][+]Docs and samples for the API reference site (P0 & P1 API)[/+]shmoradims commentedon Feb 13, 2019
I moved the trainers to a separate issue: #2522
[-]Docs and samples for the API reference site (P0 & P1 API)[/-][+]Docs and samples for the API reference site (P0 & P1 Transforms)[/+]TomFinley commentedon Apr 9, 2019
I talked with @shmoradims and @eerhardt and I will work as a sub-part of this issue an example implementation of
IDataView.[-]Docs and samples for the API reference site (P0 & P1 Transforms)[/-][+]API reference - Samples for Transforms[/+]sfilipi commentedon Apr 19, 2019
Verified that everything is documented, but the normalizer multicolumn APIs. Tracking that as a separate issue.