Repository navigation
Perf benchmarks for optimization #711
Description
We want to have a couple ML scenarios to be used for tracking performance. We can use it to detect regression from build to build, as well as improve performance.
Reacted by Shauheen and Tauheedul
Metadata
Metadata
Assignees
Labels
No labels
Activity
justinormont commentedon Sep 12, 2018
I defined a first round of benchmarks, and the implementation is in progress by @Anipik.
Tests in RSP form:
maml.exe CV k=5 data=toxicity_annotated_comments.merged.shuf.cleaned-68MB,_160k-rows.tsv loader=TextLoader{quote=- sparse=- col=Label:R4:0 col=rev_id:TX:1 col=comment:TX:2 col=logged_in:BL:4 col=ns:TX:5 col=sample:TX:6 col=split:TX:7 col=year:R4:3 header=+} xf=Convert{col=logged_in type=R4} xf=CategoricalTransform{col=ns} xf=TextTransform{col=FeaturesText:comment wordExtractor=NGramExtractorTransform{ngram=2}} xf=Concat{col=Features:FeaturesText,logged_in,ns} tr=OVA{p=AveragedPerceptron{iter=10}} out={0.model.zip}maml.exe CV k=5 data=toxicity_annotated_comments.merged.shuf.cleaned-68MB,_160k-rows.tsv loader=TextLoader{quote=- sparse=- col=Label:R4:0 col=rev_id:TX:1 col=comment:TX:2 col=logged_in:BL:4 col=ns:TX:5 col=sample:TX:6 col=split:TX:7 col=year:R4:3 header=+} xf=Convert{col=logged_in type=R4} xf=CategoricalTransform{col=ns} xf=TextTransform{col=FeaturesText:comment wordExtractor=NGramExtractorTransform{ngram=2}} xf=Concat{col=Features:FeaturesText,logged_in,ns} tr=LightGBMMulticlass{} out={1.model.zip}maml.exe CV tr=OVA{p=AveragedPerceptron{iter=10}} k=5 loader=TextLoader{quote=- sparse=- col=Label:R4:0 col=rev_id:TX:1 col=comment:TX:2 col=logged_in:BL:4 col=ns:TX:5 col=sample:TX:6 col=split:TX:7 col=year:R4:3 header=+} data=toxicity_annotated_comments.merged.shuf.cleaned-68MB,_160k-rows.tsv out=0.model.zip xf=Convert{col=logged_in type=R4} xf=CategoricalTransform{col=ns} xf=TextTransform{col=FeaturesText:comment tokens=+ wordExtractor=NGramExtractorTransform{ngram=2}} xf=WordEmbeddingsTransform{col=FeaturesWordEmbedding:FeaturesText_TransformedText model=FastTextWikipedia300D} xf=Concat{col=Features:FeaturesText,FeaturesWordEmbedding,logged_in,ns}maml.exe CV tr=SDCAMC k=5 loader=TextLoader{quote=- sparse=- col=Label:R4:0 col=rev_id:TX:1 col=comment:TX:2 col=logged_in:BL:4 col=ns:TX:5 col=sample:TX:6 col=split:TX:7 col=year:R4:3 header=+} data=toxicity_annotated_comments.merged.shuf.cleaned-68MB,_160k-rows.tsv out=2.model.zip xf=Convert{col=logged_in type=R4} xf=CategoricalTransform{col=ns} xf=TextTransform{col=FeaturesText:comment tokens=+ wordExtractor={} charExtractor={}} xf=WordEmbeddingsTransform{col=FeaturesWordEmbedding:FeaturesText_TransformedText model=FastTextWikipedia300D} xf=Concat{col=Features:FeaturesWordEmbedding,logged_in,ns}maml.exe TrainTest test=MSLR-WEB10K.VALIDATE.820MB_1.2M-rows.tsv eval=RankingEvaluator{t=10} data=MSLR-WEB10K.TRAIN.2.4GB_3.6M-rows.tsv loader=TextLoader{col=Label:R4:0 col=GroupId:TX:1 col=Features:R4:2-138} xf=HashTransform{col=GroupId} xf=NAHandleTransform{col=Features} tr=FastTreeRanking{} out={1.model.zip}maml.exe TrainTest test=MSLR-WEB10K.VALIDATE.820MB_1.2M-rows.tsv eval=RankingEvaluator{t=10} data=MSLR-WEB10K.TRAIN.2.4GB_3.6M-rows.tsv loader=TextLoader{col=Label:R4:0 col=GroupId:TX:1 col=Features:R4:2-138} xf=HashTransform{col=GroupId} xf=NAHandleTransform{col=Features} tr=LightGBMRanking{} out={3.model.zip}maml.exe Test data=toxicity_annotated_comments.merged.shuf.cleaned-68MB,_160k-rows.tsv in=0.model.fold000.zipNote: output accuracy for this one case isn't relevant as we're testing on the training set.
maml.exe Test data=MSLR-WEB10K.TEST.820MB_1.2M-rows.tsv in=1.model.zipjustinormont commentedon Sep 19, 2018
The first round of performance (speed) benchmark are complete. Thanks to @Anipik for implementing the benchmarks. cc: @Zruty0, @GalOshri
The benchmarks are chosen to be representative of user tasks and datasets. The components to test were chosen based on component usage number from internal-MSFT, what folks should be using, and tasks commonly performed.
Components covered by these tests:
Besides model training speed, bulk prediction/scoring speed is benchmarked for a text dataset w/ OVA-AveragedPerceptron, and a numeric ranking dataset w/ FastTree.
Notably absent components: (reasonable choices for the next round of benchmarks)
The focus of these benchmarks were on speed; we can also use them to track ML metrics like accuracy across builds. This would require a much larger set of datasets as to not over-fit our improvements towards a specific dataset.