Evaluation measures: undocumented / unclear

[berndbischl]

http://expdb.cs.kuleuven.be/expdb/api/?f=openml.evaluation.measures

For quite a lot of the measures it is unclear what they mean exactly, or it is, but it does not make sense to ask the client to optimize them in a task.

Examples:
a) How is kohavi_wolpert_bias_squared defined exactly?

b) Ho is the client supposed to optimize for "confusion_matrix"?

Solution:

• Document measures, at least by providing links to definitions.
• Remove the ones which do not make sense or explain to the user why they do.

[joaquinvanschoren]

Thanks, keep them coming :).

I'll try to add documentation soon, should be easy to do. Btw, this is a list of all evaluation measures, not only the ones the user can optimise for. Do you think we need a separate list for that?

Also, if you want to add new measures to the list, let me know: give me a name and description. We could add a service for adding new measures, but that may create some chaos :/.

Cheers,
Joaquin

On 27 Jan 2013, at 19:41, berndbischl notifications@github.com wrote:

http://expdb.cs.kuleuven.be/expdb/api/?f=openml.evaluation.measures

For quite a lot of the measures it is unclear what they mean exactly, or it is, but it does not make sense to ask the client to optimize them in a task.

Examples:
a) How is kohavi_wolpert_bias_squared defined exactly?

b) Ho is the client supposed to optimize for "confusion_matrix"?

Solution:

Document measures, at least by providing links to definitions.
Remove the ones which do not make sense or explain to the user why they do.
—
Reply to this email directly or view it on GitHub.

[berndbischl]

Btw, this is a list of all evaluation measures, not only the ones the user can optimise for.
Do you think we need a separate list for that?

I know, but the client programmer needs at least to know what he can possibly encounter in a task.xml.
If this is stated in the API docs somewhere, this is ok IMHO as well. Or make a 2nd list.

Also, if you want to add new measures to the list, let me know: give me a name and description

Not now, let's feature freeze as much as reasonable / doable for now.
Later on: Yes, possibly.

[joaquinvanschoren]

Added descriptions of all evaluation measures. Currently queriable from the math_function table.

[berndbischl]

Could you please add at least one sentence somewhere in the API docs how a description can currently be obtained?
I looked just now and could not find it...

[joaquinvanschoren]

Sure I still get your messages :). Do you think these descriptions should be returned by the api, to show them in your system? Or should they only be looked up on the website?
In either case, you can query them on the website:
http://expdb.cs.kuleuven.be/expdb/query.php -> SQL tab -> query "select name,description from math_function"
Or programmatically (returned as json):
http://expdb.cs.kuleuven.be/expdb/expdbsqljson.php?q=select%20name,description%20from%20math_function

If you'd rather have a dedicated API calls, let me know, this will be easy to add.

On the website there will be more intuitive pages with overviews of algorithms, datasets and evaluation measures. I will do this once Jan is finished with his refactoring on monday.

[berndbischl]

For now (and probably forever) I just want to be able to read them on the webpage, I don't care about a dedicated API call. Whether the more GUI oriented guys see this differently I don't know.

I don't know whether this is much work, but could you simply create a list / table of all results of your above mentioned queries and add a link to this tabke where the measures are covered in the API docs?

[joaquinvanschoren]

I will fold this into the work on the website (search results, overviews, tutorials). Probably early next week.

[joaquinvanschoren]

This is available from the website at openml.org, just add the metric in the search box or browse them under search>metrics