Use of masked arrays comes with huge performance drop

[fmaussion]

The change included in #787 made our test suite run in two hours instead of a couple of minutes.

I am not well placed to judge whether this was a good change or not, but I spent several hours in finding out what was going on. It would be great if these changes were documented somewhere, i.e. in a "what's new section" or similar. EDIT: just discovered the Changelog. Still, a link to it in the html documentation would be of great help.

Right now this makes v1.4 unusable for us: is there a way to set_auto_mask to False globally?

[fmaussion]

An example with indexing:

import numpy as np
import time

a = np.linspace(0, 100, 2000)
ma = np.ma.array(a)

ids = np.arange(1000, 1500)

%timeit a[ids]
631 ns ± 13.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

%timeit ma[ids]
14.3 µs ± 188 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

[davidhassell]

I, too, have been anecdotally concerned about this aspect of performance, but haven't got round to properly testing it in my code; so thanks for raising this.

An option to produce the pre-1.4 behaviour- i.e. return non-masked numpy arrays when there are no elements of fillValue, and masked numpy arrays when there are - would be great for me.

Something like set_dynamic_mask, perhaps? This would default to False, meaning the default 1.4 behaviour, or could be set to True for the pre-1.4 behaviour.

Thanks, David

[davidhassell]

... if this is of interest, I'd be happy to create pull request with some new code ....

[fmaussion]

I can't imagine that the majority of users will be happy with the current 1.4 behavior. Masked arrays are a not well understood part of numpy, beginners have troubles with them, and they come with a huge performance loss (factor 20 in the indexing example above). What's going to happen is that most users will set the masking to false per default, or wrap their own Dataset with this as default (this what we will probably do instead of enforcing netCDF v1.3).

I wonder if the path isn't to go through a painful (but rewarding on long term) deprecation of the pre 1.4 behavior: set auto_mask to False per default...

[jswhit]

You can turn auto-masking off globally using the set_auto_mask Dataset method.

I think there are only two choices that make sense - always return masked arrays or always return numpy arrays.

[jswhit]

There was a lot of discussion here about changing to the new behaviour (always returning masked arrays). The basic problem was that users could not predict what they were going to get (a numpy.ma or a numpy.ndarray), and this is generally considered bad API design.

[davidhassell]

OK, but I don't think that we would be breaking anything if we gave users who don't mind what they get the option of specifying that behaviour.

[akrherz]

@davidhassell you are asking for an attribute set at the module level to define this behavior and not just at the Dataset object? The latter already exists.

[davidhassell]

At the Dataset level is what I have in mind. How can you do it? set_auto_mask is not the right method, as setting that to True will never return masked arrays ....

[jswhit]

Re-enabling the old behavior with a Dataset method would be fine, please do submit a pull request.