Behavior of set_fill_off() changed in v1.5.1 · Issue #972 · Unidata/netcdf4-python (original) (raw)

netCDF4-1.5.1 (installed via pip 19.1.1)
Python 3.6.2 (virtualenv)
macOS Sierra (10.12.6)

Based on the documentation, my understanding ofDataset.set_fill_off() is that it leaves variables uninitialized upon creation. This is useful in cases were the variable is immediately filled with real data. With earlier versions of netCDF4, this works as expected:

""" Demonstrate fill value bug in netCDF4-1.5.1+

""" from netCDF4 import Dataset from numpy.ma import masked_all

Write masked data to netCDF, which will be replaced by _FillValue.

dataset = Dataset("test.nc", "w") dataset.set_fill_off() # comment this out and everything works as expected dim = dataset.createDimension("dim", 10) var = dataset.createVariable("var1", "f8", (dim.name,)) var[:] = masked_all((10,), "f8") dataset.close()

Read variable from netCDF. It should be masked everywhere.

dataset = Dataset("test.nc", "r") var = dataset.variables["var1"] assert var[:].mask.all()

As of v1.5.1, the final assertion fails. Inspection of the netCDF file using ncdump shows that var1 is set to _FillValue as expected. However, when var1 is read from the file, it is not masked anywhere. The value is _FillValue (9.9692e+36) everywhere, but the mask of the resulting MaskedArray is False.

Somehow, calling set_fill_off() when the variable is created is preventing it from being masked correctly when the dataset is read. This seems contrary to the documentation, and is a new behavior in v1.5.1.

A workaround is to not call set_fill_off().