Interpretation of reserved _Unsigned attribute written by netCDF-Java · Issue #656 · Unidata/netcdf4-python (original) (raw)

Unsigned integer data written by NetCDF-Java does not use an unsigned integer NetCDF type (e.g. uint16), but rather sets the reserved attribute _Unsigned on a (signed) int16 type. The result for the user of netcdf4-python is data which must be manually corrected after being read in.

It is my understanding that netCDF-C tries to "do no magic" so this is not a netcdf4-python bug per-se. #493 also notes the preference of netCDF4-python for leaving metadata interpretation to downstream applications. However, this is not a metadata convention in the CF sense, but rather a documented low level implementation decision within netCDF-Java: http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/CDM/Netcdf4.html

Differences between netCDF-4 C and Java libraries for netCDF4 files
Unsigned types
The C library uses unsigned integer types: NC_UBYTE, NC_USHORT, NC_UINT, NC_UINT64.
The Java library does not have separate types for unsigned integers, but adds the reserved attribute _Unsigned = "true" when the variable is unsigned. One can check this with Variable.isUnsigned(), Attribute.isUnsigned(), and Array.isUnsigned(). Conversions done by the library are aware of this convention. Java does not have unsigned types, and we dont want to double the internal memory requirements by widening the data.

I am in favor of a feature to assist the users in correcting for signendess on read when this attribute is present. One possible route:

• Add a method set_auto_signededness(True) to request the conversion be performed.
• If the above is set, perform the conversion if requested within the read slice method.

I would be less in favor of adding a method to handle the conversion; it means every time a user reads data they must remember to call that method. But that would be more helpful than having every user write their own when encountering unsigned data written by netCDF-Java.

Thanks to @lesserwhirls for helping me understand some of the things above; any errors remain mine!