Conversation
…Mostly working? Get 'create_cf_data_variable' to call 'create_generic_cf_array_var': Mostly working?
Rename; addin parts of old investigation; add temporary notes.
…or overlength writes.
pp-mo
force-pushed
the
chardata_plus_encoded_datasets
branch
from
4a9cbc2 to
c4b7936
Compare
Current status 2026-01-28
|
| identifying with "codecs.lookup" : This means we support the encodings in the Python | ||
| Standard Library, and the name aliases which it recognises. |
| string_width: int # string lengths when viewing as strings (i.e. "Uxx") | ||
|
|
||
| def __init__(self, cf_var): | ||
| """Get all the info from an netCDF4 variable (or similar wrapper object). |
There was a problem hiding this comment.
| """Get all the info from an netCDF4 variable (or similar wrapper object). | |
| """Get all the info from a netCDF4 variable. |
It actually must be "at least" a threadsafe wrapped variable (or real netCDF4.Variable) and not an EncodedVariable, since we inspect it's '.dtype' etc.
| read_encoding: str # *always* a valid encoding from the codecs package | ||
| write_encoding: str # *always* a valid encoding from the codecs package | ||
| n_chars_dim: int # length of associated character dimension | ||
| string_width: int # string lengths when viewing as strings (i.e. "Uxx") |
There was a problem hiding this comment.
These are now only set if "is_chardata" -- see init code
| DECODE_TO_STRINGS_ON_READ = NetcdfStringDecodeSetting() | ||
| DEFAULT_READ_ENCODING = "utf-8" | ||
| DEFAULT_WRITE_ENCODING = "ascii" |
There was a problem hiding this comment.
These should be made available in public API.
Probably by importing in iris.fileformats.netcdf and including in its __all__ ?
ukmo-ccbunney
left a comment
ukmo-ccbunney
left a comment
There was a problem hiding this comment.
Just one comment at this time.
| encoding = self.read_encoding | ||
| if "utf-16" in encoding: | ||
| # Each char needs at least 2 bytes -- including a terminator char | ||
| strlen = (strlen // 2) - 1 |
There was a problem hiding this comment.
Do we really need to account for a terminating char on "utf-32" and "utf-16" encodings?
When writing to a netCDF file, surely the terminator isn't written? This is just something that is used when storing strings in memory, is it not?
There was a problem hiding this comment.
OK - this looks to be the case. Certainly encoding a byte string to "utf-16" or "utf-32" does appear to add an extra null terminator...
2 participants
Closes #6309 + various
Successor to #6850
now incorporating #6851
+ now integrated usage with netcdf load+save, to use encoded datasets