Storing data-with-units in files

[kenahoo]

Do people have any advice for storing data in files and preserving the units, rather than having to communicate the units of everything through side-channels? I'm aware of, and have used, the following approaches, but I don't love any of them:

1. Store everything in simple files (e.g. CSV) without units, and send recipients the units via email or an external table or something. 👎
2. Store things in complex files like NetCDF or GRIB, which does preserve units explicitly, but tends to be hard for people to work with sometimes. 😑
3. Store things in simple files and embed the units as part of the variable names. 😑
4. Pickle all unit-aware data to disk and hope everything stays binary-compatible and security-bug free. 👎

There might be units support in more "modern" file formats I'm not aware of, or maybe features of formats like Parquet or Feather that I'm not aware of, but I've been googling for a while and haven't found anything useful, so I'm thinking of biting the bullet and using CSV files with units embedded in the names, but I sure don't like it.

My main use case for this question is for data interchange, e.g. giving people data that was scraped from NetCDF files so they can import it into whatever system they're using (e.g. a database or that also preserves units, or a script that wants to verify/convert units, etc.).

Any advice would be appreciated!

[dopplershift]

Tools like xarray and netCDF can store the data in a format like Zarr that gives you the benefit of storing binary data with metadata, but in a format that's more readily interrogated without the need for HDF5.

[kenahoo]

Thanks, I hadn't heard of Zarr, I'll check that out.

[andrewgsavage]

I often get data in csv with two header rows, one for column names, one for units.

example here (but transposed)
https://pint-pandas.readthedocs.io/en/docs/user/reading.html

[kenahoo]

Hmm, that might be the best for me. I wasn't looking at pint-pandas because I thought (from https://pint-pandas.readthedocs.io/en/docs) that it still didn't have a non-dev-snapshot release, but I see now that it has a 0.5. Thanks.