Use Case 1 : Discussion

[brianthomas]

Comments imported from the Wiki (from @timj, I believe)

• Spectra are a special case of multi-dimensional image so probably don't need a separate call out.
• How many dimensions need to be supported? HDF5 supports 32, HDS and Fortran 7.
• Do tables have to support N-dimensional data in cells?
• What data types? Obviously whatever FITS/HDS supports but what about quad doubles?

[juandesant]

Re the question on N-dimensional data in cells, given that FITS tables support vectors in cells, at least that should be supported. Without an additional data model supporting it, the multidimensional data in the cells will be a multidimensional opaque structure. I think it would be better to have support for referencing different tables, and have that structure in a different multidimensional entity, and not worry about supporting that in a cell.

[brianthomas]

Thats a great point. I'd like to capture that either in this use case or a new one. What do you think?

[juandesant]

Given that the Use Case is named “Standard data product capture”, I'd say that it should be a new one for referencing further dimensions in the data.

[ahincks]

I'm joining the use-case conversation quite late in the game, so apologies if what I bring up below has already been addressed elsewhere.

I work a lot on data acquisition for millimetre and radio. Our raw data products contain time-ordered data (TOD)—e.g., for CMB data, we would record the reading of each bolometer, the position of the telescope, the temperatures of the cryogenics, etc., all as a function of time—everything needed to make a map. Usually there can be more than one recording rate, so that some TOD's are sampled more frequently than others. They rates can be sychronous (i.e., all the rates are multiples of a base rate) or asynchronous. Timestamps for the TOD's need to be appropriately recorded.

So:

• Does this use case I've sketched above pertain to the kind of format we're interested in? Should an astronomy data format be used for raw data capture? In brief: is recording TOD's "standard data product capture"?
• If this use case does obtain, does it belong in Use Case 1 under the generic heading of "tables", or does it belong in its own Use Case?
• If it belongs in Use Case 1 under "tables", would it be worth fleshing it out a bit more to include the scenario where tables are used to record TOD's at (possibly) multiple rates?

[brianthomas]

Hi Adam,

Please add your use-case(s), they certainly are unique. Whether they encompass new requirements, I cannot answer (yet; not done extracting) but I don't see that as a problem.

As to where they go...this touches on a related problem. One thing I have been struggling with is how "complete" to make Use Case 1. Clearly we could spend quite a bit of time going into detail about all of the different kinds of astronomy data products (models). I think that will ultimately be needed, but for us, will be unproductive since the effort will be large and we want to make some quick progress. Discussion on the astrodataformat list has trended towards the opinion that we focus our efforts on the demands of the serialization rather than all of the data models. I tend to agree, but I think we still need some science-focused use cases (like yours).

For now, please add your use case separately. We can then review it, and discuss merging it in here later.

Thanks for contributing!

-brian