Extracting more information than in the observation seems like an interesting decision.
It makes me feel like there is a basic principle of information being violated or that you should just ne doing inference on the image rather than creating a catalog.
@crawfordsm
Yep, the catalog is larger than the coadded image!
I think there are many strange choices in this pipeline -- which is a prototype of the LSST pipeline -- for example, they seem to make coadds per band, and then measure on the coadds, and then go back and measure individual images, but there don't seem to be obvious cross-band catalogs...
And the catalogs are huge because they have a pluggable algorithm thing where it's easy to add lots of different measurement types... so they do
@crawfordsm And I agree about the funny information-theory feelings. I wonder if one could infer the image pixels based on the catalog entries! If the catalog entries aren't completely redundant then it seems like it could be possible!
@crawfordsm
Digging a bit more... the catalogs also include a ton of extra HDUs, and for the "wide" layer this includes some pixel cutouts, so the actual catalog table is 32 MB for a coadd whose uncompressed size is 64 MB (4k x 4k floats)... For the "deep/ultra-deep" layer, the catalog is 56 MB!
@dstndstn Ah being designed for the time domain/ multiple images would then make a little more sense for the size of it.
If including cutouts in the data, I can see where it can blow up in size.
It’s still an interesting choice - I’d be curious for the design constraints that led to it. I could also see depending on the use case that you might just drop 99% of the catalog.
Add comment