Really interesting! Are there any future plans to integrate this with data analysis tools/pipelines?
jonkeeley · 1d ago
Thank you! For now, I'm working on automated model fitting from experimental chromatogram data.
fuzzfactor · 1d ago
This looks really beautiful, can you import or export real-world data in .CDF format?
jonkeeley · 9h ago
For now, it can only import/export .CSV, but as I'm building out automated curve fitting, I'll definitely add .CDF importing. Thanks for the suggestion!
fuzzfactor · 3h ago
CDF files for chromatography were based on the previously-established netCDF approach, basically just a text file (structured specifically for chromatography in this case) containing metadata from a single run, followed by the delimited numerical raw data for the run, then saved as a (sometimes quite large) text file. Which can be easily manipulated however you like. For final export then the text file is losslessly compressed using the netCDF code or dll, you give it a .CDF extension, after which it can be opened or imported by suitable Chromatography Data Systems (CDS).
Download the netCDF code from its steward Unidata (while you still can during NSF chaos):
You will use their netCDF code to correspondingly uncompress the incoming .CDF file into its ordinary text form, so you can use the data in your app after that.
For import to some systems, only a limited number of the proscribed metadata items need to be present and/or populated, and some you may not even find useful to parse from the file on import, but it's good to be prepared for the whole enchilada. Some CDS ended up with a different number of metadata items in CDF files when they export them compared to the files which they can import successfully.
The ASTM Standards outline the intended aspirational structure as it stood when CDS developers best agreed on it. Implementation is not consistent between CDS vendors since some of them added CDF import/export before ASTM standardization was initiated, then never moved forward from there. Plus not everybody understands ASTM or the standard completely. Regardless it's one of the best standards you can imagine precisely because it's never been updated, simply re-approved each review cycle. You'll need these, no different than the latest 2022 version:
What's missing to complete the picture are similar more detailed articles from those deeply involved in the late 1990s, in two issues of an earlier out-of-print publication (IIRC from ACS) named "Today's Chemist at Work" (TCAW). Published in the 1990s for a few years, focusing on industrial applications more than the research subjects often seen from ACS. This was a free glossy periodical with plenty lab ads, most people threw it away after they read it, but there are two key issues with articles that provide further insight into the CDF file structure than the ASTM publications alone.
These articles were available to the chromatograph vendors (probably written by one of their people) who were involved at the time, and you can get by without them but if you can locate a copy it would be more ideal. Mine were lost in a flood.
You always have to compensate for inconsistent implementation of a standard, in this case it's not too bad since there are not that many major CDS packages available to compare both import and export structure and functionality against.
If you only start with one vendor to test CDFs against, I would say try Agilent since they are so big and popular, and the GC data handling in their 21st century ChemStation is so widespread it does set the pace to an extent and is almost exactly what you need in an IC CDF file structure. Their latest package OpenLab (not really "open") recently added CDF import in the current version, previously they had only achieved CDF export in OpenLab until now. I don't think I was the only one to complain ;)
fuzzfactor · 3h ago
Looks like the main article in TCAW is from May 1998. Just a little too old to be posted online at ACS.
Download the netCDF code from its steward Unidata (while you still can during NSF chaos):
https://downloads.unidata.ucar.edu/netcdf/
You will use their netCDF code to correspondingly uncompress the incoming .CDF file into its ordinary text form, so you can use the data in your app after that.
For import to some systems, only a limited number of the proscribed metadata items need to be present and/or populated, and some you may not even find useful to parse from the file on import, but it's good to be prepared for the whole enchilada. Some CDS ended up with a different number of metadata items in CDF files when they export them compared to the files which they can import successfully.
The ASTM Standards outline the intended aspirational structure as it stood when CDS developers best agreed on it. Implementation is not consistent between CDS vendors since some of them added CDF import/export before ASTM standardization was initiated, then never moved forward from there. Plus not everybody understands ASTM or the standard completely. Regardless it's one of the best standards you can imagine precisely because it's never been updated, simply re-approved each review cycle. You'll need these, no different than the latest 2022 version:
https://tajhizkala.ir/doc/ASTM/E1947-98%20(Reapproved%202014...
https://tajhizkala.ir/doc/ASTM/E1948-98%20(Reapproved%202014...
Here's some related info from the year 2000:
https://journals.sagepub.com/doi/pdf/10.1016/S1535-5535-04-0...
What's missing to complete the picture are similar more detailed articles from those deeply involved in the late 1990s, in two issues of an earlier out-of-print publication (IIRC from ACS) named "Today's Chemist at Work" (TCAW). Published in the 1990s for a few years, focusing on industrial applications more than the research subjects often seen from ACS. This was a free glossy periodical with plenty lab ads, most people threw it away after they read it, but there are two key issues with articles that provide further insight into the CDF file structure than the ASTM publications alone.
These articles were available to the chromatograph vendors (probably written by one of their people) who were involved at the time, and you can get by without them but if you can locate a copy it would be more ideal. Mine were lost in a flood.
You always have to compensate for inconsistent implementation of a standard, in this case it's not too bad since there are not that many major CDS packages available to compare both import and export structure and functionality against.
If you only start with one vendor to test CDFs against, I would say try Agilent since they are so big and popular, and the GC data handling in their 21st century ChemStation is so widespread it does set the pace to an extent and is almost exactly what you need in an IC CDF file structure. Their latest package OpenLab (not really "open") recently added CDF import in the current version, previously they had only achieved CDF export in OpenLab until now. I don't think I was the only one to complain ;)