4. Published Data: Working with the Pyrfume datasets

import pyrfume

Whether you want to build predictive models or simply organize data, it is essential to begin integrating across datasets and wherever possible bringing the largest datasets to bear on the problem. Most breakthroughs don’t come from new algorithms but from new, large datasets.

bushdid_molecules = pyrfume.load_data('bushdid_2014/molecules.csv')
bushdid_molecules.head()
MolecularWeight IsomericSMILES IUPACName name Odorant name C.A.S. % odorant Solvent
CID
176 60.05 CC(=O)O acetic acid acetic acid acetic acid 64-19-7 10.00 mineral oil
177 44.05 CC=O acetaldehyde acetaldehyde acetaldehyde 75-07-0 5.00 water
179 88.11 CC(C(=O)C)O 3-hydroxybutan-2-one acetoin acetoin 513-86-0 0.10 1,2-propanediol
180 58.08 CC(=O)C propan-2-one acetone propan-2-one 67-64-1 25.00 water
240 106.12 C1=CC=C(C=C1)C=O benzaldehyde benzaldehyde benzaldehyde 100-52-7 0.25 mineral oil

The above shows the first 5 molecules (all 128 are in the full dataframe) from Bushdid et al, 2014 which looked at the perceptual discriminability of random mixtures in humans. Even though this data was not on your disk, Pyrfume fell back to loading it remotely; future loads of the same data will come from your disk for speed. The first thing to note is that the index and the first 5 columns are structured identically to what we generated in Part 1 from our own data. ALL pyrfume datasets have this structure, whether there were obtained from supplemental figures and tables, excel files or pdfs, industrial databases, books, or papyrus scrolls. Additional columns (such as the final 4 shown above) may also be present, on case-by-case basis, depending on what the authors chose to include in their source materials.

What else has the Pyrfume project extracted from this data source?

pyrfume.show_files('bushdid_2014')
{'behavior': {'csv': 'Triangle test results'},
 'main': {'ipynb': 'Processing workflow'},
 'mixtures': {'csv': 'Mixtures used'},
 'molecules': {'csv': 'Molecules used'}}

The above shows a manifest of files available. Curated file names are simple and memorable (typically, “molecules”, “behavior”, “imaging”, etc.) which means you will often not even need to examine the manifest before retrieving the files you care about. Importantly, every data archive contains at least one notebook (main.ipynb) which provides the full processing workflow going from the raw data provided in the original data source to the cleaned, standardized, organized and mutually compatible datasets provided by Pyrfume.

bushdid_mixtures = pyrfume.load_data('bushdid_2014/mixtures.csv')
bushdid_behavior = pyrfume.load_data('bushdid_2014/behavior.csv')

Information about each mixture, including the CID (not provided in the original source data) is given in the mixtures file.

bushdid_mixtures.head()
Answer Components in mixtures Components that differ % mixture overlap Stimulus dilution Molecule 1 Molecule 2 Molecule 3 Molecule 4 Molecule 5 ... Molecule 21 Molecule 22 Molecule 23 Molecule 24 Molecule 25 Molecule 26 Molecule 27 Molecule 28 Molecule 29 Molecule 30
Test UID
1 right 30 30 0.0 1.00 12232 7731 7888 7966 7848 ... 8103 7344 8051 7921 460 7583 11002 12367 6590 7991
1 wrong 30 30 0.0 0.25 440917 5281515 8030 3314 31272 ... 6561 22386 11509 443162 8797 16666 7749 7793 7799 7762
1 wrong 30 30 0.0 0.50 440917 5281515 8030 3314 31272 ... 6561 22386 11509 443162 8797 16666 7749 7793 7799 7762
2 right 10 4 60.0 0.50 3314 62433 5281515 7749 6561 ... 0 0 0 0 0 0 0 0 0 0
2 wrong 10 4 60.0 0.25 3314 62433 5281515 7749 6561 ... 0 0 0 0 0 0 0 0 0 0

5 rows × 35 columns

Finally, the human behavioral results for each mixture are provided in the behavior file. How one chooses to join these tables to produce prediction target or otherwise explore the data is up to individual taste, but with standards in place it is much less work than it would have been with the source data alone!

bushdid_behavior.head()
Subject Correct
Test UID
1 1 False
1 2 True
1 3 False
1 4 True
1 5 True

Many other datasets are available now, with several dozen additional datasets ready for release in the next 12 months. Licensing issues stand in the way of some, but the Pyrfume maintainers are working this out.

pyrfume.list_archives()
['abraham_2012',
 'bushdid_2014',
 'haddad_2008',
 'keller_2016',
 'manoel_2021',
 'ravia_2020',
 'snitz_2013',
 'wakayama_2019']