Table of Contents
Overview
The GalaxiesML dataset is designed for machine learning applications in astrophysics. It includes 286,401 galaxy images from the Hyper-Suprime-Cam (HSC) Survey PDR2 across five filters: g, r, i, z, y. Spectroscopically confirmed redshifts serve as ground truth, making the dataset ideal for tasks such as redshift estimation and galaxy morphology classification. This dataset supports upcoming large-scale surveys like LSST and Euclid.
Features
- 286,401 galaxy images in five photometric bands (g, r, i, z, y).
- Spectroscopic redshifts for each galaxy, with redshift values ranging from 0.01 to 4.
- Morphological parameters such as Sérsic index, half-light radius, and ellipticity.
- Data provided in HDF5 format, along with CSV metadata.
- Training, validation, and test splits available for machine learning applications.
Examples of 64x64 pixel galaxy images from GalaxiesML in the five g, r, i, z, y filters. Images at 127x127 pixels are also available.
Data Files
The dataset is split into 127x127 pixel and 64x64 pixel image resolutions. The following files are available for each resolution:
127x127 Pixel Image Files
5x127x127_training_with_morphology.hdf5- Training set with images and morphological parameters.5x127x127_training_with_morphology.csv- Metadata and morphology for the training set.5x127x127_validation_with_morphology.hdf5- Validation set with images and morphological parameters.5x127x127_validation_with_morphology.csv- Metadata and morphology for the validation set.5x127x127_testing_with_morphology.hdf5- Testing set with images and morphological parameters.5x127x127_testing_with_morphology.csv- Metadata and morphology for the testing set.
64x64 Pixel Image Files
5x64x64_training_with_morphology.hdf5- Training set with images and morphological parameters.5x64x64_validation_with_morphology.hdf5- Validation set with images and morphological parameters.5x64x64_testing_with_morphology.hdf5- Testing set with images and morphological parameters.
Galaxy Morphology Parameters
The dataset includes several galaxy morphology parameters, such as:
- {band}_ellipticity: Ellipticity of the object, defined as 1 - B/A, where B is the semi-minor axis and A is the semi-major axis.
- {band}_half_light_radius: Radius containing 50% of the total flux.
- {band}_sersic_index: Describes the shape of the object's light profile.
- {band}_major_axis: Major axis of the detected object (pixels).
- {band}_minor_axis: Minor axis of the detected object (pixels).
- {band}_total_galaxies: Total number of galaxies detected by Source Extractor in an image.
Additional descriptions of the galaxy tabular data are provided below: Tabular Data Description.
Download the Dataset
The GalaxiesML dataset is a valuable resource for both astrophysicists and data scientists. It provides a vast collection of galaxy images along with detailed photometric data and precise redshift measurements. Whether you're working on galaxy classification, redshift estimation, or other machine learning applications in astrophysics, this dataset offers the comprehensive data you need to drive your research forward. The dataset is available on Zenodo and can be accessed using the link below.
Access DatasetCitations
If you use this dataset in your work, please cite the following references:
- GalaxiesML Dataset: Tuan Do et al., GalaxiesML: A Dataset of Galaxy Images, Photometry, Redshifts, and Structural Parameters for Machine Learning, arXiv:2410.00271 (2024).
- HSC PDR2: Aihara, H., et al., Second Data Release of the Hyper Suprime-Cam Subaru Strategic Program, Publications of the Astronomical Society of Japan, 71(6), 114 (2019). DOI
- Spectroscopic Surveys: Several spectroscopic catalogs were used, including:
- zCOSMOS Survey: Lilly, S. J., et al., The zCOSMOS 10k-Bright Spectroscopic Sample. The Astrophysical Journal Supplement Series, 184(2), 218-229 (2009). DOI: 10.1088/0067-0049/184/2/218
- VIMOS Public Extragalactic Survey (VIPERS): Garilli, B., et al., The VIMOS Public Extragalactic Survey (VIPERS): First Data Release of 57,204 Spectroscopic Measurements. Astronomy & Astrophysics, 562, A23 (2014). DOI: 10.1051/0004-6361/201322790
- DEEP2 Survey: Newman, J. A., et al., The DEEP2 Galaxy Redshift Survey: Design, Observations, Data Reduction, and Redshifts. The Astrophysical Journal Supplement Series, 208(1), 5 (2013). DOI: 10.1088/0067-0049/208/1/5
CNN Example
Explore our Convolutional Neural Network (CNN) example on GitHub to see how to leverage the GalaxiesML dataset for galaxy classification and redshift estimation tasks. This example provides a comprehensive codebase to help better understand machine learning applications in astrophysics.
View CNN Example on GitHubGenerative Model of Galaxies
Our latest research paper leverages the GalaxyML dataset to introduce a novel method using Denoising Diffusion Probabilistic Models (DDPM) conditioned on redshift data to generate realistic galaxy images. We demonstrate that DDPM effectively captures the physical characteristics and evolutionary changes of galaxies, enhancing our understanding of cosmic phenomena through machine learning.
Download Research PaperWe provide three Jupyter notebooks to help you explore our generative models for galaxy images. The Generation notebook is available as an interactive Colab notebook, while the Training and Evaluation notebooks are provided as downloadable references.
Training Notebook
Explore how we trained our Denoising Diffusion Probabilistic Model (DDPM) on the GalaxiesML dataset. This notebook details the training process and model architecture.
Download Training NotebookGenerate Notebook
Generate your own galaxy images using our pre-trained model! This interactive Colab notebook allows you to create new galaxy images by specifying redshift values and other parameters.
Open in Colab ↗Evaluate Notebook
Review our model evaluation process through various metrics and visualizations. Understand how we assessed the quality of generated images against real galaxy distributions.
Download Evaluation NotebookSupport Tools
Download the required Python scripts and utilities needed to run the notebooks. Includes modules for data management, model architecture, and evaluation tools.
Download ToolsNOTE: You must be signed in to a Google account, and the Colab extension must be installed for that account in order to access Colab.
Tip: Click the "Open With" drop down menu, search for "Colab" and click "Install". Then refresh the page and select "Open with Google Colaboratory"
Papers using GalaxiesML
-
Learning the Evolution of Physical Structure of Galaxies via Diffusion Models, Lizarraga A., Hanchen Jiang E., Nowack J., Li Y. Q., Nian Wu Y., Boscoe B., Do T., 2024,
Neurips Machine Learning and the Physical Sciences, 2024, arXiv:2411.18440
-
Using different sources of ground truths and transfer learning to improve the generalization of photometric redshift estimation, Soriano J., Saikrishnan S., Seenivasan V., Boscoe B., Singal J., Do T., 2024,
Neurips Machine Learning and the Physical Sciences, 2024, arXiv:2411.18054
-
Redshift Prediction with Images for Cosmology Using a Bayesian Convolutional Neural Network with Conformal Predictions, Jones E., Do T., Li Y. Q., Alfaro K., Singal J., Boscoe B., 2024,
The Astrophysical Journal, 974, 159
-
GalaxiesML: a dataset of galaxy images, photometry, redshifts, and structural parameters for machine learning, Do T., Boscoe B., Jones E., Li Y. Q., Alfaro K., 2024,
arXiv:2410.00271
-
Using Galaxy Evolution as Source of Physics-Based Ground Truth for Generative Models, Li Y. Q., Do T., Jones E., Boscoe B., Alfaro K., Nguyen Z., 2024,
arXiv:2407.07229
-
Improving Photometric Redshift Estimation for Cosmology with LSST Using Bayesian Neural Networks, Jones E., Do T., Boscoe B., Singal J., Wan Y., Nguyen Z., 2024,
The Astrophysical Journal, 964, 130
-
Machine Learning Classification to Identify Catastrophic Outlier Photometric Redshift Estimates, Singal J., Silverman G., Jones E., Do T., Boscoe B., Wan Y., 2022,
The Astrophysical Journal, 928, 6
-
Photometric Redshifts for Cosmology: Improving Accuracy and Uncertainty Estimates Using Bayesian Neural Networks, Jones E., Do T., Boscoe B., Wan Y., Nguyen Z., Singal J., 2022,
arXiv:2202.07121
-
Elements of effective machine learning datasets in astronomy, Bernie Boscoe, Tuan Do, Evan Jones, Yunqi Li, Kevin Alfaro, Christy Ma, 2022,
Neurips Machine Learning and the Physical Sciences 2022, arXiv:2202.07121
Tabular Data
| Column Name | Units | Description |
|---|---|---|
object_id |
Object ID from the HSC survey. Unique ID in 64-bit integer | |
coord |
(deg, deg, deg) | Coordinate used in coneSearch(coord, RA, DEC, RADIUS) |
ra |
deg | RA (J2000.0) of the image center |
dec |
deg | DEC (J2000.0) of the image center |
{band}_cmodel_mag |
mag | Magnitude of the central galaxy in filter {band} |
{band}_cmodel_magsigma |
mag | Uncertainty in the magnitude in filter {band} |
skymap_id |
Location of the galaxy in internal survey position definition (tract, patch) | |
specz_name |
Name(s) of the galaxy in the spectroscopic survey(s) | |
specz_flag_homogeneous |
Homogenized spec-z flag. (TRUE=secure, FALSE=insecure) | |
specz_mag_i |
mag | i-band magnitude of the galaxy in the spectroscopic survey |
specz_ra |
deg | RA (J2000.0) of galaxy in spectroscopic survey |
specz_dec |
deg | DEC (J2000.0) of galaxy in spectroscopic survey |
specz_redshift |
Spectroscopic redshift | |
specz_redshift_err |
Spectroscopic redshift uncertainty | |
{band}_central_image_pol_15px_rad |
Photometry within a 15-pixel radius in filter {band} |
|
{band}_central_image_pop_10px_rad |
Photometry within a 10-pixel radius in filter {band} |
|
{band}_central_image_pop_5px_rad |
Photometry within a 5-pixel radius in filter {band} |
|
{band}_ellipticity |
Ellipticity of the object in filter {band} |
|
{band}_half_light_radius |
pixels | Radius containing 50% of the total flux in filter {band} |
{band}_isophotal_area |
pixels² | Isophotal area of the object in filter {band} |
{band}_major_axis |
pixels | Major axis of the detected object in filter {band} |
{band}_minor_axis |
pixels | Minor axis of the detected object in filter {band} |
{band}_peak_surface_brightness |
mag/sq. arcsec | Peak surface brightness in filter {band} |
{band}_petro_rad |
pixels | Petrosian radius in filter {band} |
{band}_pos_angle |
deg | Position angle of the object in filter {band} |
{band}_sersic_index |
Sérsic index in filter {band} |
|
{band}_total_galaxies |
Total number of galaxies detected in filter {band} |
License
This dataset is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).