You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tutorials/cloud_access/euclid-cloud-access.md
+62-51
Original file line number
Diff line number
Diff line change
@@ -23,7 +23,7 @@ By the end of this tutorial, you will:
23
23
24
24
+++
25
25
26
-
## Introduction
26
+
## 1. Introduction
27
27
Euclid launched in July 2023 as a European Space Agency (ESA) mission with involvement by NASA. The primary science goals of Euclid are to better understand the composition and evolution of the dark Universe. The Euclid mission is providing space-based imaging and spectroscopy as well as supporting ground-based imaging to achieve these primary goals. These data will be archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics.
28
28
29
29
Euclid Quick Release 1 (Q1) consists of consists of ~30 TB of imaging, spectroscopy, and catalogs covering four non-contiguous fields: Euclid Deep Field North (22.9 sq deg), Euclid Deep Field Fornax (12.1 sq deg), Euclid Deep Field South (28.1 sq deg), and LDN1641.
@@ -32,8 +32,12 @@ Euclid Q1 data were released on-premises at IPAC and in the cloud via Amazon Web
32
32
33
33
+++
34
34
35
-
## Imports
36
-
- TODO: fill the imports explaination
35
+
## 2. Imports
36
+
-`s3fs` for browsing S3 buckets
37
+
-`astropy` for handling coordinates, units, FITS I/O, tables, images, etc.
38
+
-`astroquery` for querying Euclid data products from IRSA
39
+
-`matplotlib` for visualization
40
+
-`json` for decoding JSON strings
37
41
38
42
```{code-cell} ipython3
39
43
# Uncomment the next line to install dependencies if needed.
@@ -54,52 +58,53 @@ from matplotlib import pyplot as plt
54
58
import json
55
59
```
56
60
57
-
## Browse Euclid QR1 Bucket
61
+
## 3. Browse Euclid Q1 cloud-hosted data
58
62
59
63
```{code-cell} ipython3
60
-
BUCKET_NAME='nasa-irsa-euclid-q1' # internal to IPAC until public release (use LAN or VPN w/ Tunnel-all)
64
+
BUCKET_NAME = 'nasa-irsa-euclid-q1'
61
65
```
62
66
67
+
[s3fs](https://s3fs.readthedocs.io/en/latest/) provides a filesystem-like python interface for AWS S3 buckets. First we create a s3 client:
68
+
63
69
```{code-cell} ipython3
64
70
s3 = s3fs.S3FileSystem(anon=True)
65
71
```
66
72
67
-
TODO: link s3fs docs
73
+
Then we list the `q1` directory that contains Euclid Q1 data products:
68
74
69
75
```{code-cell} ipython3
70
76
s3.ls(f'{BUCKET_NAME}/q1')
71
77
```
72
78
73
-
## Find images for a coordinate search
74
-
75
-
+++
76
-
77
-
### Locate MER images in the bucket
79
+
Let's navigate to MER images (available as FITS files):
78
80
79
81
```{code-cell} ipython3
80
-
s3.ls(f'{BUCKET_NAME}/q1/MER')
82
+
s3.ls(f'{BUCKET_NAME}/q1/MER')[:10] # ls only top 10 to limit the long output
81
83
```
82
84
83
85
```{code-cell} ipython3
84
-
s3.ls(f'{BUCKET_NAME}/q1/MER/102018211')
86
+
s3.ls(f'{BUCKET_NAME}/q1/MER/102018211') # pick any tile ID from above
85
87
```
86
88
87
89
```{code-cell} ipython3
88
-
s3.ls(f'{BUCKET_NAME}/q1/MER/102018211/VIS')
90
+
s3.ls(f'{BUCKET_NAME}/q1/MER/102018211/VIS') # pick any instrument from above
89
91
```
90
92
91
-
As per doc specification, we need `MER/{tile_id}/{instrument}/EUC_MER_BGSUB-MOSAIC*.fits` for displaying background-subtracted mosiac images. But these images are stored under TILE IDs so first we need to find TILE ID for a coordinate search we are interested in. We will use astroquery (in next section) to retrieve FITS file paths for our coordinates.
93
+
As per "Browsable Directories" section in [user guide](https://irsa.ipac.caltech.edu/data/Euclid/docs/euclid_archive_at_irsa_user_guide.pdf), we need `MER/{tile_id}/{instrument}/EUC_MER_BGSUB-MOSAIC*.fits` for displaying background-subtracted mosiac images. But these images are stored under TILE IDs so first we need to find TILE ID for a coordinate search we are interested in. We will use astroquery (in next section) to retrieve FITS file paths for our coordinates by doing spatial search.
92
94
93
95
+++
94
96
95
-
### Get image file paths for a coordinate search of interest
97
+
## 4. Do a spatial search for MER mosaics
98
+
99
+
Pick a target and search radius:
96
100
97
101
```{code-cell} ipython3
98
-
coord = SkyCoord.from_name("TYC 4429-1677-1")
102
+
target_name = 'TYC 4429-1677-1'
103
+
coord = SkyCoord.from_name(target_name)
99
104
search_radius = 10 * u.arcsec
100
105
```
101
106
102
-
List all Simple Image Access (SIA) collections for IRSA.
107
+
List all Simple Image Access (SIA) collections for IRSA:
@@ -112,25 +117,29 @@ Filter to only those containing "euclid":
112
117
collections[['euclid' in v for v in collections['collection']]]
113
118
```
114
119
115
-
As per "Data Products Overview" in [user guide](https://irsa.ipac.caltech.edu/data/Euclid/docs/euclid_archive_at_irsa_user_guide.pdf) and the above table, we identify that MER Mosiacs are available as follows:
120
+
As per "Data Products Overview" in [user guide](https://irsa.ipac.caltech.edu/data/Euclid/docs/euclid_archive_at_irsa_user_guide.pdf) and above table, we identify that MER Mosiacs are available as the following collection:
116
121
117
122
```{code-cell} ipython3
118
123
img_collection = 'euclid_DpdMerBksMosaic'
119
124
```
120
125
126
+
Now query this collection for our target's coordinates and search radius:
We can see there's a `cloud_access` column that gives us the location info of the image files we are interested in. So let's extract the S3 bucket file path from it.
142
+
We can see there's a `cloud_access` column that gives us the location info of the image files we are interested in. So let's extract the S3 bucket file path from it:
[get_filter_name(row['instrument_name'], row['energy_bandpassname']) for row in euclid_sci_img_tbl]
157
166
```
158
167
159
-
## Retrieve image cutouts from the cloud
168
+
## 5. Efficiently retrieve mosaic cutouts
160
169
These image files are very big (~1.4GB), so we use astropy's lazy-loading capability of FITS for better performance. (See [Obtaining subsets from cloud-hosted FITS files](https://docs.astropy.org/en/stable/io/fits/usage/cloud.html#fits-io-cloud).)
161
170
162
171
```{code-cell} ipython3
@@ -194,37 +203,40 @@ for idx, ax in enumerate(axes.flat):
194
203
plt.tight_layout()
195
204
```
196
205
197
-
## Find objects for the coordinates of our interest
198
-
199
-
+++
200
-
201
-
### Locate MER catalogs in the bucket
206
+
## 6. Find the MER catalog for a given tile
207
+
Let's navigate to MER catalog in the Euclid Q1 bucket:
As per doc specification, we need `catalogs/MER_FINAL_CATALOG/{tile_id}/EUC_MER_FINAL-CAT*.fits` for listing the objects catalogued. But we only need to find objects in our coordinates of interest so we will use astroquery to do a spatial search in MER catalog (combined for all tiles).
222
+
As per "Browsable Directiories" section in [user guide](https://irsa.ipac.caltech.edu/data/Euclid/docs/euclid_archive_at_irsa_user_guide.pdf), we can use `catalogs/MER_FINAL_CATALOG/{tile_id}/EUC_MER_FINAL-CAT*.fits` for listing the objects catalogued. We can read the identified FITS file as table and do filtering on ra, dec columns to find object ID(s) only for the target we picked. But it will be an expensive operation so we will instead use astroquery (in next section) to do a spatial search in the MER catalog provided by IRSA.
223
+
224
+
```{note}
225
+
Once the catalogs are available as Parquet files in the cloud, we can efficiently do spatial filtering directly on the cloud-hosted file to identify object ID(s) for our target. But for the time being, we can use catalog VO services through astroquery to do the same.
226
+
```
216
227
217
228
+++
218
229
219
-
### Get object IDs for the coordinates of our interest
tbl_catalogs[['euclid' in v for v in tbl_catalogs['schema_name']]]
239
+
catalogs[['euclid' in v for v in catalogs['schema_name']]]
228
240
```
229
241
230
242
From this table, we can extract the MER catalog name. We also see several other interesting catalogs, let's also extract spectral file association catalog for retrieving spectra later.
Now, we do a TAP search with spatial constraints for our coordinates. We use cone of 5 arcsec around our source to pinpoint its object ID in Euclid catalog.
249
+
Now, we do a region search within a cone of 5 arcsec around our target to pinpoint its object ID in Euclid catalog:
## Find spectra for the coordinates of our interest
257
-
Using the object ID(s) we extracted above, we can narrow down the spectral file association catalog to identify spectra file path(s).
264
+
## 8. Find the spectrum of an object in the MER catalog
265
+
Using the object ID(s) we extracted above, we can narrow down the spectral file association catalog to identify spectra file path(s). So we do the following TAP search:
258
266
259
267
```{code-cell} ipython3
260
268
adql_query = f"SELECT * FROM {euclid_spec_association_catalog} \
We can see the `uri` column that gives us location of spectra file on IBE, we can map it to S3 bucket key to retrieve spectra file from the cloud. This is a very big FITS spectra file with multiple extensions where each extension contains spectrum of one object. The `hdu` column gives us the extension number for our object. So let's extract both of these.
275
+
```{warning}
276
+
If you picked a target other than what this notebook uses, it's possible that there is no spectrum associated for your target's object ID. In that case, `spec_association_tbl` will contain 0 rows.
277
+
```
278
+
279
+
In above table, we can see that the `uri` column gives us location of spectra file on IBE. We can map it to S3 bucket key to retrieve spectra file from the cloud. This is a very big FITS spectra file with multiple extensions where each extension contains spectrum of one object. The `hdu` column gives us the extension number for our object. So let's extract both of these.
0 commit comments