How to create STAC Catalogs¶
STAC Community Sprint, Arlington, November 7th 2019¶
This notebook runs through some of the basics of using PySTAC to create a static STAC. It was part of a 30 minute presentation at the community STAC sprint in Arlington, VA in November 2019.
This tutorial will require the boto3, rasterio, and shapely libraries:
[1]:
!pip install boto3
!pip install rasterio
!pip install shapely
Requirement already satisfied: boto3 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (1.10.8)
Requirement already satisfied: botocore<1.14.0,>=1.13.8 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from boto3) (1.13.8)
Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from boto3) (0.2.1)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from boto3) (0.9.4)
Requirement already satisfied: urllib3<1.26,>=1.20; python_version >= "3.4" in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from botocore<1.14.0,>=1.13.8->boto3) (1.25.6)
Requirement already satisfied: docutils<0.16,>=0.10 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from botocore<1.14.0,>=1.13.8->boto3) (0.15.2)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from botocore<1.14.0,>=1.13.8->boto3) (2.8.1)
Requirement already satisfied: six>=1.5 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.14.0,>=1.13.8->boto3) (1.12.0)
WARNING: You are using pip version 20.1.1; however, version 20.2 is available.
You should consider upgrading via the '/Users/rob/proj/stac/pystac/venv/bin/python -m pip install --upgrade pip' command.
Requirement already satisfied: rasterio in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (1.1.0)
Requirement already satisfied: numpy in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from rasterio) (1.17.3)
Requirement already satisfied: snuggs>=1.4.1 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from rasterio) (1.4.7)
Requirement already satisfied: click<8,>=4.0 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from rasterio) (7.0)
Requirement already satisfied: click-plugins in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from rasterio) (1.1.1)
Requirement already satisfied: attrs in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from rasterio) (19.3.0)
Requirement already satisfied: cligj>=0.5 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from rasterio) (0.5.0)
Requirement already satisfied: affine in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from rasterio) (2.3.0)
Requirement already satisfied: pyparsing>=2.1.6 in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (from snuggs>=1.4.1->rasterio) (2.4.2)
WARNING: You are using pip version 20.1.1; however, version 20.2 is available.
You should consider upgrading via the '/Users/rob/proj/stac/pystac/venv/bin/python -m pip install --upgrade pip' command.
Requirement already satisfied: shapely in /Users/rob/proj/stac/pystac/venv/lib/python3.6/site-packages (1.6.4.post2)
WARNING: You are using pip version 20.1.1; however, version 20.2 is available.
You should consider upgrading via the '/Users/rob/proj/stac/pystac/venv/bin/python -m pip install --upgrade pip' command.
We can import pystac and access most of the functionality we need with the single import:
[2]:
import pystac
Creating a catalog from a local file¶
To give us some material to work with, lets download a single image from the Spacenet 5 challenge. We’ll use a temporary directory to save off our single-item STAC.
[3]:
import os
import urllib.request
from tempfile import TemporaryDirectory
tmp_dir = TemporaryDirectory()
img_path = os.path.join(tmp_dir.name, 'image.tif')
[4]:
url = ('http://spacenet-dataset.s3.amazonaws.com/'
'spacenet/SN5_roads/train/AOI_7_Moscow/MS/'
'SN5_roads_train_AOI_7_Moscow_MS_chip996.tif')
urllib.request.urlretrieve(url, img_path)
[4]:
('/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/image.tif',
<http.client.HTTPMessage at 0x10e80d6a0>)
We want to create a Catalog. Let’s check the pydocs for Catalog to see what information we’ll need. (We use __doc__ instead of help() here to avoid printing out all the docs for the class.)
[5]:
print(pystac.Catalog.__doc__)
A PySTAC Catalog represents a STAC catalog in memory.
A Catalog is a :class:`~pystac.STACObject` that may contain children,
which are instances of :class:`~pystac.Catalog` or :class:`~pystac.Collection`,
as well as :class:`~pystac.Item` s.
Args:
id (str): Identifier for the catalog. Must be unique within the STAC.
description (str): Detailed multi-line description to fully explain the catalog.
`CommonMark 0.28 syntax <http://commonmark.org/>`_ MAY be used for rich text
representation.
title (str or None): Optional short descriptive one-line title for the catalog.
stac_extensions (List[str]): Optional list of extensions the Catalog implements.
href (str or None): Optional HREF for this catalog, which be set as the catalog's
self link's HREF.
Attributes:
id (str): Identifier for the catalog.
description (str): Detailed multi-line description to fully explain the catalog.
title (str or None): Optional short descriptive one-line title for the catalog.
stac_extensions (List[str] or None): Optional list of extensions the Catalog implements.
extra_fields (dict or None): Extra fields that are part of the top-level JSON properties
of the Catalog.
links (List[Link]): A list of :class:`~pystac.Link` objects representing
all links associated with this Catalog.
Let’s just give an ID and a description. We don’t have to worry about the HREF right now; that will be set later.
[6]:
catalog = pystac.Catalog(id='test-catalog', description='Tutorial catalog.')
There are no children or items in the catalog, since we haven’t added anything yet.
[7]:
print(list(catalog.get_children()))
print(list(catalog.get_items()))
[]
[]
We’ll now create an Item to represent the image. Check the pydocs to see what you need to supply:
[8]:
print(pystac.Item.__doc__)
An Item is the core granular entity in a STAC, containing the core metadata
that enables any client to search or crawl online catalogs of spatial 'assets' -
satellite imagery, derived data, DEM's, etc.
Args:
id (str): Provider identifier. Must be unique within the STAC.
geometry (dict): Defines the full footprint of the asset represented by this item,
formatted according to `RFC 7946, section 3.1 (GeoJSON)
<https://tools.ietf.org/html/rfc7946>`_.
bbox (List[float] or None): Bounding Box of the asset represented by this item using
either 2D or 3D geometries. The length of the array must be 2*n where n is the
number of dimensions. Could also be None in the case of a null geometry.
datetime (datetime or None): Datetime associated with this item. If None,
a start_datetime and end_datetime must be supplied in the properties.
properties (dict): A dictionary of additional metadata for the item.
stac_extensions (List[str]): Optional list of extensions the Item implements.
href (str or None): Optional HREF for this item, which be set as the item's
self link's HREF.
collection (Collection or str): The Collection or Collection ID that this item
belongs to.
extra_fields (dict or None): Extra fields that are part of the top-level JSON properties
of the Item.
Attributes:
id (str): Provider identifier. Unique within the STAC.
geometry (dict): Defines the full footprint of the asset represented by this item,
formatted according to `RFC 7946, section 3.1 (GeoJSON)
<https://tools.ietf.org/html/rfc7946>`_.
bbox (List[float] or None): Bounding Box of the asset represented by this item using
either 2D or 3D geometries. The length of the array is 2*n where n is the
number of dimensions. Could also be None in the case of a null geometry.
datetime (datetime or None): Datetime associated with this item. If None,
the start_datetime and end_datetime in the common_metadata
will supply the datetime range of the Item.
properties (dict): A dictionary of additional metadata for the item.
stac_extensions (List[str] or None): Optional list of extensions the Item implements.
collection (Collection or None): Collection that this item is a part of.
links (List[Link]): A list of :class:`~pystac.Link` objects representing
all links associated with this STACObject.
assets (Dict[str, Asset]): Dictionary of asset objects that can be downloaded,
each with a unique key.
collection_id (str or None): The Collection ID that this item belongs to, if any.
extra_fields (dict or None): Extra fields that are part of the top-level JSON properties
of the Item.
Using rasterio, we can pull out the bounding box of the image to use for the image metadata. If the image contained a NoData border, we would ideally pull out the footprint and save it as the geometry; in this case, we’re working with a small chip the most likely has no NoData values.
[9]:
import rasterio
from shapely.geometry import Polygon, mapping
def get_bbox_and_footprint(raster_uri):
with rasterio.open(raster_uri) as ds:
bounds = ds.bounds
bbox = [bounds.left, bounds.bottom, bounds.right, bounds.top]
footprint = Polygon([
[bounds.left, bounds.bottom],
[bounds.left, bounds.top],
[bounds.right, bounds.top],
[bounds.right, bounds.bottom]
])
return (bbox, mapping(footprint))
[10]:
bbox, footprint = get_bbox_and_footprint(img_path)
print(bbox)
print(footprint)
[37.6616853489879, 55.73478197572927, 37.66573047610874, 55.73882710285011]
{'type': 'Polygon', 'coordinates': (((37.6616853489879, 55.73478197572927), (37.6616853489879, 55.73882710285011), (37.66573047610874, 55.73882710285011), (37.66573047610874, 55.73478197572927), (37.6616853489879, 55.73478197572927)),)}
We’re also using datetime.utcnow() to supply the required datetime property for our Item. Since this is a required property, you might often find yourself making up a time to fill in if you don’t know the exact capture time.
[11]:
from datetime import datetime
item = pystac.Item(id='local-image',
geometry=footprint,
bbox=bbox,
datetime=datetime.utcnow(),
properties={})
We haven’t added it to a catalog yet, so it’s parent isn’t set. Once we add it to the catalog, we can see it correctly links to it’s parent.
[12]:
item.get_parent() is None
[12]:
True
[13]:
catalog.add_item(item)
[14]:
item.get_parent()
[14]:
<Catalog id=test-catalog>
describe() is a useful method on Catalog - but be careful when using it on large catalogs, as it will walk the entire tree of the STAC.
[15]:
catalog.describe()
* <Catalog id=test-catalog>
* <Item id=local-image>
Adding Assets¶
We’ve created an Item, but there aren’t any assets associated with it. Let’s create one:
[16]:
print(pystac.Asset.__doc__)
An object that contains a link to data associated with the Item that can be
downloaded or streamed.
Args:
href (str): Link to the asset object. Relative and absolute links are both allowed.
title (str): Optional displayed title for clients and users.
description (str): A description of the Asset providing additional details, such as
how it was processed or created. CommonMark 0.29 syntax MAY be used for rich
text representation.
media_type (str): Optional description of the media type. Registered Media Types
are preferred. See :class:`~pystac.MediaType` for common media types.
roles ([str]): Optional, Semantic roles (i.e. thumbnail, overview, data, metadata)
of the asset.
properties (dict): Optional, additional properties for this asset. This is used by
extensions as a way to serialize and deserialize properties on asset
object JSON.
Attributes:
href (str): Link to the asset object. Relative and absolute links are both allowed.
title (str): Optional displayed title for clients and users.
description (str): A description of the Asset providing additional details, such as
how it was processed or created. CommonMark 0.29 syntax MAY be used for rich
text representation.
media_type (str): Optional description of the media type. Registered Media Types
are preferred. See :class:`~pystac.MediaType` for common media types.
properties (dict): Optional, additional properties for this asset. This is used by
extensions as a way to serialize and deserialize properties on asset
object JSON.
owner (Item or None): The Item this asset belongs to.
[17]:
item.add_asset(
key='image',
asset=pystac.Asset(
href=img_path,
media_type=pystac.MediaType.GEOTIFF
)
)
[17]:
<Item id=local-image>
At any time we can call to_dict() on STAC objects to see how the STAC JSON is shaping up. Notice the asset is now set:
[18]:
import json
print(json.dumps(item.to_dict(), indent=4))
{
"type": "Feature",
"stac_version": "1.0.0-beta.2",
"id": "local-image",
"properties": {
"datetime": "2020-08-03T03:47:48.786929Z"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
37.6616853489879,
55.73478197572927
],
[
37.6616853489879,
55.73882710285011
],
[
37.66573047610874,
55.73882710285011
],
[
37.66573047610874,
55.73478197572927
],
[
37.6616853489879,
55.73478197572927
]
]
]
},
"links": [
{
"rel": "root",
"href": null,
"type": "application/json"
},
{
"rel": "parent",
"href": null,
"type": "application/json"
}
],
"assets": {
"image": {
"href": "/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/image.tif",
"type": "image/tiff; application=geotiff"
}
},
"bbox": [
37.6616853489879,
55.73478197572927,
37.66573047610874,
55.73882710285011
]
}
Note that the link href properties are null. This is OK, as we’re working with the STAC in memory. Next, we’ll talk about writing the catalog out, and how to set those HREFs.
Saving the catalog¶
As the JSON above indicates, there’s no HREFs set on these in-memory items. PySTAC uses the self link on STAC objects to track where the file lives. Because we haven’t set them, they evaluate to None:
[19]:
print(catalog.get_self_href() is None)
print(item.get_self_href() is None)
True
True
In order to set them, we can use normalize_hrefs. This method will create a normalized set of HREFs for each STAC object in the catalog, according to the best practices document’s recommendations on how to lay out a catalog.
[20]:
catalog.normalize_hrefs(os.path.join(tmp_dir.name, 'stac'))
[20]:
<Catalog id=test-catalog>
Now that we’ve normalized to a root directory (the temporary directory), we see that the self links are set:
[21]:
print(catalog.get_self_href())
print(item.get_self_href())
/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/stac/catalog.json
/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/stac/local-image/local-image.json
We can now call save on the catalog, which will recursively save all the STAC objects to their respective self HREFs.
Save requires a CatalogType to be set. You can review the API docs on CatalogType to see what each type means (unfortunately help doesn’t show docstrings for attributes).
[22]:
catalog.save(catalog_type=pystac.CatalogType.SELF_CONTAINED)
[23]:
!ls {tmp_dir.name}/stac/*
/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/stac/catalog.json
/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/stac/local-image:
local-image.json
[24]:
with open(catalog.get_self_href()) as f:
print(f.read())
{
"id": "test-catalog",
"stac_version": "1.0.0-beta.2",
"description": "Tutorial catalog.",
"links": [
{
"rel": "root",
"href": "./catalog.json",
"type": "application/json"
},
{
"rel": "item",
"href": "./local-image/local-image.json",
"type": "application/json"
}
]
}
[25]:
with open(item.get_self_href()) as f:
print(f.read())
{
"type": "Feature",
"stac_version": "1.0.0-beta.2",
"id": "local-image",
"properties": {
"datetime": "2020-08-03T03:47:48.786929Z"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
37.6616853489879,
55.73478197572927
],
[
37.6616853489879,
55.73882710285011
],
[
37.66573047610874,
55.73882710285011
],
[
37.66573047610874,
55.73478197572927
],
[
37.6616853489879,
55.73478197572927
]
]
]
},
"links": [
{
"rel": "root",
"href": "../catalog.json",
"type": "application/json"
},
{
"rel": "parent",
"href": "../catalog.json",
"type": "application/json"
}
],
"assets": {
"image": {
"href": "/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/image.tif",
"type": "image/tiff; application=geotiff"
}
},
"bbox": [
37.6616853489879,
55.73478197572927,
37.66573047610874,
55.73882710285011
]
}
As you can see, all links are saved with relative paths. That’s because we used catalog_type=CatalogType.SELF_CONTAINED. If we save an Absolute Published catalog, we’ll see absolute paths:
[26]:
catalog.save(catalog_type=pystac.CatalogType.ABSOLUTE_PUBLISHED)
Now the links included in the STAC item are all absolute:
[27]:
with open(item.get_self_href()) as f:
print(f.read())
{
"type": "Feature",
"stac_version": "1.0.0-beta.2",
"id": "local-image",
"properties": {
"datetime": "2020-08-03T03:47:48.786929Z"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
37.6616853489879,
55.73478197572927
],
[
37.6616853489879,
55.73882710285011
],
[
37.66573047610874,
55.73882710285011
],
[
37.66573047610874,
55.73478197572927
],
[
37.6616853489879,
55.73478197572927
]
]
]
},
"links": [
{
"rel": "self",
"href": "/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/stac/local-image/local-image.json",
"type": "application/json"
},
{
"rel": "root",
"href": "/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/stac/catalog.json",
"type": "application/json"
},
{
"rel": "parent",
"href": "/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/stac/catalog.json",
"type": "application/json"
}
],
"assets": {
"image": {
"href": "/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/image.tif",
"type": "image/tiff; application=geotiff"
}
},
"bbox": [
37.6616853489879,
55.73478197572927,
37.66573047610874,
55.73882710285011
]
}
Notice that the Asset HREF is absolute in both cases. We can make the Asset HREF relative to the STAC Item by using .make_all_asset_hrefs_relative():
[28]:
catalog.make_all_asset_hrefs_relative()
catalog.save(catalog_type=pystac.CatalogType.SELF_CONTAINED)
[29]:
with open(item.get_self_href()) as f:
print(f.read())
{
"type": "Feature",
"stac_version": "1.0.0-beta.2",
"id": "local-image",
"properties": {
"datetime": "2020-08-03T03:47:48.786929Z"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
37.6616853489879,
55.73478197572927
],
[
37.6616853489879,
55.73882710285011
],
[
37.66573047610874,
55.73882710285011
],
[
37.66573047610874,
55.73478197572927
],
[
37.6616853489879,
55.73478197572927
]
]
]
},
"links": [
{
"rel": "root",
"href": "../catalog.json",
"type": "application/json"
},
{
"rel": "parent",
"href": "../catalog.json",
"type": "application/json"
}
],
"assets": {
"image": {
"href": "../../image.tif",
"type": "image/tiff; application=geotiff"
}
},
"bbox": [
37.6616853489879,
55.73478197572927,
37.66573047610874,
55.73882710285011
]
}
Creating an Item that implements the EO extension¶
In the code above our item only implemented the core STAC Item specification. With extensions we can record more information and add additional functionality to the Item. Given that we know this is a World View 3 image that has earth observation data, we can enable the eo extension to add band information.
To add eo information to an item we’ll need to specify some more data. First, let’s define the bands of World View 3:
[30]:
from pystac.extensions.eo import Band
# From: https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf
wv3_bands = [Band.create(name='Coastal', description='Coastal: 400 - 450 nm', common_name='coastal'),
Band.create(name='Blue', description='Blue: 450 - 510 nm', common_name='blue'),
Band.create(name='Green', description='Green: 510 - 580 nm', common_name='green'),
Band.create(name='Yellow', description='Yellow: 585 - 625 nm', common_name='yellow'),
Band.create(name='Red', description='Red: 630 - 690 nm', common_name='red'),
Band.create(name='Red Edge', description='Red Edge: 705 - 745 nm', common_name='rededge'),
Band.create(name='Near-IR1', description='Near-IR1: 770 - 895 nm', common_name='nir08'),
Band.create(name='Near-IR2', description='Near-IR2: 860 - 1040 nm', common_name='nir09')]
Notice that we used the .create method create new band information.
We can now create an Item, enable the eo extension, add the band information and add it to our catalog:
[31]:
eo_item = pystac.Item(id='local-image-eo',
geometry=footprint,
bbox=bbox,
datetime=datetime.utcnow(),
properties={})
eo_item.ext.enable(pystac.Extensions.EO)
eo_item.ext.eo.apply(bands=wv3_bands)
There are also common metadata fields that we can use to capture additional information about the WorldView 3 imagery:
[32]:
eo_item.common_metadata.platform = "Maxar"
eo_item.common_metadata.instrument="WorldView3"
eo_item.common_metadata.gsd = 0.3
[33]:
eo_item
[33]:
<Item id=local-image-eo>
We can use the eo extension to add bands to the assets we add to the item:
[34]:
eo_ext = eo_item.ext.eo
help(eo_ext.set_bands)
#eo_item.add_asset(key='image', asset=)
Help on method set_bands in module pystac.extensions.eo:
set_bands(bands, asset=None) method of pystac.extensions.eo.EOItemExt instance
Set an Item or an Asset bands.
If an Asset is supplied, sets the property on the Asset.
Otherwise sets the Item's value.
[35]:
asset = pystac.Asset(href=img_path,
media_type=pystac.MediaType.GEOTIFF)
eo_ext.set_bands(wv3_bands, asset)
eo_item.add_asset("image", asset)
[35]:
<Item id=local-image-eo>
If we look at the asset’s JSON representation, we can see the appropriate band indexes are set:
[36]:
asset.to_dict()
[36]:
{'href': '/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/image.tif',
'type': 'image/tiff; application=geotiff',
'eo:bands': [{'name': 'Coastal',
'common_name': 'coastal',
'description': 'Coastal: 400 - 450 nm'},
{'name': 'Blue', 'common_name': 'blue', 'description': 'Blue: 450 - 510 nm'},
{'name': 'Green',
'common_name': 'green',
'description': 'Green: 510 - 580 nm'},
{'name': 'Yellow',
'common_name': 'yellow',
'description': 'Yellow: 585 - 625 nm'},
{'name': 'Red', 'common_name': 'red', 'description': 'Red: 630 - 690 nm'},
{'name': 'Red Edge',
'common_name': 'rededge',
'description': 'Red Edge: 705 - 745 nm'},
{'name': 'Near-IR1',
'common_name': 'nir08',
'description': 'Near-IR1: 770 - 895 nm'},
{'name': 'Near-IR2',
'common_name': 'nir09',
'description': 'Near-IR2: 860 - 1040 nm'}]}
Let’s clear the in-memory catalog, add the EO item, and save to a new STAC:
[37]:
catalog.clear_items()
list(catalog.get_items())
[37]:
[]
[38]:
catalog.add_item(eo_item)
list(catalog.get_items())
[38]:
[<Item id=local-image-eo>]
[39]:
catalog.normalize_and_save(root_href=os.path.join(tmp_dir.name, 'stac-eo'),
catalog_type=pystac.CatalogType.SELF_CONTAINED)
Now, if we read the catalog from the filesystem, PySTAC recognizes that the item implements eo and so use it’s functionality, e.g. getting the bands off the asset:
[40]:
catalog2 = pystac.read_file(os.path.join(tmp_dir.name, 'stac-eo', 'catalog.json'))
[41]:
list(catalog2.get_items())
[41]:
[<Item id=local-image-eo>]
[42]:
item = next(catalog2.get_all_items())
[43]:
item.ext.implements('eo')
[43]:
True
[44]:
item.ext.eo.get_bands(item.assets['image'])
[44]:
[<Band name=Coastal>,
<Band name=Blue>,
<Band name=Green>,
<Band name=Yellow>,
<Band name=Red>,
<Band name=Red Edge>,
<Band name=Near-IR1>,
<Band name=Near-IR2>]
Collections¶
Collections are a subtype of Catalog that have some additional properties to make them more searchable. They also can define common properties so that items in the collection don’t have to duplicate common data for each item. Let’s create a collection to hold common properties between two images from the Spacenet 5 challenge.
First we’ll get another image, and it’s bbox and footprint:
[45]:
url2 = ('http://spacenet-dataset.s3.amazonaws.com/'
'spacenet/SN5_roads/train/AOI_7_Moscow/MS/'
'SN5_roads_train_AOI_7_Moscow_MS_chip997.tif')
img_path2 = os.path.join(tmp_dir.name, 'image.tif')
urllib.request.urlretrieve(url2, img_path2)
[45]:
('/var/folders/sv/zr8j0t4j1f726nhlt3vb8c300000gn/T/tmpt1wuelid/image.tif',
<http.client.HTTPMessage at 0x11f9b1208>)
[46]:
bbox2, footprint2 = get_bbox_and_footprint(img_path2)
We can take a look at the pydocs for Collection to see what information we need to supply in order to satisfy the spec.
[47]:
print(pystac.Collection.__doc__)
A Collection extends the Catalog spec with additional metadata that helps
enable discovery.
Args:
id (str): Identifier for the collection. Must be unique within the STAC.
description (str): Detailed multi-line description to fully explain the collection.
`CommonMark 0.28 syntax <http://commonmark.org/>`_ MAY be used for rich text
representation.
extent (Extent): Spatial and temporal extents that describe the bounds of
all items contained within this Collection.
title (str or None): Optional short descriptive one-line title for the collection.
stac_extensions (List[str]): Optional list of extensions the Collection implements.
href (str or None): Optional HREF for this collection, which be set as the collection's
self link's HREF.
license (str): Collection's license(s) as a `SPDX License identifier
<https://spdx.org/licenses/>`_, `various`, or `proprietary`. If collection includes
data with multiple different licenses, use `various` and add a link for each.
Defaults to 'proprietary'.
keywords (List[str]): Optional list of keywords describing the collection.
providers (List[Provider]): Optional list of providers of this Collection.
properties (dict): Optional dict of common fields across referenced items.
summaries (dict): An optional map of property summaries,
either a set of values or statistics such as a range.
extra_fields (dict or None): Extra fields that are part of the top-level JSON properties
of the Collection.
Attributes:
id (str): Identifier for the collection.
description (str): Detailed multi-line description to fully explain the collection.
extent (Extent): Spatial and temporal extents that describe the bounds of
all items contained within this Collection.
title (str or None): Optional short descriptive one-line title for the collection.
stac_extensions (List[str]): Optional list of extensions the Collection implements.
keywords (List[str] or None): Optional list of keywords describing the collection.
providers (List[Provider] or None): Optional list of providers of this Collection.
properties (dict or None): Optional dict of common fields across referenced items.
summaries (dict or None): An optional map of property summaries,
either a set of values or statistics such as a range.
links (List[Link]): A list of :class:`~pystac.Link` objects representing
all links associated with this Collection.
extra_fields (dict or None): Extra fields that are part of the top-level JSON properties
of the Catalog.
Beyond what a Catalog reqiures, a Collection requires a license, and an Extent that describes the range of space and time that the items it hold occupy.
[48]:
print(pystac.Extent.__doc__)
Describes the spatio-temporal extents of a Collection.
Args:
spatial (SpatialExtent): Potential spatial extent covered by the collection.
temporal (TemporalExtent): Potential temporal extent covered by the collection.
Attributes:
spatial (SpatialExtent): Potential spatial extent covered by the collection.
temporal (TemporalExtent): Potential temporal extent covered by the collection.
An Extent is comprised of a SpatialExtent and a TemporalExtent. These hold one or more bounding boxes and time intervals, respectively, that completely cover the items contained in the collections.
Let’s start with creating two new items - these will be core Items. We can set these items to implement the eo extension by specifying them in the stac_extensions.
[49]:
collection_item = pystac.Item(id='local-image-col-1',
geometry=footprint,
bbox=bbox,
datetime=datetime.utcnow(),
properties={},
stac_extensions=[pystac.Extensions.EO])
collection_item.common_metadata.gsd = 0.3
collection_item.common_metadata.platform = 'Maxar'
collection_item.common_metadata.instruments = ['WorldView3']
asset = pystac.Asset(href=img_path,
media_type=pystac.MediaType.GEOTIFF)
collection_item.ext.eo.set_bands(wv3_bands, asset)
collection_item.add_asset('image', asset)
collection_item2 = pystac.Item(id='local-image-col-2',
geometry=footprint2,
bbox=bbox2,
datetime=datetime.utcnow(),
properties={},
stac_extensions=[pystac.Extensions.EO])
collection_item2.common_metadata.gsd = 0.3
collection_item2.common_metadata.platform = 'Maxar'
collection_item2.common_metadata.instruments = ['WorldView3']
asset2 = pystac.Asset(href=img_path,
media_type=pystac.MediaType.GEOTIFF)
collection_item2.ext.eo.set_bands([
band for band in wv3_bands if band.name in ["Red", "Green", "Blue"]
], asset2)
collection_item2.add_asset('image', asset2)
[49]:
<Item id=local-image-col-2>
We can use our two items’ metadata to find out what the proper bounds are:
[50]:
from shapely.geometry import shape
unioned_footprint = shape(footprint).union(shape(footprint2))
collection_bbox = list(unioned_footprint.bounds)
spatial_extent = pystac.SpatialExtent(bboxes=[collection_bbox])
[51]:
collection_interval = sorted([collection_item.datetime, collection_item2.datetime])
temporal_extent = pystac.TemporalExtent(intervals=[collection_interval])
[52]:
collection_extent = pystac.Extent(spatial=spatial_extent, temporal=temporal_extent)
[53]:
collection = pystac.Collection(id='wv3-images',
description='Spacenet 5 images over Moscow',
extent=collection_extent,
license='CC-BY-SA-4.0')
Now if we add our items to our Collection, and our Collection to our Catalog, we get the following STAC that can be saved:
[54]:
collection.add_items([collection_item, collection_item2])
[55]:
catalog.clear_items()
catalog.clear_children()
catalog.add_child(collection)
[56]:
catalog.describe()
* <Catalog id=test-catalog>
* <Collection id=wv3-images>
* <Item id=local-image-col-1>
* <Item id=local-image-col-2>
[57]:
catalog.normalize_and_save(root_href=os.path.join(tmp_dir.name, 'stac-collection'),
catalog_type=pystac.CatalogType.SELF_CONTAINED)
Creating a STAC of imagery from Spacenet 5 data¶
Now, let’s take what we’ve learned and create a Catalog with more data in it.
Allowing PySTAC to read from AWS S3¶
PySTAC aims to be virtually zero-dependency (notwithstanding the why-isn’t-this-in-stdlib datetime-util), so it doesn’t have the ability to read from or write to anything but the local file system. However, we can hook into PySTAC’s IO in the following way. Learn more about how to use STAC_IO in the documentation on the topic:
[59]:
from urllib.parse import urlparse
import boto3
from pystac import STAC_IO
def my_read_method(uri):
parsed = urlparse(uri)
if parsed.scheme == 's3':
bucket = parsed.netloc
key = parsed.path[1:]
s3 = boto3.resource('s3')
obj = s3.Object(bucket, key)
return obj.get()['Body'].read().decode('utf-8')
else:
return STAC_IO.default_read_text_method(uri)
def my_write_method(uri, txt):
parsed = urlparse(uri)
if parsed.scheme == 's3':
bucket = parsed.netloc
key = parsed.path[1:]
s3 = boto3.resource("s3")
s3.Object(bucket, key).put(Body=txt)
else:
STAC_IO.default_write_text_method(uri, txt)
STAC_IO.read_text_method = my_read_method
STAC_IO.write_text_method = my_write_method
We’ll need a utility to list keys for reading the lists of files from S3:
[60]:
# From https://alexwlchan.net/2017/07/listing-s3-keys/
def get_s3_keys(bucket, prefix):
"""Generate all the keys in an S3 bucket."""
s3 = boto3.client('s3')
kwargs = {'Bucket': bucket, 'Prefix': prefix}
while True:
resp = s3.list_objects_v2(**kwargs)
for obj in resp['Contents']:
yield obj['Key']
try:
kwargs['ContinuationToken'] = resp['NextContinuationToken']
except KeyError:
break
Let’s make a STAC of imagery over Moscow as part of the Spacenet 5 challenge. As a first step, we can list out the imagery and extract IDs from each of the chips.
[61]:
moscow_training_chip_uris = list(get_s3_keys(bucket='spacenet-dataset',
prefix='spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS'))
[62]:
import re
chip_id_to_data = {}
def get_chip_id(uri):
return re.search(r'.*\_chip(\d+)\.', uri).group(1)
for uri in moscow_training_chip_uris:
chip_id = get_chip_id(uri)
chip_id_to_data[chip_id] = { 'img': 's3://spacenet-dataset/{}'.format(uri) }
For this tutorial, we’ll only take a subset of the data.
[63]:
chip_id_to_data = dict(list(chip_id_to_data.items())[:10])
[64]:
chip_id_to_data
[64]:
{'0': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip0.tif'},
'1': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1.tif'},
'10': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip10.tif'},
'100': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip100.tif'},
'1000': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1000.tif'},
'1001': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1001.tif'},
'1002': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1002.tif'},
'1003': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1003.tif'},
'1004': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1004.tif'},
'1005': {'img': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1005.tif'}}
Let’s turn each of those chips into a STAC Item that represents the image.
[65]:
chip_id_to_items = {}
We’ll create core Items for our imagery, but mark them with the eo extension as we did above, and store the eo data in a Collection.
Note that the image CRS is in WGS:84 (Lat/Lng). If it wasn’t, we’d have to reproject the footprint to WGS:84 in order to be compliant with the spec (which can easily be done with pyproj).
Here we’re taking advantage of rasterio’s ability to read S3 URIs, which only grabs the GeoTIFF metadata and does not pull the whole file down.
[66]:
for chip_id in chip_id_to_data:
img_uri = chip_id_to_data[chip_id]['img']
print('Processing {}'.format(img_uri))
bbox, footprint = get_bbox_and_footprint(img_uri)
item = pystac.Item(id='img_{}'.format(chip_id),
geometry=footprint,
bbox=bbox,
datetime=datetime.utcnow(),
properties={},
stac_extensions=[pystac.Extensions.EO])
item.common_metadata.gsd = 0.3
item.common_metadata.platform = 'Maxar'
item.common_metadata.instruments = ['WorldView3']
item.ext.eo.bands = wv3_bands
asset = pystac.Asset(href=img_uri,
media_type=pystac.MediaType.COG)
item.ext.eo.set_bands(wv3_bands, asset)
item.add_asset(key='ps-ms', asset=asset)
chip_id_to_items[chip_id] = item
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip0.tif
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1.tif
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip10.tif
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip100.tif
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1000.tif
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1001.tif
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1002.tif
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1003.tif
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1004.tif
Processing s3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/PS-MS/SN5_roads_train_AOI_7_Moscow_PS-MS_chip1005.tif
Creating the Collection¶
All of these images are over Moscow. In Spacenet 5, we have a couple cities that have imagery; a good way to separate these collections of imagery. We can store all of the common eo metadata in the collection.
[67]:
from shapely.geometry import (shape, MultiPolygon)
footprints = list(map(lambda i: shape(i.geometry).envelope,
chip_id_to_items.values()))
collection_bbox = MultiPolygon(footprints).bounds
spatial_extent = pystac.SpatialExtent(bboxes=[collection_bbox])
[68]:
datetimes = sorted(list(map(lambda i: i.datetime,
chip_id_to_items.values())))
temporal_extent = pystac.TemporalExtent(intervals=[[datetimes[0], datetimes[-1]]])
[69]:
collection_extent = pystac.Extent(spatial=spatial_extent, temporal=temporal_extent)
[70]:
collection = pystac.Collection(id='wv3-images',
description='Spacenet 5 images over Moscow',
extent=collection_extent,
license='CC-BY-SA-4.0')
[71]:
collection.add_items(chip_id_to_items.values())
[72]:
collection.describe()
* <Collection id=wv3-images>
* <Item id=img_0>
* <Item id=img_1>
* <Item id=img_10>
* <Item id=img_100>
* <Item id=img_1000>
* <Item id=img_1001>
* <Item id=img_1002>
* <Item id=img_1003>
* <Item id=img_1004>
* <Item id=img_1005>
Now, we can create a Catalog and add the collection.
[73]:
catalog = pystac.Catalog(id='spacenet5', description='Spacenet 5 Data (Test)')
catalog.add_child(collection)
[74]:
catalog.describe()
* <Catalog id=spacenet5>
* <Collection id=wv3-images>
* <Item id=img_0>
* <Item id=img_1>
* <Item id=img_10>
* <Item id=img_100>
* <Item id=img_1000>
* <Item id=img_1001>
* <Item id=img_1002>
* <Item id=img_1003>
* <Item id=img_1004>
* <Item id=img_1005>
Adding items with the label extension to the Spacenet 5 catalog¶
We can use the label extension of the STAC spec to represent the training data in our STAC. For this, we need to grab the URIs of the GeoJSON of roads:
[75]:
moscow_training_geojson_uris = list(get_s3_keys(bucket='spacenet-dataset',
prefix='spacenet/SN5_roads/train/AOI_7_Moscow/geojson_roads_speed/'))
[76]:
for uri in moscow_training_geojson_uris:
chip_id = get_chip_id(uri)
if chip_id in chip_id_to_data:
chip_id_to_data[chip_id]['label'] = 's3://spacenet-dataset/{}'.format(uri)
We’ll add the items to their own subcatalog; since they don’t inherit the Collection’s eo properties, they shouldn’t go in the Collection.
[77]:
label_catalog = pystac.Catalog(id='spacenet-data-labels', description='Labels for Spacenet 5')
catalog.add_child(label_catalog)
To see the required fields for the label extension we can check the pydocs on the apply method of the extension:
[78]:
from pystac.extensions import label
print(label.LabelItemExt.apply.__doc__)
Applies label extension properties to the extended Item.
Args:
label_description (str): A description of the label, how it was created,
and what it is recommended for
label_type (str): An ENUM of either vector label type or raster label type. Use
one of :class:`~pystac.LabelType`.
label_properties (list or None): These are the names of the property field(s) in each
Feature of the label asset's FeatureCollection that contains the classes
(keywords from label:classes if the property defines classes).
If labels are rasters, this should be None.
label_classes (List[LabelClass]): Optional, but reqiured if ussing categorical data.
A list of LabelClasses defining the list of possible class names for each
label:properties. (e.g., tree, building, car, hippo)
label_tasks (List[str]): Recommended to be a subset of 'regression', 'classification',
'detection', or 'segmentation', but may be an arbitrary value.
label_methods: Recommended to be a subset of 'automated' or 'manual',
but may be an arbitrary value.
label_overviews (List[LabelOverview]): Optional list of LabelOverview classes
that store counts (for classification-type data) or summary statistics (for
continuous numerical/regression data).
This loop creates our label items and associates each to the appropriate source image Item.
[79]:
for chip_id in chip_id_to_data:
img_item = collection.get_item('img_{}'.format(chip_id))
label_uri = chip_id_to_data[chip_id]['label']
label_item = pystac.Item(id='label_{}'.format(chip_id),
geometry=img_item.geometry,
bbox=img_item.bbox,
datetime=datetime.utcnow(),
properties={},
stac_extensions=[pystac.Extensions.LABEL])
label_item.ext.label.apply(label_description="SpaceNet 5 Road labels",
label_type=label.LabelType.VECTOR,
label_tasks=['segmentation', 'regression'])
label_item.ext.label.add_source(img_item)
label_item.ext.label.add_geojson_labels(label_uri)
label_catalog.add_item(label_item)
Now we have a STAC of training data!
[80]:
catalog.describe()
* <Catalog id=spacenet5>
* <Collection id=wv3-images>
* <Item id=img_0>
* <Item id=img_1>
* <Item id=img_10>
* <Item id=img_100>
* <Item id=img_1000>
* <Item id=img_1001>
* <Item id=img_1002>
* <Item id=img_1003>
* <Item id=img_1004>
* <Item id=img_1005>
* <Catalog id=spacenet-data-labels>
* <Item id=label_0>
* <Item id=label_1>
* <Item id=label_10>
* <Item id=label_100>
* <Item id=label_1000>
* <Item id=label_1001>
* <Item id=label_1002>
* <Item id=label_1003>
* <Item id=label_1004>
* <Item id=label_1005>
[81]:
label_item = catalog.get_child('spacenet-data-labels').get_item('label_1')
label_item.to_dict()
[81]:
{'type': 'Feature',
'stac_version': '1.0.0-beta.2',
'id': 'label_1',
'properties': {'label:description': 'SpaceNet 5 Road labels',
'label:type': 'vector',
'label:properties': None,
'label:tasks': ['segmentation', 'regression'],
'datetime': '2020-08-03T03:47:56.599629Z'},
'geometry': {'type': 'Polygon',
'coordinates': (((37.68191035616281, 55.73478210707574),
(37.68191035616281, 55.73882710285011),
(37.68595535193718, 55.73882710285011),
(37.68595535193718, 55.73478210707574),
(37.68191035616281, 55.73478210707574)),)},
'links': [{'rel': 'source', 'href': None, 'type': 'application/json'},
{'rel': 'root', 'href': None, 'type': 'application/json'},
{'rel': 'parent', 'href': None, 'type': 'application/json'}],
'assets': {'labels': {'href': 's3://spacenet-dataset/spacenet/SN5_roads/train/AOI_7_Moscow/geojson_roads_speed/SN5_roads_train_AOI_7_Moscow_geojson_roads_speed_chip1.geojson',
'type': 'application/geo+json'}},
'bbox': [37.68191035616281,
55.73478210707574,
37.68595535193718,
55.73882710285011],
'stac_extensions': ['label']}
[ ]:
