Concepts¶
This page will give an overview of some important concepts to understand when working with PySTAC. If you want to check code examples, see the Tutorials.
Reading STACs¶
PySTAC can read STAC data from JSON. Generally users read in the root catalog, and then use the python objects to crawl through the data. Once you read in the root of the STAC, you can work with the STAC in memory.
from pystac import Catalog
catalog = Catalog.from_file('/some/example/catalog.json')
for root, catalogs, items in catalog.walk():
# Do interesting things with the STAC data.
To see how to hook into PySTAC for reading from alternate URIs such as cloud object storage, see Using STAC_IO.
Writing STACs¶
While working with STACs in-memory don’t require setting file paths, in order to save a STAC,
you’ll need to give each STAC object a self
link that describes the location of where
it should be saved to. Luckily, PySTAC makes it easy to create a STAC catalog with a canonical layout and with the links that follow the best practices. You simply call normalize_hrefs
with the root directory of where the STAC will be saved, and then call save
with the type of catalog (described in the Catalog Types section) that matches your use case.
from pystac import (Catalog, CatalogType)
catalog = Catalog.from_file('/some/example/catalog.json')
catalog.normalize_hrefs('/some/copy/')
catalog.save(catalog_type=CatalogType.SELF_CONTAINED)
copycat = Catalog.from_file('/some/copy/catalog.json')
Normalizing HREFs¶
The normalize_hrefs
call sets HREFs for all the links in the STAC according to the
Catalog, Collection and Items, all based off of the root URI that is passed in:
catalog.normalize_hrefs('/some/location')
catalog.save(catalog_type=CatalogType.SELF_CONTAINED)
This will lay out the HREFs of the STAC according to the best practices document.
Layouts¶
PySTAC provides a few different strategies for laying out the HREFs of a STAC. To use them you can pass in a strategy to the normalize_hrefs call.
Using templates¶
You can utilze template strings to determine the file paths of HREFs set on Catalogs, Collection or Items. These templates use python format strings, which can name the property or attribute of the item you want to use for replacing the template variable. For example:
from pystac.layout import TemplateLayoutStrategy
strategy = TemplateLayoutStrategy(item_template="${collection}/${year}/${month}")
catalog.normalize_hrefs('/some/location', strategy=strategy)
catalog.save(catalog_type=CatalogType.SELF_CONTAINED)
The above code will save items in subfolders based on the collection ID, year and month of it’s datetime (or start_datetime if a date range is defined and no datetime is defined).
You can use dot notation to specify attributes of objects or keys in dictionaries for
template variables. PySTAC will look at the object, it’s properties
and its extra_fields
for property names or dictionary keys. Some special cases, like year
, month
, day
and
date
exist for datetime on Items, as well as collection
for Item’s Collection’s ID.
See the documentation on LayoutTemplate
for more documentation on
how layout templates work.
Using custom functions¶
If you want to build your own strategy, you can subclass HrefLayoutStrategy
or use
CustomLayoutStrategy
to provide functions that work with
Catalogs, Collections or Items. Similar to the templating strategy, you can provide a
fallback strategy (which defaults to BestPracticesLayoutStrategy
)
for any stac object type that you don’t supply a function for.
Catalog Types¶
The STAC best practices document lays out different catalog types, and how their links should be formatted. A brief description is below, but check out the document for the official take on these types:
The catalog types will also dictate the asset HREF formats. Asset HREFs in any catalog type can be relative or absolute may be absolute depending on their location; see the section on Relative vs Absolute Asset HREFs below.
Self-Contained Catalogs¶
A self-contained catalog (indicated by catalog_type=CatalogType.SELF_CONTAINED
) applies
to STACs that do not have a long term location, and can be moved around. These STACs are
useful for copying data to and from locations, without having to change any link metadata.
A self-contained catalog has two important properties:
- It contains only relative links
- It contains no self links.
For a catalog that is the most easy to copy around, it’s recommended that item assets use relative links, and reside in the same directory as the item’s STAC metadata file.
Relative Published Catalogs¶
A relative published catalog (indicated by catalog_type=CatalogType.RELATIVE_PUBLISHED
) is
one that is tied at it’s root to a specific location, but otherwise contains relative links.
This is designed so that a self-contained catalog can be ‘published’ online by just adding
one field (the self link) to its root catalog.
A relative published catalog has the following properties:
- It contains only one self link: the root of the catalog contains a (necessarily absolute) link to it’s published location.
- All other objects in the STAC contain relative links, and no self links.
Absolute Published Catalogs¶
An absolute published catalog (indicated by catalog_type=CatalogType.ABSOLUTE_PUBLISHED
) uses absolute links for everything. It is preferable where possible, since it allows for the easiest provenance tracking out of all the catalog types.
An absolute published catalog has the following properties:
- Each STAC object contains only absolute links.
- Each STAC object has a self link.
It is not recommended to have relative asset HREFs in an absolute published catalog.
Relative vs Absolute HREFs¶
HREFs inside a STAC for either links or assets can be relative or absolute.
Relative vs Absolute Link HREFs¶
Absolute links point to their file locations in a fully described way. Relative links
are relative to the linking object’s file location. For example, if a catalog at
/some/location/catalog.json
has a link to an item that has an HREF set to item-id/item-id.json
,
then that link should resolve to the absolute path /some/location/item-id/item-id.json
.
Links are set as absolute or relative HREFs at save time, as determine by the root catalog’s catalog_type
:attribute:`~pystac.Catalog.catalog_type`. This means that, even if the stored HREF of the link is absolute,
if the root catalog_type=CatalogType.RELATIVE_PUBLISHED
or catalog_type=CatalogType.SELF_CONTAINED
and subsequent serializing of the any links in the catalog will produce a relative link,
based on the self link of the parent object.
You can make all the links of a catalog relative or absolute by setting the Catalog.catalog_type()
field
then resaving the entire catalog.
Relative vs Absolute Asset HREFs¶
Asset HREFs can also be relative or absolute. If an asset HREF is relative, then it is relative to the Item’s metadata file. For example, if the item at /some/location/item-id/item-id.json
had an asset with an HREF of ./image.tif
, then the fully resolved path for that image would be /some/location/item-id/image.tif
You can make all the asset HREFs of a catalog relative or absolute using the Catalog.make_all_asset_hrefs_relative
and Catalog.make_all_asset_hrefs_absolute
methods. Note that these will not move any files around, and if the file location does not share a common parent with the asset’s item’s self HREF, then the asset HREF will remain absolute as no relative path is possible.
Including a self
link¶
Every stac object has a save_object()
method, that takes as an argument whether or not to include the object’s self link. As noted in the section on Catalog Types, a self link is necessarily absolute; if an object only contains relative links, then it cannot contain the self link. PySTAC uses self links as a way of tracking the object’s file location, either what it was read from or it’s pending save location, so each object can have a self link even if you don’t ever want that self link written (e.g. if you are working with self-contained catalogs).
Using STAC_IO¶
The STAC_IO
class is the way PySTAC reads and writes text from file locations. Since PySTAC aims to be dependency-free, there is no default mechanisms to read and write from anything but the local file system. However, users of PySTAC may want to read and write from other file systems, such as HTTP or cloud object storage. STAC_IO allows users to hook into PySTAC and define their own reading and writing primitives to allow for those use cases.
To enable reading from other types of file systems, it is recommended that in the __init__.py of the client module, or at the beginning of the script using PySTAC, you overwrite the STAC_IO.read_text_method
and STAC_IO.write_text_method
members of STAC_IO with functions that read and write however you need. For example, this code will allow for reading from AWS’s S3 cloud object storage using boto3:
from urllib.parse import urlparse
import boto3
from pystac import STAC_IO
def my_read_method(uri):
parsed = urlparse(uri)
if parsed.scheme == 's3':
bucket = parsed.netloc
key = parsed.path[1:]
s3 = boto3.resource('s3')
obj = s3.Object(bucket, key)
return obj.get()['Body'].read().decode('utf-8')
else:
return STAC_IO.default_read_text_method(uri)
def my_write_method(uri, txt):
parsed = urlparse(uri)
if parsed.scheme == 's3':
bucket = parsed.netloc
key = parsed.path[1:]
s3 = boto3.resource("s3")
s3.Object(bucket, key).put(Body=txt)
else:
STAC_IO.default_write_text_method(uri, txt)
STAC_IO.read_text_method = my_read_method
STAC_IO.write_text_method = my_write_method
If you are only going to read from another source, e.g. HTTP, you could only replace the read method. For example, using the requests library:
from urllib.parse import urlparse
import requests
from pystac import STAC_IO
def my_read_method(uri):
parsed = urlparse(uri)
if parsed.scheme.startswith('http'):
return requests.get(uri).text
else:
return STAC_IO.default_read_text_method(uri)
STAC_IO.read_text_method = my_read_method
Validation¶
PySTAC includes validation functionality that allows users to validate PySTAC objects as well JSON-encoded STAC objects from STAC versions 0.8.0 and later.
Enabling validation¶
To enable the validation feature you’ll need to have installed PySTAC with the optional dependency via:
> pip install pystac[validation]
This installs the jsonschema
package which is used with the default validator. If you
define your own validation class as described below, you are not required to have this
extra dependency.
Validating PySTAC objects¶
You can validate any Catalog
, Collection
or Item
by calling the validate()
method:
item.validate()
This will validate against the latest set of JSON schemas hosted at https://schemas.stacspec.org, including any extensions that the object extends. If there are validation errors, a STACValidationError
will be raised.
You can also call validate_all()
on a Catalog or Collection to recursively walk through a catalog and validate all objects within it.
catalog.validate_all()
Validating STAC JSON¶
You can validate STAC JSON represented as a dict
using the pystac.validation.validate_dict()
method:
import json
from pystac.validation import validate_dict
with open('/path/to/item.json') as f:
js = json.load(f)
validate_dict(js)
You can also recursively validate all of the catalogs, collections and items across STAC versions
using the pystac.validation.validate_all()
method:
import json
from pystac.validation import validate_all
with open('/path/to/catalog.json') as f:
js = json.load(f)
validate_all(js)
Using your own validator¶
By default PySTAC uses the JsonSchemaSTACValidator
implementation for validation. Users can define their own implementations of STACValidator
and register it with pystac using pystac.validation.set_validator()
.
The JsonSchemaSTACValidator
takes a SchemaUriMap
, which by default uses the DefaultSchemaUriMap
. If desirable, users cn create their own implementation of SchemaUriMap
and register a new instance of JsonSchemaSTACValidator
using that schema map with pystac.validation.set_validator()
.
Extensions¶
Accessing Extension functionality¶
All STAC objects are accessed through Catalog
, Collection
and Item
, and all extension functionality
is accessed through the ext
property on those objects. For instance, to access the band information
from the eo
extension for an item that implements the extension, you use:
# All of the below are equivalent:
item.ext['eo'].bands
item.ext[pystac.Extensions.EO].bands
item.ext.eo.bands
Notice the eo
property on ext
- this utilizes the __getattr__ method to delegate the property name to the __getitem__
method, so we can access any registered extension as if it were a property on ext
.
Extensions wrap the objects they extend. Extensions hold no values of their own, but instead use Python properties to directly modify the values of the objects they wrap.
Any object that is returned by extension methods therefore also wrap components of the STAC objects.
For instance, the LabelClasses
holds a reference to the original Item
’s label:classes
property, so that
modifying the LabelClasses
properties through the setters will modify the item properties directly. For example:
from pystac.extensions import label
label_classes = item.ext.label.label_classes
label_classes[0].classes.append("other_class")
assert "other_class" in item.properties['label:classes'][0]['classes']
Because these objects wrap the object’s dictionary, the __init__ methods need to take the
dict
they wrap. Therefore to create a new object, use the class’s .create method, for example:
item.ext.label.label_classes = [label.LabelClasses.create(['class1', 'class2'], name='label')]
An apply method is available in extension wrappers and any objects that they return. This allows you to pass in property values pertaining to the extension. These will require arguments for properties required as part of the extension specification and have None default values for optional parameters:
eo_ext = item.ext.eo
eo_ext.apply(0.5, bands, cloud_cover=None) # Do not have to specify cloud_cover
If you attempt to retrieve an extension wrapper for an extension that the object doesn’t implement, PySTAC will throw a pystac.extensions.ExtensionError.
Enabling an extension¶
You’ll need to enable an extension on an object before using it. For example, if you are creating an Item and want to apply the label extension, you can do so in two ways.
You can add the extension in the list of extensions when you create the Item:
item = Item(id='Labels',
geometry=item.geometry,
bbox=item.bbox,
datetime=datetime.utcnow(),
properties={},
stac_extensions=[pystac.Extensions.LABEL])
or you can call ext.enable
on an Item (which will work for any item, whether you created it or are modifying it):
item = Item(id='Labels',
geometry=item.geometry,
bbox=item.bbox,
datetime=datetime.utcnow(),
properties={})
item.ext.enable(pystac.Extensions.LABEL)
Item Asset properties¶
Properties that apply to Items can be found in two places: the Item’s properties or in any of an Item’s Assets. If the property is on an Asset, it applies only that specific asset. For example, gsd defined for an Item represents the best Ground Sample Distance (resolution) for the data within the Item. However, some assets may be lower resolution and thus have a higher gsd. In that case, the gsd can be found on the Asset.
See the STAC documentation on Additional Fields for Assets and the relevant Best Practices for more information.
The implementation of this feature in PySTAC uses the method described here and is consistent across Item and ItemExtensions. The bare property names represent values for the Item only, but for each property where it is possible to set on both the Item or the Asset there is a get_
and set_
methods that optionally take an Asset. For the get_
methods, if the property is found on the Asset, the Asset’s value is used; otherwise the Item’s value will be used. For the set_
method, if an Asset is passed in the value will be applied to the Asset and not the Item.
For example, if we have an Item with a gsd
of 10 with three bands, and only asset “band3” having a gsd
of 20, the get_gsd
method will behave in the following way:
assert item.common_metadata.gsd == 10 assert item.common_metadata.get_gsd() == 10 assert item.common_metadata.get_gsd(item.asset['band1']) == 10 assert item.common_metadata.get_gsd(item.asset['band3']) == 20
Similarly, if we set the asset at ‘band2’ to have a gsd
of 30, it will only affect that asset:
item.common_metadata.set_gsd(30, item.assets['band2'] assert item.common_metadata.gsd == 10 assert item.common_metadata.get_gsd(item.asset['band2']) == 30
Manipulating STACs¶
PySTAC is designed to allow for STACs to be manipulated in-memory. This includes Copying STACs in-memory, walking over all objects in a STAC and mutating their properties, or using collection-style map methods for mapping over items.
Walking over a STAC¶
You can walk through all sub-catalogs and items of a catalog with a method inspired
by the Python Standard Library os.walk() method: Catalog.walk()
:
for root, subcats, items in catalog.walk():
# Root represents a catalog currently being walked in the tree
root.title = '{} has been walked!'.format(root.id)
# subcats represents any catalogs or collections owned by root
for cat in subcatalogs:
cat.title = 'About to be walked!'
# items represent all items that are contained by root
for item in items:
item.title = '{} - owned by {}'.format(item.id, root.id)
Mapping over Items¶
The Catalog.map_items
method is useful for manipulating items in a STAC. This will create a full copy of the STAC, so will leave the original catalog unmodified. In the method that manipulates and returns the modified item, you can return multiple items, in case you are generating new objects (e.g. creating a LabelItem
for image items in a stac), or splitting items into smaller chunks (e.g. tiling out large image items).
def modify_item_title(item):
item.title = 'Some new title'
return item
def create_label_item(item):
# Assumes the GeoJSON labels are in the
# same location as the image
img_href = item.assets['ortho'].href
label_href = '{}.geojson'.format(os.path.splitext(img_href)[0])
label_item = LabelItem(id='Labels',
geometry=item.geometry,
bbox=item.bbox,
datetime=datetime.utcnow(),
properties={},
label_description='labels',
label_type='vector',
label_properties='label',
label_classes=[
LabelClasses(classes=['one', 'two'],
name='label')
],
label_tasks=['classification'])
label_item.add_source(item, assets=['ortho'])
label_item.add_geojson_labels(label_href)
return [item, label_item]
c = catalog.map_items(modify_item_title)
c = c.map_items(create_label_item)
new_catalog = c
Copying STACs in-memory¶
The in-memory copying of STACs to create new ones is crucial to correct manipulations and mutations of STAC data. The STACObject.full_copy
mechanism handles this in a way that ties the elements of the copies STAC together correctly. This includes situations where there might be cycles in the graph of connected objects of the STAC (which otherwise would be a tree). For example, if a LabelItem
lists a source
that is an item also contained in the root catalog; the full copy of the STAC will ensure that the Item
instance representing the source imagery item is the same instance that is linked to by the LabelItem
.
Resolving STAC objects¶
PySTAC tries to only “resolve” STAC Objects - that is, load the metadata contained by STAC files pointed to by links into Python objects in-memory - when necessary. It also ensures that two links that point to the same object resolve to the same in-memory object.
Lazy resolution of STAC objects¶
Links are read only when they need to be. For instance, when you load a catalog using Catalog.from_file
, the catalog and all of its links are read into a Catalog
instance. If you iterate through Catalog.links
, you’ll see the target
of the Link
will refer to a string - that is the HREF of the link. However, if you call Catalog.get_items
, for instance, you’ll get back the actual Item
instances that are referred to by each item link in the Catalog. That’s because at the time you call get_items
, PySTAC is “resolving” the links for any link that represents an item in the catalog.
The resolution mechanism is accomplished through Link.resolve_stac_object
. Though this method is used extensively internally to PySTAC, ideally this is completely transparent to users of PySTAC, and you won’t have to worry about how and when links get resolved. However, one important aspect to understand is how object resolution caching happens.
Resolution Caching¶
The root Catalog
instance of a STAC (the Catalog which is linked to by every associated object’s root
link) contains a cache of resolved objects. This cache points to in-memory instances of STACObject
s that have already been resolved through PySTAC crawling links associated with that root catalog. The cache works off of the stac object’s ID, which is why it is necessary for every STAC object in the catalog to have a unique identifier, which is unique across the entire STAC.
When a link is being resolved from a STACObject that has it’s root set, that root is passed into the Link.resolve_stac_object
call. That root’s ResolvedObjectCache
will be used to ensure that if the link is pointing to an object that has already been resolved, then that link will point to the same, single instance in the cache. This ensures working with STAC objects in memory doesn’t create a situation where multiple copies of the same STAC objects are created from different links, manipulated, and written over each other.
Working with STAC JSON¶
The pystac.serialization
package has some functionality around working directly with STAC
JSON objects, without utilizing PySTAC object types. This is used internally by PySTAC, but might also be useful to users working directly with JSON (e.g. on validation).
Identifying STAC objects from JSON¶
Users can identify STAC information, including the object type, version and extensions,
from JSON. The main method for this is identify_stac_object()
,
which returns an object that contains the object type, the range of versions this object is
valid for (according to PySTAC’s best guess), the common extensions implemented by this object,
and any custom extensions (represented by URIs to JSON Schemas).
from pystac.serialization import identify_stac_object
json_dict = ...
info = identify_stac_object(json_dict)
# The object type
info.object_type
# The version range
info.version_range
# The common extensions
info.common_extensions
# The custom Extensions
info.custom_extensions
Merging common properties¶
For pre-1.0.0 STAC, The merge_common_properties()
will take a JSON dict that represents an item, and if it is associated with a collection, merge in the collection’s properties. You can pass in a dict that contains previously read collections that caches collections by the HREF of the collection link and/or the collection ID, which can help avoid multiple reads of
collection links.
Note that this feature was dropped in STAC 1.0.0-beta.1