Concepts#

This page will give an overview of some important concepts to understand when working with PySTAC. If you want to check code examples, see the Tutorials.

STAC Spec Version Support#

The latest version of PySTAC supports STAC Spec 1.0.0 and will automatically update any catalogs to this version. To work with older versions of the STAC Spec, please use an older version of PySTAC:

STAC Spec Version

PySTAC Version

>=1.0

Latest

0.9

0.4.*

0.8

0.3.*

<0.8

Not supported

Reading STACs#

PySTAC can read STAC data from JSON. Generally users read in the root catalog, and then use the python objects to crawl through the data. Once you read in the root of the STAC, you can work with the STAC in memory.

from pystac import Catalog

catalog = Catalog.from_file('/some/example/catalog.json')

for root, catalogs, items in catalog.walk():
    # Do interesting things with the STAC data.

To see how to hook into PySTAC for reading from alternate URIs such as cloud object storage, see I/O in PySTAC.

Writing STACs#

While working with STACs in-memory don’t require setting file paths, in order to save a STAC, you’ll need to give each STAC object a self link that describes the location of where it should be saved to. Luckily, PySTAC makes it easy to create a STAC catalog with a canonical layout and with the links that follow the best practices. You simply call normalize_hrefs with the root directory of where the STAC will be saved, and then call save with the type of catalog (described in the Catalog Types section) that matches your use case.

from pystac import (Catalog, CatalogType)

catalog = Catalog.from_file('/some/example/catalog.json')
catalog.normalize_hrefs('/some/copy/')
catalog.save(catalog_type=CatalogType.SELF_CONTAINED)

copycat = Catalog.from_file('/some/copy/catalog.json')

Normalizing HREFs#

The normalize_hrefs call sets HREFs for all the links in the STAC according to the Catalog, Collection and Items, all based off of the root URI that is passed in:

catalog.normalize_hrefs('/some/location')
catalog.save(catalog_type=CatalogType.SELF_CONTAINED)

This will lay out the HREFs of the STAC according to the best practices document.

Layouts#

PySTAC provides a few different strategies for laying out the HREFs of a STAC. To use them you can pass in a strategy to the normalize_hrefs call.

Using templates#

You can utilize template strings to determine the file paths of HREFs set on Catalogs, Collection or Items. These templates use python format strings, which can name the property or attribute of the item you want to use for replacing the template variable. For example:

from pystac.layout import TemplateLayoutStrategy

strategy = TemplateLayoutStrategy(item_template="${collection}/${year}/${month}")
catalog.normalize_hrefs('/some/location', strategy=strategy)
catalog.save(catalog_type=CatalogType.SELF_CONTAINED)

The above code will save items in subfolders based on the collection ID, year and month of it’s datetime (or start_datetime if a date range is defined and no datetime is defined). Note that the forward slash (/) should be used as path separator in the template string regardless of the system path separator (thus both in POSIX-compliant and Windows environments).

You can use dot notation to specify attributes of objects or keys in dictionaries for template variables. PySTAC will look at the object, it’s properties and its extra_fields for property names or dictionary keys. Some special cases, like year, month, day and date exist for datetime on Items, as well as collection for Item’s Collection’s ID.

See the documentation on LayoutTemplate for more documentation on how layout templates work.

Using custom functions#

If you want to build your own strategy, you can subclass HrefLayoutStrategy or use CustomLayoutStrategy to provide functions that work with Catalogs, Collections or Items. Similar to the templating strategy, you can provide a fallback strategy (which defaults to BestPracticesLayoutStrategy) for any stac object type that you don’t supply a function for.

Catalog Types#

The STAC best practices document lays out different catalog types, and how their links should be formatted. A brief description is below, but check out the document for the official take on these types:

The catalog types will also dictate the asset HREF formats. Asset HREFs in any catalog type can be relative or absolute may be absolute depending on their location; see the section on Relative vs Absolute Asset HREFs below.

Self-Contained Catalogs#

A self-contained catalog (indicated by catalog_type=CatalogType.SELF_CONTAINED) applies to STACs that do not have a long term location, and can be moved around. These STACs are useful for copying data to and from locations, without having to change any link metadata.

A self-contained catalog has two important properties:

  • It contains only relative links

  • It contains no self links.

For a catalog that is the most easy to copy around, it’s recommended that item assets use relative links, and reside in the same directory as the item’s STAC metadata file.

Relative Published Catalogs#

A relative published catalog (indicated by catalog_type=CatalogType.RELATIVE_PUBLISHED) is one that is tied at it’s root to a specific location, but otherwise contains relative links. This is designed so that a self-contained catalog can be ‘published’ online by just adding one field (the self link) to its root catalog.

A relative published catalog has the following properties:

  • It contains only one self link: the root of the catalog contains a (necessarily absolute) link to it’s published location.

  • All other objects in the STAC contain relative links, and no self links.

Absolute Published Catalogs#

An absolute published catalog (indicated by catalog_type=CatalogType.ABSOLUTE_PUBLISHED) uses absolute links for everything. It is preferable where possible, since it allows for the easiest provenance tracking out of all the catalog types.

An absolute published catalog has the following properties:

  • Each STAC object contains only absolute links.

  • Each STAC object has a self link.

It is not recommended to have relative asset HREFs in an absolute published catalog.

Relative vs Absolute HREFs#

HREFs inside a STAC for either links or assets can be relative or absolute.

Relative vs Absolute Asset HREFs#

Asset HREFs can also be relative or absolute. If an asset HREF is relative, then it is relative to the Item’s metadata file. For example, if the item at /some/location/item-id/item-id.json had an asset with an HREF of ./image.tif, then the fully resolved path for that image would be /some/location/item-id/image.tif

You can make all the asset HREFs of a catalog relative or absolute using the Catalog.make_all_asset_hrefs_relative and Catalog.make_all_asset_hrefs_absolute methods. Note that these will not move any files around, and if the file location does not share a common parent with the asset’s item’s self HREF, then the asset HREF will remain absolute as no relative path is possible.

I/O in PySTAC#

The pystac.StacIO class defines fundamental methods for I/O operations within PySTAC, including serialization and deserialization to and from JSON files and conversion to and from Python dictionaries. This is an abstract class and should not be instantiated directly. However, PySTAC provides a pystac.stac_io.DefaultStacIO class with minimal implementations of these methods. This default implementation provides support for reading and writing files from the local filesystem as well as HTTP URIs (using urllib). This class is created automatically by all of the object-specific I/O methods (e.g. pystac.Catalog.from_file()), so most users will not need to instantiate this class themselves.

If you are dealing with a STAC catalog with URIs that require authentication. It is possible provide auth headers (or any other customer headers) to the pystac.stac_io.DefaultStacIO.

from pystac import Catalog
from pystac import StacIO

stac_io = StacIO.default()
stac_io.headers = {"Authorization": "<some-auth-header>"}

catalog = Catalog.from_file("<URI-requiring-auth>", stac_io=stac_io)

You can double check that requests PySTAC is making by adjusting logging level so that you see all API calls.

import logging

logging.basicConfig()
logger = logging.getLogger('pystac')
logger.setLevel(logging.DEBUG)

If you require more custom logic for I/O operations or would like to use a 3rd-party library for I/O operations (e.g. requests), you can create a sub-class of pystac.StacIO (or pystac.DefaultStacIO) and customize the methods as you see fit. You can then pass instances of this custom sub-class into the stac_io argument of most object-specific I/O methods. You can also use pystac.StacIO.set_default() in your client’s __init__.py file to make this sub-class the default pystac.StacIO implementation throughout the library.

For example, this code will allow for reading from AWS’s S3 cloud object storage using boto3:

from urllib.parse import urlparse
import boto3
from pystac import Link
from pystac.stac_io import DefaultStacIO, StacIO

class CustomStacIO(DefaultStacIO):
   def __init__(self):
      self.s3 = boto3.resource("s3")

   def read_text(
      self, source: Union[str, Link], *args: Any, **kwargs: Any
   ) -> str:
      parsed = urlparse(source)
      if parsed.scheme == "s3":
         bucket = parsed.netloc
         key = parsed.path[1:]

         obj = self.s3.Object(bucket, key)
         return obj.get()["Body"].read().decode("utf-8")
      else:
         return super().read_text(source, *args, **kwargs)

   def write_text(
      self, dest: Union[str, Link], txt: str, *args: Any, **kwargs: Any
   ) -> None:
      parsed = urlparse(dest)
      if parsed.scheme == "s3":
         bucket = parsed.netloc
         key = parsed.path[1:]
         self.s3.Object(bucket, key).put(Body=txt, ContentEncoding="utf-8")
      else:
         super().write_text(dest, txt, *args, **kwargs)

StacIO.set_default(CustomStacIO)

If you only need to customize read operations you can inherit from DefaultStacIO and only overwrite the read method. For example, to take advantage of connection pooling using a requests.Session:

from urllib.parse import urlparse
import requests
from pystac.stac_io import DefaultStacIO, StacIO

class ConnectionPoolingIO(DefaultStacIO):
   def __init__(self):
      self.session = requests.Session()

   def read_text(
      self, source: Union[str, Link], *args: Any, **kwargs: Any
   ) -> str:
      parsed = urlparse(uri)
      if parsed.scheme.startswith("http"):
         return self.session.get(uri).text
      else:
         return super().read_text(source, *args, **kwargs)

StacIO.set_default(ConnectionPoolingIO)

Validation#

PySTAC includes validation functionality that allows users to validate PySTAC objects as well JSON-encoded STAC objects from STAC versions 0.8.0 and later.

Enabling validation#

To enable the validation feature you’ll need to have installed PySTAC with the optional dependency via:

> pip install pystac[validation]

This installs the jsonschema package which is used with the default validator. If you define your own validation class as described below, you are not required to have this extra dependency.

Validating PySTAC objects#

You can validate any Catalog, Collection or Item by calling the validate() method:

item.validate()

This validates against the latest set of JSON schemas (which are included with the PySTAC package) or older versions (which are hosted at https://schemas.stacspec.org). This validation includes any extensions that the object extends (these are always accessed remotely based on their URIs).

If there are validation errors, a STACValidationError is raised.

You can also call validate_all() on a Catalog or Collection to recursively walk through a catalog and validate all objects within it.

catalog.validate_all()

Validating STAC JSON#

You can validate STAC JSON represented as a dict using the pystac.validation.validate_dict() method:

import json
from pystac.validation import validate_dict

with open('/path/to/item.json') as f:
    js = json.load(f)
validate_dict(js)

You can also recursively validate all of the catalogs, collections and items across STAC versions using the pystac.validation.validate_all() method:

import json
from pystac.validation import validate_all

with open('/path/to/catalog.json') as f:
    js = json.load(f)
validate_all(js)

Using your own validator#

By default PySTAC uses the JsonSchemaSTACValidator implementation for validation. Users can define their own implementations of STACValidator and register it with pystac using pystac.validation.set_validator().

The JsonSchemaSTACValidator takes a SchemaUriMap, which by default uses the DefaultSchemaUriMap. If desirable, users cn create their own implementation of SchemaUriMap and register a new instance of JsonSchemaSTACValidator using that schema map with pystac.validation.set_validator().

Extensions#

From the documentation on STAC Spec Extensions:

Extensions to the core STAC specification provide additional JSON fields that can be used to better describe the data. Most tend to be about describing a particular domain or type of data, but some imply functionality.

This library makes an effort to support all extensions that are part of the stac-extensions GitHub org, and we are committed to supporting all STAC Extensions at the “Candidate” maturity level or above (see the Extension Maturity documentation for details).

Accessing Extension Functionality#

Extension functionality is encapsulated in classes that are specific to the STAC Extension (e.g. Electro-Optical, Projection, etc.) and STAC Object (Collection, pystac.Item, or pystac.Asset). All classes that extend these objects inherit from pystac.extensions.base.PropertiesExtension, and you can use the ext method on these classes to extend an object.

For instance, if you have an item that implements the Electro-Optical Extension, you can access the properties associated with that extension using EOExtension.ext:

import pystac
from pystac.extensions.eo import EOExtension

item = pystac.Item.from_file("tests/data-files/eo/eo-landsat-example.json")

# Check that the Item implements the EO Extension
if EOExtension.has_extension(item):
   eo_ext = EOExtension.ext(item)

   bands = eo_ext.bands
   cloud_cover = eo_ext.cloud_cover
   snow_cover = eo_ext.snow_cover
   ...

Note

The ext method will raise an ExtensionNotImplemented exception if the object does not implement that extension (e.g. if the extension URI is not in that object’s stac_extensions list). In the example above, we check that the Item implements the EO Extension before calling EOExtension.ext to handle this. See the Adding an Extension section below for details on adding an extension to an object.

See the documentation for each extension implementation for details on the supported properties and other functionality.

Instances of PropertiesExtension have a properties attribute that gives access to the properties of the extended object. This attribute is a reference to the properties of the Item or Asset being extended and can therefore mutate those properties. For instance:

item = pystac.Item.from_file("tests/data-files/eo/eo-landsat-example.json")
print(item.properties["eo:cloud_cover"])
# 78

eo_ext = EOExtension.ext(item)
print(eo_ext.cloud_cover)
# 78

eo_ext.cloud_cover = 45
print(item.properties["eo:cloud_cover"])
# 45

There is also a additional_read_properties attribute that, if present, gives read-only access to properties of any objects that own the extended object. For instance, an extended pystac.Asset instance would have read access to the properties of the pystac.Item that owns it (if there is one). If a property exists in both additional_read_properties and properties, the value in additional_read_properties will take precedence.

An apply method is available on extended objects. This allows you to pass in property values pertaining to the extension. Properties that are required by the extension will be required arguments to the apply method. Optional properties will have a default value of None:

# Can also omit cloud_cover entirely...
eo_ext.apply(0.5, bands, cloud_cover=None)

If you attempt to extend an object that is not supported by an extension, PySTAC will throw a pystac.ExtensionTypeError.

Adding an Extension#

You can add an extension to a STAC object that does not already implement that extension using the ExtensionManagementMixin.add_to method. Any concrete extension implementations that extend existing STAC objects should inherit from the ExtensionManagementMixin class, and will therefore have this method available. The add_to() adds the correct schema URI to the stac_extensions list for the object being extended.

# Load a basic item without any extensions
item = pystac.Item.from_file("tests/data-files/item/sample-item.json")
print(item.stac_extensions)
# []

# Add the Electro-Optical extension
EOExtension.add_to(item)
print(item.stac_extensions)
# ['https://stac-extensions.github.io/eo/v1.1.0/schema.json']

Extended Summaries#

Extension classes like ProjectionExtension may also provide a summaries static method that can be used to extend the Collection summaries. This method returns a class inheriting from pystac.extensions.base.SummariesExtension that provides tools for summarizing the properties defined by that extension. These classes also hold a reference to the Collection’s pystac.Summaries instance in the summaries attribute.

import pystac
from pystac.extensions.projection import ProjectionExtension

# Load a collection that does not implement the Projection extension
collection = pystac.Collection.from_file(
   "tests/data-files/examples/1.0.0/collection.json"
)

# Add Projection extension summaries to the collection
proj = ProjectionExtension.summaries(collection, add_if_missing=True)
print(collection.stac_extensions)
# [
#     ....,
#     'https://stac-extensions.github.io/projection/v1.1.0/schema.json'
# ]

# Set the values for various extension fields
proj.epsg = [4326]
collection_as_dict = collection.to_dict()
collection_as_dict["summaries"]["proj:epsg"]
# [4326]

Item Asset properties#

Properties that apply to Items can be found in two places: the Item’s properties or in any of an Item’s Assets. If the property is on an Asset, it applies only that specific asset. For example, gsd defined for an Item represents the best Ground Sample Distance (resolution) for the data within the Item. However, some assets may be lower resolution and thus have a higher gsd. In that case, the gsd can be found on the Asset.

See the STAC documentation on Additional Fields for Assets and the relevant Best Practices for more information.

The implementation of this feature in PySTAC uses the method described here and is consistent across Item and ItemExtensions. The bare property names represent values for the Item only, but for each property where it is possible to set on both the Item or the Asset there is a get_ and set_ methods that optionally take an Asset. For the get_ methods, if the property is found on the Asset, the Asset’s value is used; otherwise the Item’s value will be used. For the set_ method, if an Asset is passed in the value will be applied to the Asset and not the Item.

For example, if we have an Item with a gsd of 10 with three bands, and only asset “band3” having a gsd of 20, the get_gsd method will behave in the following way:

assert item.common_metadata.gsd == 10
assert item.common_metadata.get_gsd() == 10
assert item.common_metadata.get_gsd(item.asset['band1']) == 10
assert item.common_metadata.get_gsd(item.asset['band3']) == 20

Similarly, if we set the asset at ‘band2’ to have a gsd of 30, it will only affect that asset:

item.common_metadata.set_gsd(30, item.assets['band2']
assert item.common_metadata.gsd == 10
assert item.common_metadata.get_gsd(item.asset['band2']) == 30

Manipulating STACs#

PySTAC is designed to allow for STACs to be manipulated in-memory. This includes Copying STACs in-memory, walking over all objects in a STAC and mutating their properties, or using collection-style map methods for mapping over items.

Walking over a STAC#

You can walk through all sub-catalogs and items of a catalog with a method inspired by the Python Standard Library os.walk() method: Catalog.walk():

for root, subcats, items in catalog.walk():
    # Root represents a catalog currently being walked in the tree
    root.title = '{} has been walked!'.format(root.id)

    # subcats represents any catalogs or collections owned by root
    for cat in subcats:
        cat.title = 'About to be walked!'

    # items represent all items that are contained by root
    for item in items:
        item.title = '{} - owned by {}'.format(item.id, root.id)

Mapping over Items#

The Catalog.map_items method is useful for manipulating items in a STAC. This will create a full copy of the STAC, so will leave the original catalog unmodified. In the method that manipulates and returns the modified item, you can return multiple items, in case you are generating new objects (e.g. creating a LabelItem for image items in a stac), or splitting items into smaller chunks (e.g. tiling out large image items).

def modify_item_title(item):
    item.title = 'Some new title'
    return item

def create_label_item(item):
    # Assumes the GeoJSON labels are in the
    # same location as the image
    img_href = item.assets['ortho'].href
    label_href = '{}.geojson'.format(os.path.splitext(img_href)[0])
    label_item = LabelItem(id='Labels',
                      geometry=item.geometry,
                      bbox=item.bbox,
                      datetime=datetime.utcnow(),
                      properties={},
                      label_description='labels',
                      label_type='vector',
                      label_properties='label',
                      label_classes=[
                      LabelClasses(classes=['one', 'two'],
                                   name='label')
                      ],
                      label_tasks=['classification'])
    label_item.add_source(item, assets=['ortho'])
    label_item.add_geojson_labels(label_href)

    return [item, label_item]


c = catalog.map_items(modify_item_title)
c = c.map_items(create_label_item)
new_catalog = c

Copying STACs in-memory#

The in-memory copying of STACs to create new ones is crucial to correct manipulations and mutations of STAC data. The STACObject.full_copy mechanism handles this in a way that ties the elements of the copies STAC together correctly. This includes situations where there might be cycles in the graph of connected objects of the STAC (which otherwise would be a tree). For example, if a LabelItem lists a source that is an item also contained in the root catalog; the full copy of the STAC will ensure that the Item instance representing the source imagery item is the same instance that is linked to by the LabelItem.

Resolving STAC objects#

PySTAC tries to only “resolve” STAC Objects - that is, load the metadata contained by STAC files pointed to by links into Python objects in-memory - when necessary. It also ensures that two links that point to the same object resolve to the same in-memory object.

Lazy resolution of STAC objects#

Links are read only when they need to be. For instance, when you load a catalog using Catalog.from_file, the catalog and all of its links are read into a Catalog instance. If you iterate through Catalog.links, you’ll see the target of the Link will refer to a string - that is the HREF of the link. However, if you call Catalog.get_items, for instance, you’ll get back the actual Item instances that are referred to by each item link in the Catalog. That’s because at the time you call get_items, PySTAC is “resolving” the links for any link that represents an item in the catalog.

The resolution mechanism is accomplished through Link.resolve_stac_object. Though this method is used extensively internally to PySTAC, ideally this is completely transparent to users of PySTAC, and you won’t have to worry about how and when links get resolved. However, one important aspect to understand is how object resolution caching happens.

Resolution Caching#

The root Catalog instance of a STAC (the Catalog which is linked to by every associated object’s root link) contains a cache of resolved objects. This cache points to in-memory instances of STACObject s that have already been resolved through PySTAC crawling links associated with that root catalog. The cache works off of the stac object’s ID, which is why it is necessary for every STAC object in the catalog to have a unique identifier, which is unique across the entire STAC.

When a link is being resolved from a STACObject that has it’s root set, that root is passed into the Link.resolve_stac_object call. That root’s ResolvedObjectCache will be used to ensure that if the link is pointing to an object that has already been resolved, then that link will point to the same, single instance in the cache. This ensures working with STAC objects in memory doesn’t create a situation where multiple copies of the same STAC objects are created from different links, manipulated, and written over each other.

Working with STAC JSON#

The pystac.serialization package has some functionality around working directly with STAC JSON objects, without utilizing PySTAC object types. This is used internally by PySTAC, but might also be useful to users working directly with JSON (e.g. on validation).

Identifying STAC objects from JSON#

Users can identify STAC information, including the object type, version and extensions, from JSON. The main method for this is identify_stac_object(), which returns an object that contains the object type, the range of versions this object is valid for (according to PySTAC’s best guess), the common extensions implemented by this object, and any custom extensions (represented by URIs to JSON Schemas).

from pystac.serialization import identify_stac_object

json_dict = ...

info = identify_stac_object(json_dict)

# The object type
info.object_type

# The version range
info.version_range

# The common extensions
info.common_extensions

# The custom Extensions
info.custom_extensions

Merging common properties#

For pre-1.0.0 STAC, The merge_common_properties() will take a JSON dict that represents an item, and if it is associated with a collection, merge in the collection’s properties. You can pass in a dict that contains previously read collections that caches collections by the HREF of the collection link and/or the collection ID, which can help avoid multiple reads of collection links.

Note that this feature was dropped in STAC 1.0.0-beta.1

Geo interface#

Item implements __geo_interface__, a de-facto standard for describing geospatial objects in Python: https://gist.github.com/sgillies/2217756. Many packages can automatically use objects that implement this protocol, e.g. shapely:

>>> from pystac import Item
>>> from shapely.geometry import mapping, shape
>>> item = Item.from_file("data-files/item/sample-item.json")
>>> print(shape(item))
POLYGON ((-122.308150179 37.488035566, -122.597502109 37.538869539,
-122.576687533 37.613537207, -122.2880486 37.562818007, -122.308150179
37.488035566))