{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quickstart\n",
    "\n",
    "This notebook shows how to use PySTAC to read through the public Sentinel catalog and write a local version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading STAC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we want to hook into PySTAC to allow for reading of HTTP STAC items, as described in [the STAC_IO Concepts docs](concepts.html#using-stac-io). \n",
    "\n",
    "__Note:__ this requires the [requests](https://requests.kennethreitz.org/en/master) library be installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.parse import urlparse\n",
    "import requests\n",
    "from pystac import STAC_IO\n",
    "\n",
    "def requests_read_method(uri):\n",
    "    parsed = urlparse(uri)\n",
    "    if parsed.scheme.startswith('http'):\n",
    "        return requests.get(uri).text\n",
    "    else:\n",
    "        return STAC_IO.default_read_text_method(uri)\n",
    "\n",
    "STAC_IO.read_text_method = requests_read_method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then read the STAC catalog located at the publicly available endpoint hosted by AWS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pystac import Catalog\n",
    "\n",
    "cat = Catalog.from_file('https://sentinel-stac.s3.amazonaws.com/catalog.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of items in this catalog; crawling through it all would take a significant amount of time. Here, we lean on the fact that [link resolution is lazy](concepts.html#lazy-resolution-of-stac-objects) and get to a catalog that contains items:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Crawling through <Catalog id=sentinel-stac>\n",
      "Crawling through <Collection id=sentinel-2-l1c>\n",
      "Crawling through <Catalog id=9>\n",
      "Crawling through <Catalog id=V>\n"
     ]
    }
   ],
   "source": [
    "while len(cat.get_item_links()) == 0:\n",
    "    print('Crawling through {}'.format(cat))\n",
    "    cat = next(cat.get_children())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print some information about the catalog, including how many children it has:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "XK catalog\n",
      "Contains 388 items.\n"
     ]
    }
   ],
   "source": [
    "print(cat.description)\n",
    "print('Contains {} items.'.format(len(cat.get_item_links())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's grab the first item., check out it's cloud cover, and start exploring the assets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "item = next(cat.get_items())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access [common metadata](https://github.com/radiantearth/stac-spec/blob/v0.9.0/item-spec/common-metadata.md) fields through the common_metadata property of the item:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sentinel-2b'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item.common_metadata.platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This particular stac item implements the [eo extension](https://github.com/radiantearth/stac-spec/tree/v0.9.0/extensions/eo) extension. We can access the extension information through the \"ext\" property that's part of every Catalog, Collection and Item. For instance, to get the cloud cover, we can use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41.52"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item.ext.eo.cloud_cover"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can see the cloud cover is in it's appropriate key in the Item's properties:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41.52"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item.properties['eo:cloud_cover']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if we want to set the cloud cover, we can do that through the extension as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item.ext.eo.cloud_cover = 42.0\n",
    "item.properties['eo:cloud_cover']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the item's assets through the assets property, which is a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "thumbnail: https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/9/V/XK/2017/10/13/0/preview.jpg (None)\n",
      "info: https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/9/V/XK/2017/10/13/0/tileInfo.json (None)\n",
      "metadata: https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/9/V/XK/2017/10/13/0/metadata.xml (None)\n",
      "tki: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/TKI.jp2 (image/jp2)\n",
      "B01: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B01.jp2 (image/jp2)\n",
      "B02: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B02.jp2 (image/jp2)\n",
      "B03: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B03.jp2 (image/jp2)\n",
      "B04: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B04.jp2 (image/jp2)\n",
      "B05: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B05.jp2 (image/jp2)\n",
      "B06: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B06.jp2 (image/jp2)\n",
      "B07: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B07.jp2 (image/jp2)\n",
      "B08: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B08.jp2 (image/jp2)\n",
      "B8A: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B08.jp2 (image/jp2)\n",
      "B09: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B09.jp2 (image/jp2)\n",
      "B10: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B10.jp2 (image/jp2)\n",
      "B11: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B11.jp2 (image/jp2)\n",
      "B12: https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B11.jp2 (image/jp2)\n"
     ]
    }
   ],
   "source": [
    "for asset_key in item.assets:\n",
    "    asset = item.assets[asset_key]\n",
    "    print('{}: {} ({})'.format(asset_key, asset.href, asset.media_type))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `to_dict()` method to convert an Asset, or any PySTAC object, into a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'href': 'https://sentinel-s2-l1c.s3.amazonaws.com/tiles/9/V/XK/2017/10/13/0/B03.jp2',\n",
       " 'type': 'image/jp2',\n",
       " 'title': 'Band 3 (green)',\n",
       " 'eo:bands': [2]}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "asset = item.assets['B03']\n",
    "asset.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use the eo extension to get the band information for the asset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Band name=B03>]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bands = item.ext.eo.get_asset_bands(asset)\n",
    "bands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'B03',\n",
       " 'common_name': 'green',\n",
       " 'gsd': 10.0,\n",
       " 'center_wavelength': 0.56,\n",
       " 'full_width_half_max': 0.045}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bands[0].to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing a STAC\n",
    "\n",
    "Let's walk the catalog again, but this time create local clones of the STAC object, so we can end up with a copy that we can save off to the local file system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Crawling through <Catalog id=sentinel-stac>\n",
      "Crawling through <Collection id=sentinel-2-l1c>\n",
      "Crawling through <Catalog id=9>\n",
      "Crawling through <Catalog id=V>\n"
     ]
    }
   ],
   "source": [
    "import itertools\n",
    "\n",
    "cat = Catalog.from_file('https://sentinel-stac.s3.amazonaws.com/catalog.json')\n",
    "\n",
    "# Setup the root of our local STAC\n",
    "local_root = cat.clone()\n",
    "local_root.clear_children()\n",
    "\n",
    "# Loop over catalogs and clone\n",
    "curr_local_cat = local_root\n",
    "while len(cat.get_item_links()) == 0:\n",
    "    print('Crawling through {}'.format(cat))\n",
    "    cat = next(cat.get_children())\n",
    "    local_cat = cat.clone()\n",
    "    local_cat.clear_children()\n",
    "    curr_local_cat.add_child(local_cat)\n",
    "    curr_local_cat = local_cat\n",
    "    \n",
    "# Clear the items from the last local catalog\n",
    "curr_local_cat.clear_children()\n",
    "curr_local_cat.clear_items()\n",
    "\n",
    "# Take the first 5 items\n",
    "items = itertools.islice(cat.get_items(), 5)\n",
    "\n",
    "# Clone and add them to our local catalog\n",
    "curr_local_cat.add_items([i.clone() for i in items])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a smaller STAC, let's map over the items to reduce it even further by only including the thumbnail assets in our items:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def item_mapper(item):\n",
    "    thumbnail_asset = item.assets['thumbnail']\n",
    "    \n",
    "    new_assets = { \n",
    "        k:v \n",
    "        for k, v in item.assets.items()\n",
    "        if k == 'thumbnail'\n",
    "    }\n",
    "    item.assets = new_assets\n",
    "    return item\n",
    "\n",
    "local_root_2 = local_root.map_items(item_mapper)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now normalize our catalog and save it somewhere local:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p ./quickstart_stac"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Catalog id=sentinel-stac>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "local_root_2.normalize_hrefs('./quickstart_stac')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pystac import CatalogType\n",
    "\n",
    "local_root_2.save(catalog_type=CatalogType.SELF_CONTAINED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* <Catalog id=sentinel-stac>\n",
      "    * <Collection id=sentinel-2-l1c>\n",
      "        * <Catalog id=9>\n",
      "            * <Catalog id=V>\n",
      "                * <Catalog id=XK>\n",
      "                  * <Item id=S2B_9VXK_20171013_0>\n",
      "                  * <Item id=S2A_9VXK_20171015_0>\n",
      "                  * <Item id=S2B_9VXK_20171016_0>\n",
      "                  * <Item id=S2B_9VXK_20171017_0>\n",
      "                  * <Item id=S2A_9VXK_20171002_0>\n"
     ]
    }
   ],
   "source": [
    "local_root_2.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Item {}:\n",
      "  Assets: {'thumbnail': <Asset href=https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/9/V/XK/2017/10/13/0/preview.jpg>}\n",
      "Item {}:\n",
      "  Assets: {'thumbnail': <Asset href=https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/9/V/XK/2017/10/15/0/preview.jpg>}\n",
      "Item {}:\n",
      "  Assets: {'thumbnail': <Asset href=https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/9/V/XK/2017/10/16/0/preview.jpg>}\n",
      "Item {}:\n",
      "  Assets: {'thumbnail': <Asset href=https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/9/V/XK/2017/10/17/0/preview.jpg>}\n",
      "Item {}:\n",
      "  Assets: {'thumbnail': <Asset href=https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/9/V/XK/2017/10/2/0/preview.jpg>}\n"
     ]
    }
   ],
   "source": [
    "for item in local_root_2.get_all_items():\n",
    "    print('Item {}:')\n",
    "    print('  Assets: {}'.format(item.assets))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}