The package is now called usgsm2m, renamed from USGSm2m. The old name
read as though the package were issued by the USGS, which it is not. Update
library() calls, and reinstall from
https://github.com/stevenpawley/usgsm2m.
Scene list ids generated by the package are now prefixed usgsm2m_ rather
than USGSm2m_. Lists created under the old prefix are unaffected and can
still be retrieved by passing their id to M2MSession$scene_list().
M2MSession$remove_items() and M2MDownloadQueue$remove_items() remove
individual downloads from the queue, where M2MDownloadQueue$cancel() only
drops a whole order. Ids come from the downloadId column of
$downloads(), $ready() or $pending().
The API removes one item per request and answers the same way for an id
that does not exist, so a successful call is not evidence that anything was
removed - check $downloads() afterwards.
M2MSceneSearch$combine() merges searches, so one $products() call and
one download order can cover scenes found by several queries — two date
ranges, say, or two areas:
summer <- july$combine(sept)
summer$products()
Like $filter() it returns a new search and leaves the originals alone,
takes several at once, and chains. Scenes found by more than one search are
kept once. Previously this meant passing the same list_id to
$scene_list() from each search in turn and relying on the API appending,
which was easy to get wrong and gave no hint that it was accumulating.
$select_products() now matches exactly, against productName or
productCode, rather than as a pattern. It previously used a regular
expression, so a name copied straight out of $scene_products() could match
nothing: "Full-Resolution Browse (Natural Color) GeoTIFF" is present in
the tibble, but its parentheses were read as regex groups. Chain $filter()
for looser selection, e.g.
$select_products()$filter(grepl("Bundle", productName)).
$select_bands() matches literal substrings rather than regular
expressions, so "B4" still picks out ..._SR_B4 but punctuation in a
value is no longer read as pattern syntax.
Both selectors now warn when nothing matches, listing what was available.
Product names are not standardised across the catalogue — the bundle is
"Landsat Collection 2 Level-2 Product Bundle" for landsat_ot_c2_l2 but
"Standard Format" for corona2 — so a value carried over from another
dataset used to select an empty set silently, surfacing later as an
unexplained "No products selected".
Authorisation failures now raise a m2m_no_access condition, a subclass of
m2m_api_error, so they can be caught separately from other API errors.
M2MSceneSearch$products() no longer fails on large orders with
Can't combine \secondaryDownloads[[1]]$checksum$value` . The API reports a file's optional checksumobject unevenly - present for most files, with a null value or absent entirely for others - and the column was typed from whichever form appeared first, so the per-scene tables could not be combined. Nested objects in API records are now hoisted into scalar columns named after their path, sochecksumappears aschecksum_algorithmandchecksum_valuein$bands(), with NA` where the API gave no value.
Downloaded files now keep the extension the server gives them, taken from
the Content-Disposition header or the URL served. Scene-level products such
as a Level-2 Product Bundle were written out named after the scene's
entityId, which has no extension at all, leaving a .tar on disk that
nothing would open.
A download that fails no longer leaves a file behind holding the error body.
Asking for a dataset the account cannot access reported it as missing:
"Dataset not found: Dataset status is unavailable to this user". The
dataset exists and is spelled correctly, so this sent people hunting for a
typo. Such failures now say that access is the problem and point at
$find_datasets(), which lists exactly the datasets the account can use.
Proxied downloads could not be retrieved at all. USGS serves some products
from another host (landsatlook.usgs.gov, for instance) on a signed URL,
and lists those under $requested even though they carry a working URL.
The queue treated everything in $requested as still being prepared, so
$is_ready() never became TRUE and $retrieve() refused with "No
downloads are available yet". Readiness is now a matter of having a URL,
via the new $ready() and $pending() methods.
The M2M session token was sent to whichever host a download URL pointed at. Proxied URLs are self-authenticating and ignore it, so this handed the token to hosts that neither needed nor asked for it. It is now sent only to the M2M API's own host.
Proxied downloads are now reported back through download-complete-proxied
once fetched. The API does not serve them itself and so cannot observe the
transfer; without this they stay in the download queue indefinitely.
An expired signed URL is reported as status = "expired" rather than a bare
failure, since the fix is to $refresh() and retry.
The package is now organised around R6 objects rather than a flat set of
functions. This is a breaking change: every exported ers_* function has
been removed. See "Migrating from 0.1.0" below.
Getting data out of the M2M API takes several steps that have to happen in a particular order. That order is now expressed by the objects themselves, each stage returning an object whose methods are the valid next steps:
M2MSession → M2MDataset → M2MSceneSearch → M2MDownloadOptions → M2MDownloadQueue
Tab-completing $ shows only what you can legitimately do from where you are,
and every object prints a Next: hint. A whole download now reads as one
chain:
sess <- m2m_session()
queue <- sess$
dataset("landsat_ot_c2_l2")$
search(
spatial = filter_spatial(ll_lon = -120, ll_lat = 40, ur_lon = -119.5, ur_lat = 40.5),
temporal = filter_temporal("2020-07-01", "2020-07-31"),
cloud = filter_cloud(0, 50)
)$
products()$
select_bands(c("B4", "B5"))$
filter(bulkAvailable)$
request(label = "ndvi_july_2020")
queue$retrieve("data/")
Scene lists are now created for you. The M2M API requires one server-side
before download options can be listed; $products() handles that, where
previously you had to wire ers_scene_list_add() and ers_scene_products()
together by hand and pass a list_id between them.
| 0.1.0 | 0.2.0 |
|---|---|
| ers_session() | m2m_session() |
| ers_logout(s) | s$logout() |
| ers_dataset(s, dataset_name = x) | s$dataset(x) |
| ers_dataset_search(s, x) | s$find_datasets(x) |
| ers_dataset_filters(s, x) | s$dataset(x)$filters() |
| ers_scene_search(s, x, ...) | s$dataset(x)$search(...) |
| ers_scene_list_add(s, x, scenes, id) | search$scene_list() |
| ers_scene_list_get(s, id) | list$scenes() |
| ers_scene_list_summary(s, id) | list$summary() |
| ers_scene_list_remove(s, id) | list$remove() |
| ers_scene_products(s, x, id, ...) | search$products() |
| ers_download_request(s, p, label = l) | products$request(label = l) |
| ers_download_queue(s, label = l) | s$download_queue(l) |
| ers_download_search(s) | s$downloads() |
| ers_download_retrieve(s, r, l, dir) | queue$retrieve(dir) |
| ers_download_eula(s, code) | s$eula(code) |
| ers_download_remove_order(s, l) | queue$cancel() |
| ers_download_remove_items(s, ids) | (no direct equivalent; use queue$cancel()) |
filter_spatial(), filter_temporal() and filter_cloud() are unchanged.
The filter() S3 method for scene_products objects has been removed;
M2MDownloadOptions$filter() replaces it.
filter_metadata_value(), filter_metadata_between(),
filter_metadata_and() and filter_metadata_or() build metadata filters
from the field ids that M2MDataset$filters() reports. Those ids could be
listed in 0.1.0 but there was nothing to turn them into a filter with.M2MDownloadOptions$scene_products() and $select_products() expose whole
products — a Level-1 Product Bundle (one .tar per scene) or a
full-resolution browse image — alongside the individual band files that
$bands() lists. Browse products have no constituent files and so were
previously unreachable entirely.M2MDataset$related_scenes() wraps scene-search-secondary. Note that few
datasets define a secondary relationship; the rest raise DATASET_ERROR.M2MSession$download_labels() lists your orders, so an order can be found
again without remembering its label.M2MDownloadQueue$summary() breaks an order down by dataset, and
$prepare() moves a staged order into the queue for processing.M2MSceneList$dataset() reports which dataset a scene list holds, looking
it up from the list's summary when reattaching to a list by id and caching
the answer. M2MSession$scene_list() also takes an optional dataset_name
to skip that lookup, or to choose one when a list spans several datasets.vignette("usgsm2m").API failures now raise conditions of class m2m_error, with
m2m_http_error, m2m_api_error and m2m_not_found subclasses, instead of
printing a message and returning NULL. A failed step stops the chain where
it failed rather than surfacing later as an unrelated error:
tryCatch(
sess$dataset("not_a_real_dataset"),
m2m_not_found = function(e) message("No such dataset")
)
Seven data-coercion bugs were found and fixed by testing against the live API across 20+ datasets. Several produced plausible but wrong data rather than failing, so results obtained with 0.1.0 are worth re-checking:
"c(\"5e83d150...\", ...)") whenever a search spanned scenes with
differing metadata field counts — the common case in any multi-scene search.
Uniform searches took a different code path and worked, which hid this.secondaryDownloads under each of its product entries, and flattening them
did not de-duplicate.download-options failed outright on any dataset mixing products that have
bands with products that do not, affecting landsat_tm_c2_l1,
landsat_mss_c2_l1 and eo1_ali_pub among others.dataset-filters repeated a Select field once per permitted value, giving
22 rows for 12 fields.scene-list-add returned HTTP 500 for any single-scene list: JSON
serialisation turned a length-1 entityIds vector into a bare string
instead of an array. The same fault is fixed for eulaCodes.scene-list-summary returned a column set that varied with the data —
listTimeout vanished when null, invalidSceneCount was never returned,
and invalidScenes changed type depending on whether any existed. An
unknown or expired list also raised a confusing "Column coordinates
doesn't exist" from dplyr, rather than summarizing as empty.