Skip to contents

After Excel Review: Finalize and Publish Your Package

Source: vignettes/post-review-package-publication.Rmd
post-review-package-publication.Rmd

Overview

This guide starts after you already ran create_sdp(), opened the package in Excel (or another spreadsheet editor), and saved your reviewed metadata back to CSV.

If you have not created the package yet, start with the 5-Minute Quickstart.

If you want to assemble dataset.csv, tables.csv, and column_dictionary.csv manually instead of continuing from a reviewed package, use Publishing Data Packages.

This is the post-review path:

  1. reload the reviewed package from disk;
  2. validate the reviewed metadata and data structure;
  3. re-run semantic suggestions only for fields that are still unresolved;
  4. detect likely ontology gaps;
  5. decide whether each missing term belongs in the shared salmon-domain ontology or the DFO-specific bucket;
  6. produce a concrete list of term requests to file;
  7. rebuild EDH XML if needed;
  8. run strict final validation; and
  9. assemble the reviewed closure the EML/KNB path needs before publishing.

Step 9 is the one that surprises people. Strict validation is the last gate the Salmon Data Package has; it is not the last gate before a deposit.

1) Reload the reviewed package 

library(metasalmon)

pkg_path <- "fraser-coho-2023-2024-sdp"
pkg <- read_salmon_datapackage(pkg_path)

names(pkg$resources)
pkg$tables

read_salmon_datapackage() reloads the package from the canonical metadata/*.csv files, so you are checking the same metadata you just reviewed in Excel.

2) Run a review-state validation pass

Start with the package-level validator in non-strict mode:

review_check <- validate_salmon_datapackage(pkg_path, require_iris = FALSE)

This catches package/data mismatches now, before you worry about final ontology coverage. Since 0.2.6 the same call also enforces any declared primary_key (duplicate or missing key values are hard issues) and warns about every unresolved MISSING METADATA: placeholder — so a package fresh from Excel review will usually warn until the placeholders are filled in.

If you also want the semantic details separated out, inspect the reloaded package directly:

review_semantics <- validate_semantics(pkg$dictionary, require_iris = FALSE)

review_semantics$issues
review_semantics$missing_terms

Use this pass to answer three questions:

  • Did the Excel edits break the package structure?
  • Which required metadata is still a placeholder? (the warnings preview them, deduplicated by file and field — a long list is truncated, so treat it as a pointer, not an inventory)
  • Which measurement rows still lack a final semantic term?

Do not switch to require_iris = TRUE yet unless you believe the package is fully finalized. Strict validation is the last gate, not the first one.

3) Re-run semantic suggestions for only the unresolved pieces

The package-root semantic_suggestions.csv file is the original evidence trail from create_sdp(). After review, it is usually more useful to re-run suggest_semantics() against the current package state so you only inspect what is still unresolved.

reviewed_dict <- suggest_semantics(
  df = pkg$resources,
  dict = pkg$dictionary,
  codes = pkg$codes,
  table_meta = pkg$tables,
  dataset_meta = pkg$dataset
)

remaining_suggestions <- attr(reviewed_dict, "semantic_suggestions")

remaining_suggestions |>
  dplyr::select(
    target_scope,
    target_row_key,
    target_sdp_field,
    dictionary_role,
    search_query,
    label,
    source,
    score
  )

That usually produces a much shorter shortlist than the original package export, because rows you already resolved in Excel are no longer the target.

4) Detect likely ontology gaps

Now detect the places where metasalmon still cannot find a shared SMN match, but did find useful fallback candidates elsewhere:

gaps <- detect_semantic_term_gaps(reviewed_dict)

gaps |>
  dplyr::select(
    target_scope,
    target_row_key,
    target_sdp_field,
    dictionary_role,
    search_query,
    top_non_smn_label,
    top_non_smn_source,
    gap_detection_basis,
    llm_decision,
    llm_new_term_label,
    llm_escalated_from,
    placement_recommendation,
    placement_confidence
  )

This is the key post-review gap table:

  • rows here are still unresolved;
  • top_non_smn_* columns show the best non-SMN evidence the package found;
  • gap_detection_basis distinguishes candidate evidence, a final LLM term-gap decision, or both; and
  • placement_recommendation gives a first-pass routing guess.

When suggestions is omitted, detect_semantic_term_gaps() reads both the semantic_suggestions and semantic_llm_assessments dictionary attributes. Final request_new_term decisions are therefore included automatically, even when candidate rows contain an SMN match. An unresolved reject_shortlist escalation remains traceable through llm_escalated_from.

If you pass suggestions explicitly, only candidate evidence and the embedded llm_* columns in that table are used. The dictionary’s assessment attribute is intentionally ignored in that form.

If you only want a narrower slice, filter by role or scope:

gaps |>
  dplyr::filter(dictionary_role %in% c("variable", "property", "entity"))

Start from a commons gap register

The same request pipeline can start from a recorded domain gap rather than a dataset. From a salmon-knowledge-commons checkout, export the current register and confirm the validator exits successfully:

uv run --with pyyaml,jsonschema scripts/okf-check.py . --gaps > gaps.json

Then use the exported file in R:

commons_gaps <- detect_semantic_term_gaps(commons_gaps = "gaps.json")
commons_requests <- render_ontology_term_request(commons_gaps, ask = FALSE)
commons_preview <- submit_term_request_issues(commons_requests, dry_run = TRUE)
commons_preview

The reader retains every row, source order and repeated concepts. Fields starting with commons_ preserve the card, registry, mint target, lifecycle, draft note, dependencies, conflicts and verification state. Dataset, role, candidate and model evidence remain missing; this input cannot be combined with a dictionary, suggestions or SDP-only filters.

Only an open gap with an explicit SMN or GCDFO target and no hold reaches the issue preview. Proposed/rejected and do-not-mint records, unresolved blockers, conflicts, contested meanings, evidence needs and deprecated cards remain visible with commons_hold_reason. PSC CV, new-scheme and undecided targets are held because this pipeline has no defined route for them. scope, scope_overrides and ask cannot reopen or redirect these source declarations. Draft/unverified cards can produce drafts; they do not establish approval.

Bodies retain the exact draft gap note and identify its concept card. A curator must still supply the term definition and type. This preview makes no issue or authentication request. The matching Python reader remains owed under B-279; this R capability does not establish parity on Python main.

5) Decide shared salmon-domain vs DFO-specific routing

Use this plain-English rule:

  • Shared salmon-domain ontology (smn): use this when the term describes a reusable salmon science concept that another program, region, or organization could reasonably use too.
  • DFO Salmon Ontology (gcdfo): use this when the concept is clearly tied to DFO policy, operations, internal workflow, local identifiers, program-specific statuses, or other context that is not broadly reusable outside that setting.
  • Profile term (profile): use this for a local, program, or organization profile when neither shared SMN nor DFO-wide GCDFO governance is appropriate.

A good default test is:

  • if the term would still make sense in a non-DFO salmon dataset, lean SMN;
  • if the term only makes sense because of a DFO or local program process, lean DFO-specific.

How routing is decided

render_ontology_term_request() accepts smn, gcdfo, profile, uncertain, and skip row outcomes. Its precedence is:

  1. an explicit scope_overrides value for the row;
  2. a forced scope;
  3. a recognized llm_new_term_namespace; and
  4. the deterministic placement heuristic.

Treat llm_new_term_namespace as evidence, not authority. A human reviewer must still decide whether a DFO-specific proposal belongs in GCDFO, a narrower profile, or nowhere. Non-interactive uncertainty becomes skip; interactive use asks for a routing decision.

6) Produce a concrete term-request list

First render a dry, non-interactive plan and inspect every route:

requests <- render_ontology_term_request(
  gaps,
  scope = "auto",
  ask = FALSE,
  profile_name = "local-program"
)

request_plan <- requests |>
  dplyr::select(
    dataset_id,
    table_id,
    target_scope,
    target_row_key,
    target_sdp_field,
    dictionary_role,
    search_query,
    top_non_smn_label,
    top_non_smn_source,
    gap_detection_basis,
    llm_new_term_namespace,
    placement_recommendation,
    placement_confidence,
    request_scope,
    ontology_repo
  )

request_plan
readr::write_csv(request_plan, file.path(pkg_path, "term-request-plan.csv"))

That gives you a concrete list of which package row still needs help, the best fallback evidence, and the proposed route. Review skip rows and any route inferred from llm_new_term_namespace.

To apply reviewed row-by-row decisions, pass an override vector. Edit reviewed_scopes so every value is one of smn, gcdfo, profile, or skip:

reviewed_scopes <- request_plan$request_scope
# reviewed_scopes[2] <- "gcdfo"

requests <- render_ontology_term_request(
  gaps,
  scope = "auto",
  ask = FALSE,
  profile_name = "local-program",
  scope_overrides = reviewed_scopes
)

requests |>
  dplyr::select(
    request_scope,
    ontology_repo,
    request_title,
    target_row_key,
    search_query
  )

The override vector must have length one or one value per gap row. Rendering uses the active SMN and GCDFO repository templates and never submits an issue.

Preview issue drafts

Preview both SMN and GCDFO payloads before any submission:

issue_preview <- requests |>
  dplyr::filter(request_scope %in% c("smn", "gcdfo")) |>
  submit_term_request_issues(dry_run = TRUE)

issue_preview

The helper only posts after an explicit call with dry_run = FALSE; it never submits during detection or rendering. Keep confirm = TRUE for curator review. Profile rows remain local governance artifacts unless you provide a repository workflow for them.

7) Retain semantic mapping and decomposition evidence when available

Use SSSOM for reviewed relationships between whole vocabulary concepts. That includes an explicit sssom:NoTermFound row when a released PSC compound measurement has no defensible match in another versioned vocabulary. Do not put property/entity/unit decomposition columns into SSSOM.

write_sdp_sssom(
  pkg_path,
  mapping_sets = reviewed_sssom_paths
)
validate_sdp_sssom(pkg_path)

Store the ordered property, entity, constraint, statistical-modifier, and unit evidence in the separate SDP measurement-decomposition artifact:

write_sdp_measurement_decompositions(
  pkg_path,
  decompositions = reviewed_components
)
validate_sdp_measurement_decompositions(pkg_path)

metasalmon does not invent either record by default: both require reviewed inputs. When their manifests are present, later KNB planning validates them and includes them automatically inside the canonical SDP ZIP. This preserves the evidence as part of the package without claiming native I-ADOPT conformance or turning the component rows into concept mappings.

8) Finalize metadata and rebuild EDH XML if needed

Once the metadata is final and every surviving IRI is deliberate, regenerate EDH XML if your publication path needs it:

This is the reviewed-package wrapper around edh_build_hnap_xml(). It refuses to rebuild while obvious review-state markers still exist, including:

  • REVIEW:-prefixed IRIs; and
  • unresolved MISSING DESCRIPTION: / MISSING METADATA: placeholders in description, observation_unit, and table_label.

A blank observation_unit_iri is not one of those markers and does not block a rebuild; only a REVIEW:-prefixed value does.

9) Run strict final validation

When the package is genuinely ready, switch to strict validation:

validate_salmon_datapackage(pkg_path, require_iris = TRUE)

After this passes, the whole package folder can be shared as described in step 12. For EML/KNB export or deposit, complete the reviewed closure and EML facts in steps 10 and 11.

If strict validation still fails because a measurement term genuinely needs a new shared or DFO-specific ontology term, that is not a bug in the workflow — it means the package is not finalized yet. Keep the term-request plan with the package, resolve the ontology decision, then run strict validation again.

10) Assemble the reviewed closure

Strict validation is not the publication gate. A package that passes validate_salmon_datapackage(require_iris = TRUE) can still be refused by write_eml_from_sdp() and publish_sdp_to_knb(), because the EML/KNB path additionally requires a reviewed closure: a standing record of which vocabulary terms you accepted, on what evidence, and for which target.

Three files carry it. write_sdp_semantic_closure() writes the first two and pins the digests the third one carries; the reviewed EML facts in the third are yours (step 11):

File What it must contain Written by
metadata/semantic_vocabulary.csv One evidence row per canonical measurement IRI — exactly that set, no more and no less write_sdp_semantic_closure()
reviewed_semantic_selections.csv Exactly one accepted row per canonical review target write_sdp_semantic_closure()
metadata/eml-mapping.yml The reviewed EML facts, plus the SHA-256 of each of the two files above you, except the two SHA-256 values

The canonical measurement IRI set is every non-empty value across term_iri, property_iri, entity_iri, constraint_iri, statistical_modifier_iri and unit_iri on your measurement dictionary rows, plus any tables.csv method_iri and any procedure IRIs reached through codes.csv from an observation-structures binding.

The canonical review target set is derived independently, and it is not the measurement set plus anything. It has one target for each filled slot among these: the same six semantic fields on your measurement dictionary rows, plus each table’s observation_unit_iri and its method_iri.

The two sets differ in both directions, which is why neither can be reasoned from the other:

  • an observation_unit_iri is a review target and not a vocabulary term — a reviewed selection, but not something the EML measurement path emits; and
  • a sosa:usedProcedure reached through a code value is a vocabulary term and not a review target — the EML method path emits it, but the ledger has no slot for it, because it is reached through the code values in your data rather than through any field the review target set reads.

Getting that second direction backwards is worth avoiding rather than discovering: the ledger gate accepts exactly the canonical target set, so a row added for a code-resolved procedure is refused as unexpected. write_eml_from_sdp() and publish_sdp_to_knb() stop and name the row, while validate_salmon_datapackage(require_iris = TRUE) goes on passing: the package stops being publishable because of a row you added to make it publishable.

semantic_vocabulary.csv needs these columns, all non-empty except type_iris and source_artifact_sha256:

iri, label, definition, source, ontology, resource_kind, type_iris,
native_type, source_url, source_artifact_sha256, reviewed_snapshot_sha256

reviewed_semantic_selections.csv needs:

dataset_id, table_id, column_name, target_scope, target_sdp_field,
dictionary_role, decision, confidence, review_rationale, iri

Every decision must be accepted — a final ledger holds no open decisions — and column_name is empty only for table-scope rows.

Produce both files

write_sdp_semantic_closure() derives both canonical sets from the package, re-runs find_terms() for each IRI to resolve its evidence, computes every reviewed_snapshot_sha256, writes both files, and rewrites the two file digests in metadata/eml-mapping.yml. No SHA-256 in a reviewed package is ever hand-written.

Two kinds of value it cannot derive, and both go in evidence:

  • Vocabularies find_terms() cannot search. It resolves smn and gcdfo. A QUDT unit row is hand-authored in full. It also cannot fill native_type or source_url, which describe the ontology artifact rather than the term; those are derived from the resolved source and can be overridden.
  • confidence and review_rationale, which are your judgement. They are read from semantic_suggestions.csv where apply_sdp_semantics() recorded a decision reason, and otherwise written as a REVIEW REQUIRED: marker with a warning naming each target that got one. Replace those before publishing.
# Your judgement on each accepted IRI. The term evidence is resolved for you.
judgements <- tibble::tribble(
  ~iri, ~confidence, ~review_rationale,
  "https://w3id.org/gcdfo/salmon#SpawnerAbundance", "high",
  "The column is an escapement estimate of adult spawners.",
  "https://w3id.org/smn/Abundance", "high",
  "The measured characteristic is abundance.",
  "https://w3id.org/smn/Observation", "high",
  "Each row is one population-year observation."
)

# A QUDT row is hand-authored end to end: the search cannot reach QUDT.
qudt <- tibble::tibble(
  iri = "https://qudt.org/vocab/unit/INDIV",
  label = "Individual",
  definition = "A counting unit denoting one organism.",
  source = "qudt",
  ontology = "qudt",
  resource_kind = "Unit",
  type_iris = "http://qudt.org/schema/qudt/Unit",
  native_type = "qudt:Unit",
  source_url = "https://qudt.org/vocab/unit/",
  confidence = "high",
  review_rationale = "Values are whole counts of organisms."
)

closure <- write_sdp_semantic_closure(
  "path/to/package",
  evidence = dplyr::bind_rows(judgements, qudt)
)

Copy the sidecar template (step 11) before running this, so the two digests are pinned in the same pass. If you produce the closure first, run it again once the sidecar exists.

An IRI it cannot resolve is reported, not guessed at, and not fatal. closure$gaps is a term-gap table in the same shape detect_semantic_term_gaps() returns, so it feeds the request pipeline directly:

if (nrow(closure$gaps) > 0) {
  requests <- render_ontology_term_request(closure$gaps, ask = FALSE)
  submit_term_request_issues(requests, dry_run = TRUE)
}

The unresolved row is left out of the vocabulary rather than invented, so write_eml_from_sdp() will still refuse the package and will name the same IRI. That is the intended sequence: the gap is a term to mint or an IRI to correct, and the package is not publishable until it is one or the other.

Two things that leave a row out are not gaps, and you should not file them as term requests. A gap says the searched vocabularies do not have the term, and that is what render_ontology_term_request() acts on, so the cases that do not establish absence are reported separately:

  • A lookup that did not answer — a network or parser failure, or a find_terms() result whose "diagnostics" attribute names a source that did not reply. The call aborts and writes nothing, naming each IRI and the silent sources. Re-run when the sources answer; nothing on disk changed.
  • A term found with a required field blank, for example a class with no definition. It comes back in closure$incomplete, naming the missing field and the slot. The term exists, so supply the field through evidence or annotate it upstream rather than requesting a new term.

A worked end-to-end example is scripts/build-fraser-coho-knb-rehearsal.R in the source repository, which takes the shipped Fraser coho data from create_sdp() to a clean KNB test-node dry run using only exported functions.

11) Build reviewed EML and preview KNB publication when needed

KNB publication requires EML-specific facts that the canonical SDP does not guess, including structured parties, rights, methods, measurement scales, missing-value meanings, and access intent. Record those reviewed decisions in metadata/eml-mapping.yml, then build and validate EML 2.2.0.

Three of its fields are cross-checked against the package and will abort the build if they drift, so set them from one source rather than retyping them: source_provenance.source_citation and source_provenance.provenance_note must byte-match dataset.csv, and publication.public must equal the public argument you pass to publish_sdp_to_knb().

A live deposit also requires the sidecar to name exactly one metadata-provider orcid, and it must be the ORCID behind the DataONE token you authenticate with. A dry run accepts any syntactically valid ORCID, so the placeholder in the shipped template will pass a rehearsal and be refused by the live call.

# Copy once, then replace every example value and checksum with reviewed facts.
file.copy(
  system.file(
    "extdata",
    "eml-mapping-template.yml",
    package = "metasalmon"
  ),
  file.path(pkg_path, "metadata", "eml-mapping.yml")
)

That standalone call creates valid reviewed EML for inspection. The later KNB dry run deterministically rebuilds it once more so the final EML also describes each named canonical SDP artifact as an otherEntity.

Preview the exact immutable DataONE objects, identifiers, checksums, access decision, and OAI-ORE relationships without reading credentials or making a network request:

publish_sdp_to_knb(
  pkg_path,
  public = FALSE,
  dry_run = TRUE,
  representation = "expanded"
)

That dry run targets the KNB Test Node (urn:node:mnTestKNB), which is what knb_environment defaults to for a dry run. The golden path develops a package against the test node first and posts to production once it looks good there, so rehearse in that order:

  1. credential-free test dry run (the call above)
  2. optional authorized test deposit, with a dataone_test_token
  3. fresh production dry run, naming the environment explicitly
  4. optional private production deposit
  5. separate KNB Publish action for public release and DOI

Test objects are never promoted to production: a production deposit is planned separately, with production credentials, and mints its own identifiers.

publish_sdp_to_knb(
  pkg_path,
  public = FALSE,
  dry_run = TRUE,
  representation = "expanded",
  knb_environment = "production"
)

Switching the environment switches the DataONE network, member node, coordinating node, resolver, and catalog endpoint together — they never vary independently — along with which credential is read. Test artifacts are written under publication/test/, so the reviewed production metadata/eml.xml and publication/knb-manifest.json are left exactly as they were.

Live KNB publication is a separate explicit action. It requires a short-lived DataONE JWT in the process-local option belonging to the environment (dataone_token for production, dataone_test_token for the test node — they are separate credentials), an ORCID-authenticated subject matching the EML metadata provider, an explicitly named knb_environment, and an explicitly supplied confirm = TRUE. A live call never infers its environment. The confirmation means both that the exact pending manifest is approved and that the caller has authority to redistribute the package.

Keep the JWT out of scripts, YAML, manifests, and shell history. Enter it into the current R process only and clear it immediately after the live call:

# `dataone_token` is the production credential; the test node reads
# `dataone_test_token` instead.
options(dataone_token = rstudioapi::askForPassword("Short-lived DataONE JWT"))
on.exit(options(dataone_token = NULL), add = TRUE)

publish_sdp_to_knb(
  pkg_path,
  public = FALSE,
  dry_run = FALSE,
  confirm = TRUE,
  representation = "expanded",
  knb_environment = "production"
)

public = FALSE is the recommended review path, but it is not a server-side draft: a live call creates persistent production KNB objects. Completion requires authenticated byte-for-byte and SystemMetadata readback for every object, anonymous denial for both object bytes and SystemMetadata, a complete authenticated catalog graph, and zero planned PIDs in the anonymous catalog. The reviewed private plan also explicitly sets DataONE replication to disabled with zero replicas, and live readback must match that KNB-only policy. Changing to public = TRUE is a separate release decision: it requires public redistribution authority and explicitly requests three DataONE preservation replicas; metasalmon never infers that decision from a private deposit.

An expanded plan contains four kinds of object: the original data resources named by tables.csv, validated canonical SDP artifacts with their exact package-relative paths, the EML science-metadata record, and the OAI-ORE map. The EML record describes raw files as data tables and named SDP artifacts as supplementary entities; ORE prov:atLocation values preserve the hierarchy for exact reconstruction. A closed inventory includes SSSOM, ordered measurement decompositions, observation structures, and manifest-declared reproducibility records without scanning arbitrary files. There is no methods artifact to deposit: since sdp-0.3.0 a package carrying a metadata/methods.csv is refused outright, with a pointer to migrate_sdp_methods(). The KNB-specific metadata/eml-mapping.yml, its authorization evidence and party details, and all mutable publication/ receipts remain local and are not deposited.

KNB DOI minting is a later opt-in release action, not part of publish_sdp_to_knb(). KNB assigns the DOI to the selected science-metadata version when its Publish action makes that package public; it does not assign a separate DOI to each raw or supplementary object. Leaving a deposit private is therefore the practical review workflow. Editing after deposit creates new immutable versions rather than overwriting bytes.

For a corrected or otherwise revised private version, keep the same series_key, add a new safe publication.revision_key to the sidecar, and bind the new dry run to the verified manifest for the preceding version. Build the revision in a fresh versioned SDP directory; never reuse or overwrite the prior package directory:

publication:
  public: false
  revision_key: corrected-expanded-sdp-2026-08-04
publish_sdp_to_knb(
  revised_pkg_path,
  public = FALSE,
  manifest_path = file.path(
    revised_pkg_path,
    "publication",
    "knb-manifest.json"
  ),
  revision_manifest = previous_manifest_path,
  dry_run = TRUE,
  representation = "expanded",
  knb_environment = "production"
)

A revision continues the environment it started in; a manifest from one environment can never be revised into the other.

Review that new manifest before repeating the live call with dry_run = FALSE, confirm = TRUE. An interrupted live call is resumable: reuse the byte-identical package, revision manifest, and output manifest. A completed upload whose coordinating-network catalog check is delayed returns published_pending_catalog; rerunning the same call performs fresh checks and does not create duplicate objects.

If KNB’s separate Publish action later mints a DOI, KNB creates another science-metadata version. metasalmon does not yet import that KNB-created version into a local predecessor manifest. Do not base a later automated revision on the older pre-DOI manifest; use KNB’s editor for that next change or wait for an explicit manifest-import workflow.

Before any plan is written, metasalmon rejects referenced rows whose source or ontology label identifies them as a review-candidate. This is a deliberately offline fail-closed check; it does not resolve public IRIs or by itself prove that a vocabulary release was governed. The canonical transformation record must separately pin the approved vocabulary release and verify its public term IRIs before the SDP is deposited.

12) Publish or share the finished package

Once strict validation passes:

  • share the whole package folder (or a zip of it), not just datapackage.json;
  • keep metadata/*.csv with the data files; and
  • keep term-request-plan.csv only as working governance support, not as part of the canonical final package unless you want that audit trail to travel with it.

In short: reload -> validate -> detect gaps -> route requests -> build the required export -> strict validate -> dry-run -> publish.