After Excel Review: Finalize and Publish Your Package

Overview

This guide starts after you already ran create_sdp(), opened the package in Excel (or another spreadsheet editor), and saved your reviewed metadata back to CSV.

If you have not created the package yet, start with the 5-Minute Quickstart.

If you want to assemble dataset.csv, tables.csv, and column_dictionary.csv manually instead of continuing from a reviewed package, use Publishing Data Packages.

This is the post-review path:

reload the reviewed package from disk;
validate the reviewed metadata and data structure;
re-run semantic suggestions only for fields that are still unresolved;
detect likely ontology gaps;
decide whether each missing term belongs in the shared salmon-domain ontology or the DFO-specific bucket;
produce a concrete list of term requests to file;
rebuild EDH XML if needed; and
run strict final validation before publishing.

1) Reload the reviewed package

library(metasalmon)

pkg_path <- "fraser-coho-2023-2024-sdp"
pkg <- read_salmon_datapackage(pkg_path)

names(pkg$resources)
pkg$tables

read_salmon_datapackage() reloads the package from the canonical metadata/*.csv files, so you are checking the same metadata you just reviewed in Excel.

2) Run a review-state validation pass

Start with the package-level validator in non-strict mode:

review_check <- validate_salmon_datapackage(pkg_path, require_iris = FALSE)

This catches package/data mismatches now, before you worry about final ontology coverage. Since 0.2.6 the same call also enforces any declared primary_key (duplicate or missing key values are hard issues) and warns about every unresolved MISSING METADATA: placeholder — so a package fresh from Excel review will usually warn until the placeholders are filled in.

If you also want the semantic details separated out, inspect the reloaded package directly:

review_semantics <- validate_semantics(pkg$dictionary, require_iris = FALSE)

review_semantics$issues
review_semantics$missing_terms

Use this pass to answer three questions:

Did the Excel edits break the package structure?
Which required metadata is still a placeholder? (the warnings preview them, deduplicated by file and field — a long list is truncated, so treat it as a pointer, not an inventory)
Which measurement rows still lack a final semantic term?

Do not switch to require_iris = TRUE yet unless you believe the package is fully finalized. Strict validation is the last gate, not the first one.

3) Re-run semantic suggestions for only the unresolved pieces

The package-root semantic_suggestions.csv file is the original evidence trail from create_sdp(). After review, it is usually more useful to re-run suggest_semantics() against the current package state so you only inspect what is still unresolved.

reviewed_dict <- suggest_semantics(
  df = pkg$resources,
  dict = pkg$dictionary,
  codes = pkg$codes,
  table_meta = pkg$tables,
  dataset_meta = pkg$dataset
)

remaining_suggestions <- attr(reviewed_dict, "semantic_suggestions")

remaining_suggestions |>
  dplyr::select(
    target_scope,
    target_row_key,
    target_sdp_field,
    dictionary_role,
    search_query,
    label,
    source,
    score
  )

That usually produces a much shorter shortlist than the original package export, because rows you already resolved in Excel are no longer the target.

4) Detect likely ontology gaps

Now detect the places where metasalmon still cannot find a shared SMN match, but did find useful fallback candidates elsewhere:

gaps <- detect_semantic_term_gaps(reviewed_dict)

gaps |>
  dplyr::select(
    target_scope,
    target_row_key,
    target_sdp_field,
    dictionary_role,
    search_query,
    top_non_smn_label,
    top_non_smn_source,
    gap_detection_basis,
    llm_decision,
    llm_new_term_label,
    llm_escalated_from,
    placement_recommendation,
    placement_confidence
  )

This is the key post-review gap table:

rows here are still unresolved;
top_non_smn_* columns show the best non-SMN evidence the package found;
gap_detection_basis distinguishes candidate evidence, a final LLM term-gap decision, or both; and
placement_recommendation gives a first-pass routing guess.

When suggestions is omitted, detect_semantic_term_gaps() reads both the semantic_suggestions and semantic_llm_assessments dictionary attributes. Final request_new_term decisions are therefore included automatically, even when candidate rows contain an SMN match. An unresolved reject_shortlist escalation remains traceable through llm_escalated_from.

If you pass suggestions explicitly, only candidate evidence and the embedded llm_* columns in that table are used. The dictionary’s assessment attribute is intentionally ignored in that form.

If you only want a narrower slice, filter by role or scope:

gaps |>
  dplyr::filter(dictionary_role %in% c("variable", "property", "entity"))

5) Decide shared salmon-domain vs DFO-specific routing

Use this plain-English rule:

Shared salmon-domain ontology (smn): use this when the term describes a reusable salmon science concept that another program, region, or organization could reasonably use too.
DFO Salmon Ontology (gcdfo): use this when the concept is clearly tied to DFO policy, operations, internal workflow, local identifiers, program-specific statuses, or other context that is not broadly reusable outside that setting.
Profile term (profile): use this for a local, program, or organization profile when neither shared SMN nor DFO-wide GCDFO governance is appropriate.

A good default test is:

if the term would still make sense in a non-DFO salmon dataset, lean SMN;
if the term only makes sense because of a DFO or local program process, lean DFO-specific.

How routing is decided

render_ontology_term_request() accepts smn, gcdfo, profile, uncertain, and skip row outcomes. Its precedence is:

an explicit scope_overrides value for the row;
a forced scope;
a recognized llm_new_term_namespace; and
the deterministic placement heuristic.

Treat llm_new_term_namespace as evidence, not authority. A human reviewer must still decide whether a DFO-specific proposal belongs in GCDFO, a narrower profile, or nowhere. Non-interactive uncertainty becomes skip; interactive use asks for a routing decision.

6) Produce a concrete term-request list

First render a dry, non-interactive plan and inspect every route:

requests <- render_ontology_term_request(
  gaps,
  scope = "auto",
  ask = FALSE,
  profile_name = "local-program"
)

request_plan <- requests |>
  dplyr::select(
    dataset_id,
    table_id,
    target_scope,
    target_row_key,
    target_sdp_field,
    dictionary_role,
    search_query,
    top_non_smn_label,
    top_non_smn_source,
    gap_detection_basis,
    llm_new_term_namespace,
    placement_recommendation,
    placement_confidence,
    request_scope,
    ontology_repo
  )

request_plan
readr::write_csv(request_plan, file.path(pkg_path, "term-request-plan.csv"))

That gives you a concrete list of which package row still needs help, the best fallback evidence, and the proposed route. Review skip rows and any route inferred from llm_new_term_namespace.

To apply reviewed row-by-row decisions, pass an override vector. Edit reviewed_scopes so every value is one of smn, gcdfo, profile, or skip:

reviewed_scopes <- request_plan$request_scope
# reviewed_scopes[2] <- "gcdfo"

requests <- render_ontology_term_request(
  gaps,
  scope = "auto",
  ask = FALSE,
  profile_name = "local-program",
  scope_overrides = reviewed_scopes
)

requests |>
  dplyr::select(
    request_scope,
    ontology_repo,
    request_title,
    target_row_key,
    search_query
  )

The override vector must have length one or one value per gap row. Rendering uses the active SMN and GCDFO repository templates and never submits an issue.

Preview issue drafts

Preview both SMN and GCDFO payloads before any submission:

issue_preview <- requests |>
  dplyr::filter(request_scope %in% c("smn", "gcdfo")) |>
  submit_term_request_issues(dry_run = TRUE)

issue_preview

The helper only posts after an explicit call with dry_run = FALSE; it never submits during detection or rendering. Keep confirm = TRUE for curator review. Profile rows remain local governance artifacts unless you provide a repository workflow for them.

7) Retain semantic mapping and decomposition evidence when available

Use SSSOM for reviewed relationships between whole vocabulary concepts. That includes an explicit sssom:NoTermFound row when a released PSC compound measurement has no defensible match in another versioned vocabulary. Do not put property/entity/unit decomposition columns into SSSOM.

write_sdp_sssom(
  pkg_path,
  mapping_sets = reviewed_sssom_paths
)
validate_sdp_sssom(pkg_path)

Store the ordered property, entity, constraint, method, and unit evidence in the separate SDP measurement-decomposition artifact:

write_sdp_measurement_decompositions(
  pkg_path,
  decompositions = reviewed_components
)
validate_sdp_measurement_decompositions(pkg_path)

metasalmon does not invent either record by default: both require reviewed inputs. When their manifests are present, later KNB planning validates them and includes them automatically inside the canonical SDP ZIP. This preserves the evidence as part of the package without claiming native I-ADOPT conformance or turning the component rows into concept mappings.

8) Finalize metadata and rebuild EDH XML if needed

Once the metadata is final and every surviving IRI is deliberate, regenerate EDH XML if your publication path needs it:

write_edh_xml_from_sdp(pkg_path)

This is the reviewed-package wrapper around edh_build_hnap_xml(). It refuses to rebuild while obvious review-state markers still exist, including:

REVIEW:-prefixed IRIs;
unresolved MISSING DESCRIPTION: / MISSING METADATA: placeholders; and
blank final observation_unit_iri values in metadata/tables.csv.

9) Run strict final validation

When the package is genuinely ready, switch to strict validation:

validate_salmon_datapackage(pkg_path, require_iris = TRUE)

This should pass only when the package is actually publication-ready.

If strict validation still fails because a measurement term genuinely needs a new shared or DFO-specific ontology term, that is not a bug in the workflow — it means the package is not finalized yet. Keep the term-request plan with the package, resolve the ontology decision, then run strict validation again.

10) Build reviewed EML and preview KNB publication when needed

KNB publication requires EML-specific facts that the canonical SDP does not guess, including structured parties, rights, methods, measurement scales, missing-value meanings, and access intent. Record those reviewed decisions in metadata/eml-mapping.yml, then build and validate EML 2.2.0:

# Copy once, then replace every example value and checksum with reviewed facts.
file.copy(
  system.file(
    "extdata",
    "eml-mapping-template.yml",
    package = "metasalmon"
  ),
  file.path(pkg_path, "metadata", "eml-mapping.yml")
)

write_eml_from_sdp(pkg_path)

That standalone call creates valid reviewed EML for inspection. The later KNB dry run deterministically rebuilds it once more so the final EML also describes each named canonical SDP artifact as an otherEntity.

Preview the exact immutable DataONE objects, identifiers, checksums, access decision, and OAI-ORE relationships without reading credentials or making a network request:

publish_sdp_to_knb(
  pkg_path,
  public = FALSE,
  dry_run = TRUE,
  representation = "expanded"
)

Live KNB publication is a separate explicit action. It requires a short-lived DataONE JWT in the process-local dataone_token option, an ORCID-authenticated subject matching the EML metadata provider, and an explicitly supplied confirm = TRUE. The confirmation means both that the exact pending manifest is approved and that the caller has authority to redistribute the package.

Keep the JWT out of scripts, YAML, manifests, and shell history. Enter it into the current R process only and clear it immediately after the live call:

options(dataone_token = rstudioapi::askForPassword("Short-lived DataONE JWT"))
on.exit(options(dataone_token = NULL), add = TRUE)

publish_sdp_to_knb(
  pkg_path,
  public = FALSE,
  dry_run = FALSE,
  confirm = TRUE,
  representation = "expanded"
)

public = FALSE is the recommended review path, but it is not a server-side draft: a live call creates persistent production KNB objects. Completion requires authenticated byte-for-byte and SystemMetadata readback for every object, anonymous denial for both object bytes and SystemMetadata, a complete authenticated catalog graph, and zero planned PIDs in the anonymous catalog. The reviewed private plan also explicitly sets DataONE replication to disabled with zero replicas, and live readback must match that KNB-only policy. Changing to public = TRUE is a separate release decision: it requires public redistribution authority and explicitly requests three DataONE preservation replicas; metasalmon never infers that decision from a private deposit.

An expanded plan contains four kinds of object: the original data resources named by tables.csv, validated canonical SDP artifacts with their exact package-relative paths, the EML science-metadata record, and the OAI-ORE map. The EML record describes raw files as data tables and named SDP artifacts as supplementary entities; ORE prov:atLocation values preserve the hierarchy for exact reconstruction. A closed inventory includes SSSOM, ordered measurement decompositions, methods, observation structures, and manifest-declared reproducibility records without scanning arbitrary files. The KNB-specific metadata/eml-mapping.yml, its authorization evidence and party details, and all mutable publication/ receipts remain local and are not deposited.

KNB DOI minting is a later opt-in release action, not part of publish_sdp_to_knb(). KNB assigns the DOI to the selected science-metadata version when its Publish action makes that package public; it does not assign a separate DOI to each raw or supplementary object. Leaving a deposit private is therefore the practical review workflow. Editing after deposit creates new immutable versions rather than overwriting bytes.

For a corrected or otherwise revised private version, keep the same series_key, add a new safe publication.revision_key to the sidecar, and bind the new dry run to the verified manifest for the preceding version. Build the revision in a fresh versioned SDP directory; never reuse or overwrite the prior package directory:

publication:
  public: false
  revision_key: corrected-expanded-sdp-2026-08-04

publish_sdp_to_knb(
  revised_pkg_path,
  public = FALSE,
  manifest_path = file.path(
    revised_pkg_path,
    "publication",
    "knb-manifest.json"
  ),
  revision_manifest = previous_manifest_path,
  dry_run = TRUE,
  representation = "expanded"
)

Review that new manifest before repeating the live call with dry_run = FALSE, confirm = TRUE. An interrupted live call is resumable: reuse the byte-identical package, revision manifest, and output manifest. A completed upload whose coordinating-network catalog check is delayed returns published_pending_catalog; rerunning the same call performs fresh checks and does not create duplicate objects.

If KNB’s separate Publish action later mints a DOI, KNB creates another science-metadata version. metasalmon does not yet import that KNB-created version into a local predecessor manifest. Do not base a later automated revision on the older pre-DOI manifest; use KNB’s editor for that next change or wait for an explicit manifest-import workflow.

Before any plan is written, metasalmon rejects referenced rows whose source or ontology label identifies them as a review-candidate. This is a deliberately offline fail-closed check; it does not resolve public IRIs or by itself prove that a vocabulary release was governed. The canonical transformation record must separately pin the approved vocabulary release and verify its public term IRIs before the SDP is deposited.

Once strict validation passes:

share the whole package folder (or a zip of it), not just datapackage.json;
keep metadata/*.csv with the data files; and
keep term-request-plan.csv only as working governance support, not as part of the canonical final package unless you want that audit trail to travel with it.

In short: reload -> validate -> detect gaps -> route requests -> build the required export -> strict validate -> dry-run -> publish.