Skip to contents

metasalmon

metasalmon logo

The Problem

You’ve spent years collecting salmon data. But when you try to share it:

  • Colleagues ask “What does SPAWN_EST mean?”
  • Combining datasets fails because everyone uses different column names
  • Your future self opens old data and can’t remember what the codes mean
  • Other researchers can’t use your data without emailing you for explanations

The Solution

metasalmon wraps your salmon data with a data dictionary that travels with it—explaining every column, every code, and linking to standard scientific definitions. These definitions come from the Salmon Domain Ontology (shared layer) and the DFO Salmon Ontology (DFO-specific layer), alongside other published controlled vocabularies, and the data is packaged according to the Salmon Data Package Specification. The preferred review workflow now happens inside the R package: metasalmon can retrieve candidate terms, optionally ask an LLM to review them, write draft REVIEW-prefixed IRIs into the package metadata, and then send you back to the created package so you can confirm or edit those values directly in Excel.

Integration context: See the Salmon Data Integration System overview page (https://br-johnson.github.io/salmon-data-integration-system/) and walkthrough video (https://youtu.be/B0Zqac49zng?si=VmOjbfMDMd2xW9fH).

Think of it like adding a detailed legend to your spreadsheet that never gets lost.

What You Get

Your Data + metasalmon = Data Package
Raw CSV files Data dictionary Self-documenting dataset
Cryptic column names Clear descriptions Anyone can understand it
Inconsistent codes Linked to standards Works with other datasets

Quick Example

Before you start, do the one-time Setup and Credentials check so GitHub installs work cleanly and any optional LLM provider is ready in advance.

Install, run one function on the bundled Fraser Coho 2023-2024 example (173 rows), then review in Excel.

# Install from GitHub (recommended)
# install.packages("remotes")
# remotes::install_github("salmon-data-mobilization/metasalmon")

library(metasalmon)

data_path <- system.file("extdata", "nuseds-fraser-coho-2023-2024.csv", package = "metasalmon")
fraser_coho <- readr::read_csv(data_path, show_col_types = FALSE)

pkg_path <- create_sdp(
  fraser_coho,
  path = "fraser-coho-2023-2024-sdp",
  dataset_id = "fraser-coho-2023-2024",
  table_id = "escapement",
  check_updates = FALSE,
  overwrite = TRUE
)

# Open pkg_path and review:
# - metadata/dataset.csv
# - metadata/tables.csv
# - metadata/column_dictionary.csv
# - metadata/codes.csv (if present)
# - data/*.csv
# - semantic_suggestions.csv (if present)
# - README-review.txt

create_sdp() is the main path. It writes the canonical metadata/*.csv files plus your data/*.csv tables, adds a short review checklist, writes prefilled semantic drafts directly into metadata/column_dictionary.csv and metadata/tables.csv only where target fields were blank, and keeps semantic_suggestions.csv as a fallback shortlist when you want more context or a better match. Code-level semantic seeding stays conservative by default for factor and low-cardinality character source columns. Before SPSR/EDH upload, run validate_salmon_datapackage(pkg_path, require_iris = TRUE) to catch package/data/codes mismatches in one pass. In interactive use create_sdp() can also mention an available package update; set check_updates = FALSE to skip that check.

Bundled NuSEDS Example

The default quickstart and get-started flow use nuseds-fraser-coho-2023-2024.csv, a 173-row Fraser coho slice derived from the official Open Government Canada Fraser and BC Interior workbook.

Open Government Canada record: https://open.canada.ca/data/en/dataset/c48669a3-045b-400d-b730-48aafe8c5ee6

A smaller nuseds-fraser-coho-sample.csv file is still bundled for tiny smoke tests, but it is no longer the default walkthrough dataset.

The package also now ships a matching starter dictionary for the fuller example (system.file("extdata", "nuseds-fraser-coho-2023-2024-column_dictionary.csv", package = "metasalmon")), which is useful when you want a ready-made context file for the package-native LLM review path.

See example-data-README.md for the record/resource URLs, row counts, licensing note, and the data-raw/ script that reproduces the 2023-2024 example.

To continue:

Need the full context-file workflow? See LLM Review With Context Files.

Package-native LLM semantic review (optional)

If you want an LLM to judge the shortlisted semantic matches directly from R, keep the deterministic search path and add an opt-in review pass:

context_files <- c(
  file.path(pkg_path, "metadata", "column_dictionary.csv"),
  "README.md",
  "data-dictionary.xlsx",
  "methods-report.pdf"
)

suggested <- suggest_semantics(
  df = fraser_coho,
  dict = infer_dictionary(fraser_coho, dataset_id = "fraser-coho-2023-2024", table_id = "escapement"),
  llm_assess = TRUE,
  llm_provider = "openrouter",
  llm_context_files = context_files
)

suggestions <- attr(suggested, "semantic_suggestions")
assessments <- attr(suggested, "semantic_llm_assessments")

# In create_sdp(...), any auto-applied semantic IRI drafts are written back
# into the package metadata as REVIEW-prefixed values for manual cleanup,
# including table-level observation-unit selections in metadata/tables.csv.

This keeps find_terms() as the canonical candidate generator. For each measurement column, the LLM reviews the variable, property, entity, unit, constraint, and method shortlists together. Generic column, code, table, and dataset targets keep their existing per-target review path. Deterministic validators can downgrade an unsupported accept to review; the original model confidence remains in the assessment as provenance.

Only accepted variable, property, entity, and unit candidates can be written back automatically. Drafts are stored as REVIEW: <iri> so you can confirm or replace them in Excel rather than treating them as final. Constraint and method assessments always remain manual, as do dataset keywords and code-level suggestions. Compatible table-level observation-unit matches can still be written into metadata/tables.csv.

Omitting sources (or semantic_sources on the package-creation helpers) uses role-aware defaults. Supplying sources explicitly creates a strict allowlist for both initial retrieval and the single retry round. For example, sources = "smn" cannot introduce a QUDT unit candidate.

The target-level semantic_llm_assessments table has a stable 30-column schema. Its final two fields record whether a term request was escalated from reject_shortlist and why a retry query was rejected. An exact retry duplicate, after case and whitespace normalization, remains a retry_search decision with llm_retry_query_rejection_reason = "duplicate_original_query" and does not trigger another query-generation, search, or reassessment call.

llm_context_files must be a character vector of existing local file paths. Do not pass a tibble returned by readr::read_csv(), an xml2 document returned by read_html(), or another parsed object; those inputs now fail early instead of being silently ineffective. Supplying context also never enables an LLM call: set llm_assess = TRUE explicitly, or the package warns that the context will be ignored and continues with deterministic retrieval only. Use llm_context_text for inline text that is not stored in a file.

Supported paths include text and notes (.md, .txt, .rst), delimited/data files (.csv, .tsv, .json, .yaml, .yml), source and notebook-style files (.R, .Rmd, .qmd), HTML (.htm, .html), DOCX (.docx), Excel workbooks (.xls, .xlsx, .xlsm via readxl), and PDF reports (.pdf via pdftools). Validation should only pass after the REVIEW prefix is removed. When you use llm_provider = "openrouter" without specifying llm_model, metasalmon defaults to openrouter/free.

For the full workflow across dataset.csv, tables.csv, column_dictionary.csv, codes.csv, and the post-review EDH rebuild, use the LLM Review With Context Files guide.

The quickstart path does not require an API key. Only set up one of these providers when you want create_sdp(..., llm_assess = TRUE) or suggest_semantics(..., llm_assess = TRUE).

For DFO internal users on the internal network or VPN, open https://chapi-dev.intra.azure.cloud.dfo-mpo.gc.ca/, click the user icon in the bottom left, open Settings, click Show next to API Keys, and copy the key value. Then run:

file.edit("~/.Renviron")
CHAPI_API_KEY="paste key here"

chapi defaults to ollama2.mistral:7b and https://chapi-dev.intra.azure.cloud.dfo-mpo.gc.ca/api. Optional overrides are CHAPI_MODEL and CHAPI_BASE_URL. gpt-oss:latest is also supported, but it can be slower to warm up and now gets a longer automatic timeout.

For external users, OpenRouter is the easiest free option:

file.edit("~/.Renviron")
OPENROUTER_API_KEY="paste key here"

llm_provider = "openrouter" defaults to openrouter/free.

If you already have OpenAI API credits, use:

file.edit("~/.Renviron")
OPENAI_API_KEY="paste key here"

Then pass an OpenAI model explicitly, for example llm_model = "gpt-4.1-mini".

For the current package-native review path, use this order:

  1. Run create_sdp(...) to create the Salmon Data Package.
  2. If you want semantic review, set llm_assess = TRUE.
  3. Open README-review.txt, then review metadata/column_dictionary.csv and metadata/tables.csv first. Those files already contain the prefilled semantic values you are actually finalizing.
  4. For any prefilled or REVIEW:-prefixed IRI, click through and read the term definition before keeping it.
  5. Use semantic_suggestions.csv only as a fallback shortlist if you are unsure or want a better match.
  6. If no candidate fits, request a new term instead of forcing a bad match:
  7. Follow the After Excel Review guide to reload the package, detect unresolved semantic gaps, and produce a concrete shared-vs-DFO term-request plan.
  8. If you are preparing EDH metadata, regenerate the XML from the reviewed package with write_edh_xml_from_sdp(pkg_path) (the reviewed-package wrapper around the canonical edh_build_hnap_xml() builder). It now refuses to rebuild while REVIEW: markers or unresolved dataset/table placeholder text remain.
  9. Re-run validation with validate_salmon_datapackage(pkg_path, require_iris = TRUE).
  10. For KNB, add the reviewed metadata/eml-mapping.yml sidecar, generate schema-valid EML 2.2.0 with write_eml_from_sdp(pkg_path), and inspect the credential-free restricted-review plan with publish_sdp_to_knb(pkg_path, public = FALSE, dry_run = TRUE, representation = "expanded"). A live private review still creates persistent production objects; it verifies authenticated exact readback and anonymous non-disclosure, and explicitly disables DataONE peer replication so the restricted bytes remain KNB-only. The plan contains the original data files plus each validated canonical SDP artifact as a named object with its package-relative path. No ZIP is required, and the KNB-specific EML mapping and publication receipts remain local. KNB has no separate server-side draft state, and metasalmon does not mint a DOI. DOI minting and public release are a later, per-metadata-version decision in KNB. Referenced vocabulary rows labelled as review candidates are blocked; the transformation workflow must separately pin and verify the governed vocabulary release. Corrections use a new sidecar revision_key and the preceding verified manifest, preserving the KNB series instead of overwriting immutable objects.
  11. Publish/share only after the REVIEW: markers are gone and validation passes; send the whole package folder, not individual files.

In other words: create -> review in Excel -> reload/check gaps -> remove REVIEW: markers -> validate -> build the required export -> dry-run -> publish.

Who Is This For?

If you are… Start here
A biologist who wants to share data 5-Minute Quickstart
Finished Excel review and need to publish After Excel Review
Curious how it works How It Fits Together
A data steward standardizing datasets Data Dictionary & Publication
Reading CSVs from private GitHub repos GitHub CSV Access

Video Walkthrough

Watch: Creating Your First Data Package

Installation 

# Install from GitHub
install.packages("remotes")
remotes::install_github("salmon-data-mobilization/metasalmon")

What’s In a Data Package?

When you create a package, you get a folder containing:

my-data-package/
  +-- README-review.txt         # Step-by-step review checklist for manual Excel cleanup
  +-- semantic_suggestions.csv  # Detailed semantic evidence + LLM review trail (when present)
  +-- datapackage.json          # Machine-readable export
  +-- metadata/
  |   +-- dataset.csv           # Dataset-level metadata (canonical)
  |   +-- tables.csv            # Table-level metadata and file paths
  |   +-- column_dictionary.csv # What each column means
  |   +-- codes.csv             # What each code value means (if applicable)
  +-- data/
      +-- escapement.csv        # Your data table(s)

Anyone opening this folder - whether a colleague, a reviewer, or your future self - can immediately understand your data. The metadata/*.csv files are the canonical package metadata; datapackage.json is a derived interoperability export. When you share the package, send the whole folder (or a zip of the whole folder), not just datapackage.json.

Key Features

For everyday use:

  • Automatically generate data dictionaries from your data frames
  • Validate that your dictionary is complete and correct
  • Create shareable packages that work across R, Python, and other tools
  • Read CSVs directly from private GitHub repositories

For data stewards (optional):

  • Link columns to standard DFO Salmon Ontology terms
  • Add I-ADOPT measurement metadata (property, entity, unit, constraint)
  • Use AI assistance to help write descriptions
  • Suggest Darwin Core Data Package table/field mappings for biodiversity data
  • Opt in to DwC-DP export hints via suggest_semantics(..., include_dwc = TRUE) while keeping the Salmon Data Package as the canonical deliverable.
  • Generate HNAP-aware EDH metadata XML for DFO Enterprise Data Hub upload workflows via the canonical edh_build_hnap_xml() builder, the reviewed-package helper write_edh_xml_from_sdp(), or create_sdp(..., include_edh_xml = TRUE). Create-time XML is a draft when review markers remain; finalize the metadata and rebuild it before submission.
  • Generate deterministic, schema-validated EML 2.2.0 from a strictly reviewed SDP with write_eml_from_sdp(), then produce an offline exact-object KNB manifest or explicitly run a verified private/public DataONE deposit with publish_sdp_to_knb(). Expanded KNB plans retain the original data resources and publish the closed canonical SDP inventory as named package-relative objects without a ZIP.
  • Role-aware vocabulary search with find_terms() and sources_for_role():
    • Units: QUDT preferred, then NVS P06
    • Salmon-domain roles: shared SMN terms first, then GCDFO DFO-specific terms where needed
    • Entities/taxa: SMN and GCDFO first, then GBIF and WoRMS taxon resolvers
    • Properties/variables/methods: shared salmon-domain terms first, then broader ontology fallbacks
    • Cross-source agreement boosting for high-confidence matches
  • Per-source diagnostics, scoring, and optional rerank explain why find_terms() matches rank where they do and expose failures, so you can tune role-aware queries with confidence.
  • End-to-end semantic QA loop with fetch_salmon_ontology() + validate_semantics(), plus deduplicate_proposed_terms() to prevent term proliferation before opening ontology issues.
  • Optional package-native LLM review for semantic suggestions: suggest_semantics(..., llm_assess = TRUE) reviews all six measurement roles as one bundle, keeps explicit source lists strict across retries, and uses deterministic validators to downgrade unsupported acceptances.
  • Structured ontology-gap handling combines deterministic candidate gaps with final LLM request_new_term decisions, preserves escalation evidence, and renders curator-reviewable request bodies for shared SMN, DFO-specific GCDFO, or local profile governance.
  • NuSEDS method crosswalk helpers: nuseds_enumeration_method_crosswalk() and nuseds_estimate_method_crosswalk() for mapping legacy values to canonical method families.

Getting Help

How It Fits Together

metasalmon brings together four pieces: your raw data, the Salmon Data Package specification, the Salmon Domain Ontology (and other vocabularies), optional in-package LLM review, and the review files written into the package itself. When you finish the workflow, the dictionary, dataset/table metadata, and optional code lists are already aligned with the specification, which makes the package ready to publish. The ontology keeps the column meanings consistent, and the package-native review workflow helps draft descriptions and term choices without forcing you into a separate prompt-export side path.

The high-level flow is:

  • Start here: create_sdp() takes raw tables, infers the package metadata, writes a review-ready package, gives you a checklist, auto-fills top column/table semantic suggestions only where fields are blank, and keeps default code-level semantic seeding conservative by limiting it to factor and low-cardinality character source columns.
  • Advanced/manual path: write_salmon_datapackage() is for cases where you already assembled dataset.csv, tables.csv, column_dictionary.csv, and optional codes.csv yourself.
  • Raw tables lead into metadata/column_dictionary.csv (and metadata/codes.csv when there are categorical columns).
  • Dataset/table metadata fill the required specification fields (title, description, creator, contact, etc.), so the package folder can be shared or uploaded.
  • The Salmon Domain Ontology and published vocabularies supply term_iri/entity_iri links that describe what each column and row represents.
  • Post-review publication helpers let you reopen the package, re-run semantic checks, detect unresolved ontology gaps, and route curator-reviewed drafts to shared SMN, DFO-specific GCDFO, or a local profile.
  • write_salmon_datapackage() consumes the metadata, dictionary, codes, and data to write the files in the Salmon Data Package format; the preferred review loop is now the package itself plus README-review.txt / semantic_suggestions.csv, not an external prompt-export workflow.

For Developers

Development setup and package structure

Installation for Development 

install.packages(c("devtools", "roxygen2", "testthat", "knitr", "rmarkdown",
                   "tibble", "readr", "jsonlite", "cli", "rlang", "dplyr",
                   "tidyr", "purrr", "withr", "frictionless"))

Build and Check 

devtools::document()
devtools::test()
devtools::check()
devtools::build_vignettes()
Rscript scripts/build-pkgdown.R

# Canonical source-tarball build path (writes into the repo root, not ../)
./scripts/build-package.sh

Package Structure

  • R/: Core functions for dictionary and package operations
  • inst/extdata/: Example data files and templates
  • tests/testthat/: Automated tests
  • vignettes/: Long-form documentation
  • docs/: pkgdown site output

Internal Review Architecture

  • R/semantic-suggestions.R owns the internal semantic-suggestion row interface: target keys, candidate row normalization, column-term filtering, and LLM assessment merge rules.
  • R/llm-review-adapter.R owns shared LLM/chat review parsing, validation, and assessment row construction.
  • R/semantics-helpers.R, R/llm-semantic-helpers.R, and R/chat-decomposition.R keep the public workflows while delegating shared row and review behavior to those internal modules.

Salmon Domain Ontology

This package can link your data to the Salmon Domain Ontology for shared terms and to the DFO Salmon Ontology for DFO-specific terms. Canonical IRIs are explicit: SMN uses https://w3id.org/smn/<Term> and GCDFO uses https://w3id.org/gcdfo/salmon#<Term>. metasalmon does not silently rewrite legacy salmon: IRIs.

See the Reusing Standards for Salmon Data Terms guide for details.