deduplicate_proposed_terms
deduplicate_proposed_terms(proposed_terms, warn_threshold=30)Apply I-ADOPT compositional deduplication to a gpt_proposed_terms dataframe.
This prevents term proliferation by collapsing age and phase variants into base terms and suggesting when to use constraint_iri instead.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| proposed_terms | pd.DataFrame | DataFrame with columns: term_label, term_definition, term_type, suggested_parent_iri. Typically loaded from gpt_proposed_terms.csv. | required |
| warn_threshold | int | If the input has more than this many rows, issue a warning about potential over-engineering. | 30 |
Returns
| Name | Type | Description |
|---|---|---|
| pd.DataFrame | Deduplicated terms with additional columns: - is_base_term: True if this is the canonical base term for a pattern - needs_age_facet: True if age variants should use constraint_iri - needs_phase_facet: True if phase variants should use constraint_iri - collapsed_from: Count of how many variants were collapsed into this term - dedup_notes: Explanation of deduplication applied |
Examples
>>> import pandas as pd
>>> proposed = pd.read_csv("work/semantics/gpt_proposed_terms.csv")
>>> deduped = deduplicate_proposed_terms(proposed)
>>> # Review collapsed terms
>>> print(deduped[deduped['collapsed_from'] > 1])
>>> # Write cleaned output
>>> deduped.to_csv("work/semantics/gpt_proposed_terms_deduped.csv", index=False)Notes
Target ratio: For a dictionary with N measurement columns, expect ~N/10 to N/5 distinct base terms, NOT N terms. If the output still has >30 rows, consider further manual review.
Anti-patterns detected: - “Spawners Age 1”, “Spawners Age 2”, … patterns → collapsed to “SpawnerCount” - Duplicate term_labels across different tables → deduplicated - Phase-stratified variants (Ocean X, Terminal X) → collapsed to base term