Skip to content

METADATA: every Office custom property is reported TWICE (extractor writes it into both CustomProps and Properties) #308

Description

@sectoramen

Each custom document property in a .docx/.xlsx/.pptx produces exactly two identical findings, because the extractor stores it in two maps and the preprocessor renders both.

Mechanism

extractOfficeMetadata writes every custom property into both collections (internal/preprocessors/meta-extractors/meta-extract-officelib/office-extractor.go:369-375):

if customProps, err := extractCustomPropertiesOptimized(fileIndex); err == nil && len(customProps) > 0 {
    metadata.CustomProps = customProps
    // Also store in Properties map with "Custom_" prefix for easy scanning
    for key, value := range customProps {
        metadata.Properties["Custom_"+key] = value
    }
}

The preprocessor then renders both maps (internal/preprocessors/office_metadata_preprocessor.go:160-177): the --- Custom Properties --- block emits Custom_<key>: <value> from meta.CustomProps, and FormatPropertiesMap(meta.Properties, excludeKeys) emits the very same Custom_<key>: <value> line again. excludeKeys is {"CreationDate", "ModificationDate", "Template"}, so nothing filters the duplicated Custom_ keys.

Both lines carry the custom_ prefix, so both are classified CUSTOM_PROPERTY and both are emitted as findings.

Measured

Synthetic fixture, one custom property, benign body text: 2 CUSTOM_PROPERTY findings, identical confidence. Reproduced with three separate single-property fixtures.

Local corpus of 304 .docx files:

  • 506 CUSTOM_PROPERTY findings across 202 files
  • 0 files have an odd count
  • every (file, confidence) bucket has an even count

So the 506 findings represent ~253 real properties, each counted twice — the finding total for this type is inflated 2x.

Why it matters beyond noise

The existing dedup cannot catch this. Per #202 the dedup key is the byte span, and these two lines occupy genuinely different spans in the preprocessed text, so they are correctly treated as distinct occurrences of a duplicated input. The fix belongs upstream, at the point the same value is written into two maps.

This also inflates any per-type metric computed from finding counts, and doubles the redaction work for the affected values.

Suggested fix

Pick one home for custom properties. CustomProps is the typed field and the one the redaction path and the legacy OLE extractor both use, so the Properties["Custom_"+key] mirror looks like the redundant copy — but it is load-bearing for anything reading Properties, so removing it needs a check of its readers. Alternatively add the Custom_ prefix to excludeKeys so only the dedicated block renders them.

A regression test should assert the finding count for a one-property fixture is 1, not merely that a finding exists — the current shape passes any existence-only assertion.

No customer content is quoted; corpus figures are aggregate counts only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions