Support documented Markdown, JSON, and shell text inputs

[jioffe502]

Summary

• route .md, .json, and .sh through the existing plain-text extraction path
• support the formats across root ingest, GraphIngestor, buffers, directory/glob expansion, and service uploads
• preserve case-insensitive directory discovery and the existing text/plain service behavior
• update the bundled Retriever skill format guidance

Why

The public documentation already lists Markdown, JSON, and shell files as supported text inputs, but the shared local and service extension registries only accepted .txt. As a result, retriever ingest and GraphIngestor rejected documented formats before extraction.

This change only restores the documented routing behavior. It does not add parsers, new chunking behavior, or retrieval changes.

Validation

• focused test suite: 241 passed, 2 deselected
• configured pre-commit hooks: passed
• Retriever skill validation: passed
• live hybrid ingest with nvidia/llama-nemotron-embed-1b-v2: 3 files (.md, .json, .sh) -> 3 LanceDB rows
• two-axis Standards and Spec review: no findings

Follow-up

• Unify normalized text chunking for HTML and plain-text inputs #2320 tracks the low-priority opportunity to unify normalized text-to-chunks row construction for HTML and native text inputs.

[greptile-apps]

Greptile Summary

This PR routes .md, .json, and .sh files through the existing plain-text extraction path, aligning the implementation with public documentation that already listed these as supported inputs. The change touches the shared extension registry (input_files.py), the service file classifier (file_type.py), and the bundled Retriever skill documentation.

• Core change: Adds .md, .json, and .sh to both the CLI INPUT_TYPE_PATTERNS/INPUT_TYPE_EXTENSIONS registries and the service FileClassifier.SUFFIX_MAP, routing all three as FileCategory.TEXT / text/plain so they are processed identically to .txt.
• resolve_input_files refactor: Directory scanning now uses case-insensitive suffix matching (match.suffix.lower()) instead of per-pattern rglob calls, making the root CLI path reliably case-insensitive on Linux for all supported extensions.
• Test coverage: Five new parametrized tests and one renamed test confirm routing through manifest planner, root CLI, GraphIngestor file/buffer inputs, and service classifier; a dedicated test_file_type.py is added for the service classifier.

Confidence Score: 4/5

The change is narrowly scoped to routing three additional extensions through an existing extraction path with no new parsers or retrieval changes.

Tests are comprehensive across all affected code paths, SPDX headers are present in the new file, and the extension registry changes are self-consistent. The only gap is that resolve_input_patterns (GraphIngestor directory expansion) still uses case-sensitive glob patterns while resolve_input_files is now explicitly case-insensitive, leaving the two paths with inconsistent behavior for uppercase extensions like README.MD.

input_files.py — the interaction between resolve_input_files (case-insensitive) and resolve_input_patterns (case-sensitive globs) for directory inputs.

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/common/input_files.py	Adds .md/.json/.sh to INPUT_TYPE_PATTERNS["txt"] and rewrites resolve_input_files to use case-insensitive suffix matching; functionally correct with one minor gap noted.
nemo_retriever/src/nemo_retriever/service/utils/file_type.py	Adds .md, .json, .sh to SUFFIX_MAP as FileCategory.TEXT / text/plain; consistent with how .txt is handled and correctly flows through infer_extraction_mode_from_filename.
nemo_retriever/tests/service/utils/test_file_type.py	New file; SPDX header present, tests all three new extensions including uppercase README.MD to verify case folding in classify().
nemo_retriever/tests/test_ingest_interface.py	Three new tests covering markdown file, JSON file via extract_txt(), and shell script via buffers(); all follow existing test patterns.
nemo_retriever/tests/test_ingest_manifest.py	Parametrized test confirms manifest planner routes all three new extensions to the txt/text branch correctly.
nemo_retriever/tests/test_root_cli_workflow.py	Adds parametrized CLI routing test, case-insensitive discovery test (README.MD), glob expansion test, and updates the renamed directory-inclusion test.
nemo_retriever/tests/test_graph_pipeline_cli.py	Updates resolve_input_patterns assertion to reflect the expanded txt pattern tuple; straightforward and correct.
nemo_retriever/tests/test_service_pipeline_spec.py	Extends parametrized extraction-mode table test and replaces notes.txt with README.md in the typed-shortcuts test.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    Input["File Input\n(path / buffer / glob / directory)"]
    Input --> TypeDetect["Extension lookup\ninput_type_for_path()\nINPUT_TYPE_EXTENSIONS"]
    TypeDetect -->|".pdf / .docx / .pptx"| DocBranch["Document branch"]
    TypeDetect -->|".txt / .md / .json / .sh"| TxtBranch["Text branch\nfamily=txt / extraction_mode=text"]
    TypeDetect -->|".html"| HtmlBranch["HTML branch"]
    TypeDetect -->|"image / audio / video"| MediaBranch["Media branch"]
    TypeDetect -->|"unknown"| Reject["ValueError: Unsupported input type"]
    TxtBranch --> ServiceClassify["Service: FileClassifier.classify()\nFileCategory.TEXT / text/plain"]
    TxtBranch --> CLIResolve["CLI: resolve_input_files()\nrglob + suffix.lower() (case-insensitive)"]
    TxtBranch --> GraphPatterns["GraphIngestor: resolve_input_patterns()\n*.txt / *.md / *.json / *.sh"]
    ServiceClassify --> TextExtract["Plain-text extraction"]
    CLIResolve --> TextExtract
    GraphPatterns --> TextExtract
    TextExtract --> LanceDB["LanceDB row: text + path + metadata"]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    Input["File Input\n(path / buffer / glob / directory)"]
    Input --> TypeDetect["Extension lookup\ninput_type_for_path()\nINPUT_TYPE_EXTENSIONS"]
    TypeDetect -->|".pdf / .docx / .pptx"| DocBranch["Document branch"]
    TypeDetect -->|".txt / .md / .json / .sh"| TxtBranch["Text branch\nfamily=txt / extraction_mode=text"]
    TypeDetect -->|".html"| HtmlBranch["HTML branch"]
    TypeDetect -->|"image / audio / video"| MediaBranch["Media branch"]
    TypeDetect -->|"unknown"| Reject["ValueError: Unsupported input type"]
    TxtBranch --> ServiceClassify["Service: FileClassifier.classify()\nFileCategory.TEXT / text/plain"]
    TxtBranch --> CLIResolve["CLI: resolve_input_files()\nrglob + suffix.lower() (case-insensitive)"]
    TxtBranch --> GraphPatterns["GraphIngestor: resolve_input_patterns()\n*.txt / *.md / *.json / *.sh"]
    ServiceClassify --> TextExtract["Plain-text extraction"]
    CLIResolve --> TextExtract
    GraphPatterns --> TextExtract
    TextExtract --> LanceDB["LanceDB row: text + path + metadata"]

Loading

Comments Outside Diff (1)

1. nemo_retriever/src/nemo_retriever/common/input_files.py, line 125-126 (link)

   Case-sensitivity gap between the two directory-discovery paths

   resolve_input_files (used by the root CLI) now uses match.suffix.lower() and is correctly case-insensitive on Linux. But resolve_input_patterns (used by GraphIngestor for directory inputs) still generates raw glob patterns like **/*.md, which glob.glob resolves case-sensitively on Linux. A file named README.MD will be picked up by the root CLI path but silently skipped when the same directory is explored via GraphIngestor.files(directory). The gap existed for .txt before this PR, but the new test test_root_ingest_directory_discovers_text_extensions_case_insensitively only exercises the CLI path — there is no parallel test for the resolve_input_patterns branch, so the asymmetry is now undocumented.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: nemo_retriever/src/nemo_retriever/common/input_files.py
   Line: 125-126
   
   Comment:
   **Case-sensitivity gap between the two directory-discovery paths**
   
   `resolve_input_files` (used by the root CLI) now uses `match.suffix.lower()` and is correctly case-insensitive on Linux. But `resolve_input_patterns` (used by GraphIngestor for directory inputs) still generates raw glob patterns like `**/*.md`, which `glob.glob` resolves case-sensitively on Linux. A file named `README.MD` will be picked up by the root CLI path but silently skipped when the same directory is explored via `GraphIngestor.files(directory)`. The gap existed for `.txt` before this PR, but the new test `test_root_ingest_directory_discovers_text_extensions_case_insensitively` only exercises the CLI path — there is no parallel test for the `resolve_input_patterns` branch, so the asymmetry is now undocumented.
   
   How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
nemo_retriever/src/nemo_retriever/common/input_files.py:125-126
**Case-sensitivity gap between the two directory-discovery paths**

`resolve_input_files` (used by the root CLI) now uses `match.suffix.lower()` and is correctly case-insensitive on Linux. But `resolve_input_patterns` (used by GraphIngestor for directory inputs) still generates raw glob patterns like `**/*.md`, which `glob.glob` resolves case-sensitively on Linux. A file named `README.MD` will be picked up by the root CLI path but silently skipped when the same directory is explored via `GraphIngestor.files(directory)`. The gap existed for `.txt` before this PR, but the new test `test_root_ingest_directory_discovers_text_extensions_case_insensitively` only exercises the CLI path — there is no parallel test for the `resolve_input_patterns` branch, so the asymmetry is now undocumented.

_{Reviews (1): Last reviewed commit: "Support documented plain-text input exte..." | Re-trigger Greptile}