Remap

A lightweight Go-based CLI tool that transforms CSV and JSON files using a YAML-driven mapping configuration. Useful for:

• CSV cleanup
• System migrations
• API payload transformation
• Quick data reshaping tasks

---

Overview

Remap takes an input file (CSV or JSON), applies field-level transformations defined in a YAML config, and outputs a transformed file (CSV or JSON).

It is designed as a simple ETL-style pipeline: Input File → Reader → Validation → Mapper → Writer → Output File

Why I built this

Working in enterprise PropTech, I spent a lot of time writing one-off scripts to move data between systems: rename a field here, reformat a date there, drop a column the target system didn't expect. The logic was always simple but the setup wasn't. I wanted a single tool I could point at a file and a config and just get the output. Remap is that tool.

---

Features (MVP)

Input/Output Formats

• CSV → JSON
• JSON → CSV
• CSV → CSV
• JSON → JSON

Core Capabilities

• Field renaming
• Field dropping
• Type validation:
  • int
  • float
  • date
  • string
• Required field validation
• Date parsing with configurable input formats
• Transform system (e.g. upper/lower/date_format)
• Dry-run mode (preview output without writing files)
• Config validation (ensures YAML rules are valid before execution)

---

Installation

go build -o remap ./cmd

---

CLI Usage

remap --input data.csv --config remap.yaml --output result.csv

Dry Run Mode

remap --input data.csv --config remap.yaml --dry-run

---

YAML Configuration

Basic Example

mappings:
  - from: invoice_date
    to: date
    type: date
    input_date_format: 02-01-2006
    transform: date_format(02-01-2006,02/01/2006) #inputFormat, outputFormat

  - from: client_name
    to: customer
    transform: upper()

  - from: internal_id
    to: ~ # ~ drops the field from output

---

Mapping Fields

from

Source field name in input data.

to

Output field name. Use ~ to drop a field.

required

Marks a field as mandatory.

required: true

---

type

Defines validation type:

Supported:

• int
• float
• date
• string

---

input_date_format

Defines how to parse incoming date strings.

type: date
input_date_format: 02-01-2006

Must only be used with type: date.

---

transform

Defines transformation rules.

Examples:

transform: upper()
transform: lower()
transform: date_format(02/01/2006)

---

Transform System

Transforms are parsed and applied per field.

Example transformations:

• upper() → converts string to uppercase
• lower() → converts string to lowercase
• date_format(from, to) → reformats date strings

---

Validation System

1. Config Validation (startup)

Runs before processing data.

Checks:

• valid mapping structure
• input_date_format only used with type: date
• valid type values

---

2. Record Validation (runtime)

Runs per row of input data.

Checks:

• required fields exist and are not empty

• values match declared types:

  • int
  • float
  • date (with configurable layout)

---

Example Validation Error

invoice_amount is not a valid float

---

Dry Run Mode

Dry-run allows previewing output without writing files.

Example:

remap --input data.csv --config remap.yaml --dry-run

Output:

[
  {
    "date": "01/01/2025",
    "customer": "MANNY"
  }
]

---

Architecture

cmd/root.go
internal/
  config/        → YAML parsing + config validation
  readers/       → CSV / JSON input parsing
  mapper/        → transformation engine
  transforms/    → transform functions (upper, lower, date_format)
  validator/     → record-level validation
  writers/       → CSV / JSON output

---

Design Principles

• Readers are dumb (only parse files)
• Mapper handles one record at a time
• Writers only format output
• Validation happens before transformation
• Config drives all behavior

---

Example Flow

Input (CSV)

invoice_date,client_name,internal_id
01-06-2025,Manny,ABC123

Config

mappings:
  - from: invoice_date
    to: date
    type: date
    input_date_format: 02-01-2006
    transform: date_format(02/01/2006)

  - from: client_name
    to: customer
    transform: upper()

  - from: internal_id
    to: ~

Output (JSON)

[
  {
    "date": "01/06/2025",
    "customer": "MANNY"
  }
]

---

Current Status

Completed:

• CSV reader
• JSON reader
• CSV writer
• JSON writer
• Mapping engine
• Transform system
• Type validation
• Required field validation
• Date parsing with input format
• Dry-run mode
• Config validation (basic)

---

Next Possible Improvements

• Collect all validation errors (not fail fast)
• Streaming support for large files
• Multiple transforms per field
• Better CLI UX (logs, progress)
• Schema inference mode
• A lightweight Vue-based web preview for non-CLI users

---