Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions .github/workflows/python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,31 @@ name: CI

on:
push:
branches: ['master']
branches: ["master"]
pull_request:

jobs:
build:
test:
strategy:
matrix:
python: ['3.10', '3.11']
python: ["3.12", "3.13"]
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- uses: actions/checkout@v6

- name: Install uv
uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0
with:
version: "latest"

- name: Set up Python ${{ matrix.python }}
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python }}
cache: pip

- name: Install dependencies
run: |
python -m pip install --upgrade pip
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
run: uv sync --dev

- name: Run tests
run: pytest --cov=echolalia
run: uv run pytest
64 changes: 45 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,53 +3,79 @@
![project: prototype](https://img.shields.io/badge/project-prototype-orange.svg "Project: Prototype")
[![Build Status](https://github.com/eiri/echolalia/workflows/CI/badge.svg)](https://github.com/eiri/echolalia/actions)

Generate random data to test your application
Generate random data to test your application.

## Requirements

- Python 3.12 or 3.13
- [uv](https://docs.astral.sh/uv/)

## Installation

Clone repo with `git clone https://github.com/eiri/echolalia-prototype.git`, create and active virtual environment with `python -m venv venv` and `. venv/bin/activate`, then install requirements with `pip install -r requirements.txt`.
```bash
git clone https://github.com/eiri/echolalia.git
cd echolalia
uv sync
```

`uv sync` creates the virtual environment and installs all dependencies.

## Usage

```bash
$ ./echolalia.py -c 2 -t templates/people.json -w stdout
$ uv run echolalia -c 2 -t templates/people.json
[{"name": {"lastName": "Shannon", "firstName": "Rhonda"}, "tags": ["nihil", "fngheqnl", "impedit", "consequatur"], "age": 30, "state": "Hawaii, AR", "sex": "F", "phone": "03744269231", "single": true, "street": "4081 Sharon Ranch Apt. 197", "postcode": "ZIP: 02709-0053", "times": {"createdAt": "2017-02-13 13:14:08", "updatedAt": "2017-09-23 15:37:29"}, "email": "tiffany87@hotmail.com"}, {"name": {"lastName": "Hanson", "firstName": "Robert"}, "tags": ["quasi", "##tuesday###", "deserunt", "laborum"], "age": 104, "state": "Nevada, FL", "sex": "F", "phone": "(698)292-8761x6944", "single": false, "street": "3898 Alexandria Parkways", "postcode": "ZIP: 24439", "times": {"createdAt": "2017-05-03 03:16:21", "updatedAt": "2017-09-23 15:37:02"}, "email": "zfowler@hotmail.com"}]
```

```bash
$ ./echolalia.py -c 2 -i name -i email=free_email -f csv
$ uv run echolalia -c 2 -i name -i email=free_email -f csv
Bruce Day,lori09@yahoo.com
Janice Turner,matthew72@sanders.com
```

To skip the `uv run` prefix, activate the virtualenv first:

```bash
source .venv/bin/activate
echolalia -c 2 -i name -i email=free_email -f csv
```

## Development

```bash
uv sync --dev
uv run pytest
```

## Templates

JSON document of expected structure where keys will be used as keys for generated document and values should be methods of [faker](https://github.com/joke2k/faker) library. If method suppose to get arguments, the value block should be defined as json object with "attr" for name of method and "args" for list of provided arguments.
Templates are JSON objects. Keys become keys in the generated document; values are [faker](https://github.com/joke2k/faker) method names.

To pass arguments to a method, use an object with `"attr"` and `"args"`:

While template must be an object, the keys can take list of methods to generate arrays. For complex values mustash style of template can be used (e.g. `"{state}, {state_abbr}"`). Additional element "postprocess" can be used to run specified command over generated value.
```json
{ "birthday": { "attr": "date_of_birth", "args": [null, 18, 65] } }
```

A key's value can also be a list of methods, which produces an array. For composite strings, use mustache syntax: `"{state}, {state_abbr}"`. To transform the result after generation, add a `"postprocess"` key with a `str` method name.

Take a look at `templates/people.json` file for example.
See `templates/people.json` for a full example.

## Formatters
### Raw
Pass through, returns generated data as python object.

### JSON
Marshalls data to JSON.
**raw** — returns the data as a Python object, no serialization.

### CSV
Marshalls data to CSV format. If command line argument `--with_headers` provided adds as a first line a list of keys. Generated object more than 1 level of depth smashed into string
**json** — JSON output.

### YAML
Marshalls data in YAML. Collections always serialized in block style.
**csv** — CSV output. Pass `--with_header` to add a header row. Nested objects are flattened to strings.

**yaml** — YAML output, always in block style.

## Writers
### StdOut
This is a basic plugin that just outputs generated data on the standard output.

### File
Output to a specified with `-o` or `--output` file.
**stdout** — prints to standard output (default).

**file** — writes to a file; requires `-o`/`--output`.

## Licence

Expand Down
80 changes: 0 additions & 80 deletions echolalia.py

This file was deleted.

88 changes: 88 additions & 0 deletions echolalia/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""
Generate random data for your application
"""

import argparse
import importlib
import logging


def add_args():
parser = argparse.ArgumentParser(
description="Generate random data for your application"
)
parser.add_argument("-w", "--writer", type=str, default="stdout")
parser.add_argument("-f", "--format", type=str, default="json")
parser.add_argument("-c", "--count", type=int, default=1)
parser.add_argument("-v", "--verbose", action="store_true")
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("-t", "--template", type=str)
group.add_argument(
"-i", "--items", type=str, action="append", metavar="KEY=VALUE"
)
return parser


def init_logging(verbose: bool = False) -> logging.Logger:
level = logging.DEBUG if verbose else logging.ERROR
logging.basicConfig(
format="%(levelname)-9s %(funcName)s:%(lineno)d - %(message)s",
datefmt="%H:%M:%S",
level=level,
)
return logging.getLogger()


def main() -> None:
from echolalia.generator import Generator

parser = add_args()
args, _ = parser.parse_known_args()

writer_name = args.writer
formatter_name = args.format

writer_mod = importlib.import_module(f"echolalia.writer.{writer_name}")
writer = writer_mod.Writer()
parser = writer.add_args(parser)

formatter_mod = importlib.import_module(f"echolalia.formatter.{formatter_name}er")
formatter = formatter_mod.Formatter()
parser = formatter.add_args(parser)

args = parser.parse_args()
log = init_logging(verbose=args.verbose)
log.debug("Start")

template = args.template
count = args.count

if template is None:
log.debug("Generating %d docs with %d item(s)", count, len(args.items))
items: dict[str, str] = {}
for item in args.items:
kv = item.split("=", 1)
if len(kv) == 2:
items[kv[0]] = kv[1]
else:
items[item] = item
generator = Generator(items=items)
else:
log.debug("Generating %d docs with template %s", count, template)
generator = Generator(template=template)

data = generator.generate(count)

log.debug('Marshalling with formatter "%s"', args.format)
docs = formatter.marshall(args, data)

log.debug('Writing with writer "%s"', args.writer)
writer.write(args, docs)

log.debug("Done")
parser.exit(status=0)


if __name__ == "__main__":
main()
32 changes: 17 additions & 15 deletions echolalia/formatter/csver.py
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
import io, csv
import csv
import io


class Formatter:

def __init__(self):
return None
def __init__(self) -> None:
pass

def add_args(self, parser):
parser.add_argument('--with_header', action='store_true')
return parser
def add_args(self, parser):
parser.add_argument("--with_header", action="store_true")
return parser

def marshall(self, args, data):
output = io.StringIO()
keys = data[0].keys()
writer = csv.DictWriter(output, fieldnames=keys)
if args.with_header:
writer.writeheader()
for doc in data:
writer.writerow(doc)
return output.getvalue()
def marshall(self, args, data) -> str:
output = io.StringIO()
keys = data[0].keys()
writer = csv.DictWriter(output, fieldnames=keys)
if args.with_header:
writer.writeheader()
for doc in data:
writer.writerow(doc)
return output.getvalue()
13 changes: 7 additions & 6 deletions echolalia/formatter/jsoner.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
import json


class Formatter:

def __init__(self):
return None
def __init__(self) -> None:
pass

def add_args(self, parser):
return parser
def add_args(self, parser):
return parser

def marshall(self, args, data):
return json.dumps(data)
def marshall(self, args, data) -> str:
return json.dumps(data)
12 changes: 6 additions & 6 deletions echolalia/formatter/rawer.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
class Formatter:

def __init__(self):
return None
def __init__(self) -> None:
pass

def add_args(self, parser):
return parser
def add_args(self, parser):
return parser

def marshall(self, args, data):
return data
def marshall(self, args, data):
return data
Loading
Loading