INGEST_EVAL examples

A Splunk app containing worked examples of index-time data transformation using INGEST_EVAL, CLONE_SOURCETYPE, REGEX, ADD_META, REPEAT_MATCH, and related mechanisms. Each example ships with sample data, a commented props.conf / transforms.conf snippet, and a writeup explaining the problem, the approach, and how to search the result.

This is a reference / teaching app — not something you run standalone.

Installing

1. Copy this repository into your Splunk apps directory as ingest_eval_examples:

   cp -r ingest_eval_examples $SPLUNK_HOME/etc/apps/
   

   The directory name matters — inputs.conf uses absolute $SPLUNK_HOME/etc/apps/ingest_eval_examples/... paths to monitor the bundled sample data.

2. Restart Splunk:

   $SPLUNK_HOME/bin/splunk restart
   

3. Open the app from the Splunk nav ("INGEST_EVAL examples"). The dashboard lists each example; click through to read the description and view the underlying config.

The app creates three indexes on install (ingest_eval_examples_1, ingest_eval_examples_2, shard_data_with_splitbyindexkeys) plus a detritus last-chance index. See default/indexes.conf.

Examples

Grouped by category (from appserver/static/exampleInfo.json):

Datetime problems

• conflicting_datetime_formats — multiple datetime formats on a single sourcetype; worked around with INGEST_EVAL + strptime.
• compound_datetimes — date lives in the filename, time lives in the event; stitched together at ingest time.

Enrichment

• auto_extract_indexed_fields — discover attribute='value' pairs dynamically using REPEAT_MATCH=true + ADD_META=true.
• add_event_length — attach length(_raw) to every event as indexed metadata for back-billing license usage.
• drop_indexed_fields — blend INDEXED_CSV with search-time extraction to keep tstats useful without bloating storage.
• enrich_splunkd_component_level_thread — extract log_level, component, and thread as indexed fields on splunkd.log.
• enrich_splunkd_access_log — extract IP, wait time, URL, and REST parameters from splunkd_access.log.
• json_docker — pre-process JSON-formatted Docker logs for better search performance.
• find_ip_addresses — pull likely-looking IPs out of arbitrary text via REGEX into indexed fields.

Security

• mask_and_clone — simple double-indexing (one masked copy, one unmasked) for selective visibility.
• mask_data_and_map — SHA1-based masking with a separate "map" event for reversible decoding by privileged users.

Platform extension

• metricify_license — clone license_usage.log into _metrics via CLONE_SOURCETYPE.
• load_into_indexes — export/import tool for moving data between Splunk instances.
• shard_data_with_splitbyindexkeys — group like data into buckets to improve bucket elimination.
• data_estimation_pipeline — measure ingestion rates before onboarding new data.

Complex forwarding

• split_forwarding — break past the ~20 MB/s per-pipeline ceiling by spreading forwarder output across multiple TCP streams.
• selective_routing — fine-grained per-target routing using CLONE_SOURCETYPE.

Documentation

• Per-example writeups live in appserver/static/description/. Each covers the problem, the approach, the config, and a sample SPL search.
• extended_documentation.md covers INGEST_EVAL details that are absent from the public Splunk docs: _time vs $pd:_time$, multivalue handling ($mv:field$), := vs =, type coercion ([int], [float32-sf]), and guidance on high-cardinality string pitfalls.
• Sample data is generated by create_sample_data_sets.ipynb.

Layout

default/        app.conf, props.conf, transforms.conf, indexes.conf, inputs.conf, ...
local/          local overrides
metadata/       ACLs (local.meta)
appserver/
  static/
    description/    per-example markdown writeups
    sample_data/    sample log files used by the examples
    icons/          per-example icons shown in the dashboard
    exampleInfo.json  index of examples rendered by the in-app dashboard
create_sample_data_sets.ipynb   notebook that regenerates sample data

Author

rmorgan — app version 0.1.