Support for Ingest Pipelines

[tduesterhus]

Hey there,

I had a look through the code and tried to find a way to add ingest pipelines to index-calls.
I did not find an easy way to do so. However, it would be a great feature. E.g. for adding file-attachments to a model.

Best,
Tobias

[Jeroen-G]

I don't know what they are, but if you can find a way to incorporate them I'll be happy to take a look at it.

[tduesterhus]

The ingest Pipelines are pre-defined post-processing steps that can be used to e.g. add attachments (https://www.elastic.co/guide/en/elasticsearch/reference/8.10/ingest.html). It is simply:

{
  "index": "some_index",
  "body": {...},
  "pipeline": "pipeline-id",
}

So maybe not a big deal to implement it so that one has a pipeline-id per model or so.
I'll try to find some time and let you know

[boss554]

Any update on this topic ? I am searching for same.

[tduesterhus]

Sorry, I am too busy at the moment and totally forgot about this.

[boss554]

Hey there,

I had a look through the code and tried to find a way to add ingest pipelines to index-calls. I did not find an easy way to do so. However, it would be a great feature. E.g. for adding file-attachments to a model.

Best, Tobias

same I want to do ,I have epub file when I search than search it on file also .

[drmmr763]

For anyone else who may want to do this, there is a way. Here's what I did:

Elastic Console Command to Create my ingestion pipelines in Laravel using the PHP Client Directly:

/** @var ElasticClientFactory $esFactory */
        $esFactory = \App::make(ElasticClientFactory::class);

        /** @var Client $client */
        $client = $esFactory->client();

        // candidate enrich policy
        $candidateEnrichmentPolicy = [
            'name' => 'candidate_scoring_enrich_policy',
            'body' => [
                'match' => [
                    'indices' => 'candidates_index',
                    'match_field' => 'id',
                    'enrich_fields' => ['*']
                ]
            ]
        ];


$client->enrich()->putPolicy($candidateEnrichmentPolicy);

$client->enrich()->executePolicy(['name' => 'listing_scoring_enrich_policy']);

// ingestion pipeline
// then set up the ingestion pipelines
        $scoringIngestionPipeline = [
            'id' => 'enhance-score-data',
            'body' => [
                'description' => 'Enriches the scoring index with candidate data',
                'processors' => [
                    [
                        'enrich' => [
                            'field' => 'candidate_id',
                            'policy_name' => 'candidate_scoring_enrich_policy',
                            'target_field' => 'candidate'
                        ],
                    ]
                ]
            ]
        ];

// create the pipeline
try {
            // see if the pipeline exists
            $client->ingest()->getPipeline(['id' => 'enhance-score-data']);
        } catch (Missing404Exception $e) {
            // if we got a 404 then it doesn't exist so create it
            $client->ingest()->putPipeline($scoringIngestionPipeline);
        }

Now here's where you have some choices. You can either set a pipeline up as the default pipeline for an index. Meaning any time a document is indexed, the pipeline runs. Or you can run out of band operations to execute the pipeline against your index.

To make my pipeline run by default I just created another command to update the index mapping accordingly:

// ensure the write index has the ingestion pipeline
        $client->indices()->putSettings([
            'index' => 'all_match_data_index-write',
            'body' => [
                'default_pipeline' => 'enhance-score-data'
            ]
        ]);

This ensures that all future writes to the index pass through my new pipeline. So, regular scout-instantiated document write commands work with this natively.