siem: p1 integrations esql fixes and ingest script improvements

[JordanSh]

Summary

• Fix dataset names in azure_openai and azure_ai_foundry evaluations (were matching azure.open_ai / azure.ai_foundry; actual datasets are azure_openai.logs / azure_ai_foundry.logs)
• Add m365_defender.alert.title as event.action for the alert stream, overriding ingest-time detectionStatus with the human-readable alert name
• Fix m365_defender host.target conditions to require user.name to avoid spurious host-as-target matches on non-user events
• Extend m365_defender user.name fallback to the event stream via process.user.name
• Ingest script: handle nanosecond Unix timestamps — fixes sysdig.alerts and sysdig.event streams which were silently producing 0 docs
• Ingest script: expand dotted config keys to nested dicts — fixes salesforce pipeline routing
• Ingest script: inject data_stream.dataset when absent from fixtures — fixes Azure, ForgeRock and others invisible in Kibana Discover
• Ingest script: rewrite YAML parser to handle nested mappings — fixes microsoft_dhcp routing config
• Ingest script: inject log.file.path from fixture config — fixes cisco_umbrella routing

Test plan

• Verify azure_openai and azure_ai_foundry graph nodes appear with correct dataset matching
• Verify m365_defender.alert nodes show alert title as action label
• Verify sysdig.alerts and sysdig.event ingest successfully after nanosecond timestamp fix
• Verify salesforce pipeline routing produces actor fields correctly

🤖 Generated with Claude Code

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[github-actions]

TL;DR

Buildkite failed before uploading the pipeline because the PR branch no longer merges cleanly into siem/p1-integrations-mappings. Resolve the content conflict in dev/domain/install-p1-integrations/install_and_ingest.py, then rerun CI.

Remediation

• Rebase or merge the latest siem/p1-integrations-mappings into yarden/siem-p1-integrations-mappings and resolve dev/domain/install-p1-integrations/install_and_ingest.py.
• Preserve both sides' ingest-script behavior where needed: the target branch's config/agent metadata helpers and the PR's fixture parsing, timestamp, dataset, sample-event, and pipeline selection fixes.
• Validate by running the relevant ingest-script checks locally, then rerun the Buildkite build.

Investigation details

Root Cause

The failed step is :pipeline::arrow_up: Upload Pipeline: .buildkite/pipeline.yml, but the pipeline upload never ran successfully because the repository post-checkout hook attempted to create the PR merge branch and hit a Git content conflict:

• dev/domain/install-p1-integrations/install_and_ingest.py is modified by the PR commit bab1019eb3bbceaed1c2d108b7f123801960a6b8.
• The target branch commit 467c7093e6c12981ddcdfb3d623c4dadfd14ce7f also changed the same file in overlapping areas.
• Buildkite reports CONFLICT (content) for that file and exits the post-checkout hook with status 1.

The overlapping areas include helper/config handling near dev/domain/install-p1-integrations/install_and_ingest.py:298 on the PR head and dev/domain/install-p1-integrations/install_and_ingest.py:332 on the target branch version. The target branch has fixture_config_paths, apply_pipeline_config_fields, apply_fixture_configs, and apply_agent_metadata, while the PR introduces _parse_simple_yaml, _fixture_config_fields, sample-event discovery, nanosecond timestamp handling, data stream injection, and sample-event pipeline bypass in the same script.

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/45394
• Job/step: :pipeline::arrow_up: Upload Pipeline: .buildkite/pipeline.yml
• Key log excerpt:

Auto-merging dev/domain/install-p1-integrations/install_and_ingest.py
CONFLICT (content): Merge conflict in dev/domain/install-p1-integrations/install_and_ingest.py
Automatic merge failed; fix conflicts and then commit the result.
Merge failed: 1
Error: running "repository post-checkout" shell hook: The repository post-checkout hook exited with status 1

Verification

Not run: CI cannot reach tests until the merge conflict is resolved.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[infra-vault-gh-plugin-prod]

💔 Build Failed

• Buildkite Build
• Commit: 8e1c0dd

Failed CI Steps

• :pipeline:⬆️ Upload Pipeline: .buildkite/pipeline.yml

History

• 💔 Build #45394 failed bab1019

[teresaromero]

@JordanSh can you provide more context on this change? is this a new feature? thanks

cc @mrodm