Skip to content

Fix : misp-push event loss and timezone synchronization failures#89

Open
prince-shakyaa wants to merge 2 commits into
OWASP:masterfrom
prince-shakyaa:fix-misp-push-sync
Open

Fix : misp-push event loss and timezone synchronization failures#89
prince-shakyaa wants to merge 2 commits into
OWASP:masterfrom
prince-shakyaa:fix-misp-push-sync

Conversation

@prince-shakyaa

Copy link
Copy Markdown

Fix misp-push event loss and timezone synchronization failures

Fixes #88

What does this PR do?

This PR addresses several critical logic flaws in misp-push.py that caused attack logs to be silently dropped or duplicated, depending on the volume of events and the host system's timezone.

Changes Made:

  • Timezone Synchronization: Migrated all timestamps to use datetime.now(timezone.utc) instead of the naive local datetime.now(). This ensures the .isoformat() timestamps passed to Elasticsearch are accurately interpreted as UTC, fixing the bug where the query window shifted forward/backward relative to the local offset.
  • Pagination and Data Loss: explicitly configured the Elasticsearch size parameter to 10000 (up from the default of 10 hits). This prevents the pipeline from dropping events if more than 10 attacks arrive within the 10-second polling interval.
  • Polling Window Race Condition: Refactored the lastSearched update logic so that the now timestamp used to bound the lt condition in the Elasticsearch query is preserved and assigned to lastSearched. Previously, updating lastSearched after the blocking sendEvents step caused any events indexed during that network request to be lost.
  • Index Resolution: Simplified the set_index function to apply the "honeypot-attacks-*" wildcard directly in the query. The previous logic looped over all indices and only used the name of the final matching index, effectively ignoring all other indices if daily rolling indices were present.

How to test

  1. Deploy the misp-push docker container on a host configured to a non-UTC timezone (e.g., America/New_York or Asia/Kolkata).
  2. Trigger an automated attack burst against the honeypot (using e.g. Nikto or a custom script) generating > 15 logs in a 10-second window.
  3. Verify that MISP receives all logged events and no logs are skipped.
  4. Verify that duplicate events are not ingested.

@prince-shakyaa prince-shakyaa changed the title Fix misp-push event loss and timezone synchronization failures Fix : misp-push event loss and timezone synchronization failures Jun 5, 2026
@prince-shakyaa

Copy link
Copy Markdown
Author

Hi @fzipi @adrianwinckles,
This PR resolves the event loss issues (#88) by enforcing strict UTC boundaries and increasing the Elasticsearch query size limit so bursts of attacks aren't silently dropped. I also noticed the local Docker container was failing to build due to deprecated dependencies, so I bumped elasticsearch[async] and fixed the Pipfile environment too.

I've tested it locally and everything is polling correctly now.

Happy to adjust anything if needed.
Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Critical event loss and timezone synchronization failures in misp-push.py

1 participant