Skip to content

feat: implement API retry mechanism for Zoom data pipeline - #110

Open
PiyushSonkusare wants to merge 1 commit into
vicharanashala:mainfrom
PiyushSonkusare:feature/pipeline-retries
Open

feat: implement API retry mechanism for Zoom data pipeline#110
PiyushSonkusare wants to merge 1 commit into
vicharanashala:mainfrom
PiyushSonkusare:feature/pipeline-retries

Conversation

@PiyushSonkusare

Copy link
Copy Markdown

What does this PR do?

  • This PR improves the resilience of the backend data pipeline by introducing a reusable exponential backoff/retry utility for external API calls, specifically targeting the Zoom transcript ingestion scripts.

Specific Changes:

  1. Utility Creation: Added pipeline/retryUtils.js to handle asynchronous retries globally. It defaults to 3 max retries with a 2000ms delay between attempts.

  2. Pipeline Implementation: Wrapped the axios.post (OAuth token generation) and axios.get (Meeting summary fetch) inside pipeline/zoom-fetch-transcripts.js with the new withRetry wrapper.

Why is this needed?

  • External services like the Zoom API occasionally drop connections, timeout, or rate-limit requests. Previously, a single dropped packet would cause the cron job to fail entirely for that execution. This retry mechanism catches those transient network errors, waits, and tries again, drastically reducing false alarms and the need for manual developer intervention.

How it was tested:

  1. Created a local mock function to intentionally throw connection errors.

  2. Verified that the wrapper successfully catches the error, waits the specified delay, and increments the retry count before ultimately failing on the final attempt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant