feat/fix: Real-Time Encrypted Chat Engine, Redis Streams Event Bus, and Chatbot Backend#149
Merged
Merged
Conversation
The 60s TimeoutInterceptor killed the gRPC context mid-transaction when MB batch compute exceeded 60s, causing "context deadline exceeded" on the job completion update. Add a longRunningMethods map that gives batch compute and bulk import RPCs a 5-minute deadline instead of the default. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chat System: - Domain layer: entities, value objects, repository interfaces (AES-256-GCM) - Infrastructure: PostgreSQL repos, Redis pub/sub broadcaster, presence service - Application: 16 handlers (conversations + messaging + typing + streaming) - Delivery: gRPC handlers for ChatService + PresenceService - Migrations 000070-000076: 6 tables + menu seeds/permissions - main.go wired with conditional CHAT_MASTER_KEY activation Chatbot: - chatbot_audit_log repository for DeepSeek usage tracking Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a chat message is sent, check if each recipient is online via Redis presence. If offline, create a TypeChat notification which triggers the existing email dispatch pipeline. Debounced per conversation/user pair (5-minute TTL) to prevent email spam. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Chat menu: level-2 with NULL parent → level-1 root (constraint requires non-root menus to have a parent) - mst_permission: use actual columns (permission_name, service_name, module_name, action_type) instead of plan's assumed 'name' column - role_permissions: include required id, assigned_by columns - Permission code: iam.chatbot.assistant.use → iam.chatbot.assistant.view (action_type CHECK constraint doesn't include 'use') Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…eptor All 16 ChatService RPCs and 2 PresenceService RPCs were unmapped in methodPermissions, causing the stream/unary permission interceptor to deny them with "unmapped method denied". Added as authenticated-only (empty permission) since participant-level authorization is enforced at the handler level. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ents StreamChatEvents was sending empty proto responses (only EventId, no payload). Now parses the broadcaster JSON payload and maps to the proper oneof variant (MessageReceived, MessageEdited, Typing, etc.) so the frontend SSE bridge receives typed event data. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…restarts) - Add chat.master_key to config.yaml with a dev-only default key - Add ChatConfig struct to config loader - Bind CHAT_MASTER_KEY env var for staging/production override - main.go reads from cfg.Chat.MasterKey instead of os.Getenv Local dev: key persists in config.yaml, no need to export env var. K8s: CHAT_MASTER_KEY env var from goapps-chat-secret overrides config. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…n chat streaming Replace JSON-based chat broadcaster with typed proto events: - Event.Payload []byte → Event.Response *iamv1.StreamChatEventsResponse - All broadcast call sites build proto directly (no json.Marshal) - StreamChatEvents handler: zero-copy stream.Send(evt.Response) - Redis cross-pod serialization: proto.Marshal/Unmarshal (binary) - Delete 60 lines of JSON mapping code (broadcastPayload, buildStreamResponse) Before: 5 serialization hops per message (JSON marshal → Redis → unmarshal → proto map → gRPC) After: 2 hops (proto binary to Redis, ts-proto JSON at BFF SSE layer) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Redis Streams (replaces Pub/Sub):
- XADD with MAXLEN ~500 per user stream (durable, trimmed)
- XREAD BLOCK 5s for cross-pod event bridging
- Proto binary serialization via base64 encoding in stream values
- Events persist across pod restarts — replayable on reconnect
User info resolver for conversation participants:
- ChatUserResolver batch-resolves userIDs to {username, fullName, avatarUrl}
- convToProto now populates ParticipantProto.Username/FullName/AvatarUrl
- Fixes "Unknown User" / "UN" initials in conversation list
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…rors ListMessages: if conversation key decryption fails (master key changed between sessions), return empty list instead of 500 error. Logs the mismatch as a warning. mapChatError: log the actual error before returning "internal error" so backend logs show the root cause. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CreateHandler only persists + SSE-broadcasts notifications — it never sends email. Added EmailDispatcher to SendMessageHandler and call Dispatch() in a background goroutine after creating the offline notification. Wired notifEmailDispatcher from main.go. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- broadcastMessageEvent: accept senderName param, set MessageProto.SenderName
- resolveSenderName helper: shared between send + edit handlers
- SendMessage: resolve sender name before broadcast + offline notification
- EditMessage: resolve original sender name for edit broadcast
- Offline email: title="{name} sent you a message", body="{name}: {text}",
CTA label="Reply" pointing to /chat
- WithOfflineNotification: accept ChatUserResolver for name resolution
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ail CTA Bug A: Read receipts in ListMessages — batch-load receipts for all messages in one query (no N+1), attach to each message before return. Bug B: LastMessage + UnreadCount in conversation list — new repo methods GetUnreadCounts + GetLastMessages (batched), ListConversationsHandler returns ConversationSummary with decrypted last message body + count. Frontend conversation list now shows persistent previews from API. Bug C: GetMessageEditHistory — new application handler + gRPC method. Verifies caller is participant, decrypts old message bodies from chat_message_edit_history table. Bug D: Email CTA — ActionPayload key fixed from "url" to "path" to match NotificationDispatcher.buildCTAURL expected contract. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
msgToProto never set SenderName — messages fetched via ListMessages had empty sender names. SSE-pushed messages had names (via broadcastMessageEvent) but refetch would overwrite them with empty. Batch-resolve sender names via ChatUserResolver for the message page, same pattern as convToProto participant name resolution. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implement the backend for the three remaining chat features on top of the
regenerated proto bindings.
Clear history (per-user):
- migration 000077 adds chat_participant.history_cleared_at.
- Participant.HistoryClearedAt + ClearHistory repo method; ListByConversation
and MarkRead now honor the per-user cleared-history boundary.
- ClearHistoryHandler + ClearConversationHistory gRPC method.
Attachments:
- migration 000078 adds the chat_attachment table (message_id nullable until
the message is sent).
- domain Attachment aggregate + AttachmentRepository, postgres repository, and
MinIO storage.UploadChatAttachment ({basePath}/chat/attachments/{conv}/...).
- UploadAttachmentHandler (participant check, nil-storage guard, image
thumbnail); SendMessage links pre-uploaded attachments and ListMessages
batch-loads them; real-time broadcast carries attachments.
Both features are registered as authenticated-only in the permission
interceptor and wired in cmd/server/main.go.
Authored-By: Ilham R <ilhamram332@gmail.com>
Co-Authored-By: Ilham R <me@ilramdhan.dev>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: IT Mutugading <it@mutugading.com>
Co-Authored-By: Indra Putro <indraputro@mutugading.com>
Co-Authored-By: Ilham R <ilham.ramadhan@mutugading.com>
Co-Authored-By: IT Mutu Gading <mutugadingapps@gmail.com>
…ount GetLastMessages and GetUnreadCounts ignored chat_participant.history_cleared_at, so after a user cleared a conversation the list preview still showed the old last message and its unread badge, even though the opened thread was correctly empty. - GetLastMessages now takes viewerID and joins the viewer's participant row, excluding messages created at or before their history_cleared_at. - GetUnreadCounts applies the same history_cleared_at boundary. Authored-By: Ilham R <ilhamram332@gmail.com> Co-Authored-By: Ilham R <me@ilramdhan.dev> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: IT Mutugading <it@mutugading.com> Co-Authored-By: Indra Putro <indraputro@mutugading.com> Co-Authored-By: Ilham R <ilham.ramadhan@mutugading.com> Co-Authored-By: IT Mutu Gading <mutugadingapps@gmail.com>
ilramdhan
added
bug
CI lint (v2.3.0) caught issues the local v1.62 binary missed. Fix all 15: - errcheck: wrap defer rows.Close() in logging closures (conversation, message, user-resolver repos); nolint best-effort ResolveUsers name enrichment; nolint comma-ok map type assertions in the broadcaster. - errorlint: use errors.Is(err, redis.Nil) instead of == in the broadcaster. - gocognit/nestif: extract initChatServices from run(), and split the chat message page query into queryMessagePage/queryFirstPage/queryCursorPage. - nilnil: ResolveUsers returns an empty map (not nil,nil) on empty input. - revive package-comments: add package docs to application/chat and infrastructure/crypto. Verified locally with golangci-lint v2.3.0: 0 issues. Build, vet, and chat tests all pass. Authored-By: Ilham R <ilhamram332@gmail.com> Co-Authored-By: Ilham R <me@ilramdhan.dev> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: IT Mutugading <it@mutugading.com> Co-Authored-By: Indra Putro <indraputro@mutugading.com> Co-Authored-By: Ilham R <ilham.ramadhan@mutugading.com> Co-Authored-By: IT Mutu Gading <mutugadingapps@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This massive PR introduces the complete backend architecture for a highly scalable, real-time Chat & AI Chatbot System within the IAM Service.
It features end-to-end AES-256-GCM encryption for messages, file/image attachments via MinIO, and a production-grade event bus powered by Redis Streams (replacing standard Pub/Sub) using zero-copy Protobuf binary serialization. Additionally, it implements intelligent offline email notifications, read receipts, message edit history, and per-user conversation clear-history logic. A critical timeout extension for long-running Finance batch RPCs is also included.
Type of Change
Service(s) Affected
Changes Made
💬 Real-Time Chat Engine & Chatbot (IAM)
chatbot_audit_logrepository for tracking usage of the DeepSeek assistant.UploadAttachmentHandler) and a per-userClearHistoryfeature that perfectly syncs with unread badges and conversation list previews (Migrations 000077-000078).ChatUserResolverto automatically map participant names and avatars, eliminating "Unknown User" UI issues.🚀 Redis Streams & Performance Refactoring (IAM)
XADD/XREAD BLOCK) to ensure messages persist across pod restarts and can be replayed on reconnect.✉️ Offline Presence & Notifications (IAM)
TypeChatnotification and dispatches an email background job (debounced with a 5-minute TTL to prevent spam).🐛 Bug Fixes & Chores
TimeoutInterceptordeadline from 60s to 5 minutes specifically for heavy batch computes (TriggerMbBatch) and bulk imports to prevent "context deadline exceeded" rollbacks.CHAT_MASTER_KEYis now safely loaded via Viper config. If decryption fails due to a rotated/mismatched master key, the API gracefully returns an empty list instead of throwing a 500 error.ChatServiceandPresenceServiceproperly in the IAM Auth Interceptor and fixed schema hierarchies in the Chat menu migrations (000070-000076).Related Issues
Fixes #
Related to #
API Changes (if applicable)
Proto Changes
Breaking Changes
None.
Testing Performed
Unit Tests
Integration Tests
Manual Testing
Lint & Build
golangci-lint run ./...passesgo build ./...succeedsgo test -race ./...passesDatabase (if applicable)
Documentation
Rollback Plan
iam-serviceandfinance-service.000078down to000070) to drop chat tables and remove the seeded menus.CHAT_MASTER_KEYenvironment variable remains properly mapped if rolled back to a previous testing state.Screenshots/Logs (if applicable)
Pre-merge Checklist
Reviewer Notes
XADDcommand usesMAXLEN ~500to automatically trim the stream and prevent Redis memory bloat. This is intentional as the stream is only meant for real-time delivery and short-term reconnect replays, while long-term history is fetched via standard HTTP queries from Postgres.CHAT_MASTER_KEYis securely injected viagoapps-chat-secret. The application will fall back to a dev-key if running locally viaconfig.yaml, but production must use the secret to encrypt messages properly.