adding workflow apis for dashboard , added short bill api for dashboard and updation done in aggregated service by goutam-niua · Pull Request #1818 · niua-org/UPYOG-NIUA

goutam-niua · 2026-08-14T10:40:24Z

No description provided.

Signed-off-by: goutam-niua <goutam@niua.org>

…rd and updation done in aggregated service

Keshavmca12 · 2026-08-14T11:12:55Z

Blockers

Local dev config committed in application-dev.yaml
Debug scaffolding committed (test_client.go, test_server.go, test_url.go)
Dashboard _count vs _search use different query semantics
Status filters use pi.action while callers send application-status strings
Breaking aggregation contract (requestInfo required) without OpenAPI/client update
Provider response shape changes (recent-applications, due-renewals)
Draft search JSON key changed from DraftSearchCriteria to draftSearchCriteria
Major

Short bill API loads full bills then maps in controller
Hardcoded status lists in Go (incl. typo ASSING_DSO)
Double DB round-trip on dashboard search
Missing tests for new Java endpoints
Dead code (fetchWorkflowCount)
See inline comments for details.

Inline comments (copy per file/line on GitHub)
File: municipal-services/upyog-aggregation-service/configs/application-dev.yaml · Line 18

Blocker — revert before merge

redis.host changed from redis-headless.dev to localhost. This is a local-dev override and should not ship in shared config. Revert or move to an untracked local profile.

File: municipal-services/upyog-aggregation-service/configs/application-dev.yaml · Line 30

Blocker — revert before merge

auth.enabled: false disables authentication for the aggregation service in dev config. Please revert — this must not land in the repo default.

File: municipal-services/upyog-aggregation-service/test_client.go · Line 1

Blocker — remove before merge

This looks like local debug scaffolding (package main hitting localhost:9999). Same for test_server.go and test_url.go. Please delete these files from the PR.

File: core-services/egov-workflow-v2/src/main/java/org/egov/wf/repository/querybuilder/WorkflowQueryBuilder.java · Line 603

Blocker — count/search semantics mismatch

_count uses getDashboardProcessInstanceSearchQueryWithoutPagination() (flat join query, no latest-per-businessId dedup).

_search uses getDashboardProcessInstanceIds() (WITH clause + latest instance per business).

For a citizen dashboard, totalCount and the returned list can disagree. Both endpoints should share the same base filter logic.

File: core-services/egov-workflow-v2/src/main/java/org/egov/wf/repository/querybuilder/WorkflowQueryBuilder.java · Line 624

Blocker — verify status field

Dashboard queries filter on pi.action / pi_outer.action, but existing workflow search uses pi_outer.status (state UUID). Aggregation sends human-readable values like APPLIED, APPROVED, which may map to state.applicationStatus, not pi.action.

Please validate against real data. If needed, join eg_wf_state_v2 and filter on applicationStatus instead.

File: core-services/egov-workflow-v2/src/main/java/org/egov/wf/web/controllers/WorkflowController.java · Line 164

Major — redundant DB work

Search already returns totalCount in the response contract, but this endpoint runs a separate count query after fetching the list (3 DB round-trips total: IDs → full rows → count).

Consider returning count from the search path or reusing the count query result.

File: business-services/billing-service/src/main/java/org/egov/demand/web/controller/BillControllerv2.java · Line 77

Major — inefficient short search

short/_search calls billService.searchBill() (full bill graph from DB) and strips fields in the controller. For a dashboard endpoint, consider a repository-level projection query that selects only the six short fields.

File: municipal-services/upyog-aggregation-service/internal/dto/request.go · Line 10

Blocker — breaking API contract

requestInfo is now required on every aggregate call. Existing mobile clients that omit it will get 400.

Please update:

resources/upyog-aggregation-service-contract.yml
mobile client in the same release
validator/integration tests (validator_test.go, stack_test.go, etc.)
File: municipal-services/upyog-aggregation-service/internal/providers/draft_applications.go · Line 121

Blocker — possible JSON key regression

Changed from DraftSearchCriteria (PascalCase) to draftSearchCriteria. If the draft service expects the original key, search will silently fail or return empty results. Confirm against the draft service contract.

File: municipal-services/upyog-aggregation-service/internal/providers/due_renewals.go · Line 66

Blocker — product semantics change

Due renewals previously called TL/BPAREG search for licences nearing expiry. This now returns active bills from billing short search.

That is a different domain concept still exposed as due-renewals. Confirm with product/mobile that this is intended, and update provider docs accordingly.

File: municipal-services/upyog-aggregation-service/internal/providers/recent_applications.go · Line 126

Blocker — breaking provider response shape

Previously returned []Application. Now returns { ProcessInstances, totalCount }.

Confirm the mobile app handles this new shape, or map to the existing consumer contract before returning.

File: municipal-services/upyog-aggregation-service/internal/providers/recent_applications.go · Line 86

Major — typo in status list

ASSING_DSO looks like a typo for ASSIGN_DSO. Hardcoded status lists will also drift from workflow MDMS — consider centralizing or sourcing from config/MDMS.

File: municipal-services/upyog-aggregation-service/internal/providers/quick_summary.go · Line 152

Major — count via full search payload

Pending payments count is derived from len(searchResult.Bill) after calling short search. This works only if billing search returns all matches (no pagination cap). Prefer a dedicated count endpoint or confirm bill search has no implicit limit for mobile-number queries.

File: municipal-services/upyog-aggregation-service/internal/providers/quick_summary.go · Line 255

Minor — dead code?

fetchWorkflowCount appears unused after switching to fetchApplicationCount / fetchCompletedServicesCount. Please remove if not needed.

…w Fixes

goutam-niua · 2026-08-17T06:47:44Z

Blocker Comments

Re: redis.host and baseUrls in application-dev.yaml:
Done. Reverted redis.host to redis.backbone. I've also reverted the local baseUrls of the backend services to prevent shipping local port overrides in the shared dev config profile.

Re: auth.enabled: false in application-dev.yaml:
As aligned with the team, authentication is handled at the gateway layer in our environment. Hence, keeping auth.enabled: false in this aggregation service config is intended.

Re: Scaffolding files (test_client.go, test_server.go, test_url.go):
Done. Deleted all three debugging helper files from the repository.

Re: _count vs _search mismatch in WorkflowQueryBuilder.java:
Done. Refactored the query builder to unify query generation. Both count and search queries now utilize the same getDashboardProcessInstanceIds base query logic (incorporating latest-per-businessId deduplication), ensuring matching results.

Re: Action vs Status filter on dashboard queries:
We are keeping filtering on pi.action here as callers send action strings inside the status filter list. This is the intended behavior for this endpoint.

Re: requestInfo marked mandatory (AggregateRequest):
Yes, requestInfo is mandatory for all aggregation endpoints. I have updated the OpenAPI contract (upyog-aggregation-service-contract.yml) and adjusted integration tests to reflect this change.

Re: draftSearchCriteria JSON key regression:
Done. Restored draftSearchCriteria (camelCase) to align with the downstream draft service contract.

Re: Product Semantics Change for Due Renewals:
Yes, this is an intentional update. The mobile application now requests active bills instead of licences nearing expiry, so switching to billing short search is the correct approach.

Re: recent-applications Response Shape Change:
Done. Mapped the dashboard search response back to a flat array ([]recentAppProcessInstance) before returning, maintaining backward compatibility with the consumer app contract.

Major/Minor Comments

Re: Redundant DB queries in dashboard search (3 round-trips):
Done. Optimized the query execution path by fetching all matching IDs in a single round-trip (its length represents totalCount), slicing the list in Java memory for paging, and executing details retrieval only for the sliced page. This reduces DB round-trips down to 1 or 2.

Re: Inefficient Short Bill Search:
Done. Implemented a database projection query (BILL_SHORT_BASE_QUERY) that selects only the required 6 columns without joining the account details table, optimizing short bill searches.

Re: Hardcoded status lists (including typo ASSING_DSO):
Done. Centralized and moved all hardcoded workflow status lists in Go to the profile configuration YAMLs.

Re: Count via Full Search Payload (Pending Payments):
Since the billing service query builder does not apply pagination limits on search queries, /short/_search always returns all bills matching the mobile number. Calculating the count via slice length is correct and performant under our new projection optimization.

Re: Missing tests for new Java endpoints:
Done. Added unit tests for the dashboard count and search endpoints in WorkflowControllerTest.java, and for the short bill search endpoint in BillControllerv2Test.java. All tests pass successfully.

Re: Unused code fetchWorkflowCount:
Done. Removed the unused fetchWorkflowCount method.

Signed-off-by: goutam-niua <goutam@niua.org>

…ing , PwC PR feedback , SFTP implementation

…tor changes for SFTP & PR , removing reconcilliation service

Keshav-NIUA · 2026-09-01T11:12:59Z

+		@Override
+		public List<ShortBillV2> extractData(ResultSet rs) throws SQLException {
+			Map<String, ShortBillV2> billMap = new java.util.LinkedHashMap<>();
+			while (rs.next()) {


move this to separate row mapper

Keshav-NIUA · 2026-09-01T11:13:37Z

+
+/**
+ * Compact API response envelope wrapping the search results from the
+ * `/bill/v2/short/_search` endpoint.


add comment why we have added it on controller and to

Keshav-NIUA · 2026-09-01T11:14:28Z

+	 */
+	@PostMapping("short/_search")
+	@ResponseBody
+	public ResponseEntity<?> shortSearch(@RequestBody @Valid final RequestInfoWrapper requestInfoWrapper,


change the endpoint and function name

Keshav-NIUA · 2026-09-01T11:15:18Z

+	@PostMapping("short/_search")
+	@ResponseBody
+	public ResponseEntity<?> shortSearch(@RequestBody @Valid final RequestInfoWrapper requestInfoWrapper,
+			@ModelAttribute @Valid final BillSearchCriteria billCriteria) {


why we have used modelattribute here why not request body

this is happenning in search API so the same thing is used in this as well

Keshav-NIUA · 2026-09-01T11:18:12Z

+
    }
+
+    private IngestionResult processViaS3(DashboardPayload payload, String moduleName) {


add comment why have done this

Keshav-NIUA · 2026-09-01T11:19:27Z

+     * @param tenantId state tenant ID
+     * @return IngestionResult response from downstream ingestion engine
+     */
+    public IngestionResult uploadLegacyExcelFile(File excelFile, String moduleName, String tenantId) {


remove legacy and rename upload file

Keshav-NIUA · 2026-09-01T11:20:39Z

+        }
+
+        try {
+            String key = properties.getAwsS3Folder() + "/" + tenantId + "/" + moduleName + "/" + UUID.randomUUID().toString() + "_" + file.getName();


move this to util that anyone can use

Keshav-NIUA · 2026-09-01T11:21:53Z

     * @see org.upyog.dashboard.entity.LegacyIngestionData
     */
-    public static final String UPDATE_LEGACY_INGESTION_DETAIL = "update-adapter-module-ingestion-detail";
+    public static final String UPDATE_LEGACY_INGESTION_DETAIL = "update-dashboard-data-module-ingestion-detail";


remove this topics will come from props only

Keshav-NIUA · 2026-09-01T11:23:45Z

               "module_ingestion_id, tenant_id, module_name, push_date, request_data, " +
               "response_data, ingestion_status, exception_code, created_by, created_time, last_modified_by, last_modified_time) " +
-               "VALUES (?, ?, ?, TO_DATE(?, 'DD-MM-YYYY'), ?::jsonb, ?::jsonb, ?, ?, ?, ?, ?, ?)";
+               "VALUES (:moduleIngestionId, :tenantId, :moduleName, TO_DATE(:pushDate, 'DD-MM-YYYY'), :requestData::jsonb, :responseData::jsonb, :ingestionStatus, :exceptionCode, :createdBy, :createdTime, :lastModifiedBy, :lastModifiedTime)";


move date format in util

Signed-off-by: goutam-niua <goutam@niua.org>

goutam-niua added 5 commits August 12, 2026 16:15

Update state metric in application-local.properties

ceb9c28

Signed-off-by: goutam-niua <goutam@niua.org>

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

7d1c578

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

95cc505

Update state metric in application-local.properties

60f9951

Signed-off-by: goutam-niua <goutam@niua.org>

adding workflow apis for dashboard , added short bill api for dashboa…

067bc42

…rd and updation done in aggregated service

goutam-niua added 3 commits August 17, 2026 12:12

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

d17938d

Aggregation Service Optimizations, Workflow Alignment, and Code Revie…

16edc84

…w Fixes

Update application-dev.yaml

060358d

goutam-niua added 14 commits August 17, 2026 12:43

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

672bffc

added comments in all new files

9c30e0c

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

65f1f4c

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

366ba9a

handling multiple ULBs in data extraction

01c6631

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

b939206

handling multi-tenant hirerachy edge-cases handling

4686fce

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

c41a6bf

changes for upyog aggregated service

56f787b

Updated the structure for Finance KPIs on dashboard

678fe77

Signed-off-by: goutam-niua <goutam@niua.org>

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

b3cc51c

changes covered for extractor service regarding reflection api , lock…

eee08b7

…ing , PwC PR feedback , SFTP implementation

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

c898e8e

draft services changes in column , aggregate service changes , extrac…

6d3e867

…tor changes for SFTP & PR , removing reconcilliation service

Keshav-NIUA reviewed Sep 1, 2026

View reviewed changes

goutam-niua added 4 commits September 2, 2026 11:03

Merge branch 'niua-org:niua-dev-2.0' into niua-dev-2.0

5ac450b

feedback resolved

856631a

removed kafka topics

7d13496

Revert MasterDashboardConfig.json

35b6f34

Signed-off-by: goutam-niua <goutam@niua.org>

Keshav-NIUA merged commit 0d86651 into niua-org:niua-dev-2.0 Sep 2, 2026
0 of 2 checks passed


		}

		private IngestionResult processViaS3(DashboardPayload payload, String moduleName) {

Conversation

goutam-niua commented Aug 14, 2026

Uh oh!

Keshavmca12 commented Aug 14, 2026

Uh oh!

goutam-niua commented Aug 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants