Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions appinfo/routes.php
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@
['name' => 'schemas#patch', 'url' => '/api/schemas/{id}', 'verb' => 'PATCH', 'requirements' => ['id' => '[^/]+']],
['name' => 'sources#patch', 'url' => '/api/sources/{id}', 'verb' => 'PATCH', 'requirements' => ['id' => '[^/]+']],

// Curated MDI glyph as an SVG image (used to render a schema's icon in unified search).
['name' => 'icon#mdi', 'url' => '/api/icon/mdi/{name}', 'verb' => 'GET', 'requirements' => ['name' => '[A-Za-z0-9-]+']],

// Data sync / harvesting — manual trigger + status (data-sync-harvesting spec).
['name' => 'sources#syncNow', 'url' => '/api/sources/{id}/sync', 'verb' => 'POST', 'requirements' => ['id' => '[^/]+']],
['name' => 'sources#syncStatus', 'url' => '/api/sources/{id}/sync-status', 'verb' => 'GET', 'requirements' => ['id' => '[^/]+']],
Expand Down
12 changes: 10 additions & 2 deletions docs/features/search-and-faceting.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,19 @@

## Overview

OpenRegister provides a comprehensive, backend-agnostic search and filtering system for register objects. The system supports full-text search with relevance ranking, field-level filtering with comparison operators, faceted drill-down navigation, multi-field sorting, and cursor/offset pagination. A single unified API surface (`ObjectService.searchObjectsPaginated()`) operates transparently against PostgreSQL, Apache Solr, or Elasticsearch.
OpenRegister provides a comprehensive, backend-agnostic search and filtering system for register objects. The system supports full-text search with relevance ranking, field-level filtering with comparison operators, faceted drill-down navigation, multi-field sorting, and cursor/offset pagination. A single unified API surface (`ObjectService.searchObjectsPaginated()`) operates against the object magic tables in the database (PostgreSQL / MariaDB). Apache Solr and Elasticsearch are deprecated as search backends and are not used by unified search.

**Tender demand**: 78% of analyzed government tenders require advanced search and filtering capabilities.

## Nextcloud Unified (Top-Bar) Search

OpenRegister objects participate in Nextcloud's unified (top-bar) search through one fleet-wide provider (`lib/Search/ObjectsProvider.php`, id `openregister_objects`). When a user searches from the magnifier, the provider asks for every **searchable** schema (the `searchable` flag on the schema) with no register filter, and `MagicMapper` resolves each schema to its real owning register and queries the per-(register, schema) magic tables directly.

- **Magic tables only.** Unified search reads the magic tables; it does **not** use a secondary/denormalised index. Apache **Solr and Elasticsearch are deprecated for unified search** — the cross-schema top-bar path never touches them. (The legacy external `search-index` capability was removed in a separate change.)
- **Cross-schema is not register-scoped.** A cross-schema query (a `@self.schema` array / `_schemas`) is never narrowed to the ambient "current register"; each searched schema is paired with its own register so objects in *every* register surface, not just one.
- **Scale.** The multi-schema union projects only constant metadata columns and scopes each arm's `@self.schema` to its own schema id, so a fleet with 1000+ searchable schemas stays under the database's target-list (1664-column) and `IN`-list (1000-element) limits.
- **Security.** Results still respect RBAC, tenant isolation (active organisation), the `searchable` flag, and the published predicate; a schema with no resolvable register or missing magic table is skipped and logged, never fatal.

## Full-Text Search

Triggered via the `_search` query parameter:
Expand All @@ -15,7 +24,6 @@ Triggered via the `_search` query parameter:
- Case-insensitive matching via `ILIKE` in the database backend
- String properties with `format: date`, `format: date-time`, or `format: time` are excluded from text search
- PostgreSQL `pg_trgm` extension enables fuzzy matching when installed
- Solr and Elasticsearch backends use their native query parsers

```
GET /api/objects/meldingen-register/meldingen?_search=geluidsoverlast
Expand Down
92 changes: 92 additions & 0 deletions lib/Controller/IconController.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
<?php

/**
* Icon controller.
*
* Serves curated Material Design Icon glyphs as standalone SVG images so they
* can be referenced by a real, same-origin URL — used by the unified search
* provider to render a schema's icon (Nextcloud search renders a thumbnail only
* from a URL, not from a data: URI or a bare icon-class name).
*
* @category Controller
* @package OCA\OpenRegister\Controller
*
* @author Conduction Development Team <info@conduction.nl>
* @copyright 2026 Conduction B.V.
* @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12
*
* SPDX-FileCopyrightText: 2026 Conduction B.V. <info@conduction.nl>
* SPDX-License-Identifier: EUPL-1.2
*
* @link https://www.OpenRegister.app
*/

declare(strict_types=1);

namespace OCA\OpenRegister\Controller;

use OCA\OpenRegister\Service\MdiIconRenderer;
use OCP\AppFramework\Controller;
use OCP\AppFramework\Http;
use OCP\AppFramework\Http\Attribute\NoAdminRequired;
use OCP\AppFramework\Http\Attribute\NoCSRFRequired;
use OCP\AppFramework\Http\Attribute\PublicPage;
use OCP\AppFramework\Http\DataDisplayResponse;
use OCP\IRequest;

/**
* Renders curated MDI glyphs as SVG images.
*/
class IconController extends Controller
{
/**
* Constructor for the IconController.
*
* @param string $appName The name of the app
* @param IRequest $request The HTTP request object
*
* @return void
*/
public function __construct(string $appName, IRequest $request)
{
parent::__construct(appName: $appName, request: $request);

}//end __construct()

/**
* Serve a curated Material Design Icon as an SVG image.
*
* Public, cacheable, and read-only: it returns nothing but static glyph
* geometry from a curated allow-list, so it is safe without authentication.
* Unknown icon names return 404 so the caller falls back to its own icon.
*
* @param string $name The MDI icon reference (e.g. "Dog", "mdi-dog").
*
* @return DataDisplayResponse The SVG image, or a 404 for an unknown icon.
*
* @spec openspec/changes/unified-search-index/specs/unified-search-provider/spec.md
*/
#[PublicPage]
#[NoCSRFRequired]
public function mdi(string $name): DataDisplayResponse
{
$svg = MdiIconRenderer::svg(icon: $name);
if ($svg === null) {
return new DataDisplayResponse(
data: '',
statusCode: Http::STATUS_NOT_FOUND
);
}

$response = new DataDisplayResponse(
data: $svg,
statusCode: Http::STATUS_OK,
headers: ['Content-Type' => 'image/svg+xml']
);
// Glyph geometry is immutable for a given name — cache hard.
$response->cacheFor(86400, false, true);

return $response;

}//end mdi()
}//end class
154 changes: 111 additions & 43 deletions lib/Db/MagicMapper.php
Original file line number Diff line number Diff line change
Expand Up @@ -1398,6 +1398,7 @@ private function searchAcrossMultipleTablesWithUnion(array $query, array $regist
* @param Schema $schema Schema entity.
* @param Register $register Register entity.
* @param array $allPropertyColumns Superset of all property columns across schemas.
* @param bool $metadataOnly Project only metadata columns (no property columns) to keep wide unions under the target-list limit.
*
* @return string|null SQL SELECT statement or null if table doesn't exist.
*
Expand Down Expand Up @@ -8333,6 +8334,8 @@ public function getMaxAllowedPacketSize(): int
* @SuppressWarnings(PHPMD.CyclomaticComplexity)
* @SuppressWarnings(PHPMD.NPathComplexity)
* @SuppressWarnings(PHPMD.ExcessiveMethodLength)
*
* @spec openspec/changes/unified-search-index/specs/unified-search-provider/spec.md
*/
public function searchObjectsPaginated(
array $searchQuery=[],
Expand Down Expand Up @@ -8367,7 +8370,6 @@ public function searchObjectsPaginated(
// register filter is present, registerIds is left empty and
// searchObjectsPaginatedMultiSchema resolves each schema's real owning
// register from a schema->register map.
// @spec openspec/changes/unified-search-index/specs/unified-search-provider/spec.md
$isMultiSchemaSearch = $schemaId === null
&& $schemaIds !== null
&& is_array($schemaIds) === true
Expand All @@ -8393,7 +8395,7 @@ public function searchObjectsPaginated(
ids: $ids,
uses: $uses
);
}
}//end if

// Single schema search.
if ($registerId !== null && $schemaId !== null) {
Expand Down Expand Up @@ -8512,6 +8514,42 @@ public function searchObjectsPaginated(
];
}//end searchObjectsPaginated()

/**
* Extract integer schema ids from a register's `schemas` membership list.
*
* The list may hold ids by value or by key, as ints or numeric strings;
* this normalises all forms to a flat list of distinct integer ids.
*
* @param array $registerSchemas The register's getSchemas()/decoded schemas array.
*
* @return int[] Distinct integer schema ids.
*/
private function extractSchemaIds(array $registerSchemas): array
{
// A plain list (`[4306, 4307]`) carries ids by VALUE; its integer keys
// are positional, not ids. An id-keyed map (`{4310: "Pet"}`) carries
// ids by KEY. Only consider keys for the map shape so a list of
// non-numeric values can never inject positional indices as schema ids.
$isList = array_is_list($registerSchemas);
$ids = [];
foreach ($registerSchemas as $schemaKey => $schemaValue) {
$candidates = [$schemaKey, $schemaValue];
if ($isList === true) {
$candidates = [$schemaValue];
}

foreach ($candidates as $candidate) {
if (is_int($candidate) === true
|| (is_string($candidate) === true && ctype_digit($candidate) === true)
) {
$ids[(int) $candidate] = true;
}
}
}

return array_keys($ids);
}//end extractSchemaIds()

/**
* Search objects across multiple schemas using UNION queries.
*
Expand All @@ -8533,6 +8571,8 @@ public function searchObjectsPaginated(
* @SuppressWarnings(PHPMD.ExcessiveMethodLength)
* @psalm-suppress UnusedParam
* Parameters reserved for future per-schema security filtering.
*
* @spec openspec/changes/unified-search-index/specs/unified-search-provider/spec.md
*/
private function searchObjectsPaginatedMultiSchema(
array $searchQuery,
Expand All @@ -8548,13 +8588,37 @@ private function searchObjectsPaginatedMultiSchema(
$registersCache = [];
$schemasCache = [];

$registers = [];
// Build a schema_id -> owning register_id map so each schema is paired
// with its REAL register (correct magic table). A schema with no owning
// register is SKIPPED (logged) rather than forced onto an unrelated
// register, which produced the "Register+schema table does not exist"
// empties. Register ENTITIES are loaded lazily (find()) only for the
// registers actually matched. `$registers` caches them by id.
//
// IMPORTANT: when no register filter is given (unified search passes a
// searchable-schema set only) we read the register->schema membership
// with a DIRECT query, NOT registerMapper::findAll — findAll applies an
// organisation filter (even with _multitenancy:false the trait's active-
// org resolution can collapse the result to a single register), which
// would hide most schemas' owning registers and make cross-schema
// search return nothing.
$registers = [];
$schemaToRegisterId = [];

if (empty($registerIds) === false) {
foreach ($registerIds as $regId) {
try {
$register = $this->registerMapper->find($regId, _multitenancy: false, _rbac: false);
$registers[$register->getId()] = $register;
$registersCache[$register->getId()] = $register->jsonSerialize();
$registerSchemas = ($register->getSchemas() ?? []);
if (is_array($registerSchemas) === true) {
foreach ($this->extractSchemaIds(registerSchemas: $registerSchemas) as $sid) {
if (isset($schemaToRegisterId[$sid]) === false) {
$schemaToRegisterId[$sid] = $register->getId();
}
}
}
} catch (\Exception $e) {
$this->logger->warning(
message: '[MagicMapper] Failed to find register for multi-schema search',
Expand All @@ -8563,24 +8627,34 @@ private function searchObjectsPaginatedMultiSchema(
}
}
} else {
// No register filter (e.g. unified search passes only a
// searchable-schema set): load every register so each schema can be
// paired with its REAL owning register via the schema->register map
// below — instead of guessing one and hitting a non-existent table.
try {
foreach ($this->registerMapper->findAll(_rbac: false, _multitenancy: false) as $register) {
$registers[$register->getId()] = $register;
$registersCache[$register->getId()] = $register->jsonSerialize();
$rqb = $this->db->getQueryBuilder();
$rqb->select('id', 'schemas')->from('openregister_registers');
$res = $rqb->executeQuery();
while (($row = $res->fetch()) !== false) {
$regId = (int) $row['id'];
$schemas = json_decode((string) ($row['schemas'] ?? '[]'), true);
if (is_array($schemas) === false) {
continue;
}

foreach ($this->extractSchemaIds(registerSchemas: $schemas) as $sid) {
if (isset($schemaToRegisterId[$sid]) === false) {
$schemaToRegisterId[$sid] = $regId;
}
}
}
} catch (\Exception $e) {

$res->closeCursor();
} catch (\Throwable $e) {
$this->logger->warning(
message: '[MagicMapper] Failed to load registers for schema-only multi-schema search',
message: '[MagicMapper] Failed to build schema->register map for multi-schema search',
context: ['file' => __FILE__, 'line' => __LINE__, 'error' => $e->getMessage()]
);
}
}//end try
}//end if

if (empty($registers) === true) {
if (empty($schemaToRegisterId) === true) {
return [
'results' => [],
'total' => 0,
Expand All @@ -8589,40 +8663,34 @@ private function searchObjectsPaginatedMultiSchema(
];
}

// Build a schema_id -> owning register map once (the register whose
// getSchemas() lists the schema id — by value or key, int or numeric
// string). Each schema is then paired with its REAL register so the
// correct magic table is targeted; a schema with no owning register is
// SKIPPED (logged) rather than forced onto an unrelated register, which
// is what produced the "Register+schema table does not exist" empties.
// @spec openspec/changes/unified-search-index/specs/unified-search-provider/spec.md
$schemaToRegister = [];
foreach ($registers as $register) {
$registerSchemas = $register->getSchemas();
if (is_array($registerSchemas) === false) {
continue;
}
$registerSchemaPairs = [];
$totalCount = 0;

foreach ($registerSchemas as $schemaKey => $schemaValue) {
foreach ([$schemaValue, $schemaKey] as $candidate) {
if (is_int($candidate) === true
|| (is_string($candidate) === true && ctype_digit($candidate) === true)
) {
$mappedId = (int) $candidate;
if (isset($schemaToRegister[$mappedId]) === false) {
$schemaToRegister[$mappedId] = $register;
}
foreach ($schemaIds as $sId) {
$schemaIdInt = (int) $sId;
$matchedRegisterId = ($schemaToRegisterId[$schemaIdInt] ?? null);
$matchedRegister = null;
if ($matchedRegisterId !== null) {
if (isset($registers[$matchedRegisterId]) === false) {
// Load the owning register ENTITY lazily (only for registers
// actually matched by a searched schema). find() honours
// _multitenancy:false so it resolves regardless of the
// active organisation; on failure the schema is skipped.
try {
$reg = $this->registerMapper->find($matchedRegisterId, _multitenancy: false, _rbac: false);
$registers[$reg->getId()] = $reg;
$registersCache[$reg->getId()] = $reg->jsonSerialize();
} catch (\Throwable $e) {
$this->logger->warning(
message: '[MagicMapper] Failed to load owning register for multi-schema search',
context: ['file' => __FILE__, 'line' => __LINE__, 'registerId' => $matchedRegisterId, 'error' => $e->getMessage()]
);
}
}
}
}

$registerSchemaPairs = [];
$totalCount = 0;
$matchedRegister = ($registers[$matchedRegisterId] ?? null);
}//end if

foreach ($schemaIds as $sId) {
$schemaIdInt = (int) $sId;
$matchedRegister = ($schemaToRegister[$schemaIdInt] ?? null);
if ($matchedRegister === null) {
// No owning register -> the magic table cannot be resolved; skip
// (logged) instead of guessing a wrong register.
Expand Down
Loading