138 datasource entity ef migration

[bolsson]

1. PR Description for 138-datasource-entity-ef-migration

---

Få oversikt over objekttyper i RavenDB #138 — DataSource entity and EF migration
Summary
As part of the RavenDB → SQL Server migration analysis for Artskart3, this PR introduces the DataSource entity which replaces the SourceDataBase document type from the old RavenDB store (Artskart2Core).
The design is based on a live data analysis of all 430 SourceDataBase documents across the four provider types found in production:
• 389 × GbifIpt
• 36 × ArtskartDataDeliveryWebServiceV1
• 4 × GbifApi
• 1 × Artsobservasjoner20Api
Changes
• ProviderType enum — clean replacement for RemoteDataSourceTypeEnum (dropped GbifJson which had zero live records)
• DataSource entity — flat table inheriting BaseEntity, with:
• Doi (string?) replacing GbifDataSetId — nullable in DB, enforced at the application layer for relevant provider types
• GbifApiQuery, HarvestPropertyOverrides, ImportPropertyOverrides — stored as nvarchar(max) (large JSON payloads)
• Property names cleaned up: RemoteAddress, ArchiveName, NonValidOccurrenceIds, WebServiceVersion
• GbifPublisherId kept as Guid? (nullable — only relevant for API-based providers)
• DataSourceConfiguration — EF fluent configuration with column constraints and indexes on ProviderType and IsDeleted
• EF migration AddDataSource — generates the DataSource table
• Fixed a pre-existing bug: OnModelCreatingPartial was declared but never called from OnModelCreating

Additional changes (extended scope)
Refactored import layer into Clean Architecture
The initial single Artskart3.Import project was split into two projects following the same Clean Architecture pattern as the rest of the solution:
• Artskart3.Import.Core — domain entities, enums, repository interfaces (no EF dependency)
• Artskart3.Import.Infrastructure — EF DbContext, migrations, repository implementations, DI wiring
GbifDatasetDiscovery entity
Replaces GbifDataSetInfo from RavenDB — a staging table for datasets discovered via the GBIF API. The NonRelevant bool has been replaced with a proper DiscoveryStatus enum (Pending, Approved, Rejected). A unique index on GbifId prevents duplicate discoveries.
Architecture decisions documented
• Three-database architecture agreed: Artskart3Import, Artskart3Cache, Artskart3Index
• Taxonomy sourced from Nortaxa API — not stored as owned data; taxon tree will be cached in Artskart3Index for UI performance
• README updated with database overview and EF migration commands for all contexts

Running locally
Ensure your Artskart3.Api/appsettings.json connection string points at your local SQL Server instance, then run:

Create/update the Artskart3Import database

dotnet ef database update --project Artskart3.Import.Infrastructure
--startup-project Artskart3.Api `
--context ArtskartImportDbContext

This will create the Artskart3Import database (if it doesn't exist) and apply both migrations:
• InitialImportBaseline — creates the DataSource table
• AddGbifDatasetDiscovery — creates the GbifDatasetDiscovery table

Out of scope / future work
• RawOccurrence entity in Artskart3Cache — SQL representation of a raw Darwin Core (DwC) occurrence record as received from providers (DwC-A archives, GBIF API). Largest migration item, warrants its own issue.
• Occurrence entity in Artskart3Index
• TaxonomyChangeLog (replacement for ArtskartEvent)
• Data migration tool from RavenDB → SQL

[Kristoffer-Martinsen]

Jeg lurer litt på om det er nødvendig å ha egne prosjekt for dette? Ser at undermappene i disse nye Artskart3.Import prosjektene er helt like de som er under Core og Infrastructure allerede 🤔

[bolsson]

Seperation of concerns. Det er helt eget prosjekt, men med samme undermapper ettersom at vi følger Clean Architecture prinsippene tenker jeg.

[Kristoffer-Martinsen]

Hva er det denne configuration fila gjør egentlig?