Migrate from Newtonsoft to JsonSchema.net, other performance, memory improvementw#327
Open
spartacus-2789 wants to merge 15 commits into
Conversation
* feat: add tests for HSDS profile fallback and version extraction in OpenApiValidationService * chore: clean up .gitignore by removing specific test coverage files * feat: enhance test-validation workflow to upload additional API log and response artifacts * feat: add file sink to Serilog configuration for enhanced logging * feat: update Serilog configuration for improved logging format and file management * feat: update test-validation workflow to log validation results and enhance artifact uploads * feat: add artifact name sanitization for API log uploads * feat: add environment variables for Node.js version and step debugging in test-validation workflow * feat: update actions versions in test-validation workflow for improved stability * feat: update HSDS validation mode to SpecAndFeedRuntimeFast for improved performance * feat: implement fallback mechanism for HSDS profile spec in FullHsdsRuntime mode to avoid redundant validation * feat: add specification validation mapping and tests for OpenReferralUKValidationResponse * chore: Bump the microsoft-packages group with 1 update Bumps System.IdentityModel.Tokens.Jwt from 8.16.0 to 8.17.0 --- updated-dependencies: - dependency-name: System.IdentityModel.Tokens.Jwt dependency-version: 8.17.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: microsoft-packages ... Signed-off-by: dependabot[bot] <support@github.com> * feat(docs): Add comprehensive guides for validation options and deployment processes - Introduced `ValidationOptionsExplained.md` detailing the structure and usage of validation request options for OpenApiValidationRequest payloads. - Created `ValidatorDeploymentGuide.md` outlining the setup and deployment of the OpenReferralUK Validator, including MongoDB Atlas and Heroku configurations. - Documented branching and release workflows, CI/CD processes with GitHub Actions, and post-deployment verification steps. refactor(docs): Document phase-wise refactorings and changesets - Added `phase-1-changeset-refactorings.md` capturing non-test refactorings in controllers and services. - Introduced `phase-2-changeset-helper-tests.md` detailing unit tests for newly created helper classes. - Created `phase-3-refactor-changes.md` summarizing model consolidation and architectural improvements. - Added `phase-4-refactor-changes.md` documenting the introduction of polymorphic authentication strategies and API response documentation enhancements. * Refactor code structure for improved readability and maintainability * Refactor code structure for improved readability and maintainability * Revert "Refactor code structure for improved readability and maintainability" This reverts commit 2d565ed. * Revert "Refactor code structure for improved readability and maintainability" This reverts commit 3ef95b6. * feat(docs): Update configuration and validation documentation for OpenAPI settings * feat(config): Enable specification validation in OpenApiValidation settings * feat(validation): Add warnings for misplaced HSDS version in OpenAPI spec validation * feat(validation): Reintroduce specification validation in OpenApiValidationOptions and update related service logic * feat(validation): Enhance HSDS compliance checks with additional field reporting options * feat(validation): Refactor OpenApiValidationService to use consolidated options for validation and caching * Initial plan * Move TestEndpoints, TestOptionalEndpoints, TreatOptionalEndpointsAsWarnings to OpenApiValidationServerOptions Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/85014426-ab63-461b-a90f-8e7d97565872 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * All changes verified - 369 tests pass, 0 CodeQL alerts Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/85014426-ab63-461b-a90f-8e7d97565872 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * chore(deps): update Microsoft.IdentityModel packages to version 8.17.0 * fix(validation): comment out misplaced HSDS version warning notification * feat(validation): enhance specification validation error handling and assertions in tests * feat(validation): update validation error handling to distinguish additional fields * feat(validation): streamline validation error handling and improve service registrations * Initial plan * feat: add OwnSchemaValidation server setting to OpenApiValidation When true (default), endpoint responses are validated against the feed's own discovered openApi.json. When false, the HSDS profile openApi.json is used instead, with a fallback to the feed spec when no HSDS profile is resolvable. FullHsdsRuntime second pass is skipped when OwnSchemaValidation is false to avoid duplicate validation. Includes 4 new unit tests. Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/424baca2-cdcf-4a73-898b-a71d8ad20ec7 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * fix(validation): disable OwnSchemaValidation in OpenApiValidation settings * feat(validation): enhance OpenAPI discovery paths and improve logging in services and tests * feat(logging): change log level from Information to Debug for probing paths in OpenApiDiscoveryService and ProfileDiscoveryService * feat(validation): add notification for OpenAPI schema URL discovery failure in validation service * feat(validation): implement default HSDS profile fallback for OpenAPI schema discovery * feat(validation): enhance profile resolution logic in OpenApiValidationService and add corresponding tests * feat(validation): refactor own-schema validation settings and update documentation * feat(https): conditionally enable HTTPS redirection based on configuration settings * feat(validation): update default profile version and enhance own schema validation settings * feat(validation): refactor OpenAPI discovery methods and enhance profile version extraction logic * feat(validation): refactor specification options handling and update related services * feat(validation): streamline profile version handling and remove HSDS-UK prefix * feat(validation): update profile version extraction to return raw values and streamline normalization logic * feat(validation): enhance HSDS version extraction and add warning for misplaced HSDS version * feat(validation): enhance endpoint status enums and improve error response handling * feat(validation): add example values to OpenApiValidationRequest properties and configure OpenAPI version in appsettings * feat(validation): add examples for validation requests and responses in OpenAPI documentation * feat(validation): add response examples for feed validation endpoints in OpenAPI documentation * feat(validation): add mock response examples for API endpoints in OpenAPI documentation * feat(validation): implement Swagger documentation setup and examples for API validation endpoints * feat(validation): add operation examples for feed validation and mock responses in OpenAPI documentation * feat(validation): add unit tests for Swagger examples application in OpenReferral validation flow * feat(validation): add endpoint quality recommendations for incomplete operation metadata in OpenAPI validation * feat(telemetry): refactor OpenTelemetry configuration into a dedicated extension method * Refactor code structure for improved readability and maintainability * feat(validation): update Swagger configuration to OpenAPI 3.1 specifications * update dependancy version to latest available * feat(validation): implement ValidateAndUpdateFeedsAsync method for concurrent feed validation and update refactor: remove overall health check endpoint and update documentation * feat(validation): add ValidateAndUpdateFeedAsync method and update controller to use it * feat(validation): add ValidateAndUpdateFeedAsync test for MongoDB pipeline validation and persistence * feat(validation): update appsettings.json for UK profile version * feat(validation): update API endpoints in test-validation workflow to only tyest current live feeds * delete openapi.json as created for formatting * feat(validation): streamline validation request options in test-validation workflow * feat(validation): enhance API run command and validation request with additional flags * feat(validation): add output step for API logs in validation workflow * feat(validation): update validation workflow to log response details and adjust artifact uploads * feat(validation): normalize array indexes in validation error messages and paths * feat(validation): refactor validation error normalization and add unit tests * feat(sanitization): implement TextSanitizer for exception message and logging sanitization * Refactor OpenAPI schema handling: remove OpenApiSchema class, replace with OwnSchemaUrl in validation requests - Deleted OpenApiSchema class and its references. - Updated OpenApiValidationRequest to use OwnSchemaUrl instead of OpenApiSchema.Url. - Modified OpenApiValidationService to handle schema URL fetching and authentication based on OwnSchemaUrl. - Adjusted tests to reflect changes in schema handling and validation logic. * feat(validation): disable reporting of additional fields in validation options * feat(gitignore): add AGENTS.md to ignored files * feat(validation): replace JsonConvert with System.Text.Json for JSON parsing and serialization * feat(editorconfig): add .editorconfig for consistent coding style and rules * Refactor FeedValidationController and MockController for improved async handling and logging - Updated FeedValidationController to use ConfigureAwait(false) for async calls to improve performance. - Refactored MockController methods to include ConfigureAwait(false) for better async handling. - Enhanced logging in FeedValidationController for manual validation triggers and results. - Cleaned up unnecessary comments and improved code readability. Refactor OpenApiController and OpenReferralUkController for async handling - Added ConfigureAwait(false) to async calls in OpenApiController and OpenReferralUkController to enhance performance. Change OpenTelemetryServiceExtensions and SwaggerDocumentationExtensions to internal - Changed the accessibility of OpenTelemetryServiceExtensions and SwaggerDocumentationExtensions classes from public to internal for better encapsulation. Update CorrelationIdMiddleware and GlobalExceptionHandler for async handling - Added ConfigureAwait(false) to async calls in CorrelationIdMiddleware and GlobalExceptionHandler to improve performance. Refactor FeedValidationBackgroundService for better async handling - Improved async handling in FeedValidationBackgroundService by adding ConfigureAwait(false) to async calls. - Enhanced logging and error handling during feed validation runs. Refactor SchemaWarmupBackgroundService for async handling - Added ConfigureAwait(false) to async calls in SchemaWarmupBackgroundService for better performance. Update Instrumentation class to internal - Changed the accessibility of Instrumentation class from public to internal for better encapsulation. Refactor ApiErrorResponse and Program for consistency - Added missing newline at the end of ApiErrorResponse and Program files for consistency. * fix: suppress unused task warnings by discarding results of async calls * fix: improve string replacement in Swagger documentation for consistent behavior * fix: refactor RemoteSchemaLoader to use IHttpClientFactory for HTTP client creation * fix: enable nullable reference types for improved null-safety in RemoteSchemaLoader and SchemaResolverService * fix: refactor RemoteSchemaLoader and tests to use IHttpClientFactory for improved HTTP client management * feat: add StartupLogger for structured logging of OpenApiValidation settings * feat: implement structured logging for authentication failures in AuthenticationValidationLog * feat: enhance logging in FeedValidationBackgroundService with structured log messages * feat: implement structured logging for schema warmup process * feat: implement structured logging for mock file operations in MockController * Initial plan * Add source generator logging partial classes for all services and controllers - Create OpenApiBootstrapLog, ProfileDiscoveryLog, FeedValidationServiceLog - Create OpenApiValidationLog, ReferenceResolverLog, RemoteSchemaLoaderLog - Create RequestProcessingLog, SchemaResolverLog, EndpointTestingLog - Create JsonValidatorLog, OpenApiDiscoveryLog, OpenApiSpecFetcherLog - Create OpenApiSpecificationLog, PathParsingLog - Create web-layer logs: FeedValidationControllerLog, OpenApiControllerLog, OpenReferralUkControllerLog, GlobalExceptionHandlerLog, FeedValidationBackgroundLog - Replace all inline _logger.LogXxx calls with generated extension methods - Fix ambiguous extension method calls using fully-qualified class names - Fix NullReferenceException when baseUrl is null in PathParsingService Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * Fix missing return null and rename safeSpecUrl2 in OpenApiSpecFetcher Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * Add source generator logging partial classes and migrate inline log calls - Created 14 Core logging classes (EventId 5000-18013) in OpenReferralApi.Core/Logging/ - Created 5 web-layer logging classes (EventId 19000-23008) in OpenReferralApi/Logging/ - Updated all corresponding services/controllers to use extension methods - Covers: EndpointTesting, JsonValidator, OpenApiBootstrap, OpenApiDiscovery, OpenApiSpecFetcher, OpenApiSpecification, PathParsing, ProfileDiscovery, ReferenceResolver, RemoteSchemaLoader, RequestProcessing, SchemaResolver, FeedValidationService, OpenApiValidation, and web-layer controllers/middleware Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * correct build wanrings * feat: refactor logging for OpenApiValidation settings and migrate StartupLogger * feat: add InternalsVisibleTo attribute for testing and refactor logging calls * feat: refactor SchemaWarmupBackgroundService and SchemaWarmupExecutor for improved cancellation handling and logging * test: enable ExecuteAsync_WhenCacheDisabled_SetsSkippedStatus test and refactor execution call * test: add ExecuteWithRetryAsync setup in JsonValidatorServiceTests and enable assertions in ValidateAsync test * docs: enhance OpenApiValidation settings documentation with detailed descriptions and options * Update README with OpenAPI validation controls section * feat: enhance logging in OpenApiController and OpenReferralUkController to check log level before logging requests * feat: add InternalControllerFeatureProvider and SwaggerDocumentEndpointTests for enhanced controller discovery and validation * feat: enhance JSON serialization options in JsonValidatorService to handle cycles and nulls * feat: add logging for JSON validation errors related to circular references and depth limits * feat: enhance JSON validation logging and error reporting with source identifiers * feat: implement circular reference detection and logging in schema resolution * feat: update schema handling in EndpointTestingService to maintain component references and improve validation logic * feat: change access modifier of PrimaryTestResult to internal and add JsonIgnore attribute * feat: remove circular reference notification logic from OpenApiValidationService and update related tests * feat: update OpenApiValidation settings for stricter schema validation and enable user-supplied auth * feat: update OwnSchemaValidation setting to allow additional properties in OpenApiValidation * Add default configuration for AWS Lambda in aws-lambda-tools-defaults.json * feat: add AWS Lambda hosting configuration to Program.cs * chore: initial plan for memory optimisation Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/d43ee326-b475-43a7-b7d1-a1dd5e9588b2 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * fix: optimise memory usage across validator services Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/d43ee326-b475-43a7-b7d1-a1dd5e9588b2 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * fix: snapshot ConcurrentDictionary keys before purging to avoid enumeration race Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/d43ee326-b475-43a7-b7d1-a1dd5e9588b2 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * Refactor code structure for improved readability and maintainability * feat: increase AWS Lambda memory size and timeout settings * feat: update OpenApiValidationOptions to disable response body inclusion by default * feat: implement response body retention cap and log memory usage during OpenAPI validation * feat: add memory checkpoint logging for OpenAPI validation and endpoint testing processes * fix: remove unnecessary JsonNode.Parse for response body in validation requests * feat: enhance schema validation by implementing compiled schema caching in endpoint testing * feat: add JSON parsing and pagination extraction to endpoint testing service * feat: refactor response validation and pagination extraction logic in EndpointTestingService * feat: integrate parsed response JSON caching into endpoint testing service * feat: add MaxValidationErrorsPerResponse option to OpenApiValidationServerOptions and update validation services * feat: add linux-x64 target to packages.lock.json for net10.0 * feat: ignore API exploration for FeedValidation, Mock, and OpenReferralUk controllers * feat: add linux-x64 target to packages.lock.json for net10.0 * feat: enhance README with AWS Lambda deployment instructions and configuration details * Add lscache files for OpenReferralApi and OpenReferralApi.Tests projects to improve C# Dev Kit performance * Initial plan * Fix memory leaks in validation flow Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/ea102f30-192c-4a91-898b-5100421930f8 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * Fix validation flow memory leaks: dispose HTTP responses, release JsonDocuments eagerly, clear retained response bodies Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/ea102f30-192c-4a91-898b-5100421930f8 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * fix: update AWS Lambda deployment region from eu-west-1 to eu-west-2 * Refactor code structure for improved readability and maintainability * chore: update OpenTelemetry packages to version 1.15.2 across the project * fix: remove empty linux-x64 target from packages.lock.json * fix: optimize response handling in EndpointTestingService to reduce memory allocations * refactor: remove outdated pagination extraction logic from EndpointTestingService * fix: optimize JSON validation by reusing parsed text to reduce redundant processing * fix: enhance JSON parsing efficiency by utilizing rented buffers and stream reading * fix: improve schema caching in EndpointTestingService and enhance JSON data handling in JsonValidatorService * feat: add logging for OpenAPI cache state and memory usage during validation * fix: add missing linux-x64 target in packages.lock.json * fix: ensure proper disposal of HTTP responses in JsonValidatorService, OpenApiDiscoveryService, and ProfileDiscoveryService * fix: optimize JSON stream handling for URLs and schemas in Program.cs * fix: enhance schema caching by using SHA-256 hash for cache keys in EndpointTestingService * feat: add logging for compiled endpoint schema cache state in EndpointTestingService * fix: add missing linux-x64 target in packages.lock.json * fix: remove empty linux-x64 target from packages.lock.json * feat: add endpoint testing retention snapshot logging in EndpointTestingService * feat: enhance OpenAPI validation logging with additional GC memory metrics * fix: remove empty linux-x64 target from packages.lock.json feat: add reportAdditionalFields option to validation request payload * feat: optimize JSON validation by converting JsonDocument to JToken via UTF-8 stream * feat: add JsonSchema.Net package for enhanced JSON validation capabilities * feat: add performance-validation workflow for sequential load testing Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/5cb8f23f-562a-40f3-8a0a-b13474218021 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * fix: restrict GITHUB_TOKEN permissions in performance-validation workflow Agent-Logs-Url: https://github.com/Binary-Computer-Solutions-Limited/oruk-validator/sessions/5cb8f23f-562a-40f3-8a0a-b13474218021 Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> * Add push trigger for performance validation workflow * Update performance-validation.yml * feat: consolidate HSDS profile version detection and discovery service Co-authored-by: Copilot <copilot@github.com> * Refactor Profile Discovery Service and Tests - Removed ProfileDiscoveryServiceTests and ProfleDiscoveryServiceTests files. - Introduced FastDiscoveryService with corresponding tests. - Updated Program.cs to register FastDiscoveryService instead of ProfileDiscoveryService. - Modified appsettings.json to streamline HSDS URLs and adjust OpenApiValidation settings. - Created LegacyDiscoveryContracts for interface definitions related to profile and OpenAPI discovery. Co-authored-by: Copilot <copilot@github.com> * Refactor OpenApiValidationServiceTests to use IOpenApiBootstrapService and remove unused mocks - Replaced IProfileDiscoveryService and IOpenApiDiscoveryService with IOpenApiBootstrapService in OpenApiValidationServiceTests. - Updated test setups to use the new service and adjusted method calls accordingly. - Removed unnecessary mock setups for services that are no longer used. - Ensured all tests still pass after refactoring. Enhance TextSanitizer with additional sanitization methods - Added TextSanitizer class to handle various sanitization tasks for logging. - Implemented methods to sanitize exception messages, URLs, and general strings for safe logging. - Updated OpenApiController and OpenReferralUkController to utilize the new TextSanitizer for logging sanitized URLs. Introduce JsonElementExtensions for easier JSON path handling - Created JsonElementExtensions class with TryGetPathString method to simplify accessing nested JSON properties. Add Constants class for reusable string constants and regex patterns - Introduced Constants class to store common paths and regex patterns used throughout the application. Refactor SchemaWarmupExecutor to use TextSanitizer for logging - Updated SchemaWarmupExecutor to sanitize URLs before logging to prevent log forging. Remove unused FastDiscoveryService registration from Program.cs - Cleaned up service registrations in Program.cs by removing FastDiscoveryService, which is no longer needed. * Update OpenTelemetry packages and add ProfileDiscoveryService - Upgraded OpenTelemetry packages to versions 1.15.3 for Console, OpenTelemetryProtocol, and Extensions.Hosting, and 1.15.2 for AspNetCore instrumentation. - Introduced ProfileDiscoveryService to handle OpenAPI discovery from a base URL. - Updated Program.cs to register IProfileDiscoveryService. - Added unit tests for ProfileDiscoveryService to ensure correct functionality and error handling. Co-authored-by: Copilot <copilot@github.com> * feat: enhance OpenAPI validation and discovery flow with improved schema content handling Co-authored-by: Copilot <copilot@github.com> * Add ProfileResolverService and integrate into OpenApiValidationService - Introduced IProfileResolverService interface and ProfileResolverService class to handle profile resolution logic. - Updated OpenApiValidationService to utilize ProfileResolverService for resolving profile states. - Registered ProfileResolverService in the dependency injection container within Program.cs. - Modified OpenApiValidationService constructor to accept an optional IProfileResolverService parameter. - Updated project file to include ProfileResolverService in the build. Co-authored-by: Copilot <copilot@github.com> * feat: enhance service interfaces and implementations with detailed XML documentation for better clarity and usability Co-authored-by: Copilot <copilot@github.com> * feat: enhance OpenApiValidationService and ProfileDiscoveryService to support cached HSDS profile schema content retrieval * feat: implement memory checkpoint logging and metrics in validation services * feat: add lscache files to .gitignore for improved build cleanliness * Update package references to latest versions - Upgraded JsonSchema.Net from 9.1.4 to 9.2.0 - Upgraded Microsoft.AspNetCore.OpenApi from 10.0.5 to 10.0.7 - Upgraded MongoDB.Driver from 3.7.1 to 3.8.0 - Upgraded OpenTelemetry packages to 1.15.3 - Upgraded System.IdentityModel.Tokens.Jwt from 8.17.0 to 8.18.0 - Upgraded YamlDotNet from 16.3.0 to 17.1.0 * feat: consolidate OpenApiValidationService and ProfileDiscoveryService with a new base class for shared caching logic * feat: enhance profile discovery flow by consolidating schema retrieval and adding default profile handling * feat: update ProfileDiscoveryService to support nullable ownSchemaUrl and enhance probe path discovery * feat: update OpenApiValidationService and related tests to consolidate profile discovery and improve schema resolution handling * feat: remove unused IProfileResolverService and streamline profile resolution logic in OpenApiValidationService * feat: refactor OpenApiValidationService to use OpenApiSchemaContent and streamline profile state handling * Update package references and versions in OpenReferralApi project - Upgraded Amazon.Lambda.AspNetCoreServer.Hosting from 1.10.0 to 2.0.0 - Upgraded Microsoft.AspNetCore.OpenApi from 10.0.7 to 10.0.8 - Upgraded MongoDB.Driver from 3.8.0 to 3.8.1 - Added SharpCompress package version 1.0.0 - Updated dependencies for Amazon.Lambda packages to their latest versions - Updated package lock file to reflect the changes in package versions * feat: consolidate memory checkpoint logging and enhance payload structure for unified logging across services * feat: enhance profile discovery flow with improved error handling and schema resolution * feat: remove ProfileReason from OpenApiValidationRequest and update validation logic to handle profile discovery internally * feat: remove unused profileReason parameter and related logic from ProfileDiscoveryService * feat: consolidate profile discovery flow by removing profileReason parameter and enhancing OpenAPI spec handling * feat: streamline OpenApiValidationService by removing profileReason handling and enhancing profile version extraction * feat: refactor OpenApiValidationService to consolidate profile discovery flow and remove ProfileResolutionState * feat: remove TryGetDefaultProfileSchemaFallback method to streamline OpenApiValidationService * feat: update OpenApiValidationService to enhance profile discovery flow and handle HSDS version warnings * feat: enhance OpenApiValidationService with additional validation steps for endpoint testing and HSDS runtime validation * feat: add logging for profile discovery resolution and enhance discovery reason handling * feat: consolidate profile discovery flow by renaming variables and updating schema handling * feat: update OpenApiValidationService to use HsdsProfileSchema for improved schema handling * feat: refine error handling in OpenApiValidationService for schema discovery and HSDS profile version * feat: consolidate endpoint testing parameters by removing documentUri and updating schema handling * Refactor SchemaResolverService to use Json.Schema instead of Newtonsoft.Json.Schema - Updated ISchemaResolverService interface and its implementation in SchemaResolverService to replace JSchema with JsonSchema. - Modified methods for creating schemas from JSON strings to utilize Json.Schema library. - Adjusted related unit tests to accommodate changes in schema handling and validation. - Replaced JObject with JsonNode in various test cases to ensure compatibility with the new schema handling approach. * feat: update OpenApiSpecFetcher and OpenApiValidationService to use JsonObject for improved JSON handling * feat: consolidate JSON handling by replacing JObject with JsonObject across services and tests * feat: replace Newtonsoft.Json with System.Text.Json for improved performance and consistency in EndpointInfo and EndpointTestingService * Refactor JSON serialization to use System.Text.Json - Replaced Newtonsoft.Json with System.Text.Json in multiple model classes for improved performance and reduced dependencies. - Updated property attributes from [JsonProperty] to [JsonPropertyName] to align with the new serialization library. - Adjusted unit tests to accommodate changes in JSON serialization methods and ensure compatibility with System.Text.Json. * feat: update comments for clarity on schema validation processes in EndpointTestingService and OpenApiSpecificationService * feat: remove Newtonsoft.Json and related schema references for improved performance * feat: update documentation to reflect JSON Schema validation changes and improvements in Development Setup * feat: consolidate schema handling in JsonValidatorService and OpenApiValidationService for improved clarity and performance * Add .NET 10.0.300 user level cache and telemetry storage files - Created sentinel files for ASP.NET certificate, first use, and tool path in both .dotnet-cli-home and .dotnet directories. - Added user level cache files for Docker container status and machine ID. - Introduced telemetry storage service files with compressed JSON stream data for tracking. * feat: add JsonSchema.Net.OpenApi package and update related references for schema validation improvements * feat: integrate JsonSchema.Net.OpenApi for enhanced schema validation and performance improvements * Fix HSDS profile validation pipeline after JsonSchema.Net migration * Fix 20 OpenApiValidationService test failures: service fetches spec, schema auth, fallback mutation, exception message, HSDS_SCHEMA_VERSION_MISPLACED, revert bootstrap mock * Refactor code structure for improved readability and maintainability * Enhance services and tests with various improvements - Refactored PathParsingService to use a centralized list of allowed ports. - Updated ProfileDiscoveryService to handle unsupported profiles and fallback to default configurations. - Improved ProfileVersionNormalizer by removing regex and implementing a more efficient version extraction method. - Enhanced RemoteSchemaLoader to register schemas upon retrieval. - Optimized ValidationPathNormalizer to reduce memory allocations during array index normalization. - Modified tests across multiple services to ensure compatibility with recent changes, including adjustments to response body handling and validation logic. - Added memory leak test in OpenReferralUKValidationResponseMapperTests to ensure efficient memory usage during repeated mappings. - Improved FeedValidationBackgroundService to handle cancellation tokens more effectively during scheduled validations. * Extract root exception message for spec fetch failures - Uses `GetBaseException()` in `OpenApiValidationService.cs` to capture the underlying cause of an error. - Surfaces more accurate failure notifications for users when an OpenAPI spec cannot be retrieved or resolved. * refactor: enhance JSON schema reference resolution and caching for improved validation of nested properties and allOf constraints * refactor: update JsonValidatorService and tests to return failed field values * refactor: simplify SchemaWarmupBackgroundService and update EndpointTestingService tests * feat: add option to retain response JSON for dependency testing and fix path parameter substitution logic * feat: implement profile version mapping to support canonical HSDS version resolution * refactor: wrap service logging calls with level-specific checks and add GEMINI system configuration file * refactor: migrate logging to high-performance LoggerMessage and update schema resolution logic * feat: inject authentication headers into profile discovery requests * refactor: simplify method signatures, optimize collection initializations, and document local development guidelines in OpenApiValidationService * refactor: modernize collection initializers to use target-typed expressions across core services and tests * chore: add system configuration docs and update TextSanitizer tests to use NUnit multiple assertion scopes * docs: add GEMINI.md configuration and update architecture documentation with AI interaction guidelines * refactor: wrap assertions in Assert.EnterMultipleScope for comprehensive test failure reporting * refactor: standardize test assertions with MultipleScope and propagate cancellation tokens in tests * refactor: simplify OpenAPI validation configuration enums and naming conventions * refactor(OpenReferralApi.Core): update exception handling and error messages * feat: add explicit failure notifications to OpenApiValidationService and update tests * feat(OpenReferralApi.Core): add explicit profile support in validation - Added `Profile` property to `OpenApiValidationRequest` for overriding or bypassing automatic profile discovery. - Updated `OpenApiValidationService` to handle explicit profiles and skip discovery if provided. - Implemented new methods in `ProfileDiscoveryService` to retrieve explicit profiles. - Added unit tests to validate the behavior of explicit profile handling. * refactor(OpenReferralApi.Core): improve remote schema loading and connection resiliency - Added URL normalization for "json-everything.lib" to "json-everything.net" - Enhanced error handling in `RemoteSchemaLoader` to log warnings on connection failures and return null - Updated `SchemaResolverService` to handle "json-everything.net" schemas locally - Added unit tests for connection resiliency scenarios * refactor(JsonSchemaBuild): register meta-schemas and optimize schema loading - Register standard JSON Schema meta-schemas globally. - Optimize `JsonValidatorService` to check for pre-registered schemas before fetching remotely. - Ensure thread safety during concurrent schema registration in `RemoteSchemaLoader`. * chore: remove unused files and configurations * docs: remove outdated GEMINI.md documentation * chore: remove .DS_Store file * docs: remove all technical architecture and configuration documentation * chore: remove outdated code quality instructions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: jeffcumpsty-tpx <jeff.cumpsty@tpximpact.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: spartacus-2789 <198984858+spartacus-2789@users.noreply.github.com> Co-authored-by: Copilot <copilot@github.com>
…ading and validation - Updated `JsonValidatorService` to use asynchronous methods for detecting additional fields. - Added new `RemoteSchemaLoader` class with synchronous method `LoadRemoteSchema` for fetching and caching remote JSON schemas. - Modified `SchemaResolverService` to use the synchronous `LoadRemoteSchema` method and cache schema references.
…on improvements - Added `ProfileUri` and `RecordId` properties with JSON property names. - Refactored metadata profile resolution and updated validation result properties. - Added methods to extract record IDs from JSON locations and map them in validation responses. - Updated related logic in `ProfileDiscoveryService.cs`. - Added tests for validating OpenAPI specifications with custom metadata profiles and mapping record IDs.
…and refactor dependencies - Removed `ProfileResolverService` as it was no longer used. - Refactored `ProfileDiscoveryService` to use `IRemoteSchemaLoader` instead of `IMemoryCache`. - Updated tests for `ProfileDiscoveryService` and `SchemaResolverService` to reflect the changes.
- Removed unused methods and properties from `OptionalEndpointExtensions.cs`. - Updated test cases in `OptionalEndpointExtensionsTests.cs` to reflect the removed functionality. - Adjusted constructor parameters in `ProfileDiscoveryService.cs`.
…overyService - Replace string-based JSON content with `JsonObject` for better type safety. - Refactor method signatures to return `JsonObject?` instead of `string?`. - Update tests to assert against `JsonObject` properties.
- Refactored `JsonValidatorService` to use `JsonNode` for schema handling. - Updated `SchemaResolverService` to support asynchronous cancellation tokens. - Modified tests to reflect changes in service method signatures.
…idation result classes
…okup - Removed redundant regex for extracting profile version from schema URL. - Simplified methods for resolving schema URLs based on profile versions. - Updated service classes to use new helper methods for consistency.
- Refactored `MemoryCheckpointLog` to use structured logging with `LoggerMessage` - Updated `RemoteSchemaLoaderLog` to include a new warning message for unknown JSON schema draft URLs - Added necessary package references and updated lock files
…e unused telemetry counters
…ethods, and improve test coverage
…inference logic. Stop resolving from openapi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📋 Overview
This branch completes the implementation of the core validation engine for the OpenAPI Specification and HSDS (Human Services Data Specification) compliance checking.
Specifically, this branch:
Newtonsoft.Json/Newtonsoft.Json.SchematoSystem.Text.JsonandJsonSchema.Netfor compliance with modern Draft 2020-12 and OpenAPI 3.x schema structures, leading to substantial memory and CPU optimization.🛠️ Type of Change
🔍 Key Modifications
OpenApiValidationService: Integrates HSDS profile resolution, spec-to-profile endpoint comparisons, structural validation checks, version warnings, and live verification loops.JsonValidatorService&JsonSchemaBuild: Migrated validation logic toJsonSchema.Net. Added global registration of core Draft 2020-12 meta-schemas toSchemaRegistry.Globaland optimized cache-hits by bypassing HTTP fetches for pre-registered schemas.RemoteSchemaLoader: Implemented gated remote fetching (_activeLoadsconcurrent registry) to avoid duplicate schema downloads, added authentication header support for profile resolution, and corrected duplicate schema registrations.EndpointTestingService: Implemented runtime HTTP endpoint verification logic, supporting custom and user-supplied credentials, URL normalization, parameter extraction/substitution, and schema comparisons for API responses.HsdsComplianceService&ProfileDiscoveryService: Added discovery and probing strategies (openapi_url,/openapi.json,/swagger.json) to detect API versioning, extract HSDS profile versions, and map them to canonical versions.SchemaWarmupBackgroundService&SchemaWarmupExecutor: Implemented a startup warmup sequence that downloads, parses, and caches configured OpenAPI HSDS schemas in the background.OpenReferralApi(Configuration / Program.cs): Standardized validation server-side enums, added custom CORS/SSL settings, configured matrix-driven integration workflows, and enabled Serilog file sinks.OpenReferralApi.Tests: Standardized assertions withAssert.EnterMultipleScopeacross all tests to prevent premature test aborts on first failure, introducedTestHttpClientFactoryfor standard HTTP mock configurations, and expanded test coverage.🧪 Verification & Testing
test-validation.yml) to perform builds and trigger validations automatically on pull requests.