release: 1.0.0-preview.4 - #4
Merged
Merged
Conversation
- Introduced non-string value redaction options in JSON serialization, allowing for Null, Placeholder, and Omit strategies. - Updated `JsonRedactionAttribute` to support preserving leading characters for masked strings. - Enhanced `SensitiveJsonModifier` to handle non-string redaction modes and collections. - Added comprehensive tests for new redaction behaviors, including handling of numeric, date, boolean, and collection properties. - Improved existing tests to validate new functionality and ensure expected outcomes for various redaction scenarios.
…d retention scheduler - Implement `RedactionPerformanceProfilerTests` to validate operation timing, multiple operations, report generation, and benchmarks for email and phone masking. - Create `RedactionPolicyValidatorTests` to ensure proper validation of audit stores, token stores, and policy annotations. - Add `BulkOperationAuditInterceptorTests` to verify command detection for update and delete operations in EF Core. - Introduce `SensitiveDataNewtonsoftConverterTests` for testing JSON serialization with sensitive data redaction. - Implement `RetentionSchedulerTests` to validate options configuration and service registration for retention scheduling. - Add `RedisTokenStoreTests` (commented out) for testing Redis token store functionality and service registration. - Update project dependencies in `SensitiveFlow.Retention.Tests` and `SensitiveFlow.TokenStore.EFCore.Tests` to include necessary packages.
…ntion scheduler Replace broken BulkOperationAuditInterceptor with proper IQueryExpressionInterceptor that detects and prevents unaudited ExecuteUpdate/ExecuteDelete on annotated entities. Implement ExecuteUpdateAuditedAsync and ExecuteDeleteAuditedAsync helpers with SELECT-first approach: prefetch affected DataSubjectIds, run bulk operation, emit one AuditRecord per (subject, annotated-field) pair. Add cost guard (MaxAuditedRows default 10k) and SensitiveBulkOperationsGuardInterceptor to fail-fast on unaudited bulk ops. Fix CrossBoundarySensitiveDataAnalyzer RS1032/RS2000 diagnostics errors by centralizing descriptor in DiagnosticDescriptors, fixing message format, updating rule ID to SF0005, and adding to AnalyzerReleases.Unshipped.md. Rewrite analyzer tests using AnalyzerTestHarness to properly resolve SensitiveFlow.Core attributes. Fix RetentionScheduler test failures: change init-only properties to settable properties in RetentionSchedulerOptions, add services.AddLogging() to test setup, add required package references to csproj. Fix RedactionPerformanceProfiler tests failing in pt-BR locale by using CultureInfo.InvariantCulture in OperationSummary.ToString() formatting. Ensure library only protects data explicitly marked [PersonalData]/[SensitiveData], removing all heuristic-based field name inference. All 747 tests passing; build 0 warnings, 0 errors.
…s and enhance bulk operation auditing features
- Introduced README files for various components including: - SensitiveFlow.Audit - SensitiveFlow.Audit.Snapshots.EFCore - SensitiveFlow.Core - SensitiveFlow.Diagnostics - SensitiveFlow.EFCore - SensitiveFlow.HealthChecks - SensitiveFlow.Json - SensitiveFlow.Logging - SensitiveFlow.Retention - SensitiveFlow.SourceGenerators - SensitiveFlow.TestKit - SensitiveFlow.TokenStore.EFCore - SensitiveFlow.Tool - Each README outlines main components, usage patterns, and possible improvements. - Enhanced clarity on features like auditing, retention policies, redaction strategies, and logging integration.
…s lacking [Redaction] attribute feat(core): introduce AuditRecordDetails model for structured audit data fix(efcore): enforce DataSubjectId type validation in bulk operations and interceptors test(analyzers): implement comprehensive tests for MissingRedactionAttributeAnalyzer test(core): add unit tests for AuditRecordDetails parsing and serialization test(efcore): enhance tests for SensitiveBulkOperations and SensitiveDataAuditInterceptor with DataSubjectId validation
…and handle collections
…Flow issues - Created ANALYSIS_EXECUTIVE_SUMMARY.md detailing critical issues, impacts, and recommended actions. - Added ANALYSIS_README.md for navigation and usage guidance of analysis documents. - Introduced FIXES_IMPLEMENTATION_MAP.md outlining specific file changes, line numbers, and types of modifications needed to address identified issues. - Organized issues by severity (P0-P3) and provided detailed action plans for resolution.
…h implementation status and enhancements feat(cache): add caching for RedactionAttribute to optimize reflection feat(bulk): implement heuristic sizing for MaxAuditedRows with validation feat(json): introduce IncludeRedactionMetadata for enhanced redaction insights feat(logging): clarify redaction options and auto-detection in documentation
…mprovements and async method deprecations
- Created a comprehensive troubleshooting guide for SensitiveFlow integration in .NET applications. - Added a new Redis microservice sample demonstrating the use of SensitiveFlow with a centralized Redis token store. - Implemented PseudonymizationController for handling anonymization and deanonymization requests. - Configured Redis connection and token store in the sample application. - Updated README with instructions for running the microservice and testing endpoints. - Enhanced Redis token store functionality and added container tests for Redis integration.
…n and RedisTokenStore benchmarks, and update README - Deleted PseudonymizationBenchmarks.cs to streamline benchmark suite. - Introduced RetentionBenchmarks.cs to measure retention policy performance. - Added RedisTokenStoreBenchmarks.cs for Redis token operations benchmarking. - Updated README.md with comprehensive benchmark details and usage instructions. - Modified project file to include necessary package references for new benchmarks. - Created benchmarking guide for better understanding and execution of benchmarks.
…ation for performance insights
…with new attributes and compile-time validation
…ndling in employee and order management
…g, and searching - Introduced IAuditAlertingPolicy interface for detecting suspicious audit patterns with AuditAlert record. - Added IAuditExporter interface for exporting audit records in CSV, JSON, and Parquet formats. - Created IAuditSearchIndex interface for full-text search capabilities on audit records. - Enhanced IAuditStore interface with QueryStreamAsync method for efficient streaming of audit records. - Implemented IAuditStreamQuery interface for querying large datasets without materializing all records. - Updated InstrumentedAuditStore to support streaming queries. - Added comprehensive integration tests for audit features including streaming, anonymization, export formats, search indexing, and alerting policies. - Developed InMemoryAuditStore to support new streaming functionality. - Created CoreFeatureScenariosTests for testing core improvements including composite data subject IDs and role-based redaction contexts.
- Implement PerformanceBaselineService to manage performance baselines and detect regressions. - Introduce PerformanceBaseline and PerformanceCheckResult classes for baseline definitions and check results. - Add QueryOptimizationAdvisor to analyze query patterns and suggest index optimizations. - Create InMemoryAuditStore for testing purposes to simulate audit record storage. - Enhance README with detailed usage instructions for new features. - Implement comprehensive unit tests for performance baselines, query optimization, and custom metrics.
…ty redaction, audit trail correlation, redaction performance metrics, custom masking strategies, and log sampling
…olution, custom masking strategies, lazy redaction, and OpenAPI schema support - Added `IRedactionContextResolver` and `ClaimsPrincipalRedactionContextResolver` for role-based redaction context resolution. - Introduced `IJsonMaskingStrategy` and `JsonMaskingStrategyRegistry` for pluggable masking strategies, including built-in implementations for email, phone, credit card, SSN, and IP address. - Implemented `LazyRedactionWrapper<T>` to defer redaction until serialization, optimizing performance for large object graphs. - Created `SensitiveDataSchemaFilter` for identifying sensitive properties in OpenAPI schemas without external dependencies. - Added `IJsonRedactionMetricsCollector` and `JsonRedactionMetricsCollector` for tracking redaction metrics using OpenTelemetry. - Updated `JsonRedactionOptions` to support new features, including context resolver, masking strategies, lazy redaction, and metrics collector. - Enhanced documentation and tests to cover new functionalities and ensure reliability.
…rallel execution, and analytics - Implemented incremental scheduling to track last successful runs per policy. - Added parallel execution for retention policies to run multiple batches concurrently. - Introduced retention analytics to collect and analyze execution metrics. - Developed selective re-anonymization functionality for on-demand anonymization. - Created archive tiering for managing cold storage of expired entities. - Implemented notification templates for configurable alerts on retention completion. - Added retention analytics reporting capabilities for generating formatted reports. - Enhanced tests to cover new features and ensure functionality.
…e-based suppression, cross-assembly detection, and custom masking method recognition
- Implement ITokenAuditSink interface for recording token operations. - Create InMemoryTokenAuditSink for in-memory storage of audit records. - Define TokenAuditOperation enum for various token operations. - Introduce TokenAuditRecord record for immutable audit event representation. - Add TokenExpirationOptions for configuring token expiration behavior. - Implement TokenExpirationService for managing token expiration and cleanup. - Create TokenKeyRotationService for bulk updates of token mappings. - Introduce salting strategies with ITokenSaltStrategy interface and implementations. - Add TokenSaltStrategyRegistry for managing named salting strategies. - Enhance README with advanced features and usage examples. - Add comprehensive unit tests for new features and components.
…t isolation, custom claim extraction, and IP masking features
…ed redaction levels, and performance metrics for redaction operations
…tion, performance metrics, data quality checking, alerting integration, and audit age tracking - Implemented `RetentionPolicyValidator` for validating retention policies. - Added `HealthCheckPerformanceCollector` to track health check performance metrics. - Introduced `DataQualityChecker` for checking data integrity and quality. - Created `HealthAlertingPolicy` for configuring alerts based on health check results. - Developed `AuditAgeTracker` to monitor the age of audit records and provide recommendations. - Updated README.md to document new features and usage examples. - Added unit tests for new features to ensure functionality and reliability.
…rformance reporting features - Implement CodeGenerationConfigProvider for managing code generation snippets and setup instructions. - Introduce IncrementalGenerationTracker to track generated types and their modification status for efficient regeneration. - Create GenerationPerformanceReporter to log and report performance metrics of code generation operations. - Enhance README with advanced features including incremental generation, configuration schema, and performance reports. - Add fluent assertion extensions for audit testing in the TestKit. - Implement thread-safe audit store for concurrent testing scenarios. - Add unit tests for new features including configuration provider, performance reporter, and incremental tracker.
…functionalities - Introduced `EFCoreEnhancementsTests` to validate Raw SQL audit options, migration audit options, record compression options, and replay testing functionalities. - Added tests for `IncrementalGenerationTracker` to ensure modified types are tracked correctly. - Created `ToolEnhancementsTests` to cover setup wizard options, CI/CD exit code policies, configuration scaffolding, diff reporting, and performance analysis. - Enhanced `ThreadSafeAuditStoreTests` to ensure thread safety in mixed operations. - Added project files for `SensitiveFlow.Tool.Tests` to facilitate testing of tool-related functionalities.
…ests - Introduced ReplicationStatus class to track replication operations. - Added ImmutableSnapshot class for immutable, append-only snapshots of audit state. - Created SnapshotOptions class for configuring snapshot behavior. - Implemented unit tests for anonymization enhancements, including incremental and streaming exports. - Developed EFCore enhancements tests covering snapshots, encryption, query building, and real-time alerts. - Ensured comprehensive validation and default settings for new classes.
- Implement AlertingService for real-time alerting on suspicious patterns (bulk delete, multiple IPs). - Create ComplianceService for validating anonymization against k-anonymity and l-diversity. - Develop CustomerService for customer data validation before persistence. - Introduce ExportService for handling GDPR data export and erasure requests, including anonymization and incremental export checks. - Add OrderService for managing order creation and status updates. - Configure appsettings.json for database connection and logging settings. - Set up docker-compose.yml for SQL Server, Redis, and Prometheus services. - Update README.md with detailed project overview, architecture, API endpoints, and deployment checklist. - Add integration tests for anonymization and EF Core audit features to ensure functionality and security.
…itiveFlow integration - Updated Order and OrderItem models to enhance data subject linking and redaction attributes. - Refined Program.cs for better service registration and middleware configuration, including HMAC pseudonymization. - Revised README.md to clarify setup instructions and demonstrate API endpoints with redaction. - Enhanced AlertingService and ComplianceService with additional namespaces for better functionality. - Modified ExportService to simplify audit record handling in data exports. - Removed unnecessary Docker Compose version specification and added Prometheus configuration. - Introduced openapi.json for interactive API documentation. - Updated Anonymization and Audit service extensions to streamline service registration.
- Added DataSubjectIdAttribute to mark properties as data subject identifiers for audit records. - Updated Order model to use DataSubjectId for CustomerId. - Enhanced SensitiveDataAuditInterceptor to resolve data subject identifiers, including support for owned entities. - Refactored EfCoreAuditStore to use deferred factory resolution to avoid DI cycles. - Updated EF Core service extensions to register SensitiveDataAuditInterceptor as a singleton. - Removed openapi.json in favor of Swagger integration for API documentation. - Added Swagger generation for API documentation with GDPR compliance description. - Improved logging during application startup and migration processes. - Added tests for DataSubjectId attribute functionality and owned entity behavior in audit records.
- Added `HashChainingAuditStore` to ensure each audit record carries a SHA-256 hash linked to the previous record's hash. - Introduced `CurrentRecordHash` and `PreviousRecordHash` fields in `AuditRecordEntity` for integrity verification. - Implemented `VerifyChain` endpoint in `AuditController` to validate the integrity of the audit records. - Updated `CustomersController` to include credit card information in customer creation. - Created `ECommerceDbContextDesignTimeFactory` for EF Core migrations. - Added migration `AddAuditHashColumns` to update the database schema. - Enhanced `AuditRecordEntityTypeConfiguration` to configure new hash fields. - Updated `SensitiveDataAuditInterceptor` to improve audit detail logging. - Added unit tests for `HashChainingAuditStore` to ensure correct behavior and integrity verification.
… add changelog entries for new features
MateusjsSilva
added
bug
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Cuts the
1.0.0-preview.4release. Adds a tamper-evident audit hash chain, fixes a DI deadlock in the audit interceptor, and improves DX around audit diagnostics in the EF Core sample.Headline feature — tamper-evident hash chain:
Detects deletions, modifications, and reorderings. Appends are serialized through a semaphore (chain is inherently sequential) and the head is lazy-loaded from the inner store on first append so the chain survives process restarts.
AuditRecordEntitygained two nullable columns (PreviousRecordHash,CurrentRecordHash).DI deadlock fix: Registering both
AddDbContextandAddDbContextFactoryon the same context could captive-dep its way into a deadlock.SensitiveDataAuditInterceptoris now singleton-safe (resolvesIAuditContext/IPseudonymizerper-call viaCoreOptionsExtension.ApplicationServiceProvider) andEfCoreAuditStoreaccepts aFunc<IDbContextFactory<T>>accessor for deferred resolution.Audit details diagnostics:
AuditRecord.Detailsno longer collapses three different scenarios into[REDACTED]. New markers distinguish[NULL],[REDACTED], and[PSEUDONYMIZER_UNAVAILABLE].Misc:
[DataSubjectId]property attribute, owned-type owner traversal soCustomerProfileaudit rows correlate toCustomer, ECommerce sample fixes (Scalar wiring,CreditCard/ProfileDTOs,IDesignTimeDbContextFactory, EF tooling-generated migration).Type of change
Related regulatory/privacy requirement (if applicable)
SensitiveFlow is a toolkit that helps developers reach GDPR/LGPD-aligned auditing and data-protection patterns — it does not, by itself, make an application compliant. This release expands the building blocks developers can compose toward those obligations:
[DataSubjectId]attribute and owned-type traversal — easier to produce subject-correlated processing records (GDPR Art. 30 / LGPD Art. 37).Checklist
CHANGELOG.mdupdated