Merged spark e2e#3
Closed
aniketpalu wants to merge 27 commits into
Closed
Conversation
Reconcile server/driver Roles from an embedded YAML template so FeatureStore materialization works without manual SA/Role setup. Widen the operator ClusterRole only as needed to grant those permissions (including deletecollection for Spark driver cleanup). Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
…aterialization Adds a new batch compute engine that submits materialization jobs as SparkApplication CRDs via the Kubeflow Spark Operator. One 'feast materialize' call creates one SparkApplication pod that processes all feature views using distributed Spark, rather than running in-process on the Feast server. Key changes: - Refactor materialize()/materialize_incremental() to pass all tasks to the engine in a single batch call instead of looping per feature view. Existing engines are unaffected (base class loops tasks internally via _materialize_one). - Add public get_provider() method on FeatureStore. - New spark_application engine: config, compute, job, driver script, Dockerfile. - 12 unit tests covering config, validation, CR structure, state mapping, timeout, cleanup, and job naming. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
- Pod calls apply_materialization via gRPC after each successful FV, setting state to AVAILABLE_ONLINE. Server reads FV state post-completion to determine per-FV success/failure in batched SparkApplication runs. - registry_address is now mandatory (simplified from complex path heuristic). - Dockerfile rewritten to install feast from source (matches K8s engine pattern). - Unit tests updated: 15/15 pass (3 new tests for _build_per_fv_jobs). - E2E validated: 5 FVs x 9600 rows, 5 executors, all AVAILABLE_ONLINE. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
… stores - Config delivery: Secret → ConfigMap. Operator's ClusterRole already has full ConfigMap CRUD — avoids widening RBAC for Secrets in ODH. Matches KubernetesComputeEngine pattern. - Removed registry_address config field. Pod inherits server's registry config (SQL, Snowflake) directly and writes apply_materialization() to the same database. Eliminates TLS certificate mounting complexity. - Reject file-based offline stores (dask, file, duckdb) and registries (file) at __init__(), same as existing sqlite/faiss online store rejection. SparkApplication pod has ephemeral filesystem. - Dockerfile: added PYTHONPATH/SPARK_HOME for PySpark, added pymysql. - 20/20 unit tests pass. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
- Remove redundant get_provider(); callers use .provider property - Replace SparkSession monkey-patch with per-thread session binding - Smart retry: only 5xx/429; fail fast on 401/403 with RBAC hint - Retry ConfigMap + SparkApplication creation (transient K8s errors) - Validate env entries: require name + value or valueFrom (K8s EnvVar) - Independent per-FV job status via CompletedMaterializationJob - Exit 1 on any FV failure so SparkApp CR reflects partial failure - Cleanup logs include kubectl delete command for manual recovery - Lint: pragma allowlist on test fixture URL Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
When batch_engine.service_account is omitted, inject feast-<cr>-batch-driver so the SparkApplication matches the SA/RBAC the operator already creates (R2-2). Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Limit service_account baking to spark_application so other batch engines are not given an unexpected parameter. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
…) into merged-spark-e2e
Keep feast-dev#6550 batching; skip FV state transition when _force_local so /materialize-async (already MATERIALIZING) works with SparkApplication. Co-authored-by: Cursor <cursoragent@cursor.com>
… merged-spark-e2e Includes spark_application-only SA bake-in.
…ockerfiles Co-authored-by: Cursor <cursoragent@cursor.com>
Restore master's per-FV materialize path for engines that do not support batching; keep SparkApplication on the existing batch path via ComputeEngine.supports_batch. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Take supports_batch isolation; keep _force_local skip for remote async materialize on both batch and per-FV paths. Co-authored-by: Cursor <cursoragent@cursor.com>
… cleanup Replace __end_date__ sentinel with _MaterializationDateRange; fail fast if engine job count mismatches; map UNKNOWN SparkApp state to WAITING; always cleanup ConfigMap/CR after wait. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Defer kubeconfig load until materialize/cleanup so feast apply can construct the engine without a cluster. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Pod already writes watermarks via applies_materialization; server batch path skips the second call to avoid duplicate intervals. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Use null-safe getattr so materialize falls back to the per-FV path when batch_engine or supports_batch is absent. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Null-safe supports_batch, skip duplicate apply_materialization, lazy K8s client, review hardening.
gofmt repo_config_test map literals and refresh .secrets.baseline line numbers after batch-engine RBAC wiring in services.go. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Share getBatchEngineConfig() for engine type and driver SA resolution instead of fetching/parsing the same ConfigMap twice. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Which issue(s) this PR fixes:
Checks
git commit -s)Testing Strategy
Misc