feat(edm): implement EDM prediction framework (Simplex, S-Map, GP-EDM) by aman-raj-srivastva · Pull Request #1561 · NOAA-FIMS/FIMS

aman-raj-srivastva · 2026-06-23T14:51:12Z

Summary

This PR implements the core EDM prediction framework for Empirical Dynamic Modeling (EDM) in FIMS as part of the GSoC 2026 project:
"Add Empirical Dynamic Modeling to FIMS."

This work builds on the delay embedding infrastructure introduced in PR #1528 and implements the prediction algorithms described in Issue #1487.

Target Branch: main-edm

This PR completes the standalone prediction framework for EDM. Integration of these prediction methods into the FIMS likelihood/TMB pipeline will be addressed in a subsequent PR.

What this PR adds

Generic EDM Prediction Framework

Implemented a reusable and extensible prediction framework for EDM algorithms.

Features include:

Common prediction interface
Shared prediction result structures
Extensible architecture for additional EDM prediction methods
Configurable forecast horizon support (shared across all prediction methods)
Modular design for future integration with the FIMS objective function

Simplex Projection

Implemented the Standard Simplex Projection algorithm (Sugihara & May, 1990).

Features include:

Nearest-neighbor based prediction
Distance-weighted forecasting
Integration with the delay embedding infrastructure

Distance Weighting Infrastructure

Implemented reusable distance weighting utilities shared across prediction algorithms.

Features include:

Squared Euclidean distance calculation utilities
Modular exponential and Gaussian weighting functions
Shared infrastructure for Simplex, S-Map, and GP-EDM

Note on distance metric: This implementation uses squared Euclidean distances in all kernel functions (rather than Euclidean distances as in rEDM). This is an intentional design decision to maintain compatibility with CppAD automatic differentiation, which avoids the sqrt() operation. Predictions are numerically verified against pure-R squared-distance reference implementations.

Standard S-Map

Implemented the Standard S-Map prediction algorithm (Sugihara, 1994).

Features include:

Weighted local linear regression
Configurable exponential and Gaussian kernel functions
Shared linear algebra utilities (Gaussian elimination with partial pivoting)
Integration with delay embeddings

GP-EDM

Implemented the GP-EDM prediction algorithm following the reference GPEDM implementation (Munch et al. 2017; Rogers 2023).

Features include:

Gaussian Process regression on delay embeddings
Automatic Relevance Determination (ARD) length-scale parameters (phi)
ARD squared-exponential kernel with half-Normal priors on phi
Rprop optimizer for marginal likelihood maximization
Shared covariance kernel and linear algebra utilities
Rcpp interfaces for both hyperparameter fitting and prediction

Bug fix: Corrected a missing 0.5 factor on the log-determinant term in the GP log-marginal likelihood. This was identified during numerical verification against the GPEDM reference package.

Known intentional differences from the GPEDM reference package:

Aspect	GPEDM Reference	This Implementation	Reason
Matrix inversion	Cholesky (`chol2inv`)	Gaussian elimination	Simpler for v1; both are correct for SPD matrices
`rho` parameter	Estimated (multi-population)	Not estimated	FIMS single-population case only
Input scaling	`global`/`local`/`none`	Not implemented (user responsibility)	Scope reduction for v1

Rcpp Interfaces

Implemented R-callable interfaces for all prediction methods via Rcpp modules.

Includes:

SimplexProjectionInterface — fields: embedding_dimension, n_neighbors
SMapProjectionInterface — fields: embedding_dimension, theta, kernel
GPEdmProjectionInterface — fields: embedding_dimension, phi, sigma2, ve; methods: fit(), predict()
to_matrix() helper bridging DelayEmbeddingInterface storage to the C++ prediction framework
All interfaces registered with EDMInterfaceBase::live_objects for Rcpp memory safety

All three interfaces are callable from R via methods::new().

Validation

Numerical verification was performed for all three prediction methods:

Algorithm	Reference	Max Difference
Simplex Projection	Pure-R squared-distance implementation	`2.22e-16` (machine ε)
S-Map	Pure-R WLS squared-distance implementation	`1.30e-14`
GP-EDM (fixed params)	GPEDM R package (`predict.GP`)	`2.53e-10`

For GP-EDM, the sigma2 and ve gradients were additionally verified to match the reference package exactly (these do not depend on the per-dimension distance matrices), confirming the covariance matrix math is correct.

Testing

This PR includes comprehensive testing for the EDM prediction framework.

Current coverage:

114 GoogleTests (C++)
87 testthat tests (R)

Testing covers:

Prediction framework behavior
Simplex Projection (IO correctness, numerical agreement, edge cases, error handling)
Standard S-Map (IO correctness, numerical agreement, kernel comparison, error handling)
GP-EDM (IO correctness, fit → predict chain, hyperparameter localization, error handling)
Rcpp interfaces (construction, field access, invalid ID handling)
Numerical agreement with pure-R reference implementations

Future Work

This PR completes the standalone EDM prediction framework. Future work can focus on:

Integrating the prediction framework into the FIMS likelihood/TMB pipeline
CppAD/TMB automatic differentiation support throughout the EDM module
Observation uncertainty propagation through the delay embedding
Additional statistical model integration

Related Issues

Addresses Implement Generic EDM Prediction Functor #1487
Builds on Implement Delay Embedding Generator for Time Series #1484
Builds on Add EDM delay embedding infrastructure and tests #1528

added the comments to the following files: modified: R/distribution_formulas.R modified: R/fimsfit.R modified: R/initialize_modules.R modified: inst/include/common/information.hpp modified: inst/include/common/model.hpp modified: inst/include/distributions/functors/density_components_base.hpp modified: inst/include/distributions/functors/normal_lpdf.hpp modified: inst/include/interface/rcpp/rcpp_objects/rcpp_distribution.hpp modified: src/FIMS.cpp

- Created rcpp_edm.hpp defining EDMInterfaceBase and DelayEmbeddingInterface wrapper classes. - Exposed the construct(), construct_drop_missing(), and at() methods to R. - Exposed fields/metadata (such as embedding_dimension, time_lag, n_rows, n_cols, values, and target_indices) to R. - Registered the DelayEmbedding class in the RCPP_MODULE(fims) block within fims_modules.hpp.

- Add edm_embeddings list slot to the FIMSFrame S4 class to store delay-embedding results produced by create_edm_embedding() - Add create_edm_embedding() helper that pulls a named time series from the data slot, calls the Rcpp DelayEmbedding module, and returns a new FIMSFrame with the result stored by name - Add get_edm_embeddings() accessor following the existing get_*() pattern in fimsframe.R - Add model_edm_matrix() accessor that returns a numeric matrix (n_rows x E) ready to pass to an EDM prediction module, following the existing model_*() pattern - Register 'edm' as a valid FIMSFrame input type in data-raw/fims_input_types.R and regenerate data/fims_input_types.rda - Add tests/testthat/test-edm-fimsframe.R with 6 test blocks covering default empty slot, error handling, input validation, metadata correctness, matrix shape, and chaining multiple embeddings This completes proposal deliverable: 'Create structures in FIMSFrame for embedded matrices' (Weeks 1-3).

…gers

…r review feedback

…ution-class

…rithm

aman-raj-srivastva · 2026-06-24T18:04:20Z

Reverted the S-Map commit temporarily for further review before re-pushing

aman-raj-srivastva · 2026-06-24T19:03:53Z

@stevemunch @nathanvaughan-NOAA while implementing the S-Map predictor, I came across a few design questions where I'd appreciate feedback before I push the implementation.

I have currently implemented the standard S-Map formulation described in Section 2.1 of Esguerra & Munch (2024), i.e., the local linear model that serves as the baseline/M-step component of the full HMS-map.

For each local neighborhood, the implementation constructs a design matrix:

$$X_i = [1, x_t, x_{t-\tau}, \ldots, x_{t-(E-1)\tau}]$$

and solves the weighted least-squares system:

$$(X^T W X)\beta = X^T W y$$

using a CppAD-safe Gaussian elimination routine.

Before finalizing the implementation, I wanted to get your thoughts on a couple of design choices.

1. Weighting Kernel / Distance Metric

Classic S-Map implementations (e.g., Sugihara 1994 and rEDM) typically use an exponential kernel based on Euclidean distance:

$$w_i = \exp\left(-\theta \frac{d_i}{\bar d}\right)$$

However, Section 2.1 of Esguerra & Munch (2024) defines the weighting function as:

$$w_i = \exp\left(-\theta^2 \left(\frac{d_i}{D}\right)^2\right)$$

which is effectively a Gaussian kernel based on squared Euclidean distance.

For the generic FIMS S-Map predictor, would you prefer:

the formulation used in the HMS-map paper, or
compatibility with the more traditional rEDM-style weighting?

Using squared distances has the additional benefit of avoiding a sqrt() inside the AD tape.

2. Target Alignment in Delay Embedding

The paper formulates the delay embedding map as a one-step-ahead forecast:

$$x_{t+1} = f(x_t, \ldots, x_{t-E+1})$$

While wiring the implementation, I noticed that the current DelayEmbeddingMatrix assigns target_values[row] and embedded_values[row][0] to the same observation, which appears to correspond to a 0-step-ahead target.

Am I interpreting this correctly, or is the intended target shift handled elsewhere in the workflow?

If not, would aligning the embedding construction with the standard one-step-ahead formulation be the preferred approach?

3. Scope of This PR

My understanding is that this PR should focus on the standard S-Map predictor only:

local weighted linear regression
prediction interface integration
testing and validation infrastructure

The EM iteration and state-estimation machinery required for the full HMS-map would then be a separate follow-up effort.

Just wanted to confirm that this matches the intended project scope before I proceed further.

Thanks! I currently have the S-Map implementation and associated tests working locally, and I wanted to verify these design details before pushing the next commit.

stevemunch · 2026-06-24T19:25:05Z

Aman-In my experience using distance v. squared distance in the weighting kernel doesn’t make a huge difference in performance except in rare cases. I’d use whatever was easier/faster. In the state-estimation step of the HMS algorithm, we are updating all of the states based on the current estimates of the S-map coefficients and the observations. There’s no time lag in this step. In the S-map step, the model is fit one step ahead as usual. Regarding HMS map being a separate project - I will let others chime in. For my two cents - that’s the part that requires the most care in implementation. The rest is pretty straightforward. Critically ‘standard EDM’ does not handle observation error explicitly. The statistically minded folks at NMFS generally care about observation uncertainty, so it would be most valuable to if you completed the HMS map code or some other algorithm for handling observation uncertainty.Sent from my iPadOn Jun 24, 2026, at 12:04 PM, Aman Raj ***@***.***> wrote:aman-raj-srivastva left a comment (NOAA-FIMS/FIMS#1561) @stevemunch @nathanvaughan-NOAA while implementing the S-Map predictor, I came across a few design questions where I'd appreciate feedback before I push the implementation. I have currently implemented the standard S-Map formulation described in Section 2.1 of Esguerra & Munch (2024), i.e., the local linear model that serves as the baseline/M-step component of the full HMS-map. For each local neighborhood, the implementation constructs a design matrix: $$X_i = [1, x_t, x_{t-\tau}, \ldots, x_{t-(E-1)\tau}]$$ and solves the weighted least-squares system: $$(X^T W X)\beta = X^T W y$$ using a CppAD-safe Gaussian elimination routine. Before finalizing the implementation, I wanted to get your thoughts on a couple of design choices. 1. Weighting Kernel / Distance Metric Classic S-Map implementations (e.g., Sugihara 1994 and rEDM) typically use an exponential kernel based on Euclidean distance: $$w_i = \exp\left(-\theta \frac{d_i}{\bar d}\right)$$ However, Section 2.1 of Esguerra & Munch (2024) defines the weighting function as: $$w_i = \exp\left(-\theta^2 \left(\frac{d_i}{D}\right)^2\right)$$ which is effectively a Gaussian kernel based on squared Euclidean distance. For the generic FIMS S-Map predictor, would you prefer: the formulation used in the HMS-map paper, or compatibility with the more traditional rEDM-style weighting? Using squared distances has the additional benefit of avoiding a sqrt() inside the AD tape. 2. Target Alignment in Delay Embedding The paper formulates the delay embedding map as a one-step-ahead forecast: $$x_{t+1} = f(x_t, \ldots, x_{t-E+1})$$ While wiring the implementation, I noticed that the current DelayEmbeddingMatrix assigns target_values[row] and embedded_values[row][0] to the same observation, which appears to correspond to a 0-step-ahead target. Am I interpreting this correctly, or is the intended target shift handled elsewhere in the workflow? If not, would aligning the embedding construction with the standard one-step-ahead formulation be the preferred approach? 3. Scope of This PR My understanding is that this PR should focus on the standard S-Map predictor only: local weighted linear regression prediction interface integration testing and validation infrastructure The EM iteration and state-estimation machinery required for the full HMS-map would then be a separate follow-up effort. Just wanted to confirm that this matches the intended project scope before I proceed further. Thanks! I currently have the S-Map implementation and associated tests working locally, and I wanted to verify these design details before pushing the next commit. —Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

nathanvaughan-NOAA · 2026-06-24T19:34:50Z

Hey @aman-raj-srivastva, great job on your progress so far. For these questions I would suggest

Can you add the option for the user to choose the kernal function? That seems like it would offer the most utility for comparing with other approaches and leave options open for different functions to be added in the future. @stevemunch should be able to explain the rational for the change in kernal. I think the main difference is just that the guassian has fatter tails than the exponential but I thought both methods generally estimated a scale factor that may minimize the impact of the kernal choice. I just saw @stevemunch above say basically that same thing.
Good catch on the indexing I missed that the first time. I think it's best to change that to the 1 step ahead index. From @stevemunch 's comment above the state-estimation step would be updating the base values (why it's so beneficial to set this all up as linked pointers) and then the S-MAP step would use the embedded and target objects to make predictions.
@stevemunch is correct that the HMS uncertainty integration is critical to practically applying this and an important feature, but the mechanics will be complex and I think fairly isolated from this initial structure development so I think it's fine to compartmentalize for now and keep this pull request focused.

…rget Per mentor feedback (nathanvaughan-NOAA, stevemunch): 1. Selectable weighting kernel (SMapKernel enum): - kExponential (default): w = exp(-theta * d / d_mean) [rEDM / Sugihara 1994] - kGaussian: w = exp(-theta^2 * (d/D)^2) [Esguerra & Munch 2024] Both reduce to global OLS when theta=0. Kernel field exposed on SMapProjection<Type> so users can compare formulations. 2. One-step-ahead target alignment in MakeDelayEmbedding: Added forecast_horizon parameter (default=1). target_values[row] now points to series[target_index + h], matching the standard EDM formulation: predict x_{t+h} from the embedding state x_t. n_rows reduced by forecast_horizon to keep all target pointers in bounds. MakeDelayEmbeddingDropMissing forwards the parameter unchanged. 3. Tests: 12 S-Map + 9 Simplex = 21 passing GoogleTests.

…y compatibility

- edm.hpp: remove stale 'S-map will be added' note; SMapProjection is already included - delay_embedding.hpp: correct target_values struct description from 'x_t' to 'x_{t+h}'; fix target_uncertainty comment to say sigma_{target_index + forecast_horizon} not sigma_t

mollystevens-noaa · 2026-07-14T19:27:10Z

Hey @aman-raj-srivastva ! I met with @kellijohnson-NOAA and @Andrea-Havron-NOAA this afternoon, and we have an updated plan forward that will require much less overhead. The rebase to main conducted here will be the only full rebase we'll do until the end of the project, where we'll leave a few days for you to handle any additional merge conflicts / breaks that occur. For the remainder of the project, you can just work off this main-edm branch, and @Andrea-Havron-NOAA will let us know if we need to cherry pick any changes from the main branch that will affect your work. Bringing in @nathanvaughan-NOAA for awareness. Hope your resolution of conflicts here is going well, and please let us know if you have any questions or concerns!

…IMSFrame rebase rename

mollystevens-noaa · 2026-07-20T15:44:08Z

Thanks @aman-raj-srivastva ! I see this version is passing all checks, and looks good to me. Before I merge this pull request, I wanted to circle back with @nathanvaughan-NOAA and @Andrea-Havron-NOAA to make sure that it fits as expected within the FIMS framework.

nathanvaughan-NOAA · 2026-07-20T16:09:22Z

Thanks @aman-raj-srivastva for all the great work and @mollystevens-noaa for testing out the new functions. I had a discussion with @Andrea-Havron-NOAA who will have a more detailed comment but we are concerned that at the moment the EDM module is still isolated from the rest of the FIMS codebase. Specifically there are a lot of calculations added inside the GP-EDM for estimating scale that should be a part of the FIMS distributions module. In it's final state the EDM module should only be creating the delay embedding map, calculating distance weights for each of the embedded vectors, and calculating predicted values from the embedding map and weights which are all deterministic calculations. The multivariate normal embedding function should be a part of the distributions functor so it can be used in EDM and for other relationships in FIMS, and the scale parameters should be specified as fixed or estimable similarly to all the other parameters in FIMS. @e-perl-NOAA is working on a similar issue in pull request #1594 to add a GMRF likelihood function.

Andrea-Havron-NOAA · 2026-07-20T23:20:42Z

+    }
+
+    // --- Solve Sigma * alpha = y (alpha overwrites y) ---
+    GaussianElimination<Type>(Sigma, y, N);


It is more efficient to calculate this using the Cholesky decomposition than Gaussian Elimination, which can be evaluated using Eigen's ldlt, for example:
Given a matrix Sigma(n,n):

// Perform LLT (Cholesky) decomposition using Eigen via TMB Eigen::LDLT<Eigen::Matrix<scalartype,Dynamic,Dynamic> > ldlt(Sigma); // 1. Solve Sigma * alpha = y --> L L^T alpha = y vector<Type> alpha = ldlt.solve(y);

For reference, see TMB's MVNORM function here

Updated predict_one() to use Eigen::LDLT Cholesky factorization (Eigen::LDLT<Eigen::Matrix<Type, Dynamic, Dynamic>> ldlt(Sigma_mat); alpha = ldlt.solve(y_mat)).

Andrea-Havron-NOAA · 2026-07-20T23:33:47Z

+  }
+
+  /** @brief Sign function returning -1, 0, or +1 as a double. */
+  static double sign_d(double x) {


@msupernaw, should this use CppAD::sign (uses an atomic) instead so it is differentiable?

Added fims_math::sign() to fims_math.hpp which wraps CppAD::sign() under #ifdef TMB_MODEL so it is differentiable on the AD tape, and updated gp_edm_projection.hpp to use it.

Andrea-Havron-NOAA · 2026-07-20T23:35:01Z

+   * @param N, E    Library dimensions.
+   * @param ve_min, ve_max, s2_min, s2_max  Parameter bounds.
+   */
+  LogPosteriorResult compute_log_posterior_grad(


It's great to the see the multivariate normal function in FIMS! I would prefer you put it, however, in the distribution functors so that this likelihood can be used by the entire code base. You can use TMB's MVNORM function. If you use an explicit definition, update the inverse calculation of Sigma to use the cholesky decomposition instead of the Gaussian Elimination as mentioned above.

Done on both points:

Created MVNormLPDF<Type> in inst/include/distributions/functors/mvnorm_lpdf.hpp inheriting from DensityComponentBase<Type> and exposed it through distributions/distributions.hpp. Under TMB_MODEL, it delegates to TMB's density::MVNORM_t<Type>. For standalone execution, it evaluates using Eigen::LDLT.

Updated GaussianElimination references across GPEdmProjection (predict_one() and compute_log_posterior_grad()) and SMapProjection to use Eigen::LDLT Cholesky factorization for matrix solves, log-determinant, and inverse covariance calculations. Removed GaussianElimination from edm_linear_algebra.hpp.

Andrea-Havron-NOAA · 2026-07-20T23:35:37Z

+
+    // --- Untransform parameters ---
+    std::vector<double> phi_cur(E);
+    for (size_t d = 0; d < E; ++d) phi_cur[d] = std::exp(parst[d]);


use fims_math::exp() throughout

Updated — replaced std::exp, std::log, std::sqrt, and std::abs with fims_math:: throughout the file.

Andrea-Havron-NOAA · 2026-07-20T23:37:50Z

+      for (size_t row = 0; row < N; ++row) iKVs[row * N + col] = e_col[row];
+    }
+
+    // Log determinant via triangular factor diagonal


see TMB example for efficiently calculating the log determinant from a cholesky decomposition

Updated compute_log_posterior_grad() to use Eigen::LDLT for solving alpha (ldlt.solve(y)), inverting the covariance matrix (ldlt.solve(Identity)), and computing the log-determinant (ldlt.vectorD().array().abs().log().sum()).

Andrea-Havron-NOAA · 2026-07-20T23:39:41Z

+      lp_phi     += -0.5 * phi_cur[d] * phi_cur[d] / kLamPhi;
+      dlp_phi[d]  = -phi_cur[d] / kLamPhi;
+    }
+    // sigma2 and ve: Beta-shaped (a=2, b=2) on (0, max)


as noted above for the multivariate normal distribution, please add the beta distribution to the distribution functor folder and expose using the FIMS distribution pathway so these distribtions are available throughout the codebase.

Created BetaLPDF<Type> in inst/include/distributions/functors/beta_lpdf.hpp following the DensityComponentBase<Type> architecture, supporting shape1 ($\alpha$), shape2 ($\beta$), and scale ($s$, default 1.0). Under TMB_MODEL, it calls dbeta(x / s, alpha, beta, true) - log(s). Exposed via distributions/distributions.hpp so it's accessible across FIMS.

Andrea-Havron-NOAA · 2026-07-20T23:53:12Z

+ *  - SMapKernel::kExponential (default): w = exp(-θ · d / d̄)  [rEDM style]
+ *  - SMapKernel::kGaussian:              w = exp(-θ² · (d/D)²) [Esguerra & Munch 2024]
+ * Per Munch (pers. comm.) the performance difference is rarely large;
+ * use kGaussian to avoid a sqrt() inside the AD tape.


it is safe to use a sqrt() inside the AD tape, we use fims_math::sqrt(). Generally, when taking the sqrt of a parameter, we estimate the parameter in log space and take the exponential to keep the value positive. If you can't use this trick, however, when evaluating the s-map projections, please feel free to leave the kGaussian approach as is.

Updated — standardized all square root and exponential calls to use fims_math::sqrt() and fims_math::exp().

Andrea-Havron-NOAA · 2026-07-21T00:03:12Z

+    // Step 4: Solve A β = b via Gaussian elimination (in-place).
+    // On exit b_vec holds the solution β.
+    // -----------------------------------------------------------------------
+    GaussianElimination(A, b_vec, p);


use cholesky factorization as noted in the gp_edm_projection file.

Updated — replaced GaussianElimination in SMapProjection::predict_one() with Eigen::LDLT Cholesky factorization solve for $A \beta = b$.

Andrea-Havron-NOAA · 2026-07-21T00:09:59Z

Thanks for this PR and all your hard work! As @nathanvaughan-NOAA mentioned above, I think there are some changes that can be made to better integrate this code into the larger FIMS code base. Specifically,

distributions should all be defined in the inst/include/distributions/functors folder following the existing pattern for FIMS distributions. It is ok to use TMB macros here as long as they are wrapped in a #ifdef TMB_MODEL/#endif statement
go through the codebase and update basic functions to pull from fims_math instead of std::. You also defined a couple of functions that are already defined in fims::math. This just helps cut down redundancy and helps keep all our basic functions in one place so they are easy to access and test.

I noticed in your next PR, you have more code that helps integrate the edm piece into FIMS, I am looking forward to checking it out!

aman-raj-srivastva · 2026-07-21T17:50:55Z

Thank you all for the thorough review and the feedback. I see what you’re suggesting about architectural adjustments to help integrate the EDM implementation into the FIMS codebase. I’ll begin working through these updates and will put these changes into logical commits as I tackle each piece. Thanks again for the guidance!

stevemunch · 2026-07-21T17:55:48Z

Folks- Just a quick 2cents - While the matrices we're inverting are theoretically postive definite, they do become singular for extreme values of the local weighting parameters. So, although Cholesky is preferred for things that must be symmetric and p.d., we'll want to be sure we have some way to handle p.s.d matrices as well. Cheers, Steve

…

On Tue, Jul 21, 2026 at 10:51 AM Aman Raj ***@***.***> wrote: *aman-raj-srivastva* left a comment (NOAA-FIMS/FIMS#1561) <#1561 (comment)> Thank you all for the thorough review and the feedback. I see what you’re suggesting about architectural adjustments to help integrate the EDM implementation into the FIMS codebase. I’ll begin working through these updates and will put these changes into logical commits as I tackle each piece. Thanks again for the guidance! — Reply to this email directly, view it on GitHub <#1561?email_source=notifications&email_token=AFSAKFYMINKQ2MROW2EJXW35F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5037258824>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFSAKF7C55UVUS4WHE4TU2D5F6URLAVCNFSNUABFKJSXA33TNF2G64TZHM2DKMRQGEZDAMBUHNEXG43VMU5TINZSGY4TKMRUGI22C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AFSAKF3ATR75CITJGZGN4ZD5F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AFSAKFZNCHTWKNPHJINWMDL5F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

Andrea-Havron-NOAA · 2026-07-21T18:55:24Z

Folks- Just a quick 2cents - While the matrices we're inverting are theoretically postive definite, they do become singular for extreme values of the local weighting parameters. So, although Cholesky is preferred for things that must be symmetric and p.d., we'll want to be sure we have some way to handle p.s.d matrices as well. Cheers, Steve
…
On Tue, Jul 21, 2026 at 10:51 AM Aman Raj @.> wrote: aman-raj-srivastva left a comment (NOAA-FIMS/FIMS#1561) <#1561 (comment)> Thank you all for the thorough review and the feedback. I see what you’re suggesting about architectural adjustments to help integrate the EDM implementation into the FIMS codebase. I’ll begin working through these updates and will put these changes into logical commits as I tackle each piece. Thanks again for the guidance! — Reply to this email directly, view it on GitHub <#1561?email_source=notifications&email_token=AFSAKFYMINKQ2MROW2EJXW35F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5037258824>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFSAKF7C55UVUS4WHE4TU2D5F6URLAVCNFSNUABFKJSXA33TNF2G64TZHM2DKMRQGEZDAMBUHNEXG43VMU5TINZSGY4TKMRUGI22C5QC . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS https://github.com/notifications/mobile/ios/AFSAKF3ATR75CITJGZGN4ZD5F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y and Android https://github.com/notifications/mobile/android/AFSAKFZNCHTWKNPHJINWMDL5F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI. Download it today! You are receiving this because you were mentioned.Message ID: @.>

@stevemunch, thanks for raising these concerns! The Eigen ldlt method is the robust version of the Cholesky, it returns a solution even if A isn't positive definite. If A is singular or semi-positive definite, the ldlt method defaults to computing the least squares solution (Eigen documentation: Eigen::LDLT, solve). That being said, I'm not an expert by any means on EDMs, so I don't know if we will run into issues using ldlt in the context of these models. Is there a way to test an ill-formed use case with the ldlt method to see if it holds up?

stevemunch · 2026-07-21T22:04:11Z

Yes - just crank the locally parameter (theta or phi depending on naming convention) to something really large / really small with some artificial data where several rows of inputs are identical or nearly so.Sent from my iPadOn Jul 21, 2026, at 11:55 AM, Andrea-Havron-NOAA ***@***.***> wrote:Andrea-Havron-NOAA left a comment (NOAA-FIMS/FIMS#1561) Folks- Just a quick 2cents - While the matrices we're inverting are theoretically postive definite, they do become singular for extreme values of the local weighting parameters. So, although Cholesky is preferred for things that must be symmetric and p.d., we'll want to be sure we have some way to handle p.s.d matrices as well. Cheers, Steve … On Tue, Jul 21, 2026 at 10:51 AM Aman Raj @.> wrote: aman-raj-srivastva left a comment (NOAA-FIMS/FIMS#1561) <#1561 (comment)> Thank you all for the thorough review and the feedback. I see what you’re suggesting about architectural adjustments to help integrate the EDM implementation into the FIMS codebase. I’ll begin working through these updates and will put these changes into logical commits as I tackle each piece. Thanks again for the guidance! — Reply to this email directly, view it on GitHub <#1561?email_source=notifications&email_token=AFSAKFYMINKQ2MROW2EJXW35F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5037258824>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFSAKF7C55UVUS4WHE4TU2D5F6URLAVCNFSNUABFKJSXA33TNF2G64TZHM2DKMRQGEZDAMBUHNEXG43VMU5TINZSGY4TKMRUGI22C5QC . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS https://github.com/notifications/mobile/ios/AFSAKF3ATR75CITJGZGN4ZD5F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y and Android https://github.com/notifications/mobile/android/AFSAKFZNCHTWKNPHJINWMDL5F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI. Download it today! You are receiving this because you were mentioned.Message ID: @.> @stevemunch, thanks for raising these concerns! The Eigen ldlt method is the robust version of the Cholesky, it returns a solution even if A isn't positive definite. If A is singular or semi-positive definite, the ldlt method defaults to computing the least squares solution (Eigen documentation: Eigen::LDLT, solve). That being said, I'm not an expert by any means on EDMs, so I don't know if we will run into issues using ldlt in the context of these models. Is there a way to test an ill-formed use case with the ldlt method to see if it holds up? —Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

stevemunch · 2026-07-21T22:17:01Z

Oh yeah - when trying this for the GP, we’d also want to make sure that the process variance (Ve ?) is set close to zero. If it is large enough, it acts as a regularizer and hides / eliminates the numerical issue.Sent from my iPadOn Jul 21, 2026, at 3:04 PM, Stephan Munch ***@***.***> wrote:Yes - just crank the locally parameter (theta or phi depending on naming convention) to something really large / really small with some artificial data where several rows of inputs are identical or nearly so.Sent from my iPadOn Jul 21, 2026, at 11:55 AM, Andrea-Havron-NOAA ***@***.***> wrote:Andrea-Havron-NOAA left a comment (NOAA-FIMS/FIMS#1561) Folks- Just a quick 2cents - While the matrices we're inverting are theoretically postive definite, they do become singular for extreme values of the local weighting parameters. So, although Cholesky is preferred for things that must be symmetric and p.d., we'll want to be sure we have some way to handle p.s.d matrices as well. Cheers, Steve … On Tue, Jul 21, 2026 at 10:51 AM Aman Raj @.> wrote: aman-raj-srivastva left a comment (NOAA-FIMS/FIMS#1561) <#1561 (comment)> Thank you all for the thorough review and the feedback. I see what you’re suggesting about architectural adjustments to help integrate the EDM implementation into the FIMS codebase. I’ll begin working through these updates and will put these changes into logical commits as I tackle each piece. Thanks again for the guidance! — Reply to this email directly, view it on GitHub <#1561?email_source=notifications&email_token=AFSAKFYMINKQ2MROW2EJXW35F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5037258824>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFSAKF7C55UVUS4WHE4TU2D5F6URLAVCNFSNUABFKJSXA33TNF2G64TZHM2DKMRQGEZDAMBUHNEXG43VMU5TINZSGY4TKMRUGI22C5QC . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS https://github.com/notifications/mobile/ios/AFSAKF3ATR75CITJGZGN4ZD5F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y and Android https://github.com/notifications/mobile/android/AFSAKFZNCHTWKNPHJINWMDL5F6URLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBTG4ZDKOBYGI2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI. Download it today! You are receiving this because you were mentioned.Message ID: @.> @stevemunch, thanks for raising these concerns! The Eigen ldlt method is the robust version of the Cholesky, it returns a solution even if A isn't positive definite. If A is singular or semi-positive definite, the ldlt method defaults to computing the least squares solution (Eigen documentation: Eigen::LDLT, solve). That being said, I'm not an expert by any means on EDMs, so I don't know if we will run into issues using ldlt in the context of these models. Is there a way to test an ill-formed use case with the ldlt method to see if it holds up? —Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

…ims_math usage - Replace GaussianElimination with Eigen::LDLT Cholesky factorization in GPEdmProjection (both predict_one and compute_log_posterior_grad) and SMapProjection - Update GP log-marginal likelihood gradient solve, inverse covariance matrix (iKVs), and log-determinant calculation to use Eigen::LDLT - Add fims_math::sign() using CppAD::sign() under TMB_MODEL for AD tape differentiability and replace custom sign_d() helper - Standardize math functions (std::exp, std::log, std::sqrt, std::abs) to use fims_math:: across EDM functors - Add GoogleTest test cases for ill-conditioned systems (extreme theta = 1000 and near-zero nugget ve = 1e-6) to verify LDLT least-squares fallback

…ounds in MVNormLPDF and BetaLPDF

…st setup

Aman and others added 21 commits June 9, 2026 21:59

Add EDM delay embedding utility

8c6238b

Added R unit tests

68622fa

docs: restore Rcpp_exports.R and add DelayEmbedding documentation

dd00e76

6fc6a8f

Merge branch 'main' into feature/edm-distribution-class

34b82a3

refactor(edm): tighten E and tau validation checks to reject non-inte…

ac69d68

…gers

refactor(edm): store pointer-based target values and rename fields pe…

5bf6caf

…r review feedback

feat(edm): add uncertainty propagation to delay embedding

41aa78d

Merge branch 'main' into feature/edm-distribution-class

8612b7a

docs(edm): fix malformed doxygen comment in rcpp_edm.hpp

1442fc1

chore: add 'embs' to spelling check wordlist

51fc981

fix(ci): use dd instead of fallocate for manual swap creation

c41ca1a

fix(ci): deactivate active swap before creating swapfile with dd

896167e

fix(ci): increase build-metrics peak memory regression threshold to 25%

6a5994a

Merge remote-tracking branch 'upstream/main' into feature/edm-distrib…

f378d3e

…ution-class

Add EDM delay embedding infrastructure and tests (NOAA-FIMS#1528)

d40a9ef

feat(edm): add generic EDM predictor base and Simplex Projection algo…

1579d1a

…rithm

feat(edm): add modular distance weighting infrastructure

3b443a3

aman-raj-srivastva force-pushed the feature/edm-prediction-functors branch from 80c0d37 to 3b443a3 Compare June 24, 2026 18:02

aman-raj-srivastva added 4 commits June 27, 2026 20:43

fix: move forecast_horizon to end of signature to preserve Rcpp binar…

3374bce

…y compatibility

test(edm): update Rcpp EDM tests for forecast_horizon = 1

5492911

fix(edm): update create_edm_embedding() to use 'fleet' column after F…

55cf20c

…IMSFrame rebase rename

Andrea-Havron-NOAA reviewed Jul 20, 2026

View reviewed changes

Andrea-Havron-NOAA reviewed Jul 21, 2026

View reviewed changes

aman-raj-srivastva force-pushed the feature/edm-prediction-functors branch 2 times, most recently from 3c243dc to be959a2 Compare July 25, 2026 16:42

aman-raj-srivastva force-pushed the feature/edm-prediction-functors branch from be959a2 to c8b7161 Compare July 25, 2026 17:05

aman-raj-srivastva added 5 commits July 26, 2026 15:54

feat(distributions): add MVNormLPDF and BetaLPDF distribution functors

bca9cb6

refactor(edm): remove unused GaussianElimination routine

fdf91c8

fix(math): add fims_math::sign to STD_LIB section for gtest builds

884d16c

fix(distributions): resolve n_x size determination and get_expected b…

e79dfe7

…ounds in MVNormLPDF and BetaLPDF

fix(distributions): resolve multivariate prior vector indexing and te…

49ce4fd

…st setup

mollystevens-noaa merged commit bab1554 into NOAA-FIMS:main-edm Jul 28, 2026
1 check passed

This was referenced Jul 30, 2026

Implement EDM Model Families, R User Workflows, and Vignette Documentation #1677

Open

[WIP] feat(edm): implement EDM model families, R-level user workflows, and documentation #1679

Open

Uh oh!

Conversation

aman-raj-srivastva commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What this PR adds

Generic EDM Prediction Framework

Simplex Projection

Distance Weighting Infrastructure

Standard S-Map

GP-EDM

Rcpp Interfaces

Validation

Testing

Future Work

Related Issues

Uh oh!

aman-raj-srivastva commented Jun 24, 2026

Uh oh!

aman-raj-srivastva commented Jun 24, 2026

1. Weighting Kernel / Distance Metric

2. Target Alignment in Delay Embedding

3. Scope of This PR

Uh oh!

stevemunch commented Jun 24, 2026 via email

Uh oh!

nathanvaughan-NOAA commented Jun 24, 2026

Uh oh!

mollystevens-noaa commented Jul 14, 2026

Uh oh!

mollystevens-noaa commented Jul 20, 2026

Uh oh!

nathanvaughan-NOAA commented Jul 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andrea-Havron-NOAA Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andrea-Havron-NOAA commented Jul 21, 2026

Uh oh!

aman-raj-srivastva commented Jul 21, 2026

Uh oh!

stevemunch commented Jul 21, 2026 via email

Uh oh!

Andrea-Havron-NOAA commented Jul 21, 2026

Uh oh!

stevemunch commented Jul 21, 2026 via email

Uh oh!

stevemunch commented Jul 21, 2026 via email

Uh oh!

aman-raj-srivastva commented Jun 23, 2026 •

edited

Loading

Andrea-Havron-NOAA Jul 20, 2026 •

edited

Loading