Skip to content

cross_validate_nmf giving NAN tolerance and overfit #49

Description

@cindyfang70

Hello singlet team!

First off, thank you so much for creating this package. This and RcppML have really sped up my workflow and made my life much easier!

I have been using cross_validate_nmf to decide the number of ranks for NMF on my datasets. However, I am occasionally running into this error:

> singlet::cross_validate_nmf(A, ranks = ranks,
                                    n_replicates = 3,
                                    verbose=3)
running with dense optimization
k = 50, rep = 1 (1/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.67e-01 | 0.00e+00
   2 | 5.52e-02 |        -
   3 | 2.07e-02 |        -
   4 | 1.12e-02 |        -
   5 | 7.07e-03 |        -
   6 | 4.92e-03 | 0.00e+00
   7 | 3.63e-03 |        -
   8 | 2.79e-03 |        -
   9 | 2.23e-03 |        -
  10 | 1.84e-03 |        -
  11 | 1.57e-03 | 1.12e-03
test set error: 6.3858e-02

k = 100, rep = 1 (2/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.56e-01 | 0.00e+00
   2 | 6.45e-02 |        -
   3 | 2.49e-02 |        -
   4 | 1.44e-02 |        -
   5 | 9.87e-03 |        -
   6 | 7.48e-03 | 0.00e+00
   7 | 6.10e-03 |        -
   8 | 5.27e-03 |        -
   9 | 4.96e-03 |        -
  10 | 5.14e-03 |        -
  11 | 5.65e-03 | 1.96e-02
test set error: 6.7135e-02

overfitting detected, lower rank recommended
k = 200, rep = 1 (3/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.40e-01 | 0.00e+00
   2 | 8.00e-02 |        -
   3 | 2.89e-02 |        -
   4 | 1.73e-02 |        -
   5 | 1.27e-02 |        -
   6 | 1.08e-02 | 1.29e-02
test set error: 6.6980e-02

overfitting detected, lower rank recommended
k = 50, rep = 2 (4/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.65e-01 | 0.00e+00
   2 | 5.76e-02 |        -
   3 | 2.10e-02 |        -
   4 | 1.09e-02 |        -
   5 | 6.78e-03 |        -
   6 | 4.75e-03 | 0.00e+00
   7 | 3.60e-03 |        -
   8 | 2.82e-03 |        -
   9 | 2.29e-03 |        -
  10 | 1.92e-03 |        -
  11 | 1.63e-03 | 1.24e-03
test set error: 6.4045e-02

k = 100, rep = 2 (5/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.54e-01 | 0.00e+00
   2 | 6.63e-02 |        -
   3 | 2.49e-02 |        -
   4 | 1.41e-02 |        -
   5 | 9.59e-03 |        -
   6 | 7.17e-03 | 0.00e+00
   7 | 5.71e-03 |        -
   8 | 4.86e-03 |        -
   9 | 4.52e-03 |        -
  10 | 4.29e-03 |        -
  11 | 4.19e-03 | 1.02e-02
test set error: 6.6021e-02

overfitting detected, lower rank recommended
k = 200, rep = 2 (6/9):

iter |      tol |  overfit 
---------------------------
   1 |      nan |      nan
test set error: NaN

Error in if (model$test_mse[[length(model$test_mse)]]/model$test_mse[[1]] >  : 
  missing value where TRUE/FALSE needed

Is there any explanation for why the tolerance and overfit are nan here? I am on singlet_0.99.38, RcppML_0.5.6, and RcppEigen_0.3.3.9.4. The replicate on which the error occurs also changes between runs.

Thank you so much!

PS: another issue I have run into is having to manually set options(RcppML.threads = 0) when I am using RcppML and singlet on Mac M2. On a linux system, this seems to be set automatically when I call library(RcppML).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions