I am finetuning MACE-OMol with a custom dataset of DFT data on the same level of theory (ωB97M-V/def2-TZVPD) as OMol25. I am experiencing extremely high initial loss, RMSE_E and RMSE_F values (see the log). Additionally, RMSE_F value is fixed throughout the training to the same value.
I assumed there was a problem with the data not being loaded correctly, but from the log it seems that all energies, forces, total charges and total spins are read in by the model.
With the learning rate 0.01, the loss and RMSE_E drop to lower values immediately in Epoch 0. With a lower learning rate, the high values prevail across a couple of epochs. When run with a different seed, the initial loss is a different number, but same order of magnitude. I am using E0s: "foundation" because my data is on the same level of theory, but I also manually checked that the sum of E0s corresponds to the calculated energy. I tried to set test_foundation_filter_elements and test_foundation_model_elements to True, but it did not solve the problem. I also tested rms_forces_scaling instead of no_scaling, again with no success.
This is my first try of finetuning a MACE foundation model, so I would appreciate if anyone could help me debug this issue. I am attaching the log (I didn't let the training process finish, though, due to the high initial values), the config file and an extract from the XYZ file of the train set as an example of its format. I use mace-torch version 0.3.16.
Thanks!
config.yml
mace_omol_mel_run-123.log
xyz file example:
35
Properties=species:S:1:pos:R:3:REF_forces:R:3 REF_energy=-33054.86047582431 index=025c0e2e028e3d216254521229c25d79944a074ab297b584 total_charge=0 total_spin=0 pbc="F F F"
O 3.86163500 0.96408000 2.81672800 0.00396546 0.01474080 0.01416616
C 3.32214800 0.49167500 1.84614500 0.03817050 -0.02759437 -0.01147581
C 3.99654200 -0.52165600 0.99815400 -0.01106433 0.03753477 0.03405550
C 3.33928400 -1.02752600 -0.04030500 -0.04088425 0.02029084 0.00548694
N 3.79466100 -2.00108100 -0.96150700 0.04425080 -0.01308558 0.01014763
C 2.81212000 -2.15301900 -1.77756400 0.05158909 -0.01392504 -0.03540055
C 1.63992800 -1.30869900 -1.47881100 -0.04859858 0.01370450 -0.05174304
C 1.97207500 -0.60421100 -0.38329500 -0.05148357 0.04313365 0.05591317
C 1.27537500 0.39702900 0.40804500 0.05017246 0.07462936 0.04703782
C -0.10632800 0.76665100 0.07797000 -0.03833489 -0.05074607 -0.02304331
C -0.62601700 1.97928600 -0.22797400 -0.10650780 -0.01670430 -0.06080839
C 0.09011800 3.24385800 -0.47342500 0.02669191 0.06354215 -0.00083551
O 1.24746100 3.44885400 -0.23350600 -0.04082876 -0.03105039 0.01658346
O -0.73179800 4.15979600 -1.03265300 0.02177128 0.00330577 0.00289568
N -2.01073100 1.90109900 -0.43812400 0.10482532 0.07320744 -0.03597966
C -2.42434100 0.60433100 -0.34488900 0.02449254 -0.10586477 0.07776312
C -1.23086400 -0.17669600 0.01647500 -0.02411273 0.02582051 -0.02348317
C -1.30498700 -1.50019500 0.18043500 -0.01799824 -0.04567920 -0.02258647
C -2.64028900 -2.15806900 0.13084500 0.05043353 -0.00009199 -0.02486601
O -2.56213200 -3.45969200 -0.37671900 0.00146692 -0.00411212 0.04716514
C -3.76232300 -1.36675100 -0.55117700 0.04054208 0.04113801 -0.01883431
O -4.70053600 -1.99984400 -1.00190400 -0.06849522 -0.05982716 -0.01888424
C -3.64547700 0.07254000 -0.58632100 -0.03334798 -0.00198823 0.02880865
C 1.92186300 0.92192500 1.47186000 0.01733669 -0.08942863 -0.05819245
O 1.42500700 1.82678500 2.31999400 0.04118378 -0.02081946 -0.01516653
H 5.00339800 -0.80993900 1.26789500 -0.01468213 -0.00162447 0.00665083
H 2.87531700 -2.84783400 -2.60598700 0.00207982 -0.01351146 -0.00105677
H 0.72033600 -1.28461600 -2.04162600 -0.01798939 0.00375998 -0.00747713
H -0.20221600 4.95431700 -1.19746400 -0.00342702 0.01333220 0.02082187
H -2.55782900 2.66203800 -0.80006300 0.02431076 0.01075343 0.00474394
H -0.44074800 -2.11711000 0.39326600 -0.00435581 0.00895371 0.03457064
H -2.97832400 -2.17529100 1.18754400 -0.00573654 0.02362227 0.05948057
H -3.44623300 -3.64816500 -0.73269000 0.00646360 0.03371457 -0.02240825
H -4.47989600 0.66032200 -0.94153800 -0.01960785 -0.01033090 -0.04932358
H 0.55379800 2.10581000 2.00883900 -0.00229137 0.00120024 0.01527415
35
Properties=species:S:1:pos:R:3:REF_forces:R:3 REF_energy=-33055.72278770506 index=57a52826c6305265c1706d31ca51cc5308f2d2261c5f0380 total_charge=0 total_spin=0 pbc="F F F"
O -1.26193800 4.20767100 0.00195300 0.02794018 -0.01856342 -0.00726234
C -0.70112800 3.18905500 -0.29309100 -0.04917533 -0.00203153 0.03163912
O 0.50038700 3.15447900 -0.90027300 -0.00342394 -0.00030437 0.00411932
C -1.24728200 1.83125400 -0.04371700 -0.12819645 0.11866429 -0.04211858
C -0.58284900 0.66996300 0.10948300 0.08090398 -0.08725209 -0.00010603
C 0.85281000 0.35577900 0.08104700 -0.00910829 0.03448960 -0.00208496
C 1.74075600 0.98110100 0.93177100 -0.03549434 0.00079977 -0.01010552
O 1.29188000 1.92953300 1.80162800 -0.00020975 -0.02696558 0.01772226
C 3.11341600 0.65497000 0.90999200 0.01814422 -0.01622968 -0.00581599
O 3.87087200 1.35274700 1.81980500 0.03355655 0.00885550 -0.00484823
C 3.62773000 -0.28121500 0.04634200 -0.03291922 0.02027922 -0.00506898
C 2.72699200 -0.90366600 -0.82032200 -0.00785508 0.00625493 -0.00240650
N 2.94740400 -1.83702300 -1.80550300 -0.01094781 -0.02706786 -0.00238429
C 1.75176700 -2.12896800 -2.42104600 -0.00146651 -0.00605593 0.01169657
C 0.75476000 -1.39950000 -1.85194800 -0.00818737 -0.00593879 0.01138186
C 1.35656600 -0.60760200 -0.81606900 0.00521646 0.01604883 0.00120914
C -1.63797900 -0.31386200 0.37653900 0.03443520 -0.00539320 0.01309890
C -1.58201900 -1.61740700 0.65792400 -0.05989015 0.00662285 -0.00807902
C -2.82742700 -2.37673300 0.91600600 0.12245080 0.01266294 -0.04608255
O -2.80793700 -3.54781300 1.18657100 -0.03773326 -0.04660315 0.02777404
C -4.19599400 -1.63549100 0.81486600 -0.05863633 -0.05985236 0.01003558
O -5.19875600 -2.27398500 0.96965900 0.04474913 0.03817081 -0.00280399
C -4.23325000 -0.15075700 0.52178500 -0.02029608 0.01301050 -0.01814114
C -2.89203600 0.46791900 0.32476000 0.02739475 -0.07736069 0.02852171
N -2.66873900 1.70314300 0.08868000 0.10923950 0.05901736 0.02085031
H 0.79085100 4.07551700 -0.97362800 0.00295353 0.00758409 -0.01459878
H 2.03644000 2.23622800 2.33454500 0.00946073 -0.02251202 -0.03003820
H 4.79974900 1.12251500 1.72107400 -0.02872658 0.00504764 0.02092811
H 4.68685400 -0.51313600 0.03600500 -0.01128308 0.01238562 0.02474713
H 3.83478600 -2.24629800 -2.03002400 -0.01651152 -0.00906196 -0.02560217
H 1.70414200 -2.84046000 -3.22822500 0.03718597 -0.00274944 0.00612900
H -0.28216600 -1.41563500 -2.14248500 0.00478400 0.00081386 -0.01084409
H -0.65052200 -2.17130500 0.70151700 -0.01209761 0.02014470 0.01269421
H -4.85063100 -0.00551200 -0.36798000 -0.03221922 0.00870689 0.01373535
H -4.76750700 0.33449400 1.34236100 0.00596285 0.02438265 -0.01789128
I am finetuning MACE-OMol with a custom dataset of DFT data on the same level of theory (ωB97M-V/def2-TZVPD) as OMol25. I am experiencing extremely high initial loss, RMSE_E and RMSE_F values (see the log). Additionally, RMSE_F value is fixed throughout the training to the same value.
I assumed there was a problem with the data not being loaded correctly, but from the log it seems that all energies, forces, total charges and total spins are read in by the model.
With the learning rate 0.01, the loss and RMSE_E drop to lower values immediately in Epoch 0. With a lower learning rate, the high values prevail across a couple of epochs. When run with a different seed, the initial loss is a different number, but same order of magnitude. I am using
E0s: "foundation"because my data is on the same level of theory, but I also manually checked that the sum of E0s corresponds to the calculated energy. I tried to settest_foundation_filter_elementsandtest_foundation_model_elementstoTrue, but it did not solve the problem. I also testedrms_forces_scalinginstead ofno_scaling, again with no success.This is my first try of finetuning a MACE foundation model, so I would appreciate if anyone could help me debug this issue. I am attaching the log (I didn't let the training process finish, though, due to the high initial values), the config file and an extract from the XYZ file of the train set as an example of its format. I use
mace-torchversion 0.3.16.Thanks!
config.yml
mace_omol_mel_run-123.log
xyz file example: