Skip to content

Fix DFTB+ Hessian parsing and harden coordinate/frequency readers - #47

Merged
galjos merged 3 commits into
mainfrom
fix-dftbplus-hessian-parsing
Jun 25, 2026
Merged

Fix DFTB+ Hessian parsing and harden coordinate/frequency readers#47
galjos merged 3 commits into
mainfrom
fix-dftbplus-hessian-parsing

Conversation

@galjos

@galjos galjos commented Jun 24, 2026

Copy link
Copy Markdown
Member

Follow-up to #46 (shared parsing via PQAnalysis). Verifying that refactor surfaced one critical regression plus a couple of lower-severity issues, fixed here with regression coverage.

Critical: Hessian reading was broken on real DFTB+ output

DFTB+ writes hessian.out as a flat stream of second derivatives wrapped at a fixed number of values per line (with a ragged final line per matrix row), not as a square grid. The refactor routed it through a reader that requires an already-square numpy.loadtxt result, so it raised on every real file:

  • uniform-width output loaded as (n, 4) and failed the square check;
  • ragged output made numpy.loadtxt itself fail.

This is invisible to CI because the integration tests need the dftb+/modes binaries and are skipped, while the unit test fed a synthetic square grid DFTB+ never emits. Restored flat-token parsing that reads every value and reshapes to (3N, 3N), keeping the explicit size validation. Verified bit-for-bit against real 72x72 and 78x78 Hessians.

xyz coordinate precision

read_xyz routed coordinates through XYZFrameReader, which stores positions as float32, perturbing every value by ~1e-7 relative to the source and to read_gen (float64). Now reads the coordinate columns in float64 while keeping XYZFrameReader for validation, atom count, symbols and cell detection.

Smaller fixes

  • read_xyz no longer masks a missing file as "Invalid XYZ coordinate file."; it raises FileNotFoundError again.
  • Modes.read stops at a trailing tag section (e.g. saved_modes :integer:..) instead of casting the section header to float and crashing.

Test coverage

  • Regression test for the Hessian reader using the authentic ragged wrapped layout (asserts numpy.loadtxt cannot parse it, so the fixture can't be reduced to a square grid).
  • Small fixtures (tests/data/calculator/modes/) plus an integration test that drives Hessian.read -> Modes -> wave numbers end to end with only the modes binary mocked, closing the no-binary gap.
  • float64-precision and missing-file tests for read_xyz; a trailing-section test for Modes.read.

134 passed, 5 skipped (the 5 still need the DFTB+ binaries); pylint 7.94/10.

galjos added 3 commits June 24, 2026 16:44
DFTB+ writes hessian.out as a flat stream of second derivatives wrapped at
a fixed number of values per line (with a ragged final line per matrix row),
not as a square grid. Routing it through PQAnalysis.read_hessian_file (which
requires an already-square np.loadtxt result) raised on every real file:
uniform-width output loaded as (n, 4) and failed the square check, and ragged
output made np.loadtxt itself fail. Restore flat-token parsing that reads all
values and reshapes to (3N, 3N), keeping the explicit size validation. Add a
regression test using the real ragged wrapped layout.

read_xyz routed coordinates through XYZFrameReader, which stores positions as
float32, perturbing every value by ~1e-7 relative to the source and to
read_gen (which stays float64). Read the coordinate columns in float64 while
keeping XYZFrameReader for validation, atom count, symbols, and cell detection.

read_xyz also masked a missing file as 'Invalid XYZ coordinate file.'; move the
file read out of the parse guard so a missing file raises FileNotFoundError.
The Hessian.read -> Modes -> wave-number path was only exercised by the
integration tests that require the dftb+/modes executables, so it was skipped
in CI. Add small fixtures that reproduce the authentic DFTB+ file layouts (a
wrapped, ragged hessian.out, a frequencies-only vibrations.tag, and a matching
geo_opt.gen) and an integration test that mocks only the modes binary and
drives Modes through write -> calculate -> read end to end, asserting the parsed
wave numbers and that Hessian.read parses the same ragged layout. The test
guards the fixture against being reduced to a square grid by asserting
numpy.loadtxt cannot parse it.
Modes.read flattened every line after the header and cast all of it to float,
so a vibrations.tag that carries further tag sections after the frequencies
(e.g. 'saved_modes :integer:..') crashed on the section header. Read frequency
values line by line and stop at the first non-numeric line instead.
@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.63%. Comparing base (505074e) to head (65a9a26).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #47      +/-   ##
==========================================
+ Coverage   93.61%   93.63%   +0.01%     
==========================================
  Files          21       21              
  Lines        1191     1194       +3     
==========================================
+ Hits         1115     1118       +3     
  Misses         76       76              
Flag Coverage Δ
unittests 93.63% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@galjos

galjos commented Jun 24, 2026

Copy link
Copy Markdown
Member Author

Validated end to end against the real DFTB+ binaries (DFTB+ 24.1 + MODES 0.03, 3ob-3-1 Slater-Koster set), not just mocks.

Full suite with the binaries enabled: 139 passed, 0 skipped (the five integration tests that normally skip without dftb+/modes now run: real geometry optimization, real second-derivatives Hessian, real modes frequencies, and the full thermo pipeline).

Concrete evidence on freshly generated output for water:

  • The real hessian.out is the wrapped/ragged layout (line widths {1, 4}, 81 values -> 9x9). The reader in this PR parses it; the pre-fix square-grid reader raises ValueError: the number of columns changed from 4 to 1 at row 3 on that exact file — i.e. the regression reproduces on genuine DFTB+ output, and the fix resolves it.
  • The real vibrations.tag is frequencies-only (single :real: section, no saved_modes), matching the committed fixture; the trailing-section handling in Modes.read is therefore defensive rather than load-bearing for the default flow.
  • Resulting water frequencies are physically sensible (bend ~1462, stretches ~3604 / ~3876 cm^-1).

@galjos
galjos merged commit 9a490af into main Jun 25, 2026
3 checks passed
@galjos
galjos deleted the fix-dftbplus-hessian-parsing branch June 25, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant