Skip to content

NEWS.md

Latest commit

 

History

History
1643 lines (1208 loc) · 71.7 KB

NEWS.md

File metadata and controls

1643 lines (1208 loc) · 71.7 KB
Raw

News for 3.22.0 (July 23, 2026)

BES Updates

Security

This release includes changes that resolve CVE-2026-16637

DMR++ Improvements

  • Direct chunk IO: Added and extended a zero-copy chunk-read path for DMR++ subset/whole-variable requests — handling filled chunks, chunks larger than the array, and unlimited dimensions (including ones defined in ancestor groups) ( HYRAX-1950, 2014, 2016, 2029, 2030, 2033, 2036, 2144, 2168).
  • Buffer chunk: added a "buffer chunk" mode that reads/holds a larger chunk buffer to speed up direct IO, including whole-array support, a UseBufferChunk BES key, constraint/subset tests, and a compression-ratio/shuffle-status test macro (HYRAX-1957, 1966, 1967, 2032).
  • Superchunk: added support for superchunks to cover subsets of non-contiguous chunks (HYRAX-1641).
  • Non-contiguous-chunk read optimization: introduced a simpler, stride-aware method (find_needed_chunks_simple) to determine which chunks a subset request actually needs, with a reverted micro-optimization that didn't improve performance (HYRAX-1579, 2021, 2028).
  • Fixed-size and variable-length string array handling in dmrpp/build_dmrpp: buffer-overflow fix on ingest, subsetting support, superchunk-queue refactor, revised size calculation for max response size, 1-character string array mapping ( HYRAX-2198, 2200, 2204, 2205, 2210, 2193, 2191, 1225).
  • Vlen (variable-length) data support built into dmrpp: reading HDF5 vlen float/int arrays, dimension naming, and an option to store vlen data in a separate side-car file (HYRAX-1654, 1655, 1658, 1666, 2050).
  • Clarified Container::access() vs. the new alt_access() path used specifically for DMR++ retrieval (HYRAX-1670)
  • Removed transfer-thread concurrency from DmrppArray (HYRAX-1965).
  • Refactored the DMZ parser in the DMR++ module, removing dead code (HYRAX-1988).
  • Retired DmrppSaxParser2 (GH #1087); reformatted DmrppArray/SuperChunk sources for clang-format conformance (GH #1208).
  • Disabled DMR++ transfer threads by default after a threading bug; HYRAX-1882: fixed a memory leak from an undeleted RETURNAS_DMRPP transmitter (HYRAX-1883).
  • Fixed a dmrpp offset/length test failure caused by a mismatched test file (HYRAX-2039).
  • Added a configurable variable_sleep_interval to the dmrpp request handler.
  • .dmrpp extension detection in NgapApi; return the DMR++ via get type=dap&returnAs=dmrpp (bes#1103); build_dmrpp metadata (version/invocation/creation time) added to sidecar output by default; Unlimited_Dimensions attribute generation turned off by default (togglable via BES key, HYRAX-1771).
  • Added unlimited-dimension support to dmrpp/fileout netCDF; added byteOrder for data-less variables and made dmrpp return empty data for zero-size arrays (HYRAX-1741, 1742, HYRAX-1743).
  • Improved dmrpp generation time for many-chunk files; fixed a vlen dimension-handling bug found via valgrind ( HYRAX-1683/1652).
  • Added build_dmrpp_metadata (version/invocation/creation time) to build_dmrpp_h4 (HYRAX-1622/1623/1551);
  • Added a template string URL for sidecar files (HYRAX-1638).
  • Fixed build_dmrpp_util to correctly generate files for ECOSTRESS data (HYRAX-1947).
  • Adopted HDF5's new chunk-iteration API in build_dmrpp as part of the HDF5 1.14.6 upgrade (HYRAX-1938).
  • Fixed a double-URL-encoding bug and restored support for legacy OPeNDAP URLs in DMR++/dmr++ URI construction ( HYRAX-2148/2145).
  • Plumbed and cached S3 data-access URLs, injecting them into the DMR++ via a content filter (HYRAX-1732);
  • Presigned the DMR++ URL (HYRAX-1733).
  • Corrected error messages, including for HDF5's DisableSideCar option (HYRAX-2187).
  • Added unlimited-dimension support to the direct-IO feature (HYRAX-1232).
  • Made build_dmrpp_h4 always save the EOS projection dummy variable value (HYRAX-2043).

Handler Updates

(HDF4, HDF5, fileout netCDF, NcML, gdal, and related modules)

  • Upgraded the HDF5 handler to compile against HDF5 1.14.6, with baseline/test updates for the version change ( HYRAX-1913/1938).
  • Updated the HDF5 handler in step with libdap4's DAP4 attribute-find fix (HYRAX-2176).
  • Reduced code complexity in EOS5 dimension-name handling and DMR generation (Hyrax 2156/1577/2116)
  • Separated memory-heavy NASA tests
  • Added a config-attribute error for HDF4's EnableCF option.
  • Fixed AIRS CO2 product lat/lon by dropping reliance on the buggy HDF-EOS2 projection library; added COARDS-convention support for HDF-EOS2 Grid geo projections (HYRAX-1916).
  • Fixed HDF-EOS5 dimension-size handling for unlimited dimensions; removed a misused nc_redef() call in fileout netCDF (HYRAX-1892/1837).
  • Fixed multi-element _FillValue handling and HDF-EOS2 Grid lat/lon parameter issues (HYRAX-1884/1881).
  • Adapted to a netCDF _FillValue macro rename (HYRAX-1896).
  • Improved netCDF string-write performance (single write vs. many small writes); increased chunk cache for large string arrays (HYRAX-1697).
  • Added variable-length float/int array support in the HDF5 handler, including cross-group compound-type tests (which surfaced a netCDF-C client bug) HYRAX-1667/1695.
  • vlen float/int data support in the HDF5 handler and fileout netCDF, plus CF-convention dimension/group-naming fixes ( HYRAX-1654/1655/1658/1666).
  • Filled empty SDS datasets with _FillValue (HYRAX-1357).
  • Used the HDF5 API to detect HDF5 files (HYRAX-1656).
  • Handled groups with special characters in fileout netCDF (HYRAX-1659).
  • Added MCD43GF dimension-scale support, corrected lat/lon ranges (HYRAX-1634).
  • Reworked HDF4/HDF5 handler test suites for easier baseline generation (HYRAX-1643/2004).
  • Surveyed GDAL-based features and fixed NcML integration tests to run without GDAL (HYRAX-1961).
  • Switched to compress2/uncompress2 to handle large missing-data buffers (SWOT Level 4) (HYRAX-1994).
  • Removed unnecessary pugixml.hpp includes (HYRAX-2069).
  • Added BES keys to tune/disable floating-point data compression in fileout netCDF for large arrays (HYRAX-2182/1956).
  • Used the two-arch test-compare approach to work around a "minus zero" comparison issue seen with gdal.
  • Removed the retired w10n_handler and its build rules.
  • Fixed memory leaks from un-deleted transmitters, including in the w10n module (HYRAX-1902).

DAP4

  • Added end-to-end DAP4 Enum support (D4EnumDef, scalar/array read, group handling) (HYRAX-1488/1489/1490/1707).
  • Optional DAP4 checksums: made checksum inclusion/computation optional via D4StreamMarshaller, with test-baseline updates across nearly every handler module to keep using checksums by default.
  • Added an error message for when a DAP4 dimension size doesn't match the array dimension size (HYRAX-1242).
  • Fixed DAP4-map coordinate retrieval order; fixed null/missing dimension names for HDF-EOS5 files with dimension scales (HYRAX-1886/1880).
  • Handled DMR generation for variables with two unlimited dimensions (previously an error) (HYRAX-1993).
  • Hardwired dmrVersion to 1.0 for compatibility.
  • Checksum-flag (getdap4 -C) and DAP2-vs-DAP4 bescmd test fixes (HYRAX-1817/1544).
  • Removed a redundant string-datatype check (HYRAX-1795).
  • Fixed the AT_BESCMD_BINARY_DAP4_RESPONSE_TEST macro for OSX Intel/Apple Silicon (HYRAX-2009).
  • Mapped 64-bit acos/oco2 integer variables to the DAP4 64-bit integer type (HYRAX-1762).
  • Asciival module: Correctly throws/errors when constrained requests encounter DAP4-typed variables/attributes instead of mishandling them (HYRAX-2166).
  • Reworked dimension-name lookup to use the DAP4 group pointer instead of the HDF5 API in more cases (HYRAX-2090).

General

(CI/CD, dependencies, refactors, code organization, cross-platform/test infrastructure, ...)

  • CI/CD & Docker/RHEL9: dockerized the BES build end-to-end (HYRAX-1964, GH #1213); added RHEL9/el9 RPM production and Docker build stages (RHEL9, GH #1207, #1212); slimmed and pruned the Docker image; fixed deploy-on-merge, tag naming, and build-log-to-S3 steps; split Travis into stages; added a GHA macOS build workflow, then later added macOS 26 and dropped macos-15-intel/macos-intel; re-enabled/disabled GDAL across CI as needed (HYRAX-1968, 2012); removed --enable-developer from the bes_core Dockerfile (HYRAX-2077); removed RPM support entirely (HYRAX-2044); pushed make check/build logs to S3 (GH #1407, #1398); skipped CI runs on unrelated file changes; ran Travis on Ubuntu Jammy for Java 17.
  • Meyers singleton refactor: multi-month, systematic conversion of BES singleton classes (BESReturnManager, BESResponseHandlerList, BESRequestHandlerList, BESReporterList, BESInfoList, BESDefinitionsStorageList, BESContextManager, TheBESKeys, BESServiceRegistry, RequestServiceTimer, BESUncompressedCache, BESStoredDapResultCache, CredentialsManager, BESDapFunctionResponseCache, BESContainerStorage*, and more) to the thread-safe Meyers singleton pattern, closing with a fix for a related cache-locking bug (Hyrax-1774, 1803, 1809, 1812, 1845).
  • Error-handling refactor: replaced libdap's InternalErr with BESInternalError across handler code, improving error messages and exception-path resource cleanup (HYRAX-1418, 1800, 1362). Removed errors/warnings from the embedded gctp library; const-ified gctp() parameters (HYRAX-2076).
  • AWS/S3/CMR/credentials: replaced ad hoc libcurl S3 code with the AWS C++ SDK for GET/HEAD and presigned URLs ( HYRAX-1891, 1900, 1901); extracted and cached S3 credential URLs from CMR granules; added the EDL user id to cache keys in EffectiveURLCache/SignedURLCache (HYRAX-2177, 2178); added SignedUrlCache (HYRAX-1732); forwarded the EDL client_id to CMR (HYRAX-1888, 1898); refactored curl::http_get()/http_get_and_write_resource() to accept passed-in request headers instead of embedding EDL logic deep in the call tree; repaired AWS query-string filtering; improved STS credential-parse error messages (HYRAX-2169); skip (rather than fail) AWS tests when credentials are absent.
  • Dependency updates: upgraded to HDF5 1.14.6 then 2.1 (with linkage/rpath fixes); updated nlohmann JSON to 3.12.0; fixed shared-library-version linking (also fixing a GDAL shared-lib bug).
  • Warnings & code quality: multiple dedicated warnings-removal passes (HYRAX-1953, 2108, "warnings removal 2", "[hyrax 1895] clean autoreconf"), plus new clang-tidy modernization scripts.
  • Test infrastructure: large effort stripping the dmrVersion attribute out of test baselines for OS-agnostic, build-aware comparisons (BES-1223, 1219, 1225); added alternate (.m_proc) baselines for tests that fail on m4; improved mkchk failure reporting; added simple cleanup of old test data; updated baselines after getdap4 began printing full variable paths (Hyrax 1709).
  • Logging/profiling: added request_id/uuid ingestion and logging (GH #1071, #1063); added beslog2json.py; added profiling checkpoints across the request path (Hyrax-1835); migrated stopwatch calls to BES_STOPWATCH macros; suppressed cache hit/miss logging under NDEBUG.
  • Cleanup/consolidation: removed the last vestiges of HTTPCache (HYRAX-1184); completed removal of echo-token support; retired www-interface/builddmrpp_module; consolidated retired/unused code into a single retired/ directory; removed cmr_module tests during a CMR outage; TheBESKeys call-site refactor (GH #1305). Ongoing removal of legacy HDF5 API usage.
  • Build system: HYRAX-1921 configure.ac Makefile fix; HYRAX-1924 64-bit configure.ac OSX detection; improved cppunit-path detection in configure; HYRAX-2085 made make check work without make install first; fixed the distcheck target (Array::size() → length()); Rocky8/CentOS8 RPM packaging fix; first pass removing -test-deploy from version strings; fixed the max-compute-threads config option to enforce a non-zero value.
  • Docs: overhauled BES documentation, converting legacy Word files to Markdown and fixing content drift; added Dockerfile documentation to the README; added a default GitHub PR template.

News for 3.21.1

BES Updates

  • Streamlined BESLog i/o operator implementations.
  • Improved support for log requests for dap4 responses.
  • Fixed a bug that caused in some edge cases segmentation fault for the HDF4 handler.
  • correct the byte_per_element assignment for the fixed string handling
  • Initial support to handle the array of structure in chunking storage
  • Improved the evaluation URLs for S3 signing

DMR++ Improvements

  • Improved support to set the correct struct size to the chunk if that size can be found in the dmrpp.
  • dmrpp now reads the default fill value of a compound data type.
  • DmrppArray now returns int_64
  • DMRpp file can now represent multi-linked-blocks in a chunk.
  • Improved process to read multi-blocks chunk data and transfer of block offsets and lengths info to DMRppArray.
  • Initial support to store structure offset and structure size so that the dmrpp module can retrieve structure data.
  • Update gen_dmrpp_side_car by adding a new option for users to point to an existing sidecar file
  • New option to the gen_dmrpp_side_car python so that users can add the path to the HDF5 or the side car files.
  • Initial support of compressed compound datatype variables.
  • Now generates dmrpp files that include the CF grid variables.
  • Added new EOS2 CF grid attributes for build_dmrpp_h4 to add missing grid data.
  • Adds fake coord dimensions for NetCDF4 when missing dimension names and EnableCF option turned on.
  • Creation time is now added to the build_dmrpp metadata.
  • Now add DAP4 dimensions for the HDF4 SDS/HDF-EOS2 Grid variables. This is required to use the fileout netCDF to access HDF4 data via the HDF4 handler.
  • Includes an option to turn off the generation of the HDF4/HDF-EOS2 missing data in the dmrpp file
  • Added support to handle data subset for HDF-EOS2 lat/lon.
  • Added support for HDF-EOS2 latitude/longitude data retrieval by the HDF4 handler.
  • Added a dmrpp test that generates from the CF option of the HDF5 handler that has the group hierarchy
  • Added support to read compressed missing data from dmrpp.
  • Addeed support to add HDF-EOS2 lat/lon data to the dmrpp file.
  • Added HDF-EOS2 projection code.
  • Added support to access the HDF4 data that uses linked blocks with and without compression.

Release Details

  • This version of the BES is part of Hyrax 1.17.1. See opendap.org/software/hyrax/1.17.
  • Major changes include improved support for HDF4, HDF4-EOS, and HDF4-EOS2 data stored in S3 via the DMR++, plus much faster generation of NetCDF4 response files.
  • This version of the BES requires libdap-3.21.1.
  • DOI: https://doi.org/10.5281/zenodo.14646648

News for 3.21.0

Release Details

  • This version of the BES is part of Hyrax 1.17.0. See opendap.org/software/hyrax/1.17.
  • Major changes include improved support for HDF5 data stored in S3 via the DMR++ and much faster generation of NetCDF4 response files.
  • This version of the BES requires libdap-3.21.0.
  • DOI: https://doi.org/10.5281/zenodo.10564122

Reworked the way we implement the BES's singletons (or more of them at least) so that they use the Meyers Singleton pattern. This includes the EffectiveUrlCache and TheBESKeys classes.

We have made significant advances to supporting HDF5 files in the DMR++ builder and interpreter. Our support is close to complete with only a very few features not supported. This support now includes HDF5's Varying-length string arrays. The software now also supports scalar strings in all their forms. Note that we also added support for fixed length string arrays in the DMR++ builder and interpreter.

This version of the BES includes a new feature, dependent on the newest version of libdap, that reduces the time needed to build NetCDF4 responses by a factor of 10 or more when the variables in the response are not spatially subset (that is, when all variables in the response are returned in their entirety). It is still possible to subset at the variable level and get this performance boost. This is the 'Direct I/O' feature added ti libdap. NOTE: To use this feature, you must rebuild DMR++ files using the version of the DMR++ builder associated with this version of the BES (or get_dmrpp tool from the hyrax-1.16.8-356 build).

We improved the performance of finding the effective URL for a data item when it is accessed via a series of redirect operations, the last of which is a signed AWS URL. This is a common case for data stored in S3.

We have added generic Memory and File caching, tailored specifically toward the cases that arise when serving data from S3 using the DMR++ system.

The BES now has much better support for response size and time limits. Users are warned about responses that are too large before they are built and the BES now exits gracefully when a response takes too long to build. The limits are configurable.

We replaced a home-grown HTTP connection pool with a scheme provided by libcurl. The two have equivalent performance, but the libcurl version is much easier to maintain and might offer room for improvement in the future.

C99 compatibility improvements for the GCTP code in/used-by the HDF5 handler. Thanks ti Florian Weimer for those fixes.

We have added a BES module that can work with S3 using the DMR++ system. This provides a data flow that is similar to the one we provide for Hyrax in the Cloud as developer for NASA, but this new module does not make use of the NASA/ESDIS CMR system to resolve 'NASA Granules' to URLs. This will enable other groups to use the DMR++ system to serve data from S3.

We have adopted C++-14 as aggressively as we can, resulting in fewer lines of code, better memory management and more efficient code.

We have moved more of the handlers to build/use DAP4 as the default response format and build DAP2 responses from those. This reduces the amount of code we have to maintain.

The BES can sign S3 URLs using the AWS V4 signing scheme. This uses the Credentials Manager system.

The bundled grid() and geogrid() server side functions now support DAP4.

News for 3.20.13

  • Triggered a build to test new dependencies (hdf-4 and netcdf-4 updates).
  • Improved error messages for the response is too big, and response tool too much time to marshall.

Release Details

News for 3.20.12

  • Patched bug in HDF5Array.cc introduced by the std::vector refactor
  • Added time.h header to ppt/SocketUtilities.cc

Release Details

  • This version of the BES is part of Hyrax 1.16.7. See opendap.org/software/hyrax/1.16.
  • This version of the BES requires libdap-3.20.10.
  • DOI: https://doi.org/10.5281/zenodo.6803473
  • Note: DOI also available at zenodo.org (search for OPENDAP/bes version 3.20.12).

News for 3.20.11

  • Support for RHEL8
  • Refactored get_dmrpp, application. Some features still broken but core functionality is working now.

Continued general migration to C++11 coding norms.

  • Improved support for more GES DISC level 3 and level 4 products
  • Support for add the support of AIRS level 3 and GLDAS level products.
  • Refactored project so that all of the code that depends on the gdal library is in a single module, modules/gdal_module.
  • Retired use of auto_ptr.
  • Refactored timeout implementation and dropped the use of SIGALRM therein.
  • Modified fileout_netcdf handler to allow netcdf-3 responses to be up to 4GB in size. This behavior can be reverted by setting FONc.NC3ClassicFormat=true in the BEs configuration (aka /etc/bes/site.conf file)

DMR++ Improvements

  • Various improvements for supporting FillValue in dmr++ lifecycle.
  • Various improvements for supporting FillValue in dmr++ lifecycle.
  • Improved support for arrays of type String.
  • Fixed trusted url bug in DMZ parser.
  • Added support for "empty" valuyes scalars with associated _FillValue metadata.

Release Details

get_dmrpp Improvements

  • Added support for S3 hosted granules to get_dmrpp
  • Added regression test suite for get_dmrpp

DAP4

  • Added Map elements to DMR
  • Modify the HDF5 handler so that DAP4 Coverages have Map elements. This support extends to a number of DAAC "specials" like HDF-EOS5

News for 3.20.10

DMR++ Improvements

  • Added support for the HDF5 filter Fletcher32 to the dmr++ creation and processing code.

  • Implemented lazy evaluation of dmr++ files. This change greatly improves efficiency/speed for requests that subset a dataset that contains a large number of variables as only the variables requested will have their Chunk information read and parsed.

  • Added version and configuration information to dmr++ files built using the build_dmrpp and get_dmrpp applications. This will enable people to recreate and understand the conditions which resulted in a particular dmr++ instance. This also includes a -z switch for get_dmrpp which will return its version.

  • Performance improvement: By patching Chunk::add_tracking_query_param() so that it doesn't do anything if no parameter is submitted we eliminated a very time costly regular expression evaluation that was being performed during the read operation for every Chunk. This improved read performance by a factor of 2-3 orders of magnitude!

Handler Updates

  • Added new netcdf_handler configuration parameter: NC.PromoteByteToShort Which, when set to true will cause signed 8-bit integer values to be promoted to Int16 (Because the DAP2 data model does not support signed 8-bit integers)

  • By default the NetCDF Fileout feature will ship with FONc.ClassicModel=false

  • Added new configuration option to the HDF5 handler: EnableCFDMR=true which will allow the generation of CF compliant DMR output.

General Houskeeping

  • Code organization changes to facilitate development and compilation on different platforms.
    • Moved functions into modules directory
    • Improved Test scripts to handle version string updates.
    • Changed libdap includ paths to "libdap/*.h" to prevent include file collisions.
    • Started migrating to C++11 regex implementation (away from older less efficient GNU implementation) This is controlled by a compile time switch for now.

News for 3.20.9

NGAP & DMR++ Improvements

  • Trusted cmr (#485)

    • Modifying things into shape to use http::url anstead of std::string
    • Replaced strings with http::url objects
    • Moved AllowedHosts to http
    • Fixed implmentations of http::url::is_expired()
    • Switch RemoteSource constructor to shared_ptr
    • Changed the way that http::url interprets no protocol urls/
    • Fixed concurrency issues in EffectiveUrlCache

  • Corrected usage statement for get_dmrpp

  • Handle the "missing data" files in the NGAP system

  • Update NgapApiTest to reflect changes in CMR holdings.

  • Dropped useless call to Chunk.inflate() and added a state check to protect us from a CONTIGUOUS variable that is marked as compressed.

  • Rewrote read_contiguous() with std::async() and std::future dropping the SuperChunk idea.

  • First implementation of the new restified path with two mandatory and one optional path components.

DAP4

  • DAP4 doesn't support DAP2 Grid. The code that handles the DAP2 Grid coordinates will cause some DAP4 coordinate variables under different groups to be ignored. So this fix ensure the code NOT to call the code that handles the DAP2 Grid coordiantes for the DAP4 case. (HYRAX-477)

General

  • Added GitHub Actions to bes.

  • Stop parser EffectiveUrl resolution activity (#532)

  • Fonc refactor

    • Added support for streaming netCDF3 files that do not contain Structures.
    • Fix a small memory leak in the history attribute code at the transmitter.
    • Add history attribute is added to dap4.
    • Add NC.PromoteByteToShort=true in the configuration file. This makes it consistent with nc.conf.in. Also it make sures the value of signed 8-bit integer to be correctly represented in DAP2.
    • Remove unused getAttrType function from FONcArray.cc.
    • Dropping throttle from Fonc_transmiter

  • Added throttle to BESUtil::file_to_stream()

  • Ensure the data value correctness for the classic model.

    • When encountering a mismatch for the data type mapping mismatches,an error will be generated. (HYRAX-467)
    • For the classic model, ensure the _fillvalue datatype to be the same as the variable datatype.

  • Server handler refactor (#513)

  • Fixing duplicate CF history entries (#504)

  • Perform comprehensive check of datatype match and implementation of ensuring _FillValue attribute type the same as the variable type. (HYRAX-449)

  • Added new implementation of temp file transfer code for fileout_netcdf (#498)

  • Added config param Http.UserAgent (#493)

  • Fixed netCDF-4 and compression information is missing when A DAP2 grid maps to three netCDF variables. (#490) ( HYRAX-388)

  • Adds call to the ftruncate() function in the update cache files activity, unit tests for string replace_all()

News for 3.20.8

NGAP & DMR++ Improvements

  • The dmr++ production chain: get_dmrpp, build_dmrpp, check_dmrpp, merge_dmrpp, and reduce_mdf received the following updates:

    • Support for injecting configuration modifications to allow fine tuning of the dataset representation in the produced dmr++ file.
    • Support for HDF5 COMPACT layout data.
    • Optional creation and injection of missing (domain coordinate) data as needed.
    • Endian information carried in Chunks
    • Int64 support
    • Updated command line options and help page.

  • Improved S3 reliability by adding retry efforts for common S3 error responses that indicate a retry is worth pursuing (because S3 just fails sometimes and a retry is suggested).

  • Improved and more transparent error handling for remote access issues.

  • Migrated the service implementation making parallel requests using multi-cURL to the c++11 std:async and std:future mechanism.

  • Added caching of S3 “effective” URLs obtained from NGAP service chain.

  • Implemented support for EDL token chaining.

  • New implementation of ngap restified path parser that is (almost) impervious to the the key value content in the path.

  • Implemented the SuperChunk optimization for mass acquisition of required, consecutive chunks.

BALTO

  • Updated JSON-LD content of the server’s Data Request Form pages so that it is (once again) in keeping with the (evolving) rules enforced by the Rich Results page which is part of Google’s Dataset Search

DAP4

  • AsciiTransmit supports DAP4 functions
  • Group support in fileout netcdf-4

General

  • End Of Life for CentOS-6 Support - It’s been a long road CentOS-6, but NASA has given us the OK to drop support for you just days before your nominal end of life. On to the next std::future.

  • Dropped the “longest matching” Whitelist configuration key in favor of a multiple regular expressions configuration using the new AllowedHosts key.

  • Consolidation of internal HTTP code and caching for all services. This means more consistent behavior and error handling everywhere the server has to reach out for something.

  • Introduced log message types: request, error, info, verbose, and timing which all log to BES.LogName/. Each type is identified in the log and has a “fixed” format that can be reliably parsed by downstream software.

  • Updated GDAL handler.

  • SonarCloud and /or Snyk is now a blocking step for all Hyrax component PRs

  • Our Docker images have been updated to utilize ncWMS-2.4.2 which is compatible with current Tomcat security measures. This means ncWMS2 is working again…

  • Dynamic Configuration - This feature is currently a proof-of-concept idea and is disabled with a compiler macro. To become an actual feature it will need to be implemented in a much more thoughtful and efficient manner. Which we will be happy to do so if there is sufficient interest!

News for 3.20.7

  • Hyrax can generate signed S3 requests when processing dmr++ files whose data content live in S3 when the correct credentials are provided (injected) into the server.

  • For configurations that require Hyrax to authenticate to access various remote resources, Hyrax can now utilize ~/.netrc (or a netrc file may be specified in site.conf) and Hyrax will use the appropriate credentials, if they are present.

  • The get_dmrpp script for dmr++ production has been enhanced with better configuration options and built in QC tests for inventory along with a simple value test.

  • The get_dmrpp script will default to using a template string for the value of the dmrpp:href URL when a URL is not provided with a -u switch. This allows the URL to be injected later.

  • We fixed a problem with fill values by converting the FillValue type to the data type of the associated variable.

  • Fileout NetCDF has a new configuration key, FONc.NoGlobalAttrs that can be used to suppress global attributes.

hdf5_handler

  • The CF option support is enhanced.

  • Added support for correctly generating dmr++ files.

  • Enhanced support to handle netCDF-4 like HDF5 files and general two dimensional latitude and longitude files.

See the modules/hdf5_handler/ChangeLog for details.

News for 3.20.6

Notable Bugs Fixed:

  • Memory leak fixed and unneeded copy removed in awsv4 code
  • CovJson patch.
  • ncml_handler: When an NCML aggregation named a dataset that the server could not read... crash. Fixed.
  • Server cache stabilization

Features:

  • Added (alpha) support support for server authentication credentials:
    • Environment injection support for besstandalone and besd
    • A credentials configuration file for multiple sets, in which credentials are associated with URL prefix's and multiple creds can be defined in a bes.conf environment.
  • Combined Logs: The server can now be configured to add the OLFS request log content to the BES log.

Performance Improvements

  • No DAS construction for data response.

Tickets closed during this release https://opendap.atlassian.net/browse/HK-272 - MDS bug - LMT of data not used https://opendap.atlassian.net/browse/HK-361 - More performant handling of contiguous data for the DMR++ handler https://opendap.atlassian.net/browse/HK-376 - Have Travis add the source distribution tar balls to the S3 bucket. https://opendap.atlassian.net/browse/HK-411 - Fix the --baselines feature of the libdap DMRTest https://opendap.atlassian.net/browse/HK-413 - Persistent leaks in the libxml2-based NCML parser. https://opendap.atlassian.net/browse/HK-404 - Address operational and efficiency issues in the MDS https://opendap.atlassian.net/browse/HK-426 - Form interface bug - Structures do not work correctly - two issues https://opendap.atlassian.net/browse/HK-439 - bes source release https://opendap.atlassian.net/browse/HK-444 - Build initial version of 'ncdmr.cc' that can read the 'fnoc1.nc' and build a DMR. https://opendap.atlassian.net/browse/HK-445 - Modify the simple ncdmr.cc code so that it includes the attributes. https://opendap.atlassian.net/browse/HK-446 - Modify the ncdmr.cc code so that it correctly recognizes shared dimensions in 'fnoc1.nc' and 'coads_climatology.nc' https://opendap.atlassian.net/browse/HK-447 - Modify the ncdmr.cc code so that it can work with netCDF4 files that use groups. https://opendap.atlassian.net/browse/HK-448 - Modify the ncdmr.cc code so that it can work with netCDF4 files that contain structures. https://opendap.atlassian.net/browse/HK-449 - Integrate the ncdmr.cc code into the netCDF handler so that it is used for the DMR response. https://opendap.atlassian.net/browse/HK-454 - the dmrpp_module is unable to build a dmr++ for the test file data/dmrpp/grid_1_2d.h5 https://opendap.atlassian.net/browse/HK-456 - Install the BES RPM package built from a PR and start the BES from that install. Check for failure. https://opendap.atlassian.net/browse/HK-457 - The class BESRegex is utilized in a way that is incompatible with the underlying implementation. FIX https://opendap.atlassian.net/browse/HK-458 - Web interface bug for Structures and Sequences https://opendap.atlassian.net/browse/HK-459 - When Hyrax 1.16 runs, we see some error messages 'leaking out of stderr' https://opendap.atlassian.net/browse/HK-472 - BESInternalError Exception thrown by the NcML handler not handled properly https://opendap.atlassian.net/browse/HK-473 - Implement combined olfs/bes log. https://opendap.atlassian.net/browse/HK-474 - BES 3.20.5 memory errors https://opendap.atlassian.net/browse/HK-485 - Modify the CI/CD process to make the docker image https://opendap.atlassian.net/browse/HK-492 - Review the Travis activities for olfs, bes, and libdap https://opendap.atlassian.net/browse/HK-537 - Reported problem in fileout_netcdf associated with _FillValue in Ocean Color dataset https://opendap.atlassian.net/browse/HK-574 - Memory leak in AWSV4

News for 3.20.5

Improved and fixed bugs in the the CovJSON response format (contributed by Corey Hemphill hemphilc@oregonstate.edu).

Added support for c++11 and c++0x builds including tests in configure that look for those options and use c++11 over c++0x. This works with the stock compiler on CentOS 6 .

We have added much more support for S3. Data can now be served from S3 without transferring whole files (Hyrax supports subset in-place) and without reformatting the data. This version of Hyrax supports configuration of data already in S3 as well as files still on spinning disk. See the README.md file in bes/modules/dmrpp_handler/data

Performance improvements and bug fixes to the XML parsers.

Added a fix from Aron.Bartle@mechdyne.com. Netcdf file responses were not compressed when they should have been if the dataset had Structures. Fixed by patching FONcStructure as Aron suggested.

We have added experimental support for STARE (Spatio Temporal Adaptive-Resolution Encoding) as part of our work on NASA ACCESS grant 17-ACCESS17-0039.

We have added support for Dataset crawler/indexer systems that use JSON-LD as part of our work on NSF grant #1740704.

The FreeForm handlers now support regular expression matching when looking for format file.

News for 3.20.4

Hk-351 Improved Jpeg2000 and GeoTiff responses. HK-352 Improved Jpeg2000 and GeoTiff responses.

HK-22 Repaired maximum response size limits feature.

Ironed out a number of issues with sonar scan builds.

News for version 3.20.3

Multiple bug fixes (See ChangeLog for specifics)

HK-320 fixed - libcurl fails when called from the bes daemon on OSX

HK-275 fixed - bes cmdln tests fail on master for the Travis build

HK-285 fixed/completed - The BES Centos 7 RPM as packaged does not support systemctl enable bes

Created docker build for debian packaging

HK-304 Created docker build for debian packages.

News for version 3.20.2

HK-40 and HK-247 fixed - Renaming dimensions using NCML now works correctly for aggregations.

Added automatic generation of site maps, including 'virtual' catalogs.

HK-258 fixed/completed - Now the roi() function will work for variables with more than two dimensions. If the roi() function's bounding box has a different number of dimensions than a variable, it adapts, applying the b-box to the dimensions that correspond and not constraining the ones that don't.

News for version 3.20.1

Added server side function 'bbox_comb' that provides a way to combine to bounding box (with dimensions M and N, respectively) to a single bbox with dimensions M+N.

Fixed the roi() function so that it's more flexible about the bbox dimensions and variable dimensions (it allows some variables to have different dimensions than the bbox - the extra/unmatching dimensions are now ignored).

Fixed the MDS install error in the RPM packages

Added support for continuous delivery for CentOS 6 and 7 and Ubuntu Trusty.

Added a BES catalog for remote directories visible using Apache httpd. Currently in the developer build only.

Fixes for the DMR++ handler - it now recovers from the odd '500' errors that S3 returns (about 1 in 6,000 requests). Some other bugs in teh DMR++ code were also fixed - see the ChangeLog.

News for version 3.20.0

For more information about the new features and bug fixes, see README.md

  • Updates for the HDF4 and HDF5 handlers in addition to bug fixes for other handles as noted in the README.md file.

  • Site-specific configuration

The BES uses a number of configuration files, and until now, a site has to customize these for their server. Each server installation would overwrite those files. No more.

  • The MDS

A new cache has been added to the BES for Metadata Responses (aka, the MDS or MetaData Store). This cache is unlike the other BES caches in that it is intended to be operated as either a 'cache' or a 'store.' In the latter case, items added will never be removed - it is an open-ended place where metadata response objects will be kept indefinitely. The MDS contents (as a cache or a store) will survive Hyrax restarts.

  • COVJSON Response

For datasets that contain geo-spatial data, we now provide the option to get those data (and related metadata) encoded using the covjson format. (See https://covjson.org/). Thanks to Corey Hemphill, Riley Rimer, and Lewis McGibbney for this contribution.

  • Improved support for data stored on Amazon's S3 Web Object Store

Hyrax has been about to work with data stored on S3 by copying those datasets to a local cache for some time. With Hyrax 1.15 we have added support for subset-in-place for HDF5 and NetCDF4 data files.

We welcome feedback on this new feature.

  • Improved catalog support

We have generalized the BESCatalog system so that it is much easier to use. As an example of new Catalog sub-system's ease of use, we have implemented a new module that reads information about datasets from NASA's Common Metadata Repository (CMR) and uses that to display a Virtual Directory for NASA data

This software is currently available in source form only - contact us if you would like to extend the BES Catalog system for your own data collections.

  • Bug Fixes

About 40 bugs have been fixed for this release. See README.md

News for version 3.19.1

Logging Improvements

  • HYRAX-548: introduce copytruncate directive in besd.logrotate, remove brute-force stop/start

  • Log only one line per get command. This can be switched back to the 'three-lines-per-get' mode. using compile-time switches. There is also a way to trace which calls to the LOG() and VERBOSE() macros are responsible for which lines in the log. And there is an ERROR() macro to call out that an error is being logged. See the ChangeLog for more information.

  • As per the above, now, when logging is not verbose, there is only about one line per request (more for errors). Turning on verbose mode go back to the old way of logging where each (internal) command run by the BES is explicitly logged.

  • Switched to ISO8601 date-time; UTC is the default. Can use local time in the logs using BES.LogTimeLocal in bes.conf. The old date format is still in the code and can be used by #undef the compile-time switch ISO8601_TIME_IN_LOGS.

Bug Fixes

  • Fixed the integration tests in bes/cmdln.

HYRAX-248 Fixed the GeoTiff and JPEG2000 responses; in some cases the raster output was upside down. HYRAX-263 Geotiff output appears to have the latitude axis inverted. HYRAX-283 fileout geotiff doesn't work for NCEP dataset HYRAX-294 Make this function (scale_grid) work correctly when subsetting an array drops it down to 2D HYRAX-309 Test the geo functions with real data HYRAX-362 Make the GeoTiff (GDAL) handler work with NCML aggregations HYRAX-432 WCS-2.0 war file/ deployment is broken HYRAX-458 Use of mkstemp/mkstemps in libdap and bes is bogus HYRAX-465 The bes (travis) build is broken HYRAX-491 Missing single source of truth for supported Format HYRAX-507 Fix double quote problems with WCS constraint expression eval HYRAX-529 Hyrax fails to start using "service besd start" HYRAX-547 FreeForm handler fmt file for ASCII data with two header lines HYRAX-548 logrotate.d in the release 1.13.4 RPM restarts bes and tomcat every day

Developer fixes

Interface Refactoring

BESInterface/XMLInterface improved, easier to grok. This affects only people who write modules for the BES. It does not affect the behavior of the BES in any way.

Specifically, streamline the BESInterface and BESXMLInterface classes and how the DataHandlerInterface object is manipulated. Removed unused fields, renamed confusing fields, removed unused formal parameters. Removed useless typedefs from BESInterface. Removed init and end lists from BESInterface. (as they are not used). Simplified class field name changes Improved formatting for BESInterface and BESXMLCommand.

Added the 'hello_world' module into the BES's 'developer' build so that people using our developer documentation can reference a handler that works with the current version of the BES.

We have also added the DMR++ handler (see modules/dmrpp_module) the regular source distribution. This code is not built as part of the regular build, but it is part of the 'developer' build.

All the autotest dirs are now named 'tests'

News for version 3.19.0

Handler fixes: HDF4 and 5 Handler fixes - summarized in those projects' NEWS file

hyrax-263 GeoTiff response axis order is correct (fileout_gdal handler).

Refactored the software used to serve GeoTiff files (gdal_handler) so it no longer keeps the GDALDataset handle open. This code was keeping the GDALDataset handle open across calls to the RequestHandler when returning data responses and this breaks the NCML handler.

Server function fixes: hyrax-281: The scale_array() server function now works with 3D arrays.

hyrax-294: scale_grid() works when the grids are subset to 2D.

hyrax-417: Determine axis order for scale_grid() and scale_array() functions.

Tests for the (new) range() function. range() returns information about the max and min values of variable - much more efficient than downloading the whole variable. This function also tests if a vector is monotonically increasing or decreasing.

General fixes: There were several functions/methods that made temporary files (and used mkstemps, et c. to do so). These were refactored into just one function.

Unit tests for the code were made more usable (hyrax-391)

Tests for the 'file locking cache' sub-system were added.

make distcheck now works without the cumbersome env var for configure on Centos 7 and ubuntu 14. Centos 6 is still in the dark ages.

Removed unneeded classes in the BES framework (hyrax-378).

News for version 3.18.0

Bug Fixes:

  • Patched a number of problems with production rules for RPM builds.

  • Fixed BESFileLockingCache tests.

  • Repaired (at least some of) the broken BES error logging.

  • Repaired a number if timeout issues in the BES.

Features:

  • hdf5_handler adds support for sinusodial projections.

  • ncml_handler adds support for the aggregation and metadata manipulation of remotely located datasets. This new capanbility utilizes the exisiting gateway_handler and allows the ncml_handler to aggregate anything the gateway_handler can read.

  • Restructured the BES to a significant degree in order to streamline release and development issues.

News for version 3.17.4

Bug fixes:

A bug with logrotate was crashing the server;

Hyrax-282 fixed (Aggregations crash the BES when a response as netcdf or netcdf-4 is requested). This problem was actually that any Grid subset so that only some of the elements are sent broke the netcdf handler;

Patched the HDF4 handler code so that the source dist contains needed testing scripts/templates.

News for Version 3.17.3

Added an in-memory cache for the handlers

Added a logrotate file to the rpm package

Added the Response Cache (which is really used for function responses only at this point).

Fixed Hyrax-254. This bug caused random behavior when accessing aggregations.

Fixed a problem with ASCII responses when one or more dimensions of an N-dim array had only one element.

New edits for the HDF5 handler

News for Version 3.17.2

Minor changes for the source distribution (no changes to the software, but a change to the build scripts).

News for Version 3.17.1

Added an in-memory cache that handlers can use to cache DDS, DMR, ..., objects. The handlers have to be modified to use this, but the performance benefits are potentially huge. See bes/dap/ObjMemCache and the netcdf_handler for the code and an example of its use.

Server functions can now return multiple variables and have then appear as discrete things without being 'wrapped' in a parent Structure variable. They do this by using a structure name that ends with '_unwrap'. This means that server functions can return values in a way that is usable by clients built using the netcdf-library.

Patch for gcc-6 from Orion Poplawski.

News for Version 3.17.0

BES Error responses now better reflect the actual errors - constraint expression parse errors are reported as such, for example. The server returns sensible error responses that the newly-modified OLFS can interpret and use to send informative messages back to a (human) user.

The BES now supports a usable timeout feature along with the machinery to accept the timeout duration from the OLFS using a 'context'. The BES will (optionally) cancel the timeout period once data transmission starts so the timeout covers only the BES's computation and not the time to send a potentially large response over a slow network. The context name is 'bes_timeout' and the value is the timeout period in seconds.

New BES keys were added to control the timeout feature:

BES.CancelTimeoutOnSend=true

BES.TimeOutInSeconds=600

The BES now have a Timeout error code and uses a 'Timeout exception' to signal that the current operation has exceeded the timeout period.

We've settled on multiple value returns using a DAP Structure. In the BES/dap module we look for a Structure return with a certain name and flatten it. Pretty crude, but it works and doesn't require much fancy machinery. Rule: if a Structure's name ends with '_unwrap' then it will be flattened.

Better support for doxygen (but sadly, not more actual documentation).

News for version 3.16.0

Major fixes/improvements to the caching code - Hyrax uses a number of different caches. Now they are all correctly synchronized. Handler developers should use bes/dispatch/BESFileLockingCache when they need to make a special-purpose cache for their own use. Contact us for details if needed. Note that caching function results will be redone in the next release.

The BES now supports multi-homed hosts with a new Key that specifies the IP on which the server should listen (BES.ServerIP).

Fixed an issue where SIGPIPE was 'lost'. This fixes the mysterious 'runaway' BES issue that was happening sometimes, particularly when a client was stopped while making a large request.

UUID library issues may now be behind us for good...

An obscure bug in the XML command processing code caused elements to be returned out of order in some cases. Fixed.

We removed the BES performance timing/testing data files and put them in their own github project (opendap/bes_timing). This moves a number of huge data files out of the bes project proper, reducing the time to run git clone.

News for version 3.15.1

Bug fixes.

News for version 3.15.0

This version of the BES includes many build fixes including better support for the 'standard' build where only need modules are built. Thanks to Patrick West.

The code in bes/dap now frees data once it has been sent using libdap's serialize() methods. This can result in dramatic reduction in memory use. However, it requires libdap 3.15.0 or greater.

Combined with the above fix, other improvements in libdap in support of parallel I/O and matching improvements in the NCML handler, Hyrax using this version of the BES can build and return aggregations is less time, with far less latency and much less total memory. These improvements are directed at JoinNew aggregations.

News for version 3.14.0

This version of the BES incorporates all the 'standard' handlers and includes support for DAP4. It requires libdap 3.14. Included in this is the ability to build a RPM that, using some packages from the EPEL RedHat/CentOS/Fedora repo, provides a binary release for all of Hyrax. This version of the bes rpm contains the word 'NASA'; the plain version still uses EPEL but does not contain handlers like gdal (used to read GeoTIFF and JPEG2000) or HDF4 w/HDFEOS2. The handlers included in the 'NASA' RPM include the gdal and HDF4/HDFEOS2 handlers.

The besd script has been fixed and is now installed correctly by the RPM package so that the bes can easily be configured to start at boot time.

The code now builds using autoconf 2.63; automake 1.11.1 and libtool 2.2.6. These are the versions found on a RHEL 6 machine.

Added server functions: tabular(), roi(), bbox(), and bbox_union() for use with the new Swath Aggregation features.

The built-in 'dapreader' handler can now be used to reduce interdependencies between handlers when writing regression and integration tests.

  • Other Updates to handlers

The asciival handler was updated so that server functions will now correctly return ASCII responses.

The HTML form was patched using a contribution from LARC.

fileout_json now handles arrays of strings properly.

The fileout_netcdf handler has a contributed fix from NSIDC:

Convert OPeNDAP Byte array data to NetCDF NC_SHORT The handler's output currently interprets unsigned Bytes from OPeNDAP as signed bytes (NC_BYTE). To avoid mis-representing data and retain compatibility with NetCDF3/classic we need to convert/output this data as a NC_SHORT.

The existing test data for fits utilizes signed integers which shouldn't ever be a valid output from DAP2. The test data should be validated/regenerated and the test re-enabled.

  • New handlers

Added the new w10n handler to support w10n responses from the server.

Added the ugrid subsetting function handler. This server function can be used to subset an unstructured grid (aka irregular mesh) as long as that ugrid matches the specification being developed by ASA, inc. See the online documentation for more information at docs.opendap.org. http://docs.opendap.org/index.php/Server_Side_Processing_Functions

  • Handlers that support DAP4

csv_handler freeform_handler gdal_handler (geotiff, jpeg2000, other formats) netcdf_handler hdf5_handler fits_handler

News for version 3.13.2

Changes to the cache software.

News for version 3.13.1

Fixed a bug where too many accesses to cached files would cause the BES to 'hang'. The cached files were the result of uncompressing gziped files. This is the only change in this bug-fix release and the ABI of the BES has not changed, so there's no need to update the handlers.

News for version 3.13.0

Two significant bugs have been fixed in this version of the BES. Trac tickets 2176, 2158, 2139 and 2025 have been fixed. These correspond to problems building the code under OSX 10.9, requests with really long constraints, and a problem where the beslistener daemon failed to exit after the front end asked it to. The latter problem could result in 'zombie' daemon processes that would not go away until the server was shutdown.

Also in this update: Updates to the copyright headers and a patch from Ezequiel Lara Gomez for long options.

News for version 3.12.0

There are several changes to the BES for this release.

The DAP operations have been moved to their own module, reducing the amount of stuff hanging around in dispatch. Along with this change, new functionality for server-side functions has been added to the BES's DAP module. This supports, in conjunction with changes in the OLFS, the function syntax developed by COLA and used by the FTDS server. The BES can now run server functions and cache the results, providing a way for clients built using the netCDF library client toolkits to 'open' such a URL and then make repeated subsetting requests against the result of that function.

Note that libdap now supports new server function special forms that provide a way to pass large arrays into the server as constants in the constraint.

Also note that we've redone how server functions are defined, making it simpler to write them correctly. The default set of functions is now bundled in a separate module (see the 'functions' directory).

News for version 3.11.0

Added a new method to the BESDDSResponse and BESDataDDSResponse classes so that the contained DDS/DataDDS object can be set/changed by a handler. This is used by the GDAL handler so that it can use a version of DDS that it has specialized.

News for 3.10.4

Merged to trunk; tagged.

News for version 3.10.2

Changed the way the RPM packages install so that the 'bes' group has write permissions on both /var/log/bes and /var/run/bes. The latter is not strictly needed, but might be useful at some point.

Changed the default port for the BES to 10022 dues to conflicts on CentOS.

Also, fixed the way the server checks for conflicts at runtime and made the error messages more informative.

There is no version 3.10.1

News for version 3.10.0

Added support for the Hyrax Admin Interface (HAI). The server now supports a set of commands that can be used to start and stop the server; load/reload/edit the configuration files; look at logging output; and toggle debugging output on-the-fly.

Added support for response size limits.

News for version 3.9.2

Bug fix in PPT released in this version.

News for version 3.9.1

The CSV handler was moved out of this component and made its own handler, with all the stuff that implies (it's own RPM, source dist, etc.). At the same time a new tool for building tests has been added - look in the code for the dapreader classes. Email us for more information.

Bug & Build fixes fixes.

The default location of the bes.log file is now $prefix/var/bes.log

Support for the DAP3.3 Keywords (an experimental feature).

News for version 3.9.0, 14 Sep 2010

Moved BES.Catalog.catalog.RootDirectory parameter to bes.conf so users have to edit only one conf file for initial configuration

Allowing for conf files to include any required module configuration files to be loaded first. For example, if ncml module requires dap module, then the ncml module can include the dap module so that it gets loaded and configured first. Also takes care of a problem where a configuration file is included multiple times.

Fixed a problem where when transmitting a response any exceptions were being absorbed and a generic exception thrown in its place.

Various bug fixes released in this new version.

News for version 3.8.4, 24 May 2010

Bug fix. IPC bug triggered by an errant handler. Likely the cause of the runaway process bug which could be triggered by crawlers.

News for version 3.8.3, 04 May 2010

Fixed problem in logging where the actual command being executed wasn't being logged to the log file.

News for version 3.8.2, 06 April 2010

Fixed bug in BESKeys where = was adding to a definition instead of replacing.

Logging the received requests to the BESLog log file. Was just saying [xml document]

News for version 3.8.1, 11 March 2010

Fixed potential bug in BES Uncompression Cache code when failure to uncompress a file and checking size of current cache.

News for version 3.8.0, 2 February 2010

Changed the way BES configuration is modified for the different handlers. Before, changes were made using either a target to make (make bes-conf), or running a script (bes-x-data.sh), or making changes directly to bes.conf. This is no longer the case. The file bes.conf will have miner modifications (administrator email address, user and group BES should run as, etc...) and that's it. All module configuration will be in their own files in the etc/bes/modules directory. The bes.conf file will include module configurations in the directory that end in .conf. Any files that end with .conf in that directory will be loaded. This way, bes.conf will never need to be modified after installation.

The bes.conf files that are already out there will still work. This is the new, preferred method of configuration and we recommend that you move to this format.

Added a new besd script that can be used to run the BES at boot/reboot using chkconfig on linux systems. Please see besd script for more information and ask your system administrator about getting this installed in the proper location.

Added a new command to test error handling by clients , where x is 1-5 and represents the following:

  1. Internal Error - the error is internal to the BES Server
  2. Internal Fatal Error - error is fatal, can not continue
  3. Syntax User Error - the requester has a syntax error in request or config
  4. Forbidden Error - the requester is forbidden to see the resource
  5. Not Found Error - the resource can not be found

Removed all old OPeNDAP CGI server 3 references and files.

DataDDX support added to the BES in the dap module.

Cleaned up doxygen documentation for the BES.

Changed all tests from dejagnu to autotest. Changed bes/dispatch unit tests to use CPPUNIT.

News for version 3.7.2, 8 April 2009

Memory leaks fixed.

News for version 3.7.1, 16 March 2009

Many many memory issues resolved.

Fixes for constraint expression parsing issues.

The service interface is now supported.

News for version 3.7.0, 28 Jan 2009

The BES now supports TCP Wrappers (from a patch sent in by Gareth.Williams@csiro.au - thank you).

In the past the BES used a SQL-like command language but returned XML responses. This seemed odd, so we've adopted a new syntax where the BES accepts XML documents as commands (and still returns XML responses). The XML command/documents hold XML that will look very familiar to long-time BES users but a big change is that more than one command can be included in the document, reducing the number of messages needed to get many of the DAP responses. The upshot is that the OLFS <--> BES communication is much more efficient.

TCP window tuning is now provided. The user is allowed to set the tcp window sizes for buffer sizes by passing the receive and send buffer sizes to the tcp connection call.

The BES Chunking scheme is now set to use a really big buffer if you want - About 2 GB. There have been a number of fixes to the chunking code (See ChangeLog and/or SVN for details).

News for version 3.6.2, 25 June 2008

Changed default buffer size in PPT to 65535. This should create fewer sends and receives in the server and client code for responses making response times faster.

Added a BESStopWatch class to help with timings between client and server, and for any other debugging required in the BES.

Modified the BESReporter interface to allow for easier iteration through containers in get commands.

Other minor bug fixes and enhancements to code and configuration. Refer to the ChangeLog file for more information.

News for version 3.6.1, 11 April 2008

Bug fixed in new PPT chunking scheme

Other minor bug fixes and enhancements to code and configuration. Refer to the ChangeLog file for more information.

News for version 3.6.0, 29 February 2008

Changed a parameter name in the bes configuration file for memory management that contained a typo. If you receive an error "cannot determine memory keys" then you need to update the parameter MaximunHeapSize to MaximumHeapSize.

BES clients and servers now communicate using HTTP style chunking rather than using terminating tokens. This greatly improves performance of the clients and servers in that they no longer need to step through any data received to find the terminating token. Please refer the release notes on docs.opendap.org for Hyrax 1.4 for more information.

Exceptions/Errors have been re-written and consolidated. Changed the name of BESException to BESError and now have only five derived classes for the different types of errors (BESInternalError, BESInternalFatalError, BESForbiddenError, BESNotFoundError and BESSyntaxUserError.) Each error class has a type number associated with it to make it easier for clients to classify the given error. Please refer to the release notes on docs.opendap.org for Hyrax 1.4 for more information.

BES catalog and data access now includes an option to follow symbolic links or not, similar to what is used in Apache. In the BES configuration file is the new parameter BES.*.FollowSymLinks=No|Yes If set to yes, then container and catalog requests will follow symbolic links relative to the document root. If set to no, then we won't follow symbolic links.

The response to a 'show catalog' request will return the elements of a dataset (directory) in alphabetical order.

The 'set container' request now verifies the existence of the data container whereas before it did not, leaving the error to be handled by the data handler in the get command.

Added a new method to the container class to release the container. The access method is used to gain access to the data container. For example, the container may need to be uncompressed and cached. The release method would release locks on the cached item.

Added a stand-alone application with a bescmdln interface for testing purposes. Run just like bescmdln with the addition of a -c option for the location of the BES configuration file to use. Can pass a command via the -x option, or load commands from a file using the -i option, dump the output to a file using the -f option, or run it interactively.

Added .pc files for package configuration.

Other bug fixes. Please refer to the ChangeLog for more details and the release notes on docs.opendap.org for the list of tickets resolved in this release.

News for version 3.5.3

New option to the BES control script (besctl) to specify the location of the bes.pid file. This was needed for some binary releases as the prefix for installation is /usr, but the pid files are located not in /usr/var/run but in /var/run. If not specified, then it defaults to the prefix/var/run directory.

News for version 3.5.2

New options available to the BES control script (besctl) and therefor the bes daemon and listener applications. The new options are:

-i <install_dir> -> installation directory of the BES -c <configuration_file> -> location of the BES configuration file to use

hyraxctl script now available to run the entire hyrax installation. This Hyrax control script uses the BES control script besctl to start and stop the BES. Options passed to the hyrax control script are passed on to the BES.

The BES now determines the location of the BES configuration file in the following manner:

  1. -c option passed
  2. BES_CONF environment variable
  3. Look in default locations /usr/local/etc/bes/bes.conf, /etc/bes/bes.conf, /usr/etc/bes/bes.conf

The BES now checks the full path of a file/directory against the BES configuration parameter BES.Catalog.<catalog_name>.TypeMatch. Before it just checked the basename of the file/directory. This now allows us to have TypeMatch parameters that include a directory name as part of its regular expression. This was changed to facilitate the jgofs data handler, where the file names do not have file extensions. For example, TypeMatch could be something like "jg:./jg/.;"

A regular expression test program is now available in the BES source and binary releases called besregtest that will allow administrators/developers to test the regular expressions specified in the BES.Catalog.<catalog_name>.Include, BES.Catalog.<catalog_name>.Exclude, and BES.Catalog.<catalog_name>.TypeMatch configuration parameters. Usage is:

besregtest include|exclude|type "<regular_expression>" <string_to_test>

Multiple catalogs now works. If there is more than one catalog registered then the request must include the name of the catalog followed by a colon. For example: show catalog for "nc_catalog:/"; If there is only one catalog then works as before.

Added context to debugging in the BESDEBUG macro. This allows us to turn on and off debugging for different areas of an application, such as bes, ppt, nc (for netcdf handler), h4 (for hdf4 handler), etc... For example:

BESDEBUG( "bes", << "some debug " << endl << someobj << endl )

Also modified the -d option passed to the bes control script (besctl) and therefor the bes daemon and listener applications to take the name of the debug file plus context to debug.

-d "./bes.debug,bes,nc,h4"

Uncompression of .Z (zip) files is now available along with the already existing uncompression of bz2 (bzip2) and gz (gnu zip) files.

New regression test created under bes/cmdln to test netcdf, freeform, hdf4, hdf5, and server side functions. Must have all of these modules installed. To run the regression test you must have a BES running with the above specified modules loaded, then go to the bes/cmdln directory and type make bes-configuration-tests.

Updated the csv-handler example BES modules and the hello_world example modules. In doing so updated the besCreateModule script and all of the template files associated with it.

News for version 3.5.1

Ticket #928 fixed in ppt code to correct finding marker issue.

News for version 3.5.0

Decompressing data files and caching of the result has been completely reworked. It's now faster and secure.

Now when the BES has only a TCP or UNIX socket specified in the bes.conf file, it is not an error. Thus it's possible to configure the bes to use only UNIX or TCP sockets. When using the BES with Hyrax, use TCP and run it behind a firewall.

The BES was modified so that it's no longer possible to look at the contents of the host computer's file system when asking for catalog information.

News for version 3.4.2

Added some security notes to the documentation and to the configuration file (bes.conf). By default, data file decompression is now off.

Fixed a problem with dates for Last-Modified headers on Solaris.

Fixed some memory leaks

News Version 3.4.1

Changed the name of the release from Server4 to Hyrax.

Added two example modules under bes.

The first is the hello world module, which adds a command 'say to ;' For example, 'say hello to world;'. Adds a request handler, a response handler, a new command (the say command), and a reporter to report what is being said to who.

The second is a csv_handler, which can read comma separated files in a certain format. This module adds a new request handler that can handle the typical dap responses.

Minor bug fixes and clean ups. See the ChangeLog file for more information.

News Version 3.4.0

Added a new feature to the BES, the ability to create context within the BES. This is done through a new command 'set context to ; The dap module uses the context feature when determining how to deliver error messages. If the context 'errors' is set to 'dap2' then all exceptions are converted into libdap Error objects. The method print is then called using stdout. This way, dap2 clients can recognize errors.

Added a hello world example module. It does not handle data requests, but a simple command to say something to someone. It shows the power of creating your own module. In doing this, added besCreateModule and the templates directory to the build. Updated these files. besCreateModule is a shell script that is used to create a new module. Using template files, it creates source code, based on a few questions asked. The developer then adds their own code.

News Version 3.3.1

A new default command was added, 'get stream for ' that streams back the contents of the specified definition. A definition is created as usual, using 'set container' commands and using those containers to create the definition.

Modified the way the container creation works. Restrictions were put on what files can be used in a 'set container' command. You can not specify ../ in the path of the container. You must specify a new parameter in the BES configuration file BES.Data.RootDirectory which points to the root directory of any containers. This is used for file type containers only.

Created a BESDebug class to allow developers to run the server with debug option. The option to use with besctl is '-d cerr|' to output any debug to standard error (standard out is redirected to the socket, which would go back to the client) or to the specified file. All BES classes are derived from BESObj, where a dump method was added to dump the contents of an object (private data members, information about the instance) and added an operator<< so that developers can say strm << bes_obj << endl; and have the dump method called on that BES object.

The BES now does not require libdap. Default responses, transmitters, commands, etc... are available through the BES: showing help, status, process id, version information, keys defined in the configuration file; streaming files using the new command; creating, showing, and deleting containers; creating, showing, and deleting definitions. All DAP specific commands are now in a loaded module: retrieving the DAS, DDS, DDX, DataDDS and a transmitter to handle transmitting those response objects.

A request to the catalog now includes an attribute for the node called isData that is true if the file can be handled by one of the data handlers registered with the server. For example, if the name of the node ends with .nc and the netcdf data handler is loaded into the server, then isData would be set to true.

News Version 3.2.1

The spec file now does not list gzip and bzip2 as required. You only need them to serve compressed files.

The bes.conf.in file used 'hdf4' in some places and 'h4' in others. This meant that lazy people like me ;-) who didn't read what they were doing but just blindly uncommented lines were confused. I edited the file to use the same identifier in all places. jimg.

News Version 3.2.0

Created bes control script besctl for starting, stopping, restarting, and checking the status of the bes daemon and listener processes.

Executables are again in $prefix/bin instead of sbin and have been renamed besdaemon and beslistener. The bescmdln is also in $prefix/bin.

Modules dynamically loaded by the bes should now be built with the -avoid-version and -module flags in LDFLAGS. This will not put version information in the module names themselves. All modules should be of the form xxx_module

dap-server now has bes support for getting ascii, info, and html responses from the bes. Check the README for both bes and dap-server for these updates. To use these modules, simply add ascii_module, usage_module and www_module to the list of modules for bes to load.

Modules are now names using the name from the configuration file. The data types are also named using the name in the BES.modules list. For example, for the hdf4_module if you use BES.modules=dap,cmd,h4 then the handler will handle data of type h4. For example:

set container values c,hdf/test01.hdf,h4;

BES will then look for the data handler h4 to be loaded.

Transmitter code (derived classes of BESTransmitter) only handle BESInfo objects. For additional types of response objects, such as the OpeNDAP responses das, dds, data, etc... developers will register functions with the different transmitters to handle those types. See BESDapTransmit for examples.

News Version 3.1.1

The old killdods script has been renamed 'stopbes' and the build files have been fixed so that the script is installed in $prefix/sbin.

The dap_cmd module is now named dap_cmd_module to follow the pattern set for the other modules.

The bes.conf file in dispatch/bes is now a generated file with some of the paths filled in based on $prefix. The file also contains sample text for three common data handlers.

Fixes to the XML output/response formats. See ChangeLog

Fixes to the exception notifications. See ChangeLog.

News Version 3.1.0

This version of the BES is the first version to make it to a release status. This is an release of the 1.2 milestone.