Download Hyrax-1.16 — OPeNDAP (original) (raw)

Hyrax is open source, giving you the flexibility to choose how you install and run it. We use continuous integration and continuous delivery (CI/CD) to build our software for every change of the source code.

The latest stable version of Hyrax-1.16 is Hyrax-1.16.8, and is comprised of

We offer several options to suit your needs:

1) Docker Images

Simplest way to deploy Hyrax as it eliminates compatibility issues with your operating system and other software by creating a self-contained environment. You only need the Docker software itself, available for Mac, Windows, and Linux. There are various images options to chose from:

• Hyrax-1.16.8. The complete Hyrax service in a single Docker image.
• Hyrax-1.16.8 with ncwms viewer. (optional)
• Snapshot docker images. These builds are pushed to our Docker Hub site and as part of the CI/CD process. These are the absolute-latest version of the server and will likely include some features that are partially implemented. Use with caution (all of these have passed automated online testing).

The Docker image installation and configuration guide explains how to configure the docker images for various common data distribution cases. It also explains how the image are built. Within DockerHub you can find several options for running your own Hyrax server:

2) Linux Binaries

This requires manual installation of pre-compiled libraries for Linux (RHEL-8) available below.

Required External Dependencies

• Java > = 11.
• Tomcat >= 9.

OLFS:

• OLFS 1.18.13 Web Archive File (gpg signature) Tested against Tomcat-9.0.64.
• OLFS Automatic robots.txt generation for 1.18.13 (gpg_signature) This archive contains a web archive file that runs in the Tomcat server’s root context that returns a response for ‘/robots.txt’ so that your site can be crawled using the automatically-built site maps added in 1.15.2. This is beta software; we’d appreciate feedback on it.

BES and libdap:

Linux (RHEL-8) x86_64 RPMsOur Enterprise Linux builds are released on el8 (using rocky-8)

• libdap-3.20.11 (gpg signature).
• bes-3.20.13 (gpg signature) – This RPM includes statically linked copies of all of the modules/handlers we support, including HDF4 & 5 with HDFEOS support. There is no need to install packages from EPEL with this RPM. Other sources of RPM packages will likely provide a bes RPM that uses handlers linked (dynamically) to dependencies from their distributions (CentOS, Fedora, etc.).

Installing the binary distribution

The instructions for installing the pre-compiled binaries and testing the server can be found on the Hyrax Guide.

3) Source Code

3a) Source code for Hyrax-1.16.8 from Source

• libdap-3.20.11, gpg signature.
• bes–3.20.13, gpg signature.
• Collected dependencies for Hyrax 1.16.8 (gpg signature) – This bundles the NetCDF, HDF4, HDF5, and other libraries that the Hyrax handlers require.
• OLFS-1.18.13, gpg signature.

3b) Source code for Hyrax 1.16.8 from Github

All of our source code is available at our GitHub site. There you will find the hyrax project repository, which is a meta-project that contains scripts to clone and build all of Hyrax. You will also see all of the repos that contain the Hyrax source code (libdap4, the bes and all of its handlers, and the olfs).

Directions on building Hyrax from GitHub are available at our documentation site.

What’s New in Hyrax-1.16

OLFS:

• Setting BES timeOut to 55 seconds (thanks CloudFront) for the NGAP deployment.
• Added Attributes with DAP4 types to the “No NetCDF-3 Downloads For You” feature.
• Updated ncWMS to version 2.5.2.
• Migrated to Java-11.
• Retiring various XSLT stuff.
• Fixed dap4Contents.xsl so that the viewers link works.
• Migrated the schema.org annotations from the node_contents.xsl to dap4Contents.xsl
• Rewrote schema.org links to https.
• Removed commented out code.
• Fixed page formatting with new css type.
• Added code to detect client protoUpdated ReqInfo.getRequestUrlPath() so that it utilizes the request headers:
  • “CloudFront-Forwarded-Proto”
  • “X-Forwarded-Proto”
  • “X-Forwarded-Port”
• Patched bug where unexpected Authentication headers would trigger a redirect loop.
• Fixed broken service for delivering flat (not-data) files to clients.
• Made the “Get As NetCDF-3” an “Get As DAP2 Binary” buttons on the DAP4 Data Request form context sensitive. If the dataset in question contains variables whose data types are found in DAP4 and not in DAP2/NetCDF-3 then the buttons are disabled. A more complete solution is envisioned where the projected variables are assessed and if only DAP2/NetCDF-3 types are selected then the buttons would would be enabled. This fix is only a step in that more dynamic direction.
• Changed the way that the server responds to client requests for the unadorned Dataset URL and the way that the server generates Data Request Form links in its catalog pages. There are new configuration parameters to control these behaviors.
• Added an ngap service endpoint in support of the NASA NGAP project.
• Added (alpha) support support for server authentication credentials
  • Environment injection support for besstandalone and besd
  • A credentials configuration file for multiple sets, in which credentials are associated with URL prefixes and multiple credentials can be defined in a bes.conf environment.
• Combined Logs: The server can now be configured to add the OLFS request log content to the BES log.
• Earthdata Login User Authentication Support
• The Data Request Form requires data selection is now a configuration parameter, in the olfs.xml file.
• NoDynamicNavigation – Added a trial configuration parameter to the olfs.xml file. If this element is uncommented in the olfs.xml file then all of the dynamically generated catalog/navigation pages will be disabled. When enabled the server admin must either supply and maintain THREDDS catalogs to be read by the server, or they have to provide their own system of navigation and discovery to generate links to the dataset endpoints (ex. Data Request Form).
• Experimental support for DMR++ Aggregations in which multi file aggregations can be described in a single dmr++ file, reaping all of the efficiency benefits (and pitfalls) of dmr++.
• Configuration and behavior updates.
  • DEPRECATED: _<UseDAP2ResourceUrlResponse />_.
  • USE INSTEAD:
    * _<DatasetUrlResponse type="..." />_ (default: type="download" ) to configure the type of response that the server will generate when a client attempts to access the unadorned Dataset URL. The type of response is controlled by the value of the type attribute.
    * Allowed values
    * dsr – The dap4 DSR response will be returned for the dataset URL. Note: This setting is not compatible with DataRequestForm type of “dap2” as the DSR response URL collides with the DAP2 Data Request Form URL.
    * download – If the configuration parameter AllowDirectDataSourceAccess is set (present) then the source data file will be returned for the dataset URL. If the configuration parameter AllowDirectDataSourceAccess is not present then a 403 forbidden will be returned for the dataset URL. (This is basically a file retrieval service, any constraint expression submitted with the unadorned dataset URL will be ignored.)
    * requestForm – The Hyrax Data Request Form Page will be returned for the dataset URL. Which form is returned is controlled by the _ DataRequestForm_ configuration element.
    * _<DataRequestForm type="..." />_ (default: type="dap4") Defines the target DAP data model for the dataset links in the “blue-bar” catalog.html pages. These links point to the DAP Data Request Form for each dataset. This element also determines the type of Data request form page returned when the DatasetUrlResponse type=”requestForm” and the request is for the Dataset URL.
    * Allowed values “dap2” and “dap4”.
    * <AllowDirectDataSourceAccess /> When enabled users will be able to use Hyrax as a file server and download the underlying data files/granules/objects directly, without utilizing the DAP APIs. Default: disabled.
    * <ForceDataRequestFormLinkToHttps /> The presence of this element will cause the Data Request Form interfaces to “force” the dataset URL to HTTPS. This is useful for situations where the sever is sitting behind a connection management tool (like AWS CloudFront) whose outward facing connections are HTTPS but Hyrax is not using HTTPS. Thus the internal URLs being received by Hyrax are on HTTP. When these URLs are exposed via the Data Request Forms they can cause some client’s to have issues with session dropping because the protocols are not consistent. Default: disabled
• Dependency Library Updates
  • Updated the dependency libraries as follows:
    * Upgraded gson-2.3.1 to gson-2.8.9
    * Upgraded slf4j-1.7.16 to slf4j-1.7.32
    * Upgraded logback-core-1.1.11 to logback-core-1.2.9
    * Upgraded logback-classic-1.2.0 to logback-classic-1.2.9
• Log Sanitization
  • Log entries for User-Agent, URL path, and query string are now scrubbed.
• DAP2 Data Request Form
  • Dropped link to in DAP2 Data Request Form to the no longer supported BES generated request form.
• NASA/NGAP Updates
  • Fixed gradle dependancies according snyk scan.
  • Session serialization
  • Added session manager jars to ngap resources.
  • Updated production rules to include session manager code in the ngap application build.
  • Made history_json output as JSONArray.
  • New landing page for NGAP service.
  • Added UNIX time to OLFS logs in NGAP.
  • Changes to our packaging of the ngap-snapshot docker images.
  • Added support for EDL token chaining (in concert with BES modifications)
• Bug Fixes
  • Fixed bug in computation of dimension sizes in the dap4 ifh xsl.
  • Dropped DSR response for just the dataset_url.
  • Fixed broken json error output.
  • Changed the broken naming pattern for the PPT strings.
  • Enabled ChunkedInputStream debugging and cleaned up the messages in support of stream pollution problem.
  • Fixed broken css and deployment context links construction in DSR HTML page.
  • Improved the servers response for HEAD requests, now the server correctly returns 405 Method Not Allowed
  • Fixed bug in request routing for DAP4 requests.
  • Fixed the JSON-LD content in the Data Request Form so that the required description property is generated from the data set using a heuristic, or if that fails a generic string is used. The length of this mandatory property is now correctly limited to [50 < length < 5000]
  • Improved content encoding/scrubbing on outbound error messages.
  • Fixed bugs in bootstrap init and init for all the various init() components.
  • Stabilized the order of output in WCS-2.0
  • Memory leak fixed and unneeded copy removed in awsv4 code
  • CovJson patch.
  • ncml_handler: When an NCML aggregation named a dataset that the server could not read… crsh. Fixed..
  • Server cache stabilization
  • hdf4_handler: Fixed memory leaks
  • hdf5_handler: Fixed memory leaks
  • OPeNDAP Data Request Forms and catalog pages.
    * Corrected JSON-LD encoding in the IFH generated pages.
    * Corrected javascript code generation problems.
  • Path construction bug (causing double “/” characters) in THREDDS catalog presentation pages. (HK-338)
• Performance Improvements
  • Added file size and last modified headers to flat file transfer.
  • Added auth.log independent of debug log.
  • First pass at patching POST request handling.
  • Improved BES connection lifecycle reliability improvements.
  • No DAS construction for data response.
• Security analysis and response.
  • Sonar Scan
  • Coverity Scan

BES:

• General: What’s new?
  • Improved error messages for the response is too big, and response tool too much time to marshall.
  • Patched bug in HDF5Array.cc introduced by the std::vector refactor.
  • Added time.h header to ppt/SocketUtilities.cc
  • Support for RHEL8
  • Refactored get_dmrpp, application. Some features still broken but core functionality is working now.
  • Enhanced the fileout Covjson handler to support GES DISC AIRS level 3 and GLDAS level 4 products
  • Refactored project so that all of the code that depends on the gdal library is in a single module, modules/gdal_module.
  • Retired use of auto_ptr.
  • Refactored timeout implementation and dropped the use of SIGALRM therein.
  • Added the support of shuffling filter in the fileout_netcdf handler.
  • Modified fileout_netcdf handler to allow netcdf-3 responses to be up to 4GB in size. This behavior can be reverted by setting FONc.NC3ClassicFormat=true in the BEs configuration (aka /etc/bes/site.conf file).
  • Code organization changes to facilitate development and compilation on different platforms.
    * Moved functions into modules directory
    * Improved Test scripts to handle version string updates.
    * Changed libdap include paths to “libdap/*.h” to prevent include file collisions.
    * Started migrating to C++11 regex implementation (away from older less efficient GNU implementation) This is controlled by a compile time switch for now.
  • Added GitHub Actions to bes.
  • Stop parser EffectiveUrl resolution activity
  • Fonc refactor
    * Added support for streaming netCDF3 files that do not contain Structures
    * Fix a small memory leak in the history attribute code at the transmitter
    * Add history attribute is added to dap4
    * Add NC.PromoteByteToShort=true in the configuration file. This makes it consistent with nc.conf.in Also it make sure the value of signed 8-bit integer to be correctly represented in DAP2
    * Remove unused getAttrType function from FONcArray.cc
    * Dropping throttle from Fonc_transmiter
  • Added throttle to BESUtil::file_to_stream()
  • Ensure the data value correctness for the classic model
    * When encountering a mismatch for the data type mapping mismatches,an error will be generated
    * For the classic model, ensure the _fillvalue datatype to be the same as the variable datatype
  • Server handler refactor
  • Fixing duplicate CF history entries
  • Perform comprehensive check of datatype match and implementation of ensuring _FillValue attribute type the same as the variable type
  • Added new implementation of temp file transfer code for fileout_netcdf
  • Added config param Http.UserAgent
  • Fixed netCDF-4 and compression information is missing when A DAP2 grid maps to three netCDF variables
  • Adds call to the ftruncate() function in the update cache files activity, unit tests for string replace_all()
  • End Of Life for CentOS-6 Support – It’s been a long road CentOS-6, but NASA has given us the OK to drop support for you just days before your nominal end of life. On to the next std::future.
  • Dropped the “longest matching” Whitelist configuration key in favor of a multiple regular expressions configuration using the new AllowedHosts key.
  • Consolidation of internal HTTP code and caching for all services. This means more consistent behavior and error handling everywhere the server has to reach out for something.
  • Introduced log message types: request, error, info, verbose, and timing which all log to BES.LogName/. Each type is identified in the log and has a “fixed” format that can be reliably parsed by downstream software.
  • Updated GDAL handler.
  • SonarCloud and /or Snyk is now a blocking step for all Hyrax component PRs
  • Our Docker images have been updated to utilize ncWMS-2.4.2 which is compatible with current Tomcat security measures. This means ncWMS2 is working again…
  • Dynamic Configuration – This feature is currently a proof-of-concept idea and is disabled with a compiler macro. To become an actual feature it will need to be implemented in a much more thoughtful and efficient manner. Which we will be happy to do so if there is sufficient interest!
  • Hyrax can generate signed S3 requests when processing dmr++ files whose data content live in S3 when the correct credentials are provided (injected) into the server.
  • For configurations that require Hyrax to authenticate to access various remote resources, Hyrax can now utilize ~/.netrc (or a netrc file may be specified in site.conf) and Hyrax will use the appropriate credentials, if they are present.
  • The get_dmrpp script for dmr++ production has been enhanced with better configuration options and built in QC tests for inventory along with a simple value test.
  • The get_dmrpp script will default to using a template string for the value of the dmrpp:href URL when a URL is not provided with a -u switch. This allows the URL to be injected later.
  • We fixed a problem with fill values by converting the FillValue type to the data type of the associated variable.
  • Fileout NetCDF has a new configuration key, FONc.NoGlobalAttrs that can be used to suppress global attributes.
  • Added support for c++11 and c++0x builds including tests in configure that look for those options and use c++11 over c++0x. This works with the stock compiler on CentOS 6 .
  • We have added much more support for S3. Data can now be served from S3 without transferring whole files (Hyrax supports subset in-place) and without reformatting the data. This version of Hyrax supports configuration of data already in S3 as well as files still on spinning disk. See the README.md file in bes/modules/dmrpp_handler/data.
  • We have added experimentatl support for STARE (Spatio Temporal Adaptive-Resolution Encoding) as part of our work on NASA ACCESS grant 17-ACCESS17-0039.
  • We have added support for Dataset crawler/indexer systems that use JSON-LD as part of our work on NSF grant #1740704.
  • The FreeForm handlers now support regular expression matching when looking for format file.
• NGAP and DMR++ Improvements
  • Various improvements for supporting FillValue in dmr++ lifecycle.
  • Various improvements for supporting FillValue in dmr++ lifecycle.
  • Improved support for arrays of type String.
  • Fixed trusted url bug in DMZ parser.
  • Added support for “empty” valuyes scalars with associated _FillValue metadata.
  • Added support for the HDF5 filter Fletcher32 to the dmr++ creation and processing code.
  • Implemented lazy evaluation of dmr++ files. This change greatly improves efficiency/speed for requests that subset a dataset that contains a large number of variables as only the variables requested will have their Chunk information read and parsed.
  • Added version and configuration information to dmr++ files built using the build_dmrpp and get_dmrpp applications. This will enable people to recreate and understand the conditions which resulted in a particular dmr++ instance. This also includes a -z switch for get_dmrpp which will return its version.
  • Performance improvement: By patching Chunk::add_tracking_query_param() so that it doesn’t do anything if no parameter is submitted we eliminated a very time costly regular expression evaluation that was being performed during the read operation for every Chunk. This improved read performance by a factor of 2-3 orders of magnitude!
  • Trusted CMR
    * Modifying things into shape to use http::url instead of std::string
    * Replaced strings with http::url objects
    * Moved AllowedHosts to http
    * Fixed implementations of http::url::is_expired()
    * Switch RemoteSource constructor to shared_ptr
    * Changed the way that http::url interprets no protocol urls/
    * Fixed concurrency issues in EffectiveUrlCache
  • Corrected usage statement for get_dmrpp
  • Handle the “missing data” files in the NGAP system
  • Update NgapApiTest to reflect changes in CMR holdings.
  • Dropped useless call to Chunk.inflate() and added a state check to protect us from a CONTIGUOUS variable that is marked as compressed.
  • Rewrote read_contiguous() with std::async() and std::future dropping the SuperChunk idea.
  • First implementation of the new restified path with two mandatory and one optional path components.
  • The dmr++ production chain: get_dmrpp, build_dmrpp, check_dmrpp, merge_dmrpp, and reduce_mdf received the following updates:
    * Support for injecting configuration modifications to allow fine tuning of the dataset representation in the produced dmr++ file.
    * Support for HDF5 COMPACT layout data.
    * Optional creation and injection of missing (domain coordinate) data as needed.
    * Endian information carried in Chunks
    * Int64 support
    * Updated command line options and help page.
  • Improved S3 reliability by adding retry efforts for common S3 error responses that indicate a retry is worth pursuing (because S3 just fails sometimes and a retry is suggested).
  • Improved and more transparent error handling for remote access issues.
  • Migrated the service implementation making parallel requests using multi-cURL to the c++11 std:async and std:future mechanism.
  • Added caching of S3 “effective” URLs obtained from NGAP service chain.
  • Implemented support for EDL token chaining.
  • New implementation of ngap restified path parser that is (almost) impervious to the the key value content in the path.
  • Implemented the SuperChunk optimization for mass acquisition of required, consecutive chunks.
• get_dmrpp improvements
  • Added support for S3 hosted granules to get_dmrpp
  • Added regression test suite for get_dmrpp
• HDF5 handler and HDF4 handler Improvements
  • Continued general migration to C++11 coding norms
  • Added the DAP4 Coverage support to the CF option of the HDF5 handler.
  • Improved the CF option’s dimension name handling so that it can successfully handle NSIDC SMAP polar grid level 3 products.
  • The CF option support is enhanced.
  • Added support for correctly generating dmr++ files.
  • Enhanced support to handle netCDF-4 like HDF5 files and general two dimensional latitude and longitude files.
• Handler Updates
  • Added new netcdf_handler configuration parameter: NC.PromoteByteToShort Which, when set to true will cause signed 8-bit integer values to be promoted to Int16 (Because the DAP2 data model does not support signed 8-bit integers)
  • By default the NetCDF Fileout feature will ship with FONc.ClassicModel=false
  • Added new configuration option to the HDF5 handler: EnableCFDMR=true which will allow the generation of CF compliant DMR output
• DAP4
  • DAP4 doesn’t support DAP2 Grid. The code that handles the DAP2 Grid coordinates will cause some DAP4 coordinate variables under different groups to be ignored. So this fix ensure the code NOT to call the code that handles the DAP2 Grid coordinates for the DAP4 case.
  • AsciiTransmit supports DAP4 functions
  • Group support in fileout netcdf-4
• BALTO
  • Updated JSON-LD content of the server’s Data Request Form pages so that it is (once again) in keeping with the (evolving) rules enforced by the Rich Results page which is part of Google’s Dataset Search
• Bugs Fixed
  • Memory leak fixed and unneeded copy removed in awsv4 code
  • CovJson patch.
  • ncml_handler: When an NCML aggregation named a dataset that the server could not read… crsh. Fixed..
  • Server cache stablization
  • Added (alpha) support support for server authentication credentials:
    * Environment injection support for besstandalone and besd
    * A credentials configuration file for multiple sets, in which credentials are asscoaied with URL prefixs and multiple creds can be defined in a bes.conf environment.
  • Combined Logs: The server can now be configured to add the OLFS request log content to the BES log.
  • hdf4_handler: Fixed memory leaks
  • hdf5_handler: Fixed memory leaks
  • Improved and fixed bugs in the the CovJSON response format.
  • Added a fix to the behavior that Netcdf file responses were not compressed when they should have been if the dataset had Structures. Fixed by patching FONcStructure.

LIBDAP:

• What’s New
  • Fixed bug in computation of request_size_kb()
  • Fixed type issue in HTTPCache.cc
  • Support for RHEL8
  • Fix for bugs in the ce parser around inverted indices.
  • Fix for libdap4 github issue 147: Grid::get_map_iter() was off by one
  • Improvements to DAP4 api.
  • Fixed various memory leaks.
  • Replaced instances of &vector[0] with vector.data() (RHEL8)
  • Started migrating from (deprecated) auto_ptr to C++11 unique_ptr
  • Migrated use of regex functions from outdfated GNU implementation to C++11 implementation (uses compile time switch)
  • Modified Error so that it is more in line with C++11.
  • Added (shallow) unit64_t request size computations and max_request_size state.
  • Changed the internal representation of max response size to uint64_t and confined the overflow to just the (deprecated) functions.
  • Added code coverage (using gcov) to the CI processes.
  • Dropped support for CentOS-6
  • Improved Int64 support.
  • Corrected byte order issue with DAP4 data transmission.
  • Improved error reporting.
  • Stopped CE parse errors from returning user supplied strings in error messages.
  • Switched to README.md from README
  • Swapped out unique_ptr for auto_ptr because of CentOS6 issues.
  • Fixed memory leaks in imported aws signing code.
  • Bug in MDS in which LMT was not being utilized.
  • Fixed additional memory leaks in the libxml2-based
  • Lots of continuous integration work.
  • Fixed memory leaks in the libxml2-based DMR and DDX parsers.
  • Moved more toward C++-11. On CentOS 6 we now build using c++0x while CentOS 7 uses c++11. This means that we can use unique_ptr<> but not null_ptr, for example.
  • We have found a bug in the code that orders Dimensions, Enumeration definitions and Groups in the DMR. Listing the Groups last makes DMRs that reference Dimensions defined inside Groups fail silently.