Ship, Plane, and Anchored Radar Research and Operational Workflows
Sparrow.jl is a flexible, distributed workflow system for processing weather radar data from mobile or fixed platforms. It provides a framework for building custom data processing pipelines with built-in support for radar quality control, analysis, and visualization. Sparrow development is supported by the National Science Foundation (NSF) and National Oceanic and Atmospheric Administration (NOAA) to help improve understanding and forecasting of hazardous weather phenomena such as hurricanes and extreme rainfall. The Sparrow workflow system is designed to process data from the Colorado State University (CSU) Sea-Pol ship-borne radar and NOAA's tail Doppler radar systems aboard the Hurricane Hunter aircraft. Sparrow is part of the LROSE (Lidar Radar Open Software Environment) ecosystem and can also be used for processing data from ground-based fixed or temporarily anchored weather radar systems. The system is built in Julia to provide a high-level interface for defining complex workflows while leveraging the natively fast performance and parallel processing capabilities of the language.
- Flexible Workflow System: Process and analyze radar and other data sources with pre-defined and custom building blocks to build complex workflows
- Distributed and Real-time Processing: Built-in support for parallel, real-time processing across multiple computing cores
- Radar Data Processing: Specialized tools for quality control and analysis of weather radar data
- Pre-built Steps: Ready-to-use workflow steps for common tasks (format conversion, QC, gridding)
- Extensible Architecture: Easy to add custom processing steps and workflow types
- Message System: Configurable logging with multiple severity levels
- Integration with HPC: Support for Slurm and Sun Grid Engine cluster managers
Sparrow.jl is not an officially registered package yet, so installation takes an extra step to manually install the unregistered dependencies first.
Most of the Julia packages will be automatically installed when you add Sparrow.jl. Springsteel.jl (semi-spectral grid engine) and Ronin.jl (random forest Optimized Nonmeteorological IdentificatioN radar quality control software) are registered packages, but the following needs to be manually installed first:
- Daisho.jl - Data analysis and assimilation software
Sparrow is part of the LROSE Lidar Radar Open Software Environment. To use the core Radx tools as part of your workflow, install lrose-core and ensure binaries are in your PATH.
Future support for PyArt steps as part of the workflow is planned but is not currently implemented.
If you just want to use it, install the registered dependencies by name and the unregistered packages directly from the GitHub repositories:
using Pkg
Pkg.add("Springsteel")
Pkg.add(url="https://github.com/csu-tropical/Daisho.jl")
Pkg.add("Ronin")
Pkg.add(url="https://github.com/csu-tropical/Sparrow.jl")This will install the latest version of the code, but any updates to the code will not be reflected in your installation. You can then update the package with Pkg.update() which will update all packages in your environment.
If you want to actively develop or modify Sparrow then you can clone the repository code and install in development mode. After cloning, in the REPL, go into Package mode by pressing ]. You will see the REPL change color and indicate pkg mode. You can install the module using dev /path/to/Sparrow.jl in pkg mode. This will update the module as changes are made to the code. You should see the dependencies being installed, and then the package will be precompiled. After installing, exit Package mode with ctrl-C.
Test to make sure the precompilation was successful by running using Sparrow in the REPL. If everything is successful then you should get no errors and it will just move to a new line.
Workflows are run with the sparrow launcher script that is bundled with the package. To copy it onto your PATH (default ~/.local/bin), run:
julia -e 'using Sparrow; Sparrow.install_sparrow_script()'After that you can run workflows from any directory with sparrow my_workflow.jl .... If you cloned the repository instead, the script is at the repository root and can be run directly with julia /path/to/Sparrow.jl/sparrow or added to your PATH from there.
The smallest possible workflow uses a single pre-built step and no custom code. PassThroughStep just copies files from the data directory to the archive, so you can verify your installation and learn the mechanics without any external tools:
using Sparrow
@workflow_type SimpleWorkflow
workflow = SimpleWorkflow(
base_working_dir = "/tmp/sparrow/work",
base_archive_dir = "/tmp/sparrow/archive",
base_data_dir = "/path/to/your/radar/files",
base_plot_dir = "/tmp/sparrow/plots",
# Length of each processing window: seconds, or "20S"/"5M"/"10H"/"1D"
span_seconds = "10M",
# Format: (step_name, step_type, input_directory, archive)
steps = [
("copy", PassThroughStep, "base_data", true),
],
)Save as my_workflow.jl and run on the day you have data for:
sparrow my_workflow.jl --datetime 20260101_120000Define your own workflow types and steps for more complex processing:
using Sparrow
# Define your workflow type
@workflow_type MyRadarWorkflow
# Define workflow steps
@workflow_step QualityControl
@workflow_step Gridding
# Create the workflow
workflow = MyRadarWorkflow(
base_working_dir = "/tmp/work",
base_archive_dir = "/data/archive",
base_data_dir = "/data/raw",
base_plot_dir = "/data/plots",
# Format: (step_name, step_type, input_directory, archive)
steps = [
("qc", QualityControl, "base_data", false),
("grid", Gridding, "qc", true)
],
span_seconds = "10M",
# Add other parameters as needed
)
# Implement your step
function Sparrow.workflow_step(workflow::MyRadarWorkflow, ::Type{QualityControl},
input_dir::String, output_dir::String;
step_name::String="", step_num::Int=0, kwargs...)
msg_info("Running QC on $(input_dir)")
# Your processing logic here
endSparrow includes several ready-to-use workflow steps:
- Utility:
PassThroughStep,filterByTimeStep - Quality Control:
RadxConvertStep,RoninQCStep - Gridding:
GridRHIStep,GridCompositeStep,GridVolumeStep,GridLatlonStep,GridPPIStep,GridQVPStep
The gridding steps are configured with a Daisho TOML file referenced by the daisho_config workflow parameter. Generate a template with using Daisho; print_config("daisho.toml").
See the Provided Workflow Steps documentation for complete details.
sparrow workflow.jl [options]
Options:
--datetime DATETIME Process specific time YYYYmmdd_HHMMSS (default: "now")
--realtime Process an incoming realtime datastream
--num_workers N Number of distributed workers
--threads N Number of threads per worker
-v, --verbose LEVEL Message verbosity (0-4, default: 2)
--slurm Use Slurm cluster manager
--sge Use Sun Grid Engine
--paths_file FILE Override data paths from file- Getting Started - Installation and first workflow
- Workflow Guide - In-depth workflow concepts
- Provided Workflow Steps - Pre-built steps cookbook
- Examples - Complete workflow examples
- API Reference - Function documentation
Contributions are welcome! Please feel free to submit a Pull Request.
See LICENSE file for details.
If you use Sparrow.jl in your research, please cite:
@software{sparrow_jl,
author = {Bell, Michael M.},
title = {Sparrow.jl: Ship, Plane, and Anchored Radar Research and Operational Workflows},
url = {https://github.com/csu-tropical/Sparrow.jl},
year = {2026}
}Sparrow.jl development is supported by the NSF awards AGS-2113042, AGS-2331202, and AGS-2348448 and NOAA awards NA23OAR4590408, NA22OAR4590521, and NA25OARX459C023.
For questions or issues, please open an issue on GitHub or contact the maintainers.