This tutorial explains how to use NCO (NetCDF Operators) tools like ncks, ncatted, ncap2, ncrename, and ncrcat to manipulate NetCDF files, especially for hydrological model output like NWM.
NOTE: Netcdf Operator use has been reduced - many operations lack proper results since National Water Model
data, for the most part, cannot and won't be represented in a
CF-compliant fashion.
As a result, Netcdf Operators, such as ncks can, and will, either fail outright or produce unwanted
results.
To drop variables (e.g., nudge, velocity) and write to a new file:
ncks -O -C -x -v nudge,velocity input.nc output.nc| Part | Meaning |
|---|---|
ncks |
NetCDF Kitchen Sink — tool to extract, slice, or edit NetCDF files |
-O |
Overwrite the output file if it exists |
-C |
Coordinate output — include coordinate variables even if not requested |
-x |
Exclude mode — remove specified variables instead of extracting them |
-v nudge,velocity |
Specifies variables to operate on (nudge, velocity) |
input.nc |
Input NetCDF file |
output.nc |
Output NetCDF file (will be written or overwritten) |
In Python/xarray terms, this command:
import xarray
ds = xarray.open_dataset("input.nc")
ds = ds.drop_vars(["nudge", "velocity"])
ds.to_netcdf("output.nc")- It opens the
input.ncNetCDF file. - It removes the variables
nudgeandvelocity. - It writes the remaining dataset to
output.nc, overwriting it if it exists. - It keeps all coordinate variables, even though none were explicitly selected.
If your NetCDF file contains many variables and you want to:
- Remove problematic or unused variables
- Shrink file size
- Clean up before merging or analysis
It is a fast, CLI-based alternative to using Python/xarray.
ncatted -O -a units,streamflow,o,c,"cubic meters per second" input.nc| Part | Meaning |
|---|---|
ncatted |
NCO Attribute Editor — modifies variable or global attributes in a NetCDF file |
-O |
Overwrite the file directly (in-place edit) |
-a units,streamflow,o,c,... |
Edit the units attribute of the streamflow variable |
o |
Operation: overwrite (o = overwrite existing value) |
c |
Type: character string (NetCDF type) |
"cubic meters per second" |
New value to assign |
input.nc |
Target NetCDF file |
import xarray
ds = xarray.open_dataset("input.nc")
ds["streamflow"].attrs["units"] = "cubic meters per second"
ds.to_netcdf("input.nc") # overwrites original- Updates the units attribute for the
streamflowvariable - Useful for standardizing metadata (e.g., for CF compliance)
ncatted -O -a title,global,o,c,"Processed NWM Output" input.nc| Part | Meaning |
|---|---|
-a title,global,o,c,<value> |
Edit the global attribute title |
global |
Target the global scope (not a variable) |
"Processed NWM Output" |
New string value for the attribute |
import xarray
ds = xarray.open_dataset("input.nc")
ds.attrs["title"] = "Processed NWM Output"
ds.to_netcdf("input.nc")- Sets or updates the global title attribute
- Useful for tagging datasets with metadata, provenance, or processing history
It's:
- Faster than opening/saving with Python for minor metadata changes
- Precise and scriptable for batch workflows
- Friendly to CF-compliance workflows
ncap2 -O -s "streamflow=(streamflow!=-999900)*streamflow*1.2 + (streamflow==-999900)*streamflow" input.nc output.nc| Component | Description |
|---|---|
ncap2 |
NCO Arithmetic Processor — evaluates math expressions on NetCDF variables |
-O |
Overwrites the output file |
-s |
Script string — defines the expression to execute |
streamflow=... |
Overwrites the streamflow variable in-place |
(streamflow!=-999900) |
Boolean mask: True (1) where data is not missing |
*streamflow*1.2 |
Scales valid values by 1.2 |
+ (streamflow==-999900)*streamflow |
Preserves the fill values (leaves -999900 untouched) |
input.nc output.nc |
Input and output NetCDF files |
import xarray
ds = xarray.open_dataset("input.nc")
sf = ds["streamflow"]
ds["streamflow"] = xarray.where(sf != -999900, sf * 1.2, sf)
ds.to_netcdf("output.nc")- This scales valid
streamflowvalues by 20% (multiply by 1.2) - Missing values (
-999900) remain unchanged, not modified or scaled - This is a safe way to do arithmetic while respecting fill/missing values
The command works correctly regardless of dimensionality, because:
- NCO applies operations element-wise (like NumPy broadcasting)
- Whether
streamflowis 1D (feature_id) or 2D (time, feature_id), the boolean masks and math apply per-element
- Acts on a 1D array of 1000+ values
- The mask checks each index independently
- Acts on a 2D matrix
- Each element in the 2D array is checked and scaled if not missing
- The output type and shape will match the input unless you explicitly reshape or cast
- This does not modify
_FillValuemetadata — only raw values
Afterward, you can verify the result:
ncdump -v streamflow output.ncOr:
import xarray
print(xarray.open_dataset("output.nc")["streamflow"])ncrename -O -v feature_id,reach_id input.nc| Flag | Meaning |
|---|---|
ncrename |
NCO Rename Tool — renames variables and dimensions |
-O |
Overwrites the original file |
-v feature_id,reach_id |
Rename variable feature_id → reach_id |
input.nc |
Input file (modified in place) |
import xarray
ds = xarray.open_dataset("input.nc")
ds = ds.rename_vars({"feature_id": "reach_id"})
ds.to_netcdf("input.nc")ncrename -O -d feature_id,reach_id input.nc| Flag | Meaning |
|---|---|
-d feature_id,reach_id |
Rename dimension feature_id → reach_id |
import xarray
ds = xarray.open_dataset("input.nc")
ds = ds.rename_dims({"feature_id": "reach_id"})
ds.to_netcdf("input.nc")ncrename -O -v feature_id,reach_id -d feature_id,reach_id input.nc| Combined Flags | Meaning |
|---|---|
-v ... and -d ... |
Rename both the variable and the dimension |
This is often needed when a variable and its dimension share the same name, e.g.:
int feature_id(feature_id)
Renaming both ensures internal consistency:
int reach_id(reach_id)
import xarray
ds = xarray.open_dataset("input.nc")
ds = ds.rename({"feature_id": "reach_id"}) # handles vars & dims
ds.to_netcdf("input.nc")| Scenario | Use Command |
|---|---|
| Rename a variable only | -v feature_id,reach_id |
| Rename a dimension only | -d feature_id,reach_id |
| Rename both (dimension and associated variable) | -v ... -d ... or Python rename() |
- If the dimension name is used elsewhere (e.g. coordinate variables), renaming just the variable without the dimension can lead to inconsistent or broken metadata.
ncrcat nwm.t00z.*.nc output_concat.nc| Flag or Part | Meaning |
|---|---|
ncrcat |
NCO Concatenate along Record (unlimited) Dimension |
nwm.t00z.*.nc |
Input files to concatenate (glob pattern) |
output_concat.nc |
Output NetCDF file (concatenated) |
import xarray
ds = xarray.open_mfdataset("nwm.t00z.*.nc", concat_dim="time")
ds.to_netcdf("output_concat.nc")- Combine multiple NetCDF files into a single file along the unlimited dimension, typically
time.
| Gotcha | Explanation |
|---|---|
| ✅ Requires a record (unlimited) dimension | One of the files must have a dimension marked as UNLIMITED (e.g., time) — check with ncdump -h. |
| ✅ Variables must match | Variables across files must have the same name, shape, and metadata (e.g., streamflow(feature_id) in all files). |
| ❌ Silent dimension mismatch errors | If one file lacks a matching structure, ncrcat may skip it or corrupt output. Always validate with ncdump. |
| 🧠 Glob expands in shell, not NCO | Be sure globbing is done by the shell (*) — avoid using it in quotes. |
💡 Use ncks --mk_rec_dmn time if needed |
Preprocess files to promote time to record dimension if missing. |
ncrcat -v streamflow file1.nc file2.nc file3.nc combined.nc| Flag or Part | Meaning |
|---|---|
-v streamflow |
Include only the streamflow variable in output |
file1.nc file2.nc ... |
Files to concatenate |
combined.nc |
Output file |
import xarray
ds = xarray.open_mfdataset(["file1.nc", "file2.nc", "file3.nc"], concat_dim="time")
ds = ds[["streamflow"]] # keep only this var
ds.to_netcdf("combined.nc")- Concatenates only a subset of variables (e.g.,
streamflow) from multiple files — useful for trimming down file size and focusing on what matters.
| Gotcha | Explanation |
|---|---|
| ✅ Must match in structure | All streamflow variables must have the same dimension structure (e.g., time, feature_id) across all files. |
| ✅ Coordinate vars are included automatically | Even if not requested, dimensions and coords used by streamflow are kept. |
| ❌ Metadata conflicts | If streamflow has different attributes across files, ncrcat may fail or issue warnings. |
| 🧠 Efficient for post-processed output | This is a great way to extract and merge a single variable from many model runs or forecasts. |
| Use Case | Use Command |
|---|---|
Combine full files along time |
ncrcat nwm.t00z.*.nc output_concat.nc |
Combine only a single variable (e.g., streamflow) |
ncrcat -v streamflow file*.nc combined.nc |
ncdump -h output_concat.ncLook for:
time = UNLIMITED ; // (N records)
double streamflow(time, feature_id)
Or inspect in Python:
import xarray
xarray.open_dataset("combined.nc")| Tool | Purpose | Behavior |
|---|---|---|
ncrcat |
Concatenate along an existing record dimension | Requires an existing unlimited (record) dimension (e.g. time) |
ncecat |
Concatenate by adding a new record dimension | Adds a new outermost dimension called record |
import xarray
ds1 = xarray.load_dataset("/path/to/one.nc")
ds2 = xarray.load_dataset("/path/to/two.nc")
ds3 = xarray.load_dataset("/path/to/three.nc")
xarray.concat([ds1, ds2, ds3], dim="time") # ← ncrcat
xarray.concat([ds1, ds2, ds3], dim="record") # ← ncecat- Files already have a record (unlimited) dimension, such as
time - You want to combine them into a longer time-series
ncrcat file1.nc file2.nc file3.nc output.nc- Input:
streamflow(time, feature_id)in each file - Output:
streamflow(time=3, feature_id)
- Input files lack a record dimension (e.g., just
feature_id) - You want to simulate stacking over a new axis (like
time,run,realization)
ncecat file1.nc file2.nc file3.nc output.nc- Input:
streamflow(feature_id) - Output:
streamflow(record=3, feature_id)
| Situation | Use ncrcat |
Use ncecat |
|---|---|---|
Files have time dimension |
✅ Yes | ❌ No |
| Files lack any record dim | ❌ No | ✅ Yes |
| You want to simulate an ensemble | ❌ No | ✅ Yes |
| You want to merge time series data | ✅ Yes | ❌ No |
You’ll run ncrcat later |
✅ Yes (after --mk_rec_dmn) |
✅ (pre-processing step) |
| Feature | ncrcat |
ncecat |
|---|---|---|
| Record dim req? | ✅ Yes (e.g., time must be unlimited) |
❌ No (adds new record dim) |
| Output dim name | Original (e.g., time) |
Always record (unless renamed) |
Compatible with ncrcat? |
✅ Concatenated again over time |
✅ If you rename record → time |
# Step 1: Add record dim
ncecat *.nc temp.nc
# Step 2: Rename 'record' to 'time'
ncrename -d record,time temp.nc
# Step 3: Reorder dimensions (optional)
ncpdq -a time,feature_id temp.nc final.nc| Tool | Requires record dim | Adds new dim | Best for |
|---|---|---|---|
ncrcat |
✅ Yes | ❌ No | Time-series or unlimited-dim stacking |
ncecat |
❌ No | ✅ Yes | Ensemble/replica stacking |
Your NetCDF file looks like this (via ncdump -h):
dimensions:
time = UNLIMITED ; // (1 currently)
feature_id = 1000 ;
variables:
int streamflow(feature_id) ;
int64 time(time) ;
So:
- ✅
timeexists and is a record/unlimited dimension - ❌
streamflowdoes not usetime— it’s only defined overfeature_id
ncrcat file1.nc file2.nc file3.nc output.ncncrcatonly stacks variables that use the record dimension (e.g.,streamflow(time, feature_id))- Since
streamflowis juststreamflow(feature_id), it will not be stacked - Result:
streamflowis simply taken from the first file only (no time series created!)
import xarray
ds1 = xarray.load_dataset("/path/to/one.nc")
ds2 = xarray.load_dataset("/path/to/two.nc")
ds3 = xarray.load_dataset("/path/to/three.nc")
xarray.concat([ds1, ds2, ds3], dim="time") # but streamflow lacks time → not stackedncecat file1.nc file2.nc file3.nc output.nc- Adds a new dimension
recordand forces all variables to use it streamflow(feature_id)→ becomes →streamflow(record, feature_id)- Each input file becomes a new slice along the
recordaxis
import xarray
ds1 = xarray.load_dataset("/path/to/one.nc")
ds2 = xarray.load_dataset("/path/to/two.nc")
ds3 = xarray.load_dataset("/path/to/three.nc")
xarray.concat([ds1, ds2, ds3], dim="record") # even if streamflow lacks time, it's stackedYou must promote time as a dimension of streamflow, e.g. make:
streamflow(time, feature_id)
Instead of:
streamflow(feature_id)
You can do this with ncap2:
ncap2 -s "streamflow_new[time,feature_id]=streamflow" file.nc temp.nc
ncrename -v streamflow_new,streamflow temp.ncThen ncrcat will concatenate as expected.
| Structure | ncrcat |
ncecat |
|---|---|---|
streamflow(time, ...) |
✅ works | ✅ works |
streamflow(feature_id) |
❌ skips or duplicates | ✅ stacks across files |
ncks -O -d feature_id,10,50 input.nc subset.nc| Flag | Meaning |
|---|---|
ncks |
NetCDF Kitchen Sink — extracts slices from datasets |
-O |
Overwrite the output file |
-d feature_id,10,50 |
Slice the feature_id dimension from index 10 to 50 |
input.nc subset.nc |
Read from input.nc, write to subset.nc |
import xarray
ds = xarray.open_dataset("input.nc")
subset = ds.isel(feature_id=slice(10, 51)) # 10 to 50 inclusive
subset.to_netcdf("subset.nc")- You want to subset a contiguous range of indices for a given dimension (
feature_id) - Index-based: This is positional, not value-based
If feature_id = 0..999, streamflow(feature_id) shape = (1000,)
After subsetting 10–50:
feature_id→(41,)streamflow→(41,)(values from index 10 to 50)
ncks -O -X -d feature_id,, --cmp ids_subset.nc input.nc output.nc| Flag | Meaning |
|---|---|
-X |
Activate external file-based subsetting |
-d feature_id,, |
Subset on feature_id, but use external values instead of range |
--cmp ids_subset.nc |
Use coordinate values from this file to define what to keep |
input.nc output.nc |
Input and output NetCDF files |
- Extracts the values of
feature_idfromids_subset.nc - Then matches those values to the
feature_iddimension ininput.nc - Only keeps the entries that match
import xarray
subset_ids = xarray.open_dataset("ids_subset.nc")["feature_id"].values
ds = xarray.open_dataset("input.nc")
filtered = ds.sel(feature_id=subset_ids)
filtered.to_netcdf("output.nc")- You have a non-contiguous list of IDs to keep (e.g.,
[101, 205, 999]) - You want semantic filtering instead of index slicing
- Useful for spatial subsetting based on external masks or station lists
If:
input.nc has: feature_id = [0, 1, 2, ..., 999]
ids_subset.nc has: feature_id = [2, 9, 88, 550]
Then:
output.ncwill keep only entries wherefeature_id∈[2, 9, 88, 550]streamflow→ shape =(4,)
| Feature | -d feature_id,10,50 |
-X -d feature_id,, --cmp file.nc |
|---|---|---|
| Based on | Index (slicing) | Value match (content-driven) |
| Input | One file | Two files (reference + target) |
| Output shape | Fixed-size slice | Based on matched values |
| Handles non-contiguous | ❌ No | ✅ Yes |
| Python analog | isel() |
sel() with np.isin() |
| Gotcha | Description |
|---|---|
❗ --cmp requires matching coordinate names |
The variable in both files must be called the same (e.g., feature_id) |
❗ -d feature_id,, is not a typo |
That means “use all values, but subset by matching from --cmp” |
✅ ncks -X is faster than scripting in Python |
Especially for large files or workflows run on clusters |
In NetCDF (and xarray), a coordinate variable:
- Has the same name as a dimension (
feature_id) - Contains coordinate values used to label and align data
- Used to subset or align across datasets
Example:
dimensions:
feature_id = 1000 ;
variables:
int feature_id(feature_id) ; // coordinate variable
float streamflow(feature_id) ; // data variable
-
feature_iddimension is sliced by index (from 10 to 50) -
All variables that depend on
feature_idwill be subset in the same slice -
This includes:
streamflow(feature_id)velocity(feature_id)- any variable defined over
feature_id
- All relevant data variables are reduced in size
feature_idcoordinate variable is sliced to match
feature_id = 1000 ;
streamflow(feature_id) → shape (1000,)
feature_id = 41 ; // (50 - 10 + 1)
streamflow(feature_id) → shape (41,)
feature_idcoordinate values fromids_subset.ncare used as a filter- All data variables that use
feature_idare subset to only include matching rows - Result: only data for those
feature_idvalues are retained
- So if
ids_subset.nchasfeature_id = [101, 205, 999] - Then
streamflow(feature_id)will be filtered to include only entries wherefeature_id == 101,205, or999
feature_id = 1000 ;
streamflow(feature_id) → shape (1000,)
feature_id = 3 ; // if 3 IDs matched
streamflow(feature_id) → shape (3,)
| Condition | What Happens to Data Variables |
|---|---|
Data var uses feature_id as a dimension |
✅ Will be subset accordingly |
Data var does not use feature_id |
❌ Will be retained in full (unchanged) |
feature_id is a coordinate variable |
✅ It will be sliced/matched in sync |
| Matching fails (no overlaps) | 🔥 Output variable will be empty or missing |
| Use Case | Best Command |
|---|---|
| You want a block from index 10 to 50 | ncks -d feature_id,10,50 |
| You want exact IDs from a region file | ncks -X -d feature_id,, --cmp ids_subset.nc |
| Tool | Purpose |
|---|---|
ncks |
Extract or exclude variables, dimensions |
ncatted |
Add, modify, or delete attributes |
ncap2 |
Arithmetic operations on variables |
ncrename |
Rename variables and dimensions |
ncrcat |
Concatenate along record (time) dimension |
ncpdq |
Reorder dimensions (transpose) |
ncecat |
Add a new record dimension |
Zender, C. (2025). NCO User Guide. Departments of Earth System Science and Computer Science, University of California, Irvine. https://nco.sourceforge.net/nco.pdf