Clément Cherblanc

Proper reading and reprojection of model output on a non-regular grid with no CRS

Fri, 04 Oct 2024 00:00:00 +0000

This work benefited from the valuable help, sources and great advice of Fredrick Bunt (University of Montana). Big thank you Fred!

import xarray as xr
import rioxarray # for the .rio functions
import matplotlib.pyplot as plt
import numpy as np

from pyproj import Transformer
from scipy.interpolate import griddata

Procedures and good habits when writting and reading large geo-spatio-temporal datasets

We deal with large datasets, often time in netCDF format, these datasets span space and time, with associated coordinate systems. Proper handling of those coordinates is essential to reduce error and facilitate user-friendliness, which remains a key component of our work, our science needs to be useful, in perticular to others. Knowledge of the global temperature in ten years is only useful if you can use it to draw conclusions on other processes, in other fields of sciences. This document focuses on proper encoding of the spatial Coordinate Reference System (CRS) and data formatting, based on pre-existing conventions, specifically the NetCDF Climate and Forecast (CF) Metadata Convention.

TL/DR: when encoding a dataset, this one needs a CRS (Coordinate Reference System) encoded along with coordinates in that CRS. A CRS_WKT (Well Known Text) is an efficient way to encode key information in all files. Make sure your datasets have that CRS/proper grid/WKT encoded in, whether it is a model output or a dataset you created for your own use. If you read someone else’s file with a compromised/missing CRS, here are some methods to help you.

ds = xr.open_dataset('/dmidata/projects/nckf/smb_hirham/ANT11_eval_ERA5/1999/01/01/00/ts_sfx_ANT11_ANT11_eval_ERA5_1hr_199901010000-199902010000.nc')
ds

This is an example dataset from an 11km reanalysis of ERA5. I would like to attract your attention on the Dimensions and Coordinates. There are two dimensions here: x and y. Coordinates are that x and y, time and lat and lon. We notice right away that x and y are 1D arrays, vectors in other words, of size 739 and 635 respectively. This indicates a “regular grid”, that can be simply be defined by these two vectors. In other words, one row of that grid will have a single y value, and one column will have a single x value. This regular grid may look exactly like what we need, but we need to make sure that this grid is connected to reality somehow. Indeed, we need to locate this grid with respect to some standards, imagine a coordinate reference system in high school mathematics, you need the coordinates of the origin and its unit vectors. This is the information carried by the CRS, it is a standard way of encoding some coordinate system’s key information, it will usually be latitudes/longitudes or in meters, both having their advantages and inconvenients. Thousands of CRS exist, you can find them on EPSG.io, some will be globally oriented (ex: the modern definition of latitude and longitude is a CRS), some will be more regionally oriented, peticularily the metered CRS (a CRS that uses a unit of meters), they are really accurate at small scale but start being wrong further away from the CRS’s origin due to the curvature of the Earth. There are very simple ways to check one datatset’s CRS:

print(f"Dataset CRS: {ds.rio.crs}")

Dataset CRS: None

This is problematic, there is no CRS encoded here. A common procedure is to have an Attribute called CRS_WKT, standing for Coordinate Reference System Well Known Text, summarizing all the information about the CRS in one spot. More information can be found in the NetCDF CF Metadata Convention document. Here, there is no such attribute because this x/y grid is actually the grid on which the model was run. The format of this dataset is standard (Section 5.2 of the NetCDF CF Metadata Conventions), it is output this way such that no alteration of the data or grid is done and the user can choose to make their own changes. We only have one last source of information: the lat and lon, let’s look at them.

fig,axs = plt.subplots(ncols=2, figsize=(15,6))
ds.lat.plot(ax=axs[0])
ds.lon.plot(ax=axs[1])
axs[0].set_title("Latitute")
axs[1].set_title("Longitude")

Text(0.5, 1.0, 'Longitude')

We are in a complex situation where the coordinates that make physical sense, that matter to us, are a 2D grid, which does not function well with xarray and netCDFs. Indeed, we cannot just erase x and y, they are Dimensions and can therefore not be removed, although they have no meaning to anyone else than the person running the model. There are several ways to resolve this.

1: Manually create a grid with appropriate coordinates

We will simply reproject the grid of lat/lon onto a grid that suits us better, but without altering the original dataset. For that exercice, I chose to reproject everything onto a metered CRS centered around Antarctica, the EPSG:3031. This method has the advantage of not compromising the data. Interpolation indeed would affect the data, while reprojecting the grid only affect the location of these data.

transformer = Transformer.from_crs(4326, 3031, always_xy=True)
transformer

<Concatenated Operation Transformer: pipeline>
Description: axis order change (2D) + Antarctic Polar Stereographic
Area of Use:
- name: World
- bounds: (-180.0, -90.0, 180.0, 90.0)

x_reprojected, y_reprojected = transformer.transform(ds.lon, ds.lat)

fig,axs = plt.subplots(ncols=2, figsize=(15, 6))
ds.ts[0].plot(ax=axs[0])
axs[1].pcolormesh(x_reprojected, y_reprojected, ds.ts[0])

## Bonus: finding the x/y of the south pole on the new grid
t = np.unravel_index(np.argmin(ds.lat.values), ds.lat.values.shape)
print(r'Coordinates of the cell closest to the south pole (our grid is 11km spaced, so we are ~4km away from the south pole):', x_reprojected[t], y_reprojected[t])

Coordinates of the cell closest to the south pole (our grid is 11km spaced, so we are ~4km away from the south pole): 4005.7661470373446 -4382.867657798594

I would like to bring your attention on the following:

Notice the location of the origin for both grids: 0 is somewhere off in the ocean for the left plot, it is at the south pole in our reprojected grid.
The orientation: it changed, both Antarctica has been turned but also our grid coordinates are different, shown by the white triangles on the side.
The scale along one axis: Antarctica has somewhat shrinked on the right plot, although this could be a plotting artifact, this is also the result of distances being appropriately represented after using the lat/lon to meters conversion.

The right plot is closer to a usable grid, we can actual start performing certain tasks using the ´x_reprojected´ and ´y_reprojected´ grids, for example extracting the data at a given location (whether it is lat/lon or meters) very easily.

This method does not affect the data itself, only the location of it, but it takes us away from xarray (example of an extremely powerful and optimised library for this kind of problems), we would like to have a properly set dataset instead of this hybrid numpy/xarray solution, but this only comes at the cost of resampling the data.

######## Bonus: most efficient way to pull data from a list of coordinates:
lat_stations = [-89, -88, -85]
lon_stations = [10,-50,150] # mock numbers

x_stations, y_stations = transformer.transform(lon_stations, lat_stations)

arg_stations = []

for x_, y_ in zip(x_stations, y_stations):
    arg_station = np.argmin(np.ravel(( (x_reprojected-x_)**2 + (y_reprojected-y_)**2)**.5) )   # find the argument of the cell closest to the desired location
    arg_stations.append(np.unravel_index(arg_station, x_reprojected.shape)) 

print('Station indices:', arg_stations)
print('Data at those indices (first time step):', [ds.ts.values[0, *s_] for s_ in arg_stations])

Station indices: [(289, 400), (286, 383), (247, 440)]
Data at those indices (first time step): [254.09996, 256.95547, 255.72516]

2: Resampling the data

If altering the data is not a problem for you; if you want to save the data into your own simple file; if you want to use the full power of xarray on a well organized dataset, you may need to resample the data. To do so, we use the SciPy griddata function.

aspect = ds.y.size / ds.x.size
nx = 700
ny = int(aspect * nx)
shape = (1, ny, nx) # the first dimension is for time
print(f"Target Grid Shape: {shape}")

Target Grid Shape: (1, 603, 700)

A) Building the Target Grid as a Regular Grid

We want a regular grid where every pixel is the same size in x and y. The y coordinates are flipped to follow the common convention of having a decreasing y coordinate for raster data.

x_target = np.linspace(x_reprojected.min(), x_reprojected.max(), nx)
y_target = np.linspace(y_reprojected.min(), y_reprojected.max(), ny)[::-1]
X_target, Y_target = np.meshgrid(x_target, y_target)

fig, axs = plt.subplots(ncols=2, figsize=(15, 6))
axs[0].imshow(X_target)
axs[0].set_title("X")
axs[1].imshow(Y_target)
axs[1].set_title("Y")

Text(0.5, 1.0, 'Y')

B) Regridding the Data with `griddata`

We need to interpolate the data onto the target grid.

regridded_data = griddata(
    list(zip(x_reprojected.ravel(), y_reprojected.ravel())),
    ds.ts[0].to_numpy().ravel(),
    list(zip(X_target.ravel(), Y_target.ravel())),
    method="cubic"
)
regridded_data = regridded_data.reshape(shape)

plt.figure(figsize=(10, 10))
plt.pcolormesh(X_target, Y_target, regridded_data[0])

C) Building a New Xarray Object

xdata = xr.DataArray(
    regridded_data,
    dims=("time", "y", "x"),
    coords=(ds.time[:1], y_target, x_target)
).rio.write_crs("EPSG:3031")
xdata

plt.figure(figsize=(10, 8))
xdata.plot(robust=True, cmap="viridis")

print(f"Regridded data CRS: {xdata.rio.crs}")

Regridded data CRS: EPSG:3031

The key point here is that you are able to save this data in a new netCDF file because it has a regular grid. Regular examples from the netCDF or xarray library would suffice to do so.

Keeping your science and your papers safe (MEGAsync)

Thu, 20 Jun 2024 00:00:00 +0000

Keeping your scientific (or not) data and others safe is a key part of efficient science and work. Here I describe my setup to have my important documents, my research code, my writing versions, etc, all synced on a cloud and on my computers in real time.

Disclaimer 1: I am “grandfathered” in MEGAsync, meaning that I signed up to the service when it offered a greater free storage. I have 50Gb for free but signing up for it now will provide you with 20 Gb.
Disclaimer 2: I am not paid by MEGA, I just describe the setup that works amazing for me.

MEGAsync is a sub-product of the MEGA cloud service, it allows for real time syncing of a folder on your computer, to the cloud folder. Any altered file on a given computer will be updated near-instantly in the drive, and in all other accessible computers. This makes up for several extremely practical uses:

Your files are stored online, so whatever happens to your computer now, you can recover your files. If you get a new computer, just add MEGAsync and you will have all of your files.
Any deleted file ends in the “.debris” folder, meaning you can recover lost files, even months afterwards, including past versions of some of your files.
Perfect computer-switching. I have a laptop that I carry around all the time, its useful in classes, during travels, at home, etc; but I also have a big computer in my lab, for bigger computation and more comfortable work. I can switch between the two computers instantaneously, finding all of my code and datasets where I want them to be.
This is very compatible with coding: your python scripts and notebooks, your datasets and other files are always available at the same location in your file hierarchy. I do not sync my python environments but can access all of my code with no compatibility issues.
Because you can decide which of you cloud folder is synced on each one of your computers, I keep most of my files synced on all my computers, but I only keep my “important documents” synced on my phone. This folder contains pdfs of important things for me, like scans of ID, university transcripts, diplomas, etc, and I can always access the latest version of all of those documents from my phone or my computers when needed.
I sync my scientific paper library on the cloud. I use zotero and make the software pull from a specific folder of my synced folder. I can download a paper on one computer, and access it somewhere else instantaneously.

Keep in mind that your are limited in storage amount, and to a daily quota of file transfer per computer. Keep also in mind that large datasets may need to not be in the synced folder. Finally, if your code generates a heavy files, even temporarily, your transfer quota may exceed.

Automatic computation of shadows over a landscape using QGIS and python (miniconda on Linux)

Fri, 23 Feb 2024 00:00:00 +0000

In my research for my Master’s thesis, I had to generate the shadows of mountains over a landscape, not just hillshade but actually the projected shadow of mountains over valley bottoms, to get a little closer to the actual solar radiation hitting the ground. I needed to do this computation for each day of the year and multiple times a day. I also needed a high enough resolution because it was for scientific purposes, the higher resolution, the better results in theory. This post is a description of my method, such that it could be replicated.

Generating the sun’s location;
QGIS shadow plugin;
Automatization of the shadow computation.

1. Terrain properties and sun azimuth and altitude

In order to compute the incident solar radiation on a surface, we need both the surface information and the incidence of the sun. The surface information is a DEM of your choice, QGIS will know how to use it. I used a 30m DEM of my study area in northern Montana. The sun location requires a bit more work. At a given date, time and location, the sun will have a given altitude (smallest angle between the horizon and the sun) and a given azimuth (angle of the projection of the sun on the Earth surface to the north). For a large area (define large …). You may need to compute the sun’s location at every location because in the northern hemisphere, the sun will appear lower in the sky (smaller altitude) the further north you go, but in a small enough area like the one I considered, at mid-latitudes, I made the simplification of only considering one sun azimuth and altitude.

I computed the sun azimuth at hourly resolution for a whole year using the code at this address, using the python library pysolar. I investigated the importance of time resolution and found about 0.5% of cumulative sun radiation difference between hourly and 10-minute computation (code for this comparison). To verify that the numbers output by pysolar are not absurd, I plotted all the sun azimuths, altitudes and colored them with the intensity of the incident solar radiation, for a given hour of the day, we find what resembles analemma, as can be seen on the following image. This is encouraging, but should be considered with caution still.

2. QGIS Shadow plugin

I am no QGIS expert, but I have come to use it on occasion, particularly I have found this cool plugin called “Terrain Shading” developed by Zoran Čučković. He talks in depth about his plugin, its methods and outputs on his website and with further explanation on this blog, I will now solely focus on the automatization of the call of this plugin. /!\ IMPORTANT: For reasons mentioned in this article, the DEM must be in a metered CRS, this is was the plugins can use, the use of latitude and longitude will result in distortions and will probably not even work for this project. The idea is simple: the function “Shadow depth” of the plugin computes the height of the shadow column at a given location, you may keep this information, I simply picked a threshold of 5m and assigned 1 to any cell with >5m shadow depth, and 0 otherwise. I then up-sampled my map to go from 30m to 1km resolution for the sake of my work.

3. Automatization of the shadow computation

This plugin is great, but at hourly resolution for a whole year, no way I will repeat this task manually thousands of times! This is where the QGIS console comes in handy. This is where things become a little bit tricky too. You need the “processing” and “pandas” libraries, accessible from the QGIS console. There may be ways that I do not know about, but the following wat worked for me: I am on Linux, I use miniconda to handle python environments, one of them is dedicated to this work, the following steps apply:

Create a miniconda environment dedicated to QGIS;
Install QGIS in this environment, as well as pandas;
Open QGIS from this environment and console;
Open your DEM;
Open the python console by going into: Plugins → python console;
You are ready to execute the code!

The python script that I wrote must then be called manually by you from the console, you can find it at this link. This is a very short code that will produce one raster (“.tif” file) per time step considered. Remember to take a buffer outside your exact working area to consider the shadows of landscape outside your working area.

The code does the following things:

It imports the 2 necessary libraries (again, make sure that they are installed in the right environment, that QGIS was opened from a terminal using this environment)
It loads the csv file containing the sun altitudes and azimuths and starts looping through all the values:
- If the altitude is <0, there is no sun, let’s spare the computation time and skip it;
- if the altitude is >0, then I call the “Shadow depth” function of the plugin, and save the output with a number (increment I), which basically is the i-th hour of the year. This allows me to them transform those thousands of rasters later on to be in whatever format I desire.

The call of the function should be quite straightforward, the parameters used too. You can also see the organization of my folders and how I save the computed shadows.

You end up with numerous files that are ready for processing! If you have any question about this work, would like guidance or make suggestions or corrections, feel free to reach out at my contact email that can be found on this website. I am no expert and might have done mistakes, I would welcome your expertise and help.