Basic GRASS GIS with BASH, plus GDAL

As a follow-up to the last blog, I thought it would be helpful to demonstrate how next to break up the elevation example into individual watersheds. The reason being, in my last example I demonstrated the process using a square raster tile. Hydrological processes are not accurate when run on square tiles. It is best to run further processes, like stream order extraction, on complete watersheds.

If you are reading this and want to follow along, it is very helpful to go back and read this blog. The following builds off that last example.

I’ll break this into two parts:

1. Creating multiple watershed boundaries of different sizes with GRASS and using a basic loop in BASH for the process.
2. Clipping the original raster by the watershed boundaries using GDAL and SQL with a basic loop in BASH.

Creating Watershed Boundaries with Varying Sizes

This section will modify the script created in the last post.

Notice in the previous script, when I used r.watershed, I set the threshold value to 100000. You can use this value to set the minimum size of your watershed based on cell units. I see this as minimum number of pixels per watershed since your cell unit is the size of your pixel.

threshold=100000
accumulation=${rasterName}_accumulation
drainage=${rasterName}_drainage
stream=${rasterName}_stream
basin=${rasterName}_basin
r.watershed elevation=$fillDEM threshold=$threshold accumulation=$accumulation drainage=$drainage stream=$stream basin=$basin --overwrite

Have a play setting the threshold value to different values, like 2000000 or 5000. You will get very different outputs in your watersheds vector layer. Try to determine the upper most threshold value you can input before GRASS can no longer determine a watershed.

Warning: really small threshold values will take very long to process and give very poor results, like thousands of very tiny watersheds of little to no value in analysis.

Once you have found you upper most limit for you threshold, determine a few lesser values and build them into a list. Maybe take the upper limit, half it, then half it again.

Make that into a list into a variable:

$ list=$(echo 2000000 1000000 500000)

Then loop the list to iterate through your threshold values when running r.watershed in GRASS. All we change from the last script is the threshold value to use the item from the list. Note we add the threshold value as part of the name too.

Use the standard form of a BASH loop:

for i in $list
do
    echo $i 
done

Loop with GRASS watershed commands:

list=$(echo 2000000 1000000 500000)

for i in $list
do
    # Run watershed operation on fill sink raster
    threshold=$i
    accumulation=${rasterName}_accumulation_${i}
    drainage=${rasterName}_drainage_${i}
    stream=${rasterName}_stream_${i}
    basin=${rasterName}_basin_${i}
    r.watershed elevation=$fillDEM threshold=$threshold accumulation=$accumulation drainage=$drainage stream=$stream basin=$basin --overwrite

    # Convert Basin (watershed) to vector format
    basinVect=${rasterName}_basinVect_${i}
    r.to.vect input=$basin output=$basinVect type=area column=bnum --overwrite

    # Export catchment to vector format
    basinVectOut=${outDir}/${rasterName}_basinVectOut_${i}.shp
    v.out.ogr input=$basinVect output=$basinVectOut type=area format=ESRI_Shapefile --overwrite
done

Full script example here

Your outputs this time will be a collection of shapefile watersheds constructed by your threshold value.

Clipping your Raster by Individual Watersheds using GDAL, OGR and SQL

The watersheds layers are full of nice watersheds, however you cannot simply clip the elevation raster by the full vector file. For the clip, you need to select each individual watershed first. The following will demonstrate how to list each of those watersheds and use them as a clip file via a loop.

To start, set up a test directory for your processed raster outputs:

$ mkdir grass_test/raster_watersheds

The basic command for clipping a raster with a vector in GDAL is:

$ gdalwarp -of GTiff -cutline input_shapefile_for cut.shp input_raster_to_be_cut.tif output_raster.tif

You could create a shapefile for each watershed and clip your raster by that watershed individually using the command above, but the process would be fraught with errors and take a very long time.

It is much easier to automate the process using a combination of ogrinfo, sql and gdal.

We’ll do this with an example using JM_basinVectOut_2000000.shp you just created.

JM_basinVectOut_2000000.shp

In the example, we’ll:

1. Read and list individual watersheds from the vector layer, by listing all the ‘cat’ values leveraging the ogrinfo -sql switch.
2. Using the list, we’ll clip the original raster by each individual watershed

First, list the ‘cat’ values in the vector watershed layer.

The basic ogrinfo command is:

$ ogrinfo /grass_test/JM_basinVectOut_2000000.shp

For this operation; however, we need to list the information of each row in the attributes:

$ ogrinfo -geom=NO -q -sql "SELECT cat FROM JM_basinVectOut_2000000" /grass_test/JM_basinVectOut_2000000.shp

In the above ogrinfo command lists all of the rows for the ‘cat’ column, excluding any geographic information for the file. To do this we use the ‘-sql’ switch available in ogrinfo.

Note when using the ‘-sql’ switch you need a layer name, ‘JM_basinVectOut_2000000’ from the shapefile. The layer name is just the name of the shapefile with the .shp removed.

Use grep to select the parts from the output you want and sed to clean up the outputs for a tidy list.

$ ogrinfo -geom=NO -q -sql "SELECT cat FROM JM_basinVectOut_2000000" /home/ireese/grass_test/JM_basinVectOut_2000000.shp | grep 'cat (Integer)' | sed s/'cat (Integer) =//g'

Make it a list variable:

$ watershedList=$(ogrinfo -geom=NO -q -sql "SELECT cat FROM JM_basinVectOut_2000000" /home/ireese/grass_test/JM_basinVectOut_2000000.shp | grep 'cat (Integer)' | sed s/'cat (Integer) =//')

Before we loop the process in gdal lets look quickly at why this list was created.

In gdalwarp we are going to again leverage the ‘-sql’ switch in order to select each individual watershed row and clip only by that row. We’ll be writing the a basic ‘sql’ similar to the above, but instead filtering by each watershed:

SELECT cat FROM watershed WHERE cat=$i

So, putting all the bits together, using clip and SQL, in gdalwarp we get:

$ gdalwarp -of GTiff -dstnodata -9999 -cutline $inputVector -csql "SELECT cat FROM $inputVectorLayerName where cat='$i'" -crop_to_cutline input_raster_to_be_cut.tif output_raster.tif

Note, I am using ‘-crop_to_cutline’ and ‘-dstnodata’. Basically I’m saying, crop the output raster the bounding box of the input vector and set a ‘nodata’ value of -9999 to the pixels outside the clip area.

Now we need put this all together in a script to loop the process.


#!/bin/bash

# set base path
outDir=grass_test
outDirRast=grass_test/raster_watersheds

# Set raster as variable
raster=${outDir}/lds-tile-jm-GTiff/JM.tif
rasterName=$( basename $raster | sed 's/.tif//g' )

#prep your input vectors
inputVector=/home/ireese/grass_test/JM_basinVectOut_2000000.shp
inputVectorLayerName=$(basename $inputVector | sed 's/.shp//')

#create your watersheds list
watershedList=$(ogrinfo -geom=NO -q -sql "SELECT cat FROM $inputVectorLayerName" $inputVector | grep 'cat (Integer)' | sed s/'cat (Integer) = //')

for i in $watershedList
do
    gdalwarp -of GTiff -dstnodata -9999 -cutline $inputVector -csql "SELECT cat FROM $inputVectorLayerName where cat='$i'" -crop_to_cutline $raster $outDirRast/${rasterName}_${i}.tif
done

Copy the above script, check your file paths, and save it in a text editor as:

rasterClipByWatershed.sh

 

To run the script:

$ bash grass_test/rasterClipByWatershed.sh

Final Note
In the above process, it may be necessary to run some clean up operations to remove invalid geometries in the watersheds vector layer. I did not include this, but it is helpful to resave the watersheds shapefile first by running a buffer operation with a buffer value of ‘0’. This will remove the invalid geometries. You can run this in GRASS or using ogr2ogr with PostGIS. Here is an ogr2ogr with PostGIS example:

$ ogr2ogr -f "ESRI Shapefile" output_vector.shp input_vector.shp -dialect sqlite -sql "select id, ST_buffer(Geometry,0) as geom from input_vector" -overwrite

All the scripts from the past two posts can be found here.

Let me know of this helpful. I could potentially move on to doing some more basic hydro analysis with GRASS and BASH, but these things take time and energy. It would be good to know if these sorts of blogs have value to the wider geospatial world.

Basic GRASS GIS with BASH

I love GRASS… GIS

But this wasn’t always the case.

GRASS GIS was, for a long time, something I dismissed as ‘too complex’ for my everyday geospatial operations. I formulated any number of excuses to work around the software and could not be convinced it had practical use in my daily work. It was ‘too hard to set-up’, ‘never worked well with QGIS’, and ‘made my scripting processes a nightmare’.

I am here to officially say I’ve been very wrong for a very long time. GRASS GIS is pretty amazing and is a wonderfully easy tool to script.

Recently I scripted a model for extracting river centrelines from high resolution elevation data. During the process I thought how useful it would have been, way back, to have a simple example for setting up an environment and scripting a BASH process using GRASS. So, I built one for myself.

The following example provides the steps for a simple catchment extraction on a piece of LINZ 8m elevation data. The example is only for demonstration of running a basic BASH/GRASS set up with a hydrology command and NOT a demonstration for how to do hydrology using GRASS. The 8m elevation data is not the best data for hydrological extraction; however, these data do provide a nicely sized dataset to use as practice and will give results. Also please note, catchment extraction on a square raster is will not give accurate results at the edges of the raster.

In this example we will:

1. Download a small piece of elevation data from the LINZ Data Service
2. Build a GRASS environment to process these data
3. Build a BASH script to process the catchments
4. Import the elevation into the GRASS environment
5. Perform some basic GRASS operations (fill and watershed)
6. Export raster format for viewing
7. Export the vector catchments to shapefile

The following assumes you are working in a Linux environment and have a basic knowledge of BASH scripting. I tested this process using Ubuntu 18.04 and GRASS 7.4. I have 16GB RAM on my machine.

If you do not all ready have it, you can install GRASS as follows:

$ sudo apt-get install grass-core

If interested, but not necessary, you can install functionality for building GRASS plug-ins:

$ sudo apt-get install grass-dev

Create yourself a directory to work in:

$ mkdir grass_test

Download a piece of elevation data from the LINZ Data Service, place in your directory:

https://data.linz.govt.nz/layer/51768-nz-8m-digital-elevation-model-2012/data/

From the Tiles Table, I downloaded the JM tile in EPSG:2193.

Within your directory you will need to build a ‘PERMANENT’ folder in order for GRASS to do its magic. This folder will be set to operate in the projection of your data. Be sure your data is in the same projection as the environment you built. The data is downloaded in EPSG:2193, New Zealand Transverse Mercator (NZTM), so we set the GRASS environment to work in this projection:

$ grass -c epsg:2193 -e grass_test/GRASS_ENV

‘-c’ will create your directory using epsg:2193 and ‘-e’ will exit once this operation is complete.

Running this command will build a PERMANENT folder in:

grass_test/GRASS_ENV

providing all the bits GRASS needs to operate. Your folder structure will look like:

grass_test/GRASS_ENV/PERMANENT

Have a look inside the folder and see what GRASS built for itself.

The key from now on is to always run your BASH script through this environment. Running your script in this environment will allow you to perform BASH and GRASS commands at the same time.

First, let’s look at the the basic command to run a BASH script using a GRASS environment.

$ grass grass_test/GRASS_ENV/PERMANENT --exec sh grass_test/catchment.sh

The above says, launch GRASS using this environment

grass_test/GRASS_ENV/PERMANENT

and execute, –exec, this script

/grass_test/catchment.sh

Let’s build the BASH script. In this we will:

1. Import the elevation data. GRASS likes to work in its own data formats
2. Set the region for where the operation will be performed. GRASS needs to know where the operation is going to be performed.
3. Perform the operations, r.fill.dir and r.watershed
4. Export a raster to .tif format
5. Export the vector outputs to .shp format


#!/bin/bash

# set base path
outDir=grass_test

# Set raster as variable
raster=${outDir}/JM.tif

# Set a base name for the data. This is used to demonstrate that normal
# BASH commands can be used in this process, along side GRASS
rasterName=$( basename $raster | sed 's/.tif//g' )

# Import raster data
r.in.gdal input=$raster output=$rasterName --overwrite

# Set region. IMPORTANT so GRASS knows where the data is located.
# This region is set for the duration of the following commands
g.region rast=$rasterName

# Fill sinks
fillDEM=${rasterName}_filldem
directionDEM=${rasterName}_directiondem
areasDEM=${rasterName}_areasDEM
r.fill.dir input=$rasterName output=$fillDEM direction=$directionDEM areas=$areasDEM --overwrite

# Export a raster for viewing
areaOut=${outDir}/${rasterName}_areas.tif
r.out.gdal input=$areasDEM output=$areaOut

# Run watershed operation on fill sink raster
threshold=100000
accumulation=${rasterName}_accumulation
drainage=${rasterName}_drainage
stream=${rasterName}_stream
basin=${rasterName}_basin
r.watershed elevation=$fillDEM threshold=$threshold accumulation=$accumulation drainage=$drainage stream=$stream basin=$basin --overwrite

# Convert Basin (watershed) to vector format
basinVect=${rasterName}_basinVect
r.to.vect input=$basin output=$basinVect type=area column=bnum --overwrite

# Export catchment to vector format
basinVectOut=${outDir}/${rasterName}_basinVectOut.shp
v.out.ogr input=$basinVect output=$basinVectOut type=area format=ESRI_Shapefile --overwrite

There you have it. A simple BASH script, set up for running some GRASS commands, running on your LINUX machine.

Putting it all together

Set up you environment, copy the above script, paste it into a text editor, save it as:

grass_test/catchment.sh

Now run

$ grass grass_test/GRASS_ENV/PERMANENT —exec sh grass_test/catchment.sh

Your output catchments shapefile should be similar to the image below

wshed_output

 

Click here for the next step in this process; how to clip your raster using the watershed layer using GDAL and loops.

Let me know if this was helpful or if you would like to see any changes.

Wellington Elevations: Interpolating the Bathymetry

It is important to note something from the very beginning. The interpolated bathymetry developed in this project does not reflect the actual bathymetry of the Wellington Harbour. It is my best guess based on the tools I had and the data I worked with. Furthermore, this interpolation is NOT the official product of any institution. It is an interpolation created by me only for the purposes of visualization.

welly_harbour-colour-and-aerial_FULLVIEW

Part of the goal when visualizing the Wellington landscape was to incorporate a better idea about what may be happening below the surface of the harbor. Various bathymetric scans in the past have gathered much of the information and institutions like NIWA have done the work visualizing that data. As for myself, I did not have access to those bathymetries; however, I did have a sounding point data set to work with, so I set about interpolating those points.

The data set, in CSV format, was over a million points; too dense for a single interpolation. I worked out a basic plan for the interpolation based on splitting the points into a grid, interpolate the smaller bits, then reassemble the grid tiles into a uniform bathymetry.

Conversion from CSV to shp
Using the open option (-oo) switch, OGR will convert CSV to shp seamlessly

ogr2ogr -s_srs EPSG:4167 -t_srs EPSG:4167 -oo X_POSSIBLE_NAMES=$xname* -oo Y_POSSIBLE_NAMES=$yname*  -f "ESRI Shapefile" $outputshapepath/$basenme.shp $i

Gridding the Shapefile
With the shapefile in place, I next needed to break it into smaller pieces for interpolation. For now, I create the grid by hand in QGIS using the ‘Create Grid’ function. This is found under Vector>Reasearch Tools>Create Grid. Determining a grid size that works best for the interpolation is a bit of trial and error. You want the largest size your interpolation can manage without crashing. Using the grid tool from QGIS in very convenient, in that it creates an attribute table of the xmin, xmax, ymin, ymax corrodinates for each tile in the grid. These attributes become very helpful during the interpolation process.

Interpolating the Points
I switched things up in the interpolation methods this time and tried out SAGA GIS. I have been looking for a while now for a fast and efficient method of interpolation that I could easily build into a scripted process. SAGA seemed like a good tool for this. The only drawback, I had a very hard time finding examples online about how to use this tool. My work around to was to test the tool in QGIS first. I noticed when the command would run, QGIS saved the last command in a log file. I found that log, copied out the command line function, and began to build my SAGA command for my script from there.

Here is look at the command I used:


saga_cmd grid_spline "Multilevel B-Spline Interpolation" -TARGET_DEFINITION 0 -SHAPES "$inputpoints" -FIELD "depth" -METHOD 0 -EPSILON 0.0001 -TARGET_USER_XMIN $xmin -TARGET_USER_XMAX $xmax -TARGET_USER_YMIN $ymin -TARGET_USER_YMAX $ymax -TARGET_USER_SIZE $reso -TARGET_USER_FITS 0 -TARGET_OUT_GRID "$rasteroutput/sdat/spline_${i}"

I tested a number of methods and landed on ‘grid_spline’ as producing the best results for the project. It was useful because it did a smooth interpolation across the large ‘nodata’ spaces.

Once the initial interpolation was complete, I needed to convert the output to GeoTIFF since SAGA exports in an .sdat format. Easy enough since GDAL_TRANSLATE recognizes the .sdat format. I then did my standard prepping and formatting for visualization:


gdal_translate "$iupput_sdat/IDW_${i}.sdat" "$output_tif/IDW_${i}.tif"
gdaldem hillshade -multidirectional -compute_edges "$output_tif/IDW_${i}.tif" "$ouput_hs/IDW_${i}.tif"
gdaladdo -ro "$output_tif/IDW_${i}.tif" 2 4 8 16 32 64 128
gdaladdo -ro "$ouput_hs/IDW_${i}.tif"2 4 8 16 32 64 128

Here is look at the interpolated harbour bathymetry, hillshaded, with Wellington 1m DEM hillshade added over top
welly_harbour_bw_all

And here is a look at the same bathy hillshade with coloring
welly_harbour_bw-and-aerial

Visualizing the Bathymetry
With the bathymetry, complete it was simply a matter of building it into the existing visualization I built for the Wellington Region. Learn more about the project here. The visualization was four steps:

Hillshade
addedbathy_bathyonlypng
Color
addedbathy_bathyonly_withcolor
Aerial Imagery
addedbathy_bathyonly_withcoloraerial
Then merge the models together
addedbathy_final

Easy as, eh? Let me know what you think!

Note: All imagery was produced during my time at Land Information New Zealand. Imagery licensing can be found here:
“Source: Land Information New Zealand (LINZ) and licensed by LINZ for re-use under the Creative Commons Attribution 4.0 International licence."

Building the Wellington Model with 1m DEM and DSM

As interest in LiDAR derived elevation increases, so grows the interest in the capabilities. LiDAR derived elevation data has been great for my visualization game and in helping me communicate the story out about what LiDAR can do. It all starts with a picture to get the imagination going.

wellyvation

The Wellington model derived for this project is part of an ongoing project to help increase the exposure of the Wellington 1m DEM/DSM elevation data derived from LiDAR. Step one for me is getting a working model built in QGIS, capturing still images, and increasing interest in the data.

I’ve talked about the processing of the elevation data for Wellington visualizations in the past, so for this post I’m only focusing on the blending of the data sets in building the model. This project is a good model since it encompasses a number of subtle techniques to get the model to stand out. This post is one of a two part series; the second post discusses the techniques used to derive and visualize the bathymetry for the surrounding harbor.

Let’s start with the base, Aerial Imagery.
wellyhabour_aerialonly

Blended with a hillshade
wellyhabour_aerial_withHS

DSM added for texture and context
wellyhabour_aerial_withDSMHS

Slope added to define some edges
wellyhabour_aerial_withDSMDEMSLOPEHS

Some darker shading added to the bathymetry to frame the elevation data
wellyhabour_aerial_withDSMDEMSLOPEHS_darkenframe

And finally some added bathymetry to lighten the edges at the shoreline enhancing the frame a bit more.
wellyhabour_aerial_withDSMDEMSLOPEHS_edgeframe

In the end there is some post-processing in Photoshop to lighten up the image. Honestly, this could have been done in QGIS, but I was being lazy. For the images produced, there was no need to retain the georeferencing, and when that is the case, I rely on Photoshop for color and light balancing.

The greatest difficultly in this project so far has been trying to create a universal model for the data set. I’m finding that as I visualize different regions using this model, I need to adjust the hillshading quite significantly to draw out different features. Take a look at the images here. It is the same model, but with the noticeably different gradients used in the hillshades. The techniques used for the images in this post worked well for the urban region shown, but fall apart as you move further out into the more mountainous regions. Much of the blending is too harsh and turns the mountains into a black muddled mess. I am almost there, but like any project, it takes a good bit of subtle tweaking of the blending to get a universal image to work.

The entire base mapping work is completed in QGIS. The elevation data was processed using GDAL and the bathymetric interpolations were produced SAGA GIS. There are no color palettes for this project. The aerial imagery does all the work in that department.

Base data can be found here:
DEM: https://data.linz.govt.nz/layer/53621-wellington-lidar-1m-dem-2013/
DSM: https://data.linz.govt.nz/layer/53592-wellington-lidar-1m-dsm-2013/
Aerial Imagery: https://data.linz.govt.nz/layer/51870-wellington-03m-rural-aerial-photos-2012-2013/

The next post covers the development of the bathymetry for the surrounding harbor. Thanks for having a look and let me know what you think.

Note: All imagery was produced during my time at Land Information New Zealand. Imagery licensing can be found here:
“Source: Land Information New Zealand (LINZ) and licensed by LINZ for re-use under the Creative Commons Attribution 4.0 International licence.”

The Rejects

Sometimes there is simply not enough room for all the ideas. Sometimes you want all the images to make it to the final round.

wairarapa

In a recent project to promote some of our elevation data, I was asked to present a number of ideas for a 2000mm x 900mm wall hanging. The piece was to act as a conversation starter and demonstrate some of the finer details elevation from LiDAR possesses.

In the end, the image above was the chosen candidate. Below are the drafts I initially presented for review. You can see the difference in treatment from the original ideas to the final product. Personally, I really enjoyed the images developed for the draft series, I liked the silvery undertones, and I thought it was a shame to merely let these images sit on my hard drive.
Below, you’ll find a brief description about a few challenges faced in the image development.

near_lake_ferry
nice_farm
masterton_region
random
draft_wairarapa

Artifacts and Finer Details
The hardest part of this job was drawing out the finer details of the chosen location. There was a strong interest in showing the ancient river bed; however, without a good bit of tweaking in the hillshades, the image is quite flat. After some trial and error, I found I could get a good contrast by limiting the hillshade values range to 170-190. That’s it, but the readability of the project really hinged on the simple tweak. It really made the details stand out.
That said, the gain in detail also revealed a significant artifact in the data. If you go back up and have a closer look, you will find diagonal depressions running across the images in equal intervals. These are lines from where the LiDAR scans overlap. I haven’t quite had the time to figure out how to remove these from the original data source, so for now I leave them in as conversational piece around improving LiDAR capture practices.
As usual, all map layout work was completed on QGIS, with the bulk of the data processing done using GDAL. The ‘Reject’ images for this post are direct exports from QGIS, with no manipulation apart from some down-sampling and cropping in Photoshop.

Base data can be found here:
DEM: https://data.linz.govt.nz/layer/53621-wellington-lidar-1m-dem-2013/
DSM: https://data.linz.govt.nz/layer/53592-wellington-lidar-1m-dsm-2013/
Aerial Imagery: https://data.linz.govt.nz/layer/51870-wellington-03m-rural-aerial-photos-2012-2013/

Hope you like and thanks for checking in!

Note: All imagery was produced during my time at Land Information New Zealand. Imagery licensing can be found here:
“Source: Land Information New Zealand (LINZ) and licensed by LINZ for re-use under the Creative Commons Attribution 4.0 International licence.”