GDAL Conversion Tips

thareUSGS · March 14, 2019, 10:10pm

Note: this is a post from 2009, transferred to this new board and update for 2019.

GDAL. This open source raster library (used in many apps) has some support for map projected PDS3, PDS4, ISIS2, ISIS3, VICAR, FITS reading (and some write capabilities). It can export raw, ASCII or a tons of other formats like GeoTiff. It tries to maintain map projection information and some metadata to output formats that allow it. The GDAL utility applications have stretching (scaling options), map projection transformation support, clipping, resampling and mosaic capabilities.

There are many binary versions available, but our current favorite is to use anaconda. First download and install miniconda installed or anaconda installed, then to get GDAL, run conda install -c conda-forge gdal libgdal. While this works across all OSs, some Mac users still prefer the Mac GDAL binary Frameworks and Windows users like OSGeo4W

for the main gdal_translate help: http://gdal.org/gdal_translate.html

to see what formats are currently supported:
>gdal_translate --formats

convert input bit type = same output type in Tiff (ISIS2/3 cub example ):
>gdal_translate -of GTiff input.cub output.tif
- GTiff is the default format so you can actually skip “-of GTiff” in this case

16bit Tiff (truncate floats):
>gdal_translate -of GTiff -ot Int16 input.cub output.tif

16bit UInt Tiff (truncate ALL negative values and floats):
gdal_translate -ot UInt16 -of GTiff -a_nodata 0 input.cub output.tif

scaled 16bit Uint Tiff Good for 3D applications like Blender, Bryce, Vue, changes pixel values across a positive 16bit range:
gdal_translate -ot UInt16 -of GTiff -a_nodata 0 -scale -767 2296 1 65536 moon11s308_8.cub moon.tif
–output range has been scaled to: 1 65536. To set the same integer range for this example try (where 2296 - -767 + 1 = 3064) or:
gdal_translate -ot UInt16 -of GTiff -a_nodata 0 -scale -767 2296 1 3064 moon11s308_8.cub moon.tif

output raw (ENVI compatible):
>gdal_translate -of ENVI input.cub out.raw
-This will create a ENVI header (*.hdr) with parameters spelled out

stretch to PNG 8bits and save out PNG worldfile (recommended):
>gdal_translate -of PNG -scale -co worldfile=yes input.cub out.png

manual stretch to PNG 8bits (no GIS worldfile):
>gdal_translate -of PNG -scale in_min in_max out_min out_max input.cub out.png
*notes I use output as 1 to 255 so that 0 is maintained as NULL - example:
>gdal_translate -of PNG -scale 0.0232 0.12 1 255 -a_nodata 0 input.cub out.png

scale (resize) to PNG 8bits at 10% original size for browse image:
>gdal_translate -of PNG -scale -size 10% 10% input.cub out_browse.png

scale (resize) to JPG 8bits at 100x200 pixels for web image (aspect not maintained):
>gdal_translate -of JPEG -scale -size 100 200 input.cub out100x200.jpg

ESRI ASCII Grid format:
>gdal_translate -of AAGrid input.cub output.asc

information about image:
> gdalinfo input.cub

stats about image:
>gdalinfo -stats input.cub

min/max range:
>gdalinfo -mm input.cub

To project to new projection use the program gdalwarp .

thareUSGS · April 18, 2019, 3:36pm

Just to pile on. If you are still in ISIS3 format (cube) and using 32bit floating point, I generally recommend stretching to 8bit integer for visualization purposes for images. Here’s how in ISIS3

Stretch method 1
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs=“0:1 100:254” null=0 lis=0 lrs=0 his=255 hrs=255

This allows you to specify input percentages for the mapping pairs. Thus when USEPERCENTAGES=true is set pairs=“0:1 100:254” means:
map 0% to 1 (or the file’s min value to 1) and 100% to 254 (file’s max value).

Stretch method 2
This also means you can apply a recommended 0.5% clip to remove the potential extraneous lows and highs like:
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs=“0:1 0.5:1 99.5:254 100:254” null=0 lis=0 lrs=0 his=255 hrs=255

now use gdal_translate to convert your 8bit cube to Geotiff or other (or try to use the *.cub directly).

mengtingpan · September 2, 2020, 9:09am

Hi, I tried ‘conda install -c conda-forge gdal libgdal’, but error occurred:

Found conflicts! Looking for incompatible packages.

I have tried several times both on windows and on ubuntu 18.04, the error still exists.

Is there any other way to install gdal?

Best,
Yu

jlaura · September 2, 2020, 6:47pm

I would do conda install -c conda-forge gdal. libgdal has been archived by conda-forge, so that install should be good to go simply specifying gdal. Let us know how that works out.

thareUSGS · September 3, 2020, 7:23pm

I bet you are running into conflicts with other installed packages. I would recommend making your own environment area just for GDAL (and other spatial tools).

Here is a recent presentation which might help:
https://github.com/pds-data-dictionaries/ldd-cart/wiki/Introduction-to-Planetary-Updates-and-Tips-for-using-GDAL-3.x

As listed there, but just for GDAL, you can try:

Installation environment for workshop notebooks (for all OSes)

(1) install Anaconda (or mini-conda): https://www.anaconda.com/products/individual

(2) Start a conda terminal and install GDAL:

creates a stand alone environment

conda create -n gdal3

activates the environment

conda activate gdal3

for only GDAL, use this line:

conda install -c conda-forge gdal=3

Now whenever you need GDAL, first “conda activate gdal3” the environment. To move away from that environment, either activate a different environment or “conda deactivate” to get out.

mengtingpan · September 4, 2020, 3:41am

Yes, you are right. I did as you said, and it worked! Thank you very much!

mystyphy · March 8, 2022, 9:11am

Hi, were the GDAL_scripts removed from the github or do we need permissions access to see them?

jessemapel · March 16, 2022, 3:45pm

Unfortunately, we’ve had to pull a lot of our code off of GitHub to comply with USGS requirements around releasing code funded. We’re gradually working to get everything officially released so that it can be made publicly available again.

jeanlucmargot · January 7, 2022, 11:43pm

It appears that GDAL does not support the Logical Cube Format Guide documented at:

https://isis.astrogeology.usgs.gov/documents/LogicalCubeFormatGuide/LogicalCubeFormatGuide.html

However, it works if one replaces the EndGroup with End_Group and the EndObject with End_Object.

Please consider updating the documentation. Thank you.

jessemapel · January 12, 2022, 5:05pm

I’ve added an issue to get this fixed over on the repo.

github.com/USGS-Astrogeology/ISIS3

Logical Cube Format Docs use wrong PVL end format

opened 05:03PM - 12 Jan 22 UTC

jessemapel

documentation

**Description** The [logical cube format guide](https://isis.astrogeology.u…sgs.gov/documents/LogicalCubeFormatGuide/LogicalCubeFormatGuide.html) uses the incorrect end keywords for the end of groups and objects in its Cube label PVL examples. Instead of using `EndGroup` and `EndObject` it should use `End_Group` and `End_Object`. For example: ``` Object = IsisCube Object = Core StartByte = 65537 Format = Tile TileSamples = 128 TileLines = 128 Group = Dimensions Samples = 2468 Lines = 11499 Bands = 1 EndGroup Group = Pixels Type = Real ByteOrder = Lsb Base = 0.0 Multiplier = 1.0 EndGroup EndObject EndObject End ``` should instead be ``` Object = IsisCube Object = Core StartByte = 65537 Format = Tile TileSamples = 128 TileLines = 128 Group = Dimensions Samples = 2468 Lines = 11499 Bands = 1 End_Group Group = Pixels Type = Real ByteOrder = Lsb Base = 0.0 Multiplier = 1.0 End_Group End_Object End_Object End ``` See the [cube format template](https://github.com/USGS-Astrogeology/ISIS3/blob/dev/isis/appdata/templates/labels/CubeFormatTemplate.pft) for the proper end keywords in a cube label. This was first reported on [astrodiscuss](https://astrodiscuss.usgs.gov/t/gdal-conversion-tips/101/7?u=jessemapel)