Re: GMT 5 default format

Hi,

Sorry I missed all the discussion because of my time zone :(

I did some investigation yesterday and I found that  some of the GMT 5 
NC files
are in NetCDF format, and some of them are not.  This is very easy to 
see at the beginning of the file:
the first bytes says "CDF" or  0x89"HDF".

MagPick was just checking for "CDF" and rejecting the rest.

However from point of view of the NetCDF library these GMT 5  HDF files 
behave exactly as CDF, for instance
one could use "ncdump" program on them.

The examples are "BigIsland.nc" (CDF) and "tsu_chile60.nc" (HDF) files 
from grdcontour example.
You can use "ncdump" easily on both of files.

This comes from the fact that modern NetCDF is build on top of HDF( see 
for instance
http://www.unidata.ucar.edu/software/netcdf/docs/netCDF-CMake.html)

Under Linux, after I removed "CDF" string check from magpick, it was 
able to open HDF file
"tsu_chile60.nc" perfectly without any additional changes in the code. 
The reason was
because Linux magpick is compiled with stock netCDF 4 which is turn has 
HDF and everything else
enabled.

This did not work however on Windows because I was using static and some 
what old netCDF
with no further dependencies.

So I went online and got latest HDF 5 and CDF  4 and compiled them with 
CMake and VC 2010.
But I did not enable SZip in HDF. I also did not enable libcurl in netCDF.

Now under windows it was possible to open tsu_chile60.nc file but all 
values are reported as 0.
The header values were reported correctly.

Looking further I found that file "tsu_chile60.nc" uses compression.

At this point I just tried to remove compression with "h5repack" This 
created
valid HDF file with no compression, but neither GMT nor ncgump was able 
to read it.
But I could see it with HDFView.

I will follow with re-compilation of HDF with SZip and other features.

It would be nice I think to make difference between HDF and CDF clearly 
visible,
for instance with file extension, or reported by grdinfo.

Regarding the compression, sole netCDF library supports it on its own.

I hope you find this information useful,

Best Regards
Mikhail






















On 2/6/2014 6:31 AM, Lester Anderson wrote:
> Thanks for the information
> Cheers
> Lester
>
> > Date: Thu, 6 Feb 2014 11:52:21 +0000
> > From: [email protected]
> > Subject: Re: [GMT-HELP] GMT 5 default format
> > To: [email protected]
> >
> > Hi Remko and all
> >
> > That is all correct but I want also warn you guys about this GDAL
> > bug/limitation
> >
> > http://trac.osgeo.org/gdal/ticket/5291
> >
> > that will pop up (hidden) for instance to QGIS users
> >
> > Joaquim
> >
> > > Dear Lester,
> > >
> > > One option is to use
> > > gmt set IO_NC4_CHUNK_SIZE classic
> > >
> > > But I also want to urge you to either update the tools you use, or 
> if no update is available to urge the makers of those tools to finally 
> update to link against the netCDF 4 library.
> > >
> > > NetCDF 4 was introduced already some FIVE years ago and is now at 
> version 4.3.1.1. To hold off on the very useful functionalities of the 
> netCDF 4 library (particularly internal compression) would be like 
> trying to move the clock many years back.
> > >
> > > Remko
> > >
> > > On 6 Feb 2014, at 08:04, Lester 
> Anderson<[email protected]> wrote:
> > >
> > >> Hello,
> > >>
> > >> I have spotted a slight difference in the default grid formats 
> from v4 to v5 and this appears to give some issues loading into other 
> applications:
> > >>
> > >> VERSION 5
> > >> black1_emag2b_RTP.grd: Title: Anomaly reducted to the pole
> > >> black1_emag2b_RTP.grd: Command: grdredpol black1_emag2a.grd 
> -Gblack1_emag2b.grd -T2013 -W1 -F45/45 -V
> > >> black1_emag2b_RTP.grd: Remark:
> > >> black1_emag2b_RTP.grd: Gridline node registration used 
> [Geographic grid]
> > >> black1_emag2b_RTP.grd: Grid file format: nf = GMT netCDF format 
> (32-bit float),COARDS, CF-1.5
> > >> black1_emag2b_RTP.grd: x_min: 20 x_max: 50 x_inc: 0.0333333333333 
> name: longitude [degrees_east] nx: 901
> > >> black1_emag2b_RTP.grd: y_min: 35 y_max: 50 y_inc: 0.0333333333333 
> name: latitude [degrees_north] ny: 451
> > >> black1_emag2b_RTP.grd: z_min: -1385.10412598 z_max: 3309.81860352 
> name: nT
> > >> black1_emag2b_RTP.grd: scale_factor: 1 add_offset: 0
> > >> black1_emag2b_RTP.grd: format: netCDF-4 chunk_size: 129,151 
> shuffle: on deflation_level: 3<------
> > >>
> > >> VERSION 4
> > >> black1_emag2b_RTP_v2.grd: Title: Grid computed inside Mirone
> > >> black1_emag2b_RTP_v2.grd: Command:
> > >> black1_emag2b_RTP_v2.grd: Remark: File written from Matlab
> > >> black1_emag2b_RTP_v2.grd: Gridline node registration used 
> [Geographic grid]
> > >> black1_emag2b_RTP_v2.grd: Grid file format: nf = GMT netCDF 
> format (32-bit float), COARDS, CF-1.5
> > >> black1_emag2b_RTP_v2.grd: x_min: 20 x_max: 50 x_inc: 
> 0.0333333333333 name: longitude [degrees_east] nx: 901
> > >> black1_emag2b_RTP_v2.grd: y_min: 35 y_max: 50 y_inc: 
> 0.0333333333333 name: latitude [degrees_north] ny: 451
> > >> black1_emag2b_RTP_v2.grd: z_min: -1385.10412598 z_max: 
> 3309.81860352 name: z [unknown]
> > >> black1_emag2b_RTP_v2.grd: scale_factor: 1 add_offset: 0
> > >> black1_emag2b_RTP_v2.grd: format: classic<------
> > >>
> > >> The format reference from the headers are different. With version 
> 4 there are no issues loading GMT (netCDF) grids into other 
> applications like MagPick or QGIS (1.8 to 2.1), but with version 5 
> they are considered an unknown format, yet still the default format (=nf).
> > >>
> > >> From gmtdefaults:
> > >>
> > >> # I/O Parameters
> > >> #
> > >> IO_COL_SEPARATOR = tab
> > >> IO_GRIDFILE_FORMAT = nf
> > >> IO_GRIDFILE_SHORTHAND = false
> > >> IO_HEADER = false
> > >> IO_N_HEADER_RECS = 0
> > >> IO_NAN_RECORDS = pass
> > >> IO_NC4_CHUNK_SIZE = auto
> > >> IO_NC4_DEFLATION_LEVEL = 3
> > >> IO_LONLAT_TOGGLE = false
> > >> IO_SEGMENT_MARKER =>
> > >> #
> > >>
> > >> Should any parameters above be tweaked?
> > >>
> > >> It is not a big issue as it is easy to change to a format that 
> works in other applications, but it is worth noting that the subtle 
> change in the format seems to have a big impact, given that everything 
> else is identical. In terms of formats, what constitutes the "classic" 
> format in the list under grdreformat?
> > >>
> > >> Cheers
> > >> Lester
> > >> To unsubscribe, send the message "signoff gmt-help" to 
> [email protected]
> > > To unsubscribe, send the message "signoff gmt-help" to 
> [email protected]
> > >
> > >
> >
> > To unsubscribe, send the message "signoff gmt-help" to 
> [email protected]
> To unsubscribe, send the message "signoff gmt-help" to 
> [email protected] 




PLEASE NOTE: This message, including any attachments, may include privileged,
confidential and/or inside information. Any dissemination, distribution or copy
of this communication by   anyone other than the intended recipient is strictly
prohibited. If you are not the intended recipient, please notify the sender by
replying to this message and then delete it from your system. Information
provided via electronic media is not guaranteed against defects including
translation and transmission errors. The company accepts no liability for any
damage caused by any virus transmitted by this email.

Geometrics Inc. | 2190 Fortune Drive | San Jose, CA 95131 USA

To unsubscribe, send the message "signoff gmt-help" to [email protected]