Input/Output and Manipulation with Meta Data

• Functions for manipulation with meta data are in "contributed.ncl", which comes along with the NCL installation. To call these functions one needs to first load file "contributed.ncl" in the NCL.
  
  ; load contributed.ncl:
  load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
  
  
• NCL supports various formats of data. Here, we only provide examples of NetCDF format, though. For the info on the other formats, go to NCL web page.
  
  ; open the file to read data from it
  fi=addfile("rain-profiles.cdf","r")
  
  
• An advantage of using NCL is that it supports and allows for an easy manipulation with the meta data, which include the following:
  
  1. dimensions (which can be variables on their own, with names and values)
     
  2. attributes (like name, unit, missing values, etc.)
     
     
  • When reading data from the file, one can read either the variable with its meta data, or only the values of the variable.
    
    ; read time from the file as a whole variable,
    ; including both the values and the meta data
    t0 = fi->time
    printVarSummary(t0)
    
    ; read only values of time
    t1 = (/fi->time/)
    printVarSummary(t1)
    
    ; copy this variable for future use
    t2 = t1
    t5 = t1
    
    
  • When only values have been read, one can read and assign the meata data as a whole, or as only dimensions or only attributes afterwards.
    
    ; copy all meta data from t0 to t1
    copy_VarMeta(t0,t1)
    printVarSummary(t1)
    
    ; copy dimensions from t0 to t2
    copy_VarCoords(t0,t2)
    printVarSummary(t2)
    
    ; copy attributes from t0 to t2
    copy_VarAtts(t0,t2)
    printVarSummary(t2)
    
    
  • One can also assign additional attributes.
    
    ; assign fill or missing value
    t0@_FillValue = getFillValue(t0) ; used in NCL
    t0@missing_value = getFillValue(t0) ; used by NCO
    printVarSummary(t0)
    
    
  • When doing calculations, one can either be careless of metadata during the proces and assign meta data to the final product, or pay attention not to lose the meta data along the road.
    
    ; use data without worring about its meta data and copy them at the end
    t3 = t0+10.
    printVarSummary(t3)
    copy_VarMeta(t0,t3)
    printVarSummary(t3)
    
    ; assign metadata in the beginning and do calculations without deleting them
    t4 = t0
    t4 = (/t0+10./)
    printVarSummary(t4)
    
    
  • When completely new variable is produced in the calculation, one can quickly assign dimensions names and only two attributes (long_name and units). Coordinate variables and additional attributes must be assigned individually.
    
    ; assign dimension name, long_name and units attribute
    t5 = nameDim (t5,"time","Time","s")
    printVarSummary(t5)
    
    ; now assign coordinate variables
    t5&time = t0
    printVarSummary(t5)
    
    ; add missing or fill in value:
    t5@_FillValue = getFillValue(t5) ; used in NCL
    t5@missing_value = getFillValue(t5) ; used by NCO
    printVarSummary(t5)
  
  
• One of nice properties of NCL is that it allows writing of an output either as a "closed" file, where one cannot extend the lenght of the series, or as an "open"/"unlimited" file, where one dimension can be extended as much as needed (there is an upper limit, but I have never reached it and have no clue what it is).
  
  ; open several files
  system("rm output1.nc")
  fo1 = addfile("output1.nc","c")
  
  system("rm output2.nc")
  fo2 = addfile("output2.nc","c")
  
  system("rm output3.nc")
  fo3 = addfile("output3.nc","c")
  
  
  • When storing data the way they are in the program, there is no possibility for the unlimited length of the coordinate variables.
    
    ; store the coordinate variable
    fo1->time = t0
    
    ; store the variables
    fo1->t1 = t1
    fo1->t3 = t3
    
    
  • Faster way to store data, which also allows for unlimited length of one of the coordinate variables:
    
    ; define coordinate variable, its size and if it is unlimited length
    dimNames = "time"
    dimSizes = -1
    dimUnlim = True
    
    ; define variables in the output file
    filedimdef(fo2,dimNames,dimSizes,dimUnlim)
    filevardef(fo2,"time",typeof(t0),"time")
    filevardef(fo2,"t1",typeof(t1),"time")
    filevardef(fo2,"t3",typeof(t3),"time")
    
    ; define variables' attributes
    filevarattdef(fo2,"time",t0)
    filevarattdef(fo2,"t3",t3)
    
    ; store the values of the variables in the output file
    fo2->time = (/t0/)
    fo2->t1 = (/t1/)
    fo2->t3 = (/t3/)
    
    
• NetCDF file format allow keeping the file attributes, which is an extreemly cool thing - it allows one to store directly in the file all the information regarding the data (e.g. when, where, how and who produced them).
  
  ; add file attributes
  fo3->time=t0
  fAtt = True
  fAtt@creation_date = systemfunc("date")
  fAtt@author = "Verica"
  fAtt@title = "example"
  fileattdef(fo3,fAtt)