Filename: \FOI2009\FOIA\documents\HARRY_READ_ME.txt 1:READ ME for Harry's work on the CRU TS2.1/3.0 datasets, 2006-2009! 2: 3:1. Two main filesystems relevant to the work: 4: 5:/cru/dpe1a/f014 6:/cru/tyn1/f014 7: 8:Both systems copied in their entirety to /cru/cruts/ 9: 10:Nearly 11,000 files! And about a dozen assorted 'read me' files addressing 11:individual issues, the most useful being: 12: 13:fromdpe1a/data/stnmon/doc/oldmethod/f90_READ_ME.txt 14:fromdpe1a/code/linux/cruts/_READ_ME.txt 15:fromdpe1a/code/idl/pro/README_GRIDDING.txt 16: 17:(yes, they all have different name formats, and yes, one does begin '_'!) 18: 19:2. After considerable searching, identified the latest database files for 20:tmean: 21: 22:fromdpe1a/data/cruts/database/+norm/tmp.0311051552.dtb 23:fromdpe1a/data/cruts/database/+norm/tmp.0311051552.dts 24: 25:(yes.. that is a directory beginning with '+'!) 26: 27:3. Successfully ran anomdtb.f90 to produce anomaly files (as per item 7 28:in the '_READ_ME.txt' file). Had to make some changes to allow for the 29:move back to alphas (different field length from the 'wc -l' command). 30: 31:4. Successfully ran the IDL regridding routine quick_interp_tdm.pro 32:(why IDL?! Why not F90?!) to produce '.glo' files. 33: 34:5. Currently trying to convert .glo files to .grim files so that we can 35:compare with previous output. However the progam suite headed by 36:globulk.f90 is not playing nicely - problems with it expecting a defunct 37:file system (all path widths were 80ch, have been globally changed to 160ch) 38:and also no guidance on which reference files to choose. It also doesn't 39:seem to like files being in any directory other than the current one!! 40: 41:6. Temporarily abandoned 5., getting closer but there's always another 42:problem to be evaded. Instead, will try using rawtogrim.f90 to convert 43:straight to GRIM. This will include non-land cells but for comparison 44:purposes that shouldn't be a big problem... [edit] noo, that's not gonna 45:work either, it asks for a 'template grim filepath', no idea what it wants 46:(as usual) and a serach for files with 'grim' or 'template' in them does 47:not bear useful fruit. As per usual. Giving up on this approach altogether. 48: 49:7. Removed 4-line header from a couple of .glo files and loaded them into 50:Matlab. Reshaped to 360r x 720c and plotted; looks OK for global temp 51:(anomalies) data. Deduce that .glo files, after the header, contain data 52:taken row-by-row starting with the Northernmost, and presented as '8E12.4'. 53:The grid is from -180 to +180 rather than 0 to 360. 54:This should allow us to deduce the meaning of the co-ordinate pairs used to 55:describe each cell in a .grim file (we know the first number is the lon or 56:column, the second the lat or row - but which way up are the latitudes? And 57:where do the longitudes break? 58:There is another problem: the values are anomalies, wheras the 'public' 59:.grim files are actual values. So Tim's explanations (in _READ_ME.txt) are 60:incorrect.. 61: 62:8. Had a hunt and found an identically-named temperature database file which 63:did include normals lines at the start of every station. How handy - naming 64:two different files with exactly the same name and relying on their location 65:to differentiate! Aaarrgghh!! Re-ran anomdtb: 66: 67:crua6[/cru/cruts/rerun1/data/cruts/rerun1work] ./anomdtb 68: 69: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 70: 71: > Enter the suffix of the variable required: 72:.tmp 73: > Select the .cts or .dtb file to load: 74:tmp.0311051552.dtb 75: > Specify the start,end of the normals period: 76:1961,1990 77: > Specify the missing percentage permitted: 78:25 79: > Data required for a normal: 23 80: > Specify the no. of stdevs at which to reject data: 81:3 82: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 83:3 84: > Check for duplicate stns after anomalising? (0=no,>0=km range) 85:8 86: > Select the generic .txt file to save (yy.mm=auto): 87:rr2.txt 88: > Select the first,last years AD to save: 89:1901,2002 90: > Operating... 91: > Failed to find file. 92: > Enter the file, with suffix: .dts 93:tmp.0311051552.dts 94:Values loaded: 1255171542; No. Stations: 12155 95: > NORMALS MEAN percent STDEV percent 96: > .dtb 5910325 86.6 97: > .cts 575661 8.4 6485986 95.0 98: > PROCESS DECISION percent %of-chk 99: > no lat/lon 12043 0.2 0.2 100: > no normal 335741 4.9 4.9 101: > out-of-range 31951 0.5 0.5 102: > duplicated 341323 5.0 5.3 103: > accepted 6107721 89.4 104: > Dumping years 1901-2002 to .txt files... 105: 106:crua6[/cru/cruts/rerun1/data/cruts/rerun1work] 107: 108: 109:9. Ran the IDL function: 110:IDL> quick_interp_tdm2,1901,2002,'rr2glofiles/rr2grid.',1200,gs=0.5,dumpglo='dumpglo',pts_prefix='rr2txtfiles/rr2.' 111:% Compiled module: QUICK_INTERP_TDM2. 112:% Compiled module: GLIMIT. 113:Defaults set 114: 1901 115:% Compiled module: MAP_SET. 116:% Compiled module: CROSSP. 117:% Compiled module: STRIP. 118:% Compiled module: SAVEGLO. 119:% Compiled module: SELECTMODEL. 120: 1902 121: (etc) 122: 2002 123:IDL> 124: 125:This produces anomoly files even when given a normals-added 126:database.. doesn't create the CLIMATOLOGY. However we do have 127:it, both in the 'normals' directory of the user data 128:directory, and in the dpe1a 'cru_cl_1.0' folder! The relevant 129:file is 'clim.6190.lan.tmp'. Obviously this is for land 130:only. 131: 132:10. Trying to compare .glo and .grim 133:Wrote several programs to assist with this process. Tried 134:creating anomalies from the .grim files, using the 135:published climatology. Then tried to compare with the glo 136:files I'd produced (this is all for 1961-1970). Couldn't 137:get a sensible grid layout for the glo files! Eventually 138:resorted to visualisation - looks like the .glo files are 139:'regular' grid format after all (longitudes change 140:fastest). Don't understand why the comparison program had 141:so much trouble getting matched cells! 142: 143:11. Decided to concentrate on Norwich. Tim M uses Norwich 144:as the example on the website, so we know it's at (363,286). 145:Wrote a prog to extract the relevant 1961-1970 series from 146:the published output, the generated .glo files, and the 147:published climatology. Prog is norwichtest.for. Prog also 148:creates anomalies from the published data, and raw data 149:from the generated .glo data. Then Matlab prog plotnorwich.m 150:plots the data to allow comparisons. 151:First result: works perfectly, except that the .glo data is 152:all zeros. This means I still don't understand the structure 153:of the .glo files. Argh! 154: 155:12. Trying something *else*. Will write a prog to convert 156:the 1961-1970 .glo files to a single file with 120 columns 157:and a row for each non-zero cell. It will be slow. It is a 158:nuisance because the site power os off this weekend (and 159:it's Friday afternoon) so I will get it running at home. 160:Program is glo2vec.for, and yup it is slow. Started a second 161:copy on uealogin1 and it's showing signs of overtaking the 162:crua6 version that started on Friday (it's Tuesday now). I'm 163:about halfway through and the best correlation so far (as 164:tested by norwichcorr.for) is 0.39 at (170,135) (lon,lat). 165: 166:13. Success! I would crack open a bottle of bubbly but it's 167:only 11.25am. The program norwichcorr.for found a correlation 168:for the norwich series at (363, 286) of 1.00! So we have 169:found the published Norwich series in the grids I produced. A 170:palpable sense of relief pervades the office :-) It's also the 171:grid reference given by Tim for Norwich. So how did I miss it 172:earlier?? 173: 174:14. Wrote a program ('glo2grim.for') to do what I cannot get 175:Tim's 'raw2grim.f90', ie, convert .glo files to GRIM format. 176:It's slow but sure. In parallel, a quick prog called grimcmp.for 177:which compares two GRIM-format files. It produces brief stats. 178:At time of writing, just over 4000 cells have been converted, 179:and the output of grimcmp is: 180: 181:uealogin1[/cru/cruts/rerun1/data/cruts/rerun1work] ./grimcmp 182: 183:Welcome to the GRIM Comparer 184: 185:Please enter the first grim file (must be complete!): cru_ts_2_10.1961-1970.tmp 186:Please enter the second grim file (may be incomplete): glo2grim1.out 187: 188:File glo2grim1.out terminated prematurely after 4037 records. 189: 190:SUMMARY FROM GRIMCMP 191: 192:Files compared: 193:1. cru_ts_2_10.1961-1970.tmp 194:2. glo2grim1.out 195: 196:Total Cells Compared 4037 197:Total 100% Matches 0 198:Cells with Corr. == 1.00 0 ( 0.0%) 199:Cells with 0.90<=Corr<=0.99 3858 (95.6%) 200:Cells with 0.80<=Corr<=0.89 119 ( 2.9%) 201:Cells with 0.70<=Corr<=0.79 25 ( 0.6%) 202: 203:..which is good news! Not brilliant because the data should be 204:identical.. but good because the correlations are so high! This 205:could be a result of my mis-setting of the parameters on Tim's 206:programs (although I have followed his recommendations wherever 207:possible), or it could be a result of Tim using the Beowulf 1 208:cluster for the f90 work. Beowulf 1 is now integrated in to the 209:latest Beowulf cluster so it may not be practical to test that 210:theory. 211: 212:15. All change! My 'glo2grim1' program was presciently named as 213:it's now up to v3! My attempt to speed up early iterations by 214:only reading as much of each glo file as was needed was really 215:stupidly coded and hence the poor results. Actually they're 216:worryingly good as the data was effectively random :-0 217:We are now on-beam and initial results are very very promising: 218: 219:uealogin1[/cru/cruts/rerun1/data/cruts/rerun1work] ./grimcmp3x 220: 221:File glo2grim3.out terminated prematurely after 143 records. 222: 223:SUMMARY FROM GRIMCMP 224: 225:Files compared: 226:1. cru_ts_2_10.1961-1970.tmp 227:2. glo2grim3.out 228: 229:Total Cells Compared 143 230:Total 100% Matches 12 231:Cells with Corr. == 1.00 12 ( 8.4%) 232:Cells with 0.96<=Corr<=0.99 130 (90.9%) 233:Cells with 0.90<=Corr<=0.95 1 ( 0.7%) 234:Cells with 0.80<=Corr<=0.89 0 ( 0.0%) 235:Cells with 0.70<=Corr<=0.79 0 ( 0.0%) 236: 237:..so all correlations are >= 0.9 and all but one are >=0.96! 238:with 12 complete (100% identical) matches I think we can safely 239:say we are producing the data Tim produced. The variations can 240:be accounted for as rounding errors due to different hardware 241:and compilers, I reckon.. 242: 243:16. So, it seemed like a good time to start a Precip run. With 244:a bit of luck this would go as smoothly as the Temperature run, 245:ho, ho, ho. The first problem was that anomdtb kept crashing: 246: 247:crua6[/cru/cruts/rerun1/data/cruts/rerun2work] ./anomdtb 248: 249: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 250: 251: > Enter the suffix of the variable required: 252:.pre 253: > Will calculate percentage anomalies. 254: > Select the .cts or .dtb file to load: 255:pre.0312031600.dtb 256: > Specify the start,end of the normals period: 257:1961,1990 258: > Specify the missing percentage permitted: 259:25 260: > Data required for a normal: 23 261: > Specify the no. of stdevs at which to reject data: 262:3 263: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 264:3 265: > Check for duplicate stns after anomalising? (0=no,>0=km range) 266:8 267: > Select the generic .txt file to save (yy.mm=auto): 268:rr2pre.txt 269: > Select the first,last years AD to save: 270:1901,2002 271: > Operating... 272:Values loaded: 1258818288; No. Stations: 12732 273: > NORMALS MEAN percent STDEV percent 274: > .dtb 2635549 29.6 275:forrtl: error (75): floating point exception 276:IOT trap (core dumped) 277:crua6[/cru/cruts/rerun1/data/cruts/rerun2work] 278: 279:..not good! Tried recompiling for uealogin1.. AARGGHHH!!! Tim's 280:code is not 'good' enough for bloody Sun!! Pages of warnings and 281:27 errors! (full results in 'anomdtb.uealogin1.compile.results'). 282: 283:17. Inserted debug statements into anomdtb.f90, discovered that 284:a sum-of-squared variable is becoming very, very negative! Key 285:output from the debug statements: 286: 287:OpEn= 16.00, OpTotSq= 4142182.00, OpTot= 7126.00 288:DataA val = 93, OpTotSq= 8649.00 289:DataA val = 172, OpTotSq= 38233.00 290:DataA val = 950, OpTotSq= 940733.00 291:DataA val = 797, OpTotSq= 1575942.00 292:DataA val = 293, OpTotSq= 1661791.00 293:DataA val = 83, OpTotSq= 1668680.00 294:DataA val = 860, OpTotSq= 2408280.00 295:DataA val = 222, OpTotSq= 2457564.00 296:DataA val = 452, OpTotSq= 2661868.00 297:DataA val = 561, OpTotSq= 2976589.00 298:DataA val = 49920, OpTotSq=-1799984256.00 299:DataA val = 547, OpTotSq=-1799684992.00 300:DataA val = 672, OpTotSq=-1799233408.00 301:DataA val = 710, OpTotSq=-1798729344.00 302:DataA val = 211, OpTotSq=-1798684800.00 303:DataA val = 403, OpTotSq=-1798522368.00 304:OpEn= 16.00, OpTotSq=-1798522368.00, OpTot=56946.00 305:forrtl: error (75): floating point exception 306:IOT trap (core dumped) 307: 308:..so the data value is unbfeasibly large, but why does the 309:sum-of-squares parameter OpTotSq go negative?!! 310: 311:Probable answer: the high value is pushing beyond the single- 312:precision default for Fortran reals? 313: 314:Value located in pre.0312031600.dtb: 315: 316:-400002 3513 3672 309 HAMA SYRIA 1985 2002 -999 -999 317:6190 842 479 3485 339 170 135 106 0 9 243 387 737 318:1985 887 582 93 16 17 0 0 0 0 352 221 627 319:1986 899 252 172 527 173 30 0 0 0 84 496 570 320:1987 578 349 950 191 4 0 0 0 0 343 462 929 321:1988 1044 769 797 399 11 903 218 0 0 163 517 1181 322:1989 269 62 293 3 13 0 0 0 0 101 292 342 323:1990 328 276 83 135 224 0 0 0 0 87 343 230 324:1991 1297 292 860 320 70 0 0 0 0 206 298 835 325:1992 712 1130 222 39 339 301 0 0 0 0 909 351 326:1993 726 609 452 82 672 3 0 0 0 34 183 351 327:1994 625 661 561 41 155 0 0 0 22 345 953 1072 328:1995 488-9999-9999 182-9999 0-9999 0 0 0 754-9999 329:1996-9999 40949920-9999 82 0-9999 0 36 414 112 312 330:1997-9999 339 547-9999 561-9999 0 0 54 155 265 962 331:1998 1148 289 672 496-9999 0 0-9999 9 21-9999 1206 332:1999 343 379 710 111 0 0 0-9999-9999-9999 132 285 333:2000 1518 399 211 354 27 0-9999 0 27 269 316 1057 334:2001 370-9999-9999 273 452 0-9999-9999-9999 290 356-9999 335:2002 871 329 403 111 233-9999 0 0-9999-9999 377 1287 336: 337:(value is for March 1996) 338: 339:Action: value replaced with -9999 and file renamed: 340: 341:pre.0312031600H.dtb (to indicate I've fixed it) 342: 343:.dts file also renamed for consistency. 344: 345:anomdtb then runs fine!! Producing the usual txt files. 346: 347:18. Ran the IDL gridding routine for the precip files: 348: 349:quick_interp_tdm2,1901,2002,'rr2preglofiles/rr2pregrid.',450,gs=0.5,dumpglo='dumpglo',pts_prefix='rr2pretxtfiles/rr2pre.' 350: 351:..and this is where it gets CRAZY. Instead of running normally, 352:this time I get: 353: 354:IDL> quick_interp_tdm2,1901,1910,'rr2glofiles2/rr2grid.',1200,gs=0.5,dumpglo='dumpglo',pts_prefix='rr2txtfiles/rr2.' 355: 356:limit=glimit(/all) ; sets limit to global field 357: ^ 358:% Syntax error. 359: At: /cru/cruts/fromdpe1a/code/idl/pro/quick_interp_tdm2.pro, Line 38 360: 361: lim=glimit(/all) 362: ^ 363:% Syntax error. 364: At: /cru/cruts/fromdpe1a/code/idl/pro/quick_interp_tdm2.pro, Line 122 365: 366: r=area_grid(pts2(n,1),pts2(n,0),pts2(n,2),gs*2.0,bounds,dist,angular=angular) 367: ^ 368:% Syntax error. 369: At: /cru/cruts/fromdpe1a/code/idl/pro/quick_interp_tdm2.pro, Line 183 370:% Compiled module: QUICK_INTERP_TDM2. 371:% Attempt to call undefined procedure/function: 'QUICK_INTERP_TDM2'. 372:% Execution halted at: $MAIN$ 373:IDL> 374: 375:.. WHAT?! Now it's not precompiling its functions for some reason! 376:What's more - I cannot find the 'glimit' function anywhere!! 377: 378:Eventually (the following day) I found glimit and area_grid, they are 379:in Mark New's folder: /cru/u2/f080/Idl. Since this is in $IDL_PATH I 380:have no idea why they're not compiling! I manually compiled them with 381:.compile, and the errors vanished! Though not for long: 382: 383:IDL> .compile /cru/u2/f080/Idl/glimit.pro 384:% Compiled module: GLIMIT. 385:IDL> .compile /cru/u2/f080/Idl/area_grid.pro 386:% Compiled module: AREA_GRID. 387:IDL> quick_interp_tdm2,1901,1910,'rr2glofiles2/rr2grid.',1200,gs=0.5,dumpglo='dumpglo',pts_prefix='rr2txtfiles/rr2.' 388:% Compiled module: QUICK_INTERP_TDM2. 389:Defaults set 390: 1901 391:% Compiled module: MAP_SET. 392:% Compiled module: CROSSP. 393:% Variable is undefined: STRIP. 394:% Execution halted at: QUICK_INTERP_TDM2 215 /cru/cruts/fromdpe1a/code/idl/pro/quick_interp_tdm2.pro 395:% $MAIN$ 396:IDL> 397: 398:Was this a similar problem? Unfortunately not: 399: 400:IDL> .compile /cru/u2/f080/Idl/strip.pro 401:% Compiled module: STRIP. 402:IDL> quick_interp_tdm2,1901,1910,'rr2glofiles2/rr2grid.',1200,gs=0.5,dumpglo='dumpglo',pts_prefix='rr2txtfiles/rr2.' 403:Defaults set 404: 1901 405:% Variable is undefined: STRIP. 406:% Execution halted at: QUICK_INTERP_TDM2 215 /cru/cruts/fromdpe1a/code/idl/pro/quick_interp_tdm2.pro 407:% QUICK_INTERP_TDM2 215 /cru/cruts/fromdpe1a/code/idl/pro/quick_interp_tdm2.pro 408:% $MAIN$ 409:IDL> 410: 411:..so it looks like a path problem. I wondered if the NFS errors that have 412:been plagueing crua6 work for some time now might have prevented IDL from 413:adding the correct directories to the path? After all the help file does 414:mention that IDL discards any path entries that are inaccessible.. so if 415:the timeout is a few seconds that would explain it. So I restarted IDL, 416:and PRESTO! It worked. I then tried the precip veriosn - and it worked 417:too! 418: 419:IDL> quick_interp_tdm2,1901,2002,'rr2preglofiles/rr2pregrid.',450,gs=0.5,dumpglo='dumpglo',pts_prefix='rr2pretxtfiles/rr2pre.' 420:% Compiled module: QUICK_INTERP_TDM2. 421:% Compiled module: GLIMIT. 422:Defaults set 423: 1901 424:% Compiled module: MAP_SET. 425:% Compiled module: CROSSP. 426:% Compiled module: STRIP. 427:% Compiled module: SAVEGLO. 428:% Compiled module: SELECTMODEL. 429: 1902 430:(etc) 431: 2001 432: 2002 433:IDL> 434: 435:I then ran glo2grim4.for to convert from percentage anomalies to real 436:(10ths of a mm) values. Initial results are not as good as temperature, 437:but mainly above 0.96 so obviously on the right track. 438: 439:However.. 440: 441:19. Here is a little puzzle. If the latest precipitation database file 442:contained a fatal data error (see 17. above), then surely it has been 443:altered since Tim last used it to produce the precipitation grids? But 444:if that's the case, why is it dated so early? Here are the dates: 445: 446:/cru/dpe1a/f014/data/cruts/database/+norm/pre.0312031600.dtb 447:- directory date is 23 Dec 2003 448: 449:/cru/tyn1/f014/ftpfudge/data/cru_ts_2.10/data_dec/cru_ts_2_10.1961-1970.pre.Z 450:- directory date is 22 Jan 2004 (original date not preserved in zipped file) 451:- internal (header) date is also '22.01.2004 at 17:57' 452: 453:So what's going on? I don't see how the 'final' precip file can have been 454:produced from the 'final' precipitation database, even though the dates 455:imply that. The obvious conclusion is that the precip file must have been 456:produced before 23 Dec 2003, and then redated (to match others?) in Jan 04. 457: 458:20. Secondary Variables - Eeeeeek!! Yes the time has come to attack what even 459:Tim seems to have been unhappy about (reading between the lines). To assist 460:me I have 12 lines in the gridding ReadMe file.. so par for the course. 461:Almost immediately I hit that familiar feeling of ambiguity: the text 462:suggests using the following three IDL programs: 463: frs_gts_tdm.pro 464: rd0_gts_tdm.pro 465: vap_gts_anom.pro 466:So.. when I look in the code/idl/pro/ folder, what do I find? Well: 467: 468: 3447 Jan 22 2004 fromdpe1a/code/idl/pro/frs_gts_anom.pro 469: 2774 Jun 12 2002 fromdpe1a/code/idl/pro/frs_gts_tdm.pro 470: 471: 2917 Jan 8 2004 fromdpe1a/code/idl/pro/rd0_gts_anom.pro 472: 2355 Jun 12 2002 fromdpe1a/code/idl/pro/rd0_gts_tdm.pro 473: 474: 5880 Jan 8 2004 fromdpe1a/code/idl/pro/vap_gts_anom.pro 475: 476:In other words, the *anom.pro scripts are much more recent than the *tdm 477:scripts. There is no way of knowing which Tim used to produce the current 478:public files. The scripts differ internally but - you guessed it! - the 479:descriptions at the start are identical. WHAT IS GOING ON? Given that the 480:'README_GRIDDING.txt' file is dated 'Mar 30 2004' we will have to assume 481:that the originally-stated scripts must be used. 482: 483:To begin with, we need binary output from quick_interp_tdm2, so it's run 484:again for tmp and pre, and (for the first time) for dtr. This time, the 485:command line looks like this for tmp: 486:IDL> quick_interp_tdm2,1901,2002,'idlbinout/idlbin',1200,gs=2.5,dumpbin='dumpbin',pts_prefix='tmp_txt_4idl/tmp.' 487:This gives screen output for each year, typically: 488: 1991 489:grid 1991 non-zero 0.9605 2.0878 2.1849 cells= 27048 490:And produces output files (in, in this case, 'idlbinout/'), like this: 491:-rw------- 1 f098 cru 248832 Sep 21 12:20 idlbin_tmp/idlbin_tmp1991 492: 493:At this point, did some logical renaming. So.. 494:.txt files (pre-IDL) are typically 'tmp.1901.01.txt' in 'tmp_txt_4idl/' 495:binary files (post-IDL) are typically 'idlbin_tmp1991' in 'idlbin_tmp/'. 496:These changes rolled back to the quoted command lines, to avoid confusion. 497: 498:Next, precip command line: 499:IDL> quick_interp_tdm2,1901,2002,'idlbin_pre/idlbin_pre',450,gs=2.5,dumpbin='dumpbin',pts_prefix='pre_txt_4idl/pre.' 500:(note new filenaming schema) 501:This gives example screen output: 502: 1991 503:grid 1991 non-zero -4.8533 36.2155 51.0738 cells= 51060 504:And produces output files like: 505:-rw------- 1 f098 cru 248832 Sep 21 12:50 idlbin_pre/idlbin_pre1991 506: 507:Finally for the primaries, the first stab at dtr. Ran anomdtb with the 508:database file dtr.0312221128.dtb, and the standard/recommended responses. 509:Screen output: 510: > NORMALS MEAN percent STDEV percent 511: > .dtb 0 0.0 512: > .cts 3375441 84.1 3375441 84.1 513: > PROCESS DECISION percent %of-chk 514: > no lat/lon 3088 0.1 0.1 515: > no normal 638538 15.9 15.9 516: > out-of-range 70225 1.7 2.1 517: > duplicated 135457 3.4 4.1 518: > accepted 3167636 78.9 519: > Dumping years 1901-2002 to .txt files... 520: 521:Then for the gridding: 522:IDL> quick_interp_tdm2,1901,2002,'idlbin_dtr/idlbin_dtr',750,gs=2.5,dumpbin='dumpbin',pts_prefix='dtr_txt_4idl/dtr.' 523:Giving screen output: 524: 1991 525:grid 1991 non-zero -0.3378 1.6587 1.7496 cells= 3546 526:And files such as: 527:-rw------- 1 f098 cru 248832 Sep 21 13:39 idlbin_dtr/idlbin_dtr1991 528: 529:And.. at this point, I read the ReadMe file properly. I should be gridding at 530:2.5 degrees not 0.5 degrees! For some reason, secondary variables are not 531:derived from the 0.5 degree grids. Re-did all three generations (the sample 532:command lines and outputs above have been altered to reflect this, to avoid 533:confusion). 534: 535:So, to the generation of the synthetic grids. 536: 537:Tried running frs_gts_tdm but it complained it couldn't find the normals file: 538: 539:IDL> frs_gts_tdm,dtr_prefix='idlbin_dtr/idlbin_dtr',tmp_prefix='idlbin_tmp/idlbin_tmp',1901,2002,outprefix='syngrid_frs/syngrid_frs' 540:% Compiled module: FRS_GTS_TDM. 541:% Attempt to call undefined procedure/function: 'FRS_GTS_TDM'. 542:% Execution halted at: $MAIN$ 543:IDL> frs_gts,dtr_prefix='idlbin_dtr/idlbin_dtr',tmp_prefix='idlbin_tmp/idlbin_tmp',1901,2002,outprefix='syngrid_frs/syngrid_frs' 544:% Compiled module: RDBIN. 545:% Compiled module: STRIP. 546:ls: /home/cru/f098/m1/gts/frs/glo/glo.frs.norm not found 547:ls: /home/cru/f098/m1/gts/frs/glo/glo.frs.norm.Z not found 548:ls: /home/cru/f098/m1/gts/frs/glo/glo.frs.norm.gz not found 549:% READF: End of file encountered. Unit: 99, File: foo 550:% Execution halted at: RDBIN 25 /cru/u2/f080/Idl/rdbin.pro 551:% FRS_GTS 18 /cru/cruts/fromdpe1a/code/idl/pro/frs_gts_tdm.pro 552:% $MAIN$ 553:IDL> 554: 555:However when I eventually found what I hope is the normals file: 556: 557:/cru/cruts/fromdpe1a/data/grid/twohalf/glo25.frs.6190 558: 559:..and altered the IDL prog to read it.. same error! Turns out it's preferring 560:to pick up Mark N's version so tried explicitly compiling, 561:('.compile xxxxxx.pro') that worked, in that the error changed: 562: 563:IDL> frs_gts,dtr_prefix='idlbin_dtr/idlbin_dtr',tmp_prefix='idlbin_tmp/idlbin_tmp',1901,2002,outprefix='syngrid_frs/syngrid_frs' 564:% Compiled module: RDBIN. 565:% Compiled module: STRIP. 566:yes 567:% Variable is undefined: NF. 568:% Execution halted at: RDBIN 68 /cru/u2/f080/Idl/rdbin.pro 569:% FRS_GTS 21 /cru/cruts/fromdpe1a/code/idl/pro/frs_gts_tdm.pro 570:% $MAIN$ 571:IDL> 572: 573:So what is this mysterious variable 'nf' that isn't being set? Well strangely, 574:it's in Mark N's 'rdbin.pro'. I say strangely because this is a generic prog 575:that's used all over the place! Nonetheless it does have what certainly looks 576:like a bug: 577: 578: 38 if keyword_set(gridsize) eq 0 then begin 579: 39 info=fstat(lun) 580: 40 if keyword_set(seas) then info.size=info.size*2.0 581: 41 if keyword_set(ann) then info.size=info.size*12.0 582: 42 nlat=sqrt(info.size/48.0) 583: 43 gridsize=180.0/nlat 584: 44 if keyword_set(quiet) eq 0 then print,'filesize=',info.size 585: 45 if keyword_set(quiet) eq 0 then print,'gridsize=',gridsize 586: 46 endif 587: 47 if keyword_set(had) then had=1 else had=0 588: 48 if keyword_set(echam) then echam=1 else echam=0 589: 49 if keyword_set(gfdl) then gfdl=1 else gfdl=0 590: 50 if keyword_set(ccm) then ccm=1 else ccm=0 591: 51 if keyword_set(csiro) then csiro=1 else csiro=0 592: 52 ;create array to read data into 593: 53 if keyword_set(seas) then nf=6 else nf=12 594: 54 if keyword_set(ann) then nf=1 595: 55 defxyz,lon,lat,gridsize,grid=grid,nf=nf,had=had,echam=echam,gfdl=gfdl,ccm=ccm,csiro=csiro 596: 56 if keyword_set(quiet) eq 0 then help,grid 597: 57 grid=fix(grid) 598: 58 ;read data 599: 59 readu,lun,grid 600: 60 close,lun 601: 61 spawn,string('rm -f ',fff) 602: 62 endif else begin 603: 63 openr,lun,fname 604: 64 ; check file size and work out grid spacing if gridsize isn't set 605: 65 if keyword_set(gridsize) eq 0 then begin 606: 66 info=fstat(lun) 607: 67 if keyword_set(quiet) eq 0 then print,'yes' 608: 68 nlat=sqrt((info.size/nf)/4.0) 609: 69 gridsize=180.0/nlat 610: 70 if keyword_set(quiet) eq 0 then print,'filesize=',info.size 611: 71 if keyword_set(quiet) eq 0 then print,'gridsize=',gridsize 612: 72 endif 613: 73 if keyword_set(seas) then nf=6.0 else nf=12.0 614: 74 if keyword_set(ann) then nf=1 615: 616:In other words, 'nf' is set in the first conditional set of statements, but in 617:the alternative (starting on #62) it is only set AFTER it's used 618:(set #73,#74; used #68). So I shifted #73 and #74 to between #64 and #65, and.. 619:with precompiling to pick up the local version of rdbin, too.. it worked! 620:Er, perhaps. 621: 622:Lots of screen output, and lots of files. A set of synthetic grids in 'syngrid_frs/' as requested, typically: 623: 624:-rw------- 1 f098 cru 20816 Sep 17 22:10 syngrid_frs/syngrid_frs1991.Z 625: 626:..but also a set of some binariy files in the working directory! They look like this: 627: 628:-rw------- 1 f098 cru 51542 Sep 17 22:10 glo.frs.1991.Z 629: 630:Having read the program it looks as though the latter files are absolutes, 631:whereas the former are anomalies. With this in mind, they are renamed: 632: 633:glo.frs.1991 -> glo.frs.abs.1991 634: 635:..and put into folder syngrid_frs_abs/ 636: 637:Then - a real setback. Looked for a database file for frost.. nothing. Is 638:this a real secondary parameter? Answer: yes. Further digging revealed that 639:quick_interp_tdm2.pro has a 'nostn' command line option. It's undocumented, 640:as usual, but it does seem to avoid the use of the 'pts_prefix' option.. so 641:I set it, and it at least *ran* for the full term (though very slow compared 642:to primary variables)! 643: 644:IDL> quick_interp_tdm2,1901,2002,'glo_frs_grids/frs.grid.',750,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='syngrid_frs/syngrid_frs' 645: 646:It does produce output grids. Without converting to absolutes with the normals file, 647:it's hard to know if they're realistic. 648: 649:Then, I moved on to rd0 (wet-day frequency). This time, when I searched for the 650:normals files required ('glo.pre.norm' and 'glo.rd0.norm'), I could not (as before) 651:find exact matches. The difference this time is that the program checks that the 652:normals file supplied is a 0.5-degree grid, so glo25.pre.6190 failed. This implies 653:to me that my approach to frs (above) was wrong as well. Where is the documenatation 654:to explain all this?! 655: 656:Finally - a breakthrough. A search of Mark New's old directory hierarchy revealed 657:what look like the required files: 658: 659:crua6[/cru/mark1/f080] find . -name 'glo.*.norm*' 660:./gts/cld/glo/glo.cld.norm.Z 661:./gts/dtr/glo_old/glo.dtr.norm.Z 662:./gts/frs/glo.frs.norm.Z 663:./gts/frs/glo/glo.frs.norm.Z 664:find: cannot open < ./gts/frs/glo_txt > 665:./gts/pre/glo_quick_abs/glo.pre.norm.Z 666:./gts/pre/glo_quick_log/glo.pre.norm.Z 667:./gts/pre/glo_spl/glo.pre.norm.Z 668:find: cannot open < ./gts/pre_perc/station_list > 669:./gts/rad/glo/glo.rad.norm.Z 670:./gts/rd0/glo/glo.rd0.norm.Z 671:./gts/rd0/glo_old/glo.rd0.norm.Z 672:./gts/sunp/glo/glo.sunp.norm 673:./gts/sunp/means/glo.sunp.norm.Z 674:./gts/tmp/glo/glo.tmp.norm.Z 675:./gts/tmp/glo_old/glo.tmp.norm.Z 676:find: cannot open < ./gts/tmp/station_list > 677:./gts/vap/glo/glo.vap.norm.Z 678:./gts/wnd/glo/glo.wnd.norm.Z 679: 680:A listing of /cru/mark1/f080/gts gives: 681: 682:drwxr-x--- 2 f080 cru 1024 Sep 12 2005 cdrom 683:drwxr-x--- 10 f080 cru 57344 Nov 1 2001 cld 684:drwxr-xr-x 19 f080 cru 24576 Feb 27 2001 dtr 685:drwxr-x--- 2 f080 cru 8192 Feb 25 1998 elev 686:drwxr-x--- 2 f080 cru 8192 Jun 8 1998 euroclivar 687:-rw-r----- 1 f080 cru 0 Aug 3 1999 foo 688:drwxr-x--- 6 f080 cru 8192 Aug 6 2002 frs 689:-rw-r-x--- 1 f080 cru 438 May 12 1998 gts.errors 690:-rw-r----- 1 f080 cru 10 Jul 21 1999 in 691:drwxr-x--- 5 f080 cru 8192 Jan 6 1999 jiang 692:drwxr-x--- 2 f080 cru 8192 Apr 7 1998 landsea 693:-rw-r----- 1 f080 cru 240 May 12 1998 normal.errors 694:drwxr-x--- 5 f080 cru 8192 Aug 6 2002 plots 695:drwxr-xr-x 12 f080 cru 106496 May 22 2000 pre 696:drwxr-x--- 9 f080 cru 114688 Aug 6 2002 pre_perc 697:drwxr-x--- 4 f080 cru 1024 Jan 6 1999 rad 698:drwxr-x--x 6 f080 cru 8192 Nov 1 2001 rd0 699:-rwxr-xr-- 1 f080 cru 1779 Dec 5 1997 readme.txt 700:drwxr-x--- 8 f080 cru 1024 Apr 5 2000 reg_series 701:drwxr-x--- 3 f080 cru 1024 Oct 18 1999 reh 702:drwxr-x--- 2 f080 cru 8192 Jan 19 2000 scengen 703:drwxr-x--- 5 f080 cru 24576 Nov 5 1998 sunp 704:drwxr-x--- 2 f080 cru 1024 Aug 6 2002 test 705:drwxr-x--- 4 f080 cru 1024 Aug 3 1999 tmn 706:drwxr-xr-x 20 f080 cru 122880 Mar 19 2002 tmp 707:drwxr-x--- 4 f080 cru 1024 Aug 3 1999 tmx 708:drwxr-x--- 6 f080 cru 1024 Jul 8 1998 ukcip 709:drwxr-x--- 5 f080 cru 8192 Nov 5 2001 vap 710:drwxr-x--- 4 f080 cru 1024 Jul 2 1998 wnd 711: 712:And a listing of, for example, the 'frs' directory: 713: 714:drwxr-x--- 2 f080 cru 16384 Jul 18 2002 glo 715:-rw-r-x--- 1 f080 cru 433393 Aug 12 1998 glo.frs.1961.Z 716:-rw-r-x--- 1 f080 cru 321185 Aug 12 1998 glo.frs.ano.1961.Z 717:-rw-r-x--- 1 f080 cru 740431 Aug 12 1998 glo.frs.norm.Z 718:drwxr-xr-x 2 f080 cru 16384 Jul 27 1999 glo25 719:drwx------ 2 f080 cru 8192 Jul 18 2002 glo_txt 720:drwxr-xr-x 2 f080 cru 8192 Aug 28 1998 means 721: 722:So, the following were copied to the working area: 723: 724:cp /cru/mark1/f080/gts/frs/glo.frs.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 725:cp /cru/mark1/f080/gts/cld/glo/glo.cld.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 726:cp /cru/mark1/f080/gts/dtr/glo_old/glo.dtr.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 727: 728:precip looked like it might be a problem (3 matching files, see above), 729:but on investigation they were found to be identical! Wonderful. 730: 731:cp /cru/mark1/f080/gts/pre/glo_quick_log/glo.pre.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 732:cp /cru/mark1/f080/gts/rad/glo/glo.rad.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 733:cp /cru/mark1/f080/gts/rd0/glo/glo.rd0.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 734: 735:There were two 'sunp' norm files, but one was 0 bytes in length. 736: 737:cp /cru/mark1/f080/gts/sunp/means/glo.sunp.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 738:cp /cru/mark1/f080/gts/tmp/glo/glo.tmp.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 739:cp /cru/mark1/f080/gts/vap/glo/glo.vap.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 740:cp /cru/mark1/f080/gts/wnd/glo/glo.wnd.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/ 741: 742:The synthetics generation was then re-run for frs (records above have 743:been modified to reflect this). 744: 745:Next, rd0. Synthetics generated OK.. 746: 747:IDL> rd0_gts,1901,2002,1961,1990,outprefix="syngrid_rd0/syngrid_rd0",pre_prefix="idlbin_pre/idlbin_pre" 748: 749:..until the end: 750: 751: 2001 752:yes 753:filesize= 248832 754:gridsize= 2.50000 755: 2002 756:yes 757:filesize= 248832 758:gridsize= 2.50000 759:% Program caused arithmetic error: Floating divide by 0 760:% Program caused arithmetic error: Floating illegal operand 761:IDL> 762: 763:However, all synthetic grids appear to have been written OK, including 2002. 764: 765:Grid generation proceeded without error: 766: 767:IDL> quick_interp_tdm2,1901,2002,'glo_rd0_grids/rd0.grid.',450,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='syngrid_rd0/syngrid_rd0' 768: 769: 770:Onto vapour pressure, and the crunch. For here, the recommended program for 771:synthetic grid production is 'vap_gts_anom.pro'. In fact, there is no sign 772:of a 'vap_gts_tdm.pro'. And, in the program notes, it reads: 773: 774:; required inputs are: 775:; ** vapour pressure and temperature normals on 2.5deg grid 776:; (these come ready-supplied for a 1961-90 normal period) 777:; ** temp and dtr monthly anomalies on 2.5deg grid, including normal period 778: 779:So, we face a situation where some synthetics are built with 0.5-degree 780:normals, and others are built with 2.5-degree normals. I can find no 781:documentation of this. There are '*_anom.pro' versions of the frs and rd0 782:programs, both of which use 2.5-degree normals, however they are dated 783:Jan 2004, and Tim's Read_Me (which refers to the '*_tdm.pro' 0.5-degree 784:versions) is dated end March 2004, so we have to assume these are his 785:best suggestions. 786: 787:The 2.5 normals are found here: 788: 789:> ls -l /cru/cruts/fromdpe1a/data/grid/twohalf/ 790:total 1248 791:-rwxr-xr-x 1 f098 cru 248832 Jan 9 2004 glo25.frs.6190 792:-rwxr-xr-x 1 f098 cru 248832 Jan 8 2004 glo25.pre.6190 793:-rwxr-xr-x 1 f098 cru 248832 Jan 8 2004 glo25.rd0.6190 794:-rwxr-xr-x 1 f098 cru 248832 Jan 7 2004 glo25.tmp.6190 795:-rwxr-xr-x 1 f098 cru 248832 Jan 6 2004 glo25.vap.6190 796:-rwxr-xr-x 1 f098 cru 86 Feb 25 2004 readme.txt 797: 798:readme.txt: 799:2.5deg climatology files 800:Tim Mitchell, 25.2.04 801: 802:These are in Mark New's binary format 803:(end) 804: 805:Set up the required inputs, and ran it: 806: 807:IDL> vap_gts_anom,dtr_prefix='idlbin_dtr/idlbin_dtr',tmp_prefix='idlbin_tmp/idlbin_tmp',1901,2002,outprefix='syngrid_vap/syngrid_vap',dumpbin=1 808: 809:Producing screen output like this: 810:1991 vap (x,s2,<<,>>): 0.000493031 0.000742087 -0.0595093 1.86497 811: 812:And output files like this: 813:-rw------- 1 f098 cru 248832 Sep 22 10:56 syngrid_vap/syngrid_vap1991 814: 815:On, without further ado, to the gridding. For this secondary, there *are* database 816:files, so the 'nostn' option is not used, and anomdtb.f is wheeled out again 817:to construct .txt files for the run: 818: 819:crua6[/cru/cruts/rerun1/data/cruts/rerun_vap] ./anomdtb 820: 821: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 822: 823: > Enter the suffix of the variable required: 824:.vap 825: > Select the .cts or .dtb file to load: 826:vap.0311181410.dtb 827: > Specify the start,end of the normals period: 828:1961,1990 829: > Specify the missing percentage permitted: 830:25 831: > Data required for a normal: 23 832: > Specify the no. of stdevs at which to reject data: 833:3 834: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 835:3 836: > Check for duplicate stns after anomalising? (0=no,>0=km range) 837:8 838: > Select the generic .txt file to save (yy.mm=auto): 839:vap.txt 840: > Select the first,last years AD to save: 841:1901,2002 842: > Operating... 843:Values loaded: 1239868112; No. Stations: 7691 844: > NORMALS MEAN percent STDEV percent 845: > .dtb 887754 46.9 846: > .cts 34175 1.8 921929 48.7 847: > PROCESS DECISION percent %of-chk 848: > no lat/lon 105 0.0 0.0 849: > no normal 969384 51.3 51.3 850: > out-of-range 2661 0.1 0.3 851: > duplicated 25557 1.4 2.8 852: > accepted 893711 47.3 853: > Dumping years 1901-2002 to .txt files... 854: 855:crua6[/cru/cruts/rerun1/data/cruts/rerun_vap] 856: 857:Moved straight onto the gridding, which, of course, failed: 858: 859:IDL> quick_interp_tdm2,1901,2002,'glo_vap_grids/vap.grid.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='syngrid_vap/syngrid_vap',pts_prefix='../rerun_vap/vap_txt_4idl/vap.' 860:Defaults set 861: 1901 862: 1902 863:% Array dimensions must be greater than 0. 864:% Execution halted at: QUICK_INTERP_TDM2 88 /cru/cruts/fromdpe1a/code/idl/pro/quick_interp_tdm2.pro 865:% QUICK_INTERP_TDM2 88 /cru/cruts/fromdpe1a/code/idl/pro/quick_interp_tdm2.pro 866:% $MAIN$ 867:IDL> 868: 869:This turns out to be because of the sparcity of VAP station measurements in the 870:early years. The program cannot handle anom files of 0 length, even though it 871:checks the length! Bizarre. The culprit is 'vap.1902.03.txt', the only month to 872:have no station reading at all (45 months have only 1 however). I decided to mod 873:the program to use the 'nostn' option if the length is 0. Hope that's right - the 874:synthetics are read in first and the station data is added to that grid so this 875:should be OK.. and it looks OK: 876: 877:IDL> quick_interp_tdm2,1901,2002,'vap.grid.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='syngrid_vap/syngrid_vap',pts_prefix='../rerun_vap/vap_txt_4idl/vap.' 878:% Compiled module: GLIMIT. 879:Defaults set 880: 1901 881: 1902 882:no stations found in: ../rerun_vap/vap_txt_4idl/vap.1902.03.txt 883: 1903 884: 885:(..etc..) 886: 887:Pause for reflection: the list of CRU_TS_2.1 parameters is as follows: 888:pre primary, done 889:tmp primary, done 890:tmx derived, not done 891:tmn derived, not done 892:dtr primary, done 893:vap secondary, done 894:cld/spc secondary, not done 895:wet secondary, done 896:frs secondary, done 897: 898:Now the interesting thing is that the 'Read Me' file for gridding only 899:mentions frs, rd0 (which I'm assuming == wet) and vap. How, then, do I 900:produce cld/spc and the two derived vars?? 901: 902:Well, there's a /cru/cruts/fromdpe1a/code/idl/pro/cal_cld_gts_tdm.pro, 903:also: 904:/cru/cruts/fromdpe1a/code/idl/pro/cloudcorrspc.pro 905:/cru/cruts/fromdpe1a/code/idl/pro/cloudcorrspcann.pro 906:/cru/cruts/fromdpe1a/code/idl/pro/cloudcorrspcann9196.pro 907: 908:Loading just the first program opens up another huge can o' worms. The 909:program description reads: 910: 911:pro cal_cld_gts_tdm,dtr_prefix,outprefix,year1,year2,info=info 912:; calculates cld anomalies using relationship with dtr anomalies 913:; reads coefficients from predefined files (*1000) 914:; reads DTR data from binary output files from quick_interp_tdm2.pro (binfac=1000) 915:; creates cld anomaly grids at dtr grid resolution 916:; output can then be used as dummy input to splining program that also 917:; includes real cloud anomaly data 918: 919:So, to me this identifies it as the program we cannot use any more because 920:the coefficients were lost. As it says in the gridding read_me: 921: 922:Bear in mind that there is no working synthetic method for cloud, because Mark New 923: lost the coefficients file and never found it again (despite searching on tape 924: archives at UEA) and never recreated it. This hasn't mattered too much, because 925: the synthetic cloud grids had not been discarded for 1901-95, and after 1995 926: sunshine data is used instead of cloud data anyway. 927: 928:But, (Lord how many times have I used 'however' or 'but' in this file?!!), when 929:you look in the program you find that the coefficient files are called: 930: 931:rdbin,a,'/cru/tyn1/f709762/cru_ts_2.0/_constants/_7190/a.25.7190',gridsize=2.5 932:rdbin,b,'/cru/tyn1/f709762/cru_ts_2.0/_constants/_7190/b.25.7190',gridsize=2.5 933: 934:And, if you do a search over the filesystems, you get: 935: 936:crua6[/cru/cruts] ls fromdpe1a/data/grid/cru_ts_2.0/_makecld/_constants/_7190/spc2cld/_ann/ 937:a.25.01.7190.glo.Z a.25.05.7190.glo.Z a.25.09.7190.glo.Z a.25.7190.eps.Z b.25.04.7190.glo.Z b.25.08.7190.glo.Z b.25.12.7190.glo.Z 938:a.25.02.7190.glo.Z a.25.06.7190.glo.Z a.25.10.7190.glo.Z b.25.01.7190.glo.Z b.25.05.7190.glo.Z b.25.09.7190.glo.Z b.25.7190.eps.Z 939:a.25.03.7190.glo.Z a.25.07.7190.glo.Z a.25.11.7190.glo.Z b.25.02.7190.glo.Z b.25.06.7190.glo.Z b.25.10.7190.glo.Z 940:a.25.04.7190.glo.Z a.25.08.7190.glo.Z a.25.12.7190.glo.Z b.25.03.7190.glo.Z b.25.07.7190.glo.Z b.25.11.7190.glo.Z 941:crua6[/cru/cruts] ls fromdpe1a/data/grid/cru_ts_2.0/_makecld/_constants/_7190/spc2cld/_mon/ 942:a.25.01.7190.glo.Z a.25.05.7190.glo.Z a.25.09.7190.glo.Z a.25.7190.eps.Z b.25.04.7190.glo.Z b.25.08.7190.glo.Z b.25.12.7190.glo.Z 943:a.25.02.7190.glo.Z a.25.06.7190.glo.Z a.25.10.7190.glo.Z b.25.01.7190.glo.Z b.25.05.7190.glo.Z b.25.09.7190.glo.Z b.25.7190.eps.Z 944:a.25.03.7190.glo.Z a.25.07.7190.glo.Z a.25.11.7190.glo.Z b.25.02.7190.glo.Z b.25.06.7190.glo.Z b.25.10.7190.glo.Z 945:a.25.04.7190.glo.Z a.25.08.7190.glo.Z a.25.12.7190.glo.Z b.25.03.7190.glo.Z b.25.07.7190.glo.Z b.25.11.7190.glo.Z 946: 947:So.. we don't have the coefficients files (just .eps plots of something). But 948:what are all those monthly files? DON'T KNOW, UNDOCUMENTED. Wherever I look, 949:there are data files, no info about what they are other than their names. And 950:that's useless.. take the above example, the filenames in the _mon and _ann 951:directories are identical, but the contents are not. And the only difference 952:is that one directory is apparently 'monthly' and the other 'annual' - yet 953:both contain monthly files. 954: 955:Lots of further investigation.. probably the most useful program found is 956:cal_cld_gts_tdm.pro, the description of which reads as follows: 957: 958:pro cal_cld_gts_tdm,dtr_prefix,outprefix,year1,year2,info=info 959:; calculates cld anomalies using relationship with dtr anomalies 960:; reads coefficients from predefined files (*1000) 961:; reads DTR data from binary output files from quick_interp_tdm2.pro (binfac=1000) 962:; creates cld anomaly grids at dtr grid resolution 963:; output can then be used as dummy input to splining program that also 964:; includes real cloud anomaly data 965: 966:It also tellingly contains: 967:; unnecessary because 61-90 normals have already been created 968:; print, "@@@@@ looking for 2.5 deg DTR 1961-90 @@@@@" 969:; mean_gts,'~/m1/gts/dtr/glo25/glo25.dtr.',nor1,nor2 970:; mean_gts_tdm,'/cru/mark1/f080/gts/dtr/glo25/glo25.dtr.',nor1,nor2 971:;print, "@@@@@ looking for 2.5 deg DTR normal @@@@@" 972:;; rdbin,dtrnor,'~/m1/gts/dtr/glo25/glo25.dtr.'+string(nor1-1900,nor2-1900,form='(2i2.2)') 973:;dtrnorstr='/cru/mark1/f080/gts/dtr/glo25/glo25.dtr.'+string(nor1-1900,nor2-1900,form='(2i2.2)') 974:;rdbin,dtrnor,dtrnorstr 975: 976:The above has seemingly been replaced with: 977:rdbin,a,'/cru/tyn1/f709762/cru_ts_2.0/_constants/_7190/a.25.7190',gridsize=2.5 978:rdbin,b,'/cru/tyn1/f709762/cru_ts_2.0/_constants/_7190/b.25.7190',gridsize=2.5 979: 980:These are the files that have been lost according to the gridding read_me 981:(see above). 982: 983:The conclusion of a lot of investigation is that the synthetic cloud grids 984:for 1901-1995 have now been discarded. This means that the cloud data prior 985:to 1996 are static. 986: 987:Edit: have just located a 'cld' directory in Mark New's disk, containing 988:over 2000 files. Most however are binary and undocumented.. 989: 990:Eventually find fortran (f77) programs to convert sun to cloud: 991: 992:sh2cld_tdm.for converts sun hours monthly time series to cloud percent 993:sp2cld_m.for converts sun percent monthly time series to cloud oktas 994: 995:There are also programs to convert sun parameters: 996: 997:sh2sp_m.for sun hours to sun percent 998:sh2sp_normal.for sun hours monthly .nrm to sunshine percent 999:sh2sp_tdm.for sun hours monthly time series to sunshine percent 1000: 1001:AGREED APPROACH for cloud (5 Oct 06). 1002: 1003:For 1901 to 1995 - stay with published data. No clear way to replicate 1004:process as undocumented. 1005: 1006:For 1996 to 2002: 1007: 1. convert sun database to pseudo-cloud using the f77 programs; 1008: 2. anomalise wrt 96-00 with anomdtb.f; 1009: 3. grid using quick_interp_tdm.pro (which will use 6190 norms); 1010: 4. calculate (mean9600 - mean6190) for monthly grids, using the 1011: published cru_ts_2.0 cloud data; 1012: 5. add to gridded data from step 3. 1013: 1014:This should approximate the correction needed. 1015: 1016:On we go.. firstly, examined the spc database.. seems to be in % x10. 1017:Looked at published data.. cloud is in % x10, too. 1018:First problem: there is no program to convert sun percentage to 1019:cloud percentage. I can do sun percentage to cloud oktas or sun hours 1020:to cloud percentage! So what the hell did Tim do?!! As I keep asking. 1021: 1022:Examined the program that converts sun % to cloud oktas. It is 1023:complicated! Have inserted a line to multiple the result by 12.5 (the 1024:result is in oktas*10 and ranges from 0 to 80, so the new result will 1025:range from 0 to 1000). 1026: 1027:Next problem - which database to use? The one with the normals included 1028:is not appropriate (the conversion progs do not look for that line so 1029:obviously are not intended to be used on +norm databases). The non 1030:normals databases are either Jan 03 (in the '_ateam' directory) or 1031:Dec 03 (in the regular database directory). The newer database is 1032:smaller! So more weeding than planting in 2003. Unfortunately both 1033:databases contain the 6190 normals line, just unpopulated. So I will 1034:go with the 'spc.0312221624.dtb' database, and modify the already- 1035:modified conversion program to process the 6190 line. 1036: 1037:Then - comparing the two candidate spc databases: 1038: 1039:spc.0312221624.dtb 1040:spc.94-00.0312221624.dtb 1041: 1042:I find that they are broadly similar, except the normals lines (which 1043:both start with '6190') are very different. I was expecting that maybe 1044:the latter contained 94-00 normals, what I wasn't expecting was that 1045:thet are in % x10 not %! Unbelievable - even here the conventions have 1046:not been followed. It's botch after botch after botch. Modified the 1047:conversion program to process either kind of normals line. 1048: 1049:Decided to go with the 'spc.94-00.0312221624.dtb' database, as it 1050:hopefully has some of the 94-00 normals in. I just wish I knew more. 1051: 1052:Conversion was hampered by the discovery that some stations have a mix 1053:of % and % x10 values! So more mods to Hsp2cldp_m.for. Then conversion, 1054:producing cldfromspc.94000312221624.dtb. Copied the .dts file across 1055:as is, not sure what it does unfortunately (or can't remember!). 1056: 1057:After conversion, ran anomdtb: 1058: 1059:crua6[/cru/cruts/rerun1/data/cruts/rerun_cld] ./anomdtb 1060: 1061: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 1062: 1063: > Enter the suffix of the variable required: 1064:.cld 1065: > Select the .cts or .dtb file to load: 1066:cldfromspc.94000312221624.dtb 1067: 1068: > Specify the start,end of the normals period: 1069:1994,2000 1070: > Specify the missing percentage permitted: 1071:25 1072: > Data required for a normal: 6 1073: > Specify the no. of stdevs at which to reject data: 1074:3 1075: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 1076:3 1077: > Check for duplicate stns after anomalising? (0=no,>0=km range) 1078:8 1079: > Select the generic .txt file to save (yy.mm=auto): 1080:cldfromspc.txt 1081: > Select the first,last years AD to save: 1082:1994,2002 1083: > Operating... 1084: 1085: > .cts 96309 19.6 280712 57.2 1086: > PROCESS DECISION percent %of-chk 1087: > no lat/lon 0 0.0 0.0 1088: > no normal 209619 42.8 42.8 1089: > out-of-range 177298 36.2 63.2 1090: > duplicated 154 0.0 0.1 1091: > accepted 103260 21.1 1092: > Dumping years 1994-2002 to .txt files... 1093: 1094:crua6[/cru/cruts/rerun1/data/cruts/rerun_cld] 1095: 1096:Then ran quick_interp_tdm2: 1097: 1098:IDL> .compile /cru/cruts/fromdpe1a/code/idl/pro/quick_interp_tdm2.pro 1099:% Compiled module: QUICK_INTERP_TDM2. 1100:IDL> .compile /cru/cruts/fromdpe1a/code/idl/pro/rdbin.pro 1101:% Compiled module: RDBIN. 1102:IDL> quick_interp_tdm2,1994,2002,'glo_from_idl/cld.',600,gs=0.5,pts_prefix='txt_4_idl/cldfromspc.',dumpglo='dumpglo' 1103:Defaults set 1104: 1994 1105:% Compiled module: MAP_SET. 1106:% Compiled module: CROSSP. 1107:% Compiled module: STRIP. 1108:% Compiled module: SAVEGLO. 1109:% Compiled module: SELECTMODEL. 1110: 1995 1111: 1996 1112: 1997 1113: 1998 1114: 1999 1115: 2000 1116: 2001 1117: 2002 1118:IDL> 1119: 1120:Tadaa: .glo files produced for 1994 to 2002. 1121: 1122:Then retracked to produce regular 0.5-degree grids for dtr (having only 1123:produced 2.5-degree binaries for synthetics earlier): 1124: 1125:IDL> quick_interp_tdm2,1901,2002,'glo_dtr_grids/dtr.',750,gs=0.5,pts_prefix='dtr_txt_4idl/dtr.',dumpglo='dumpglo' 1126: 1127:That went off without any apparent hitches, so I wrote a fortran prog, 1128:'maxminmaker.for', to produce tmn and tmx grids from tmp and dtr. It ran. 1129: 1130:However - yup, more problems - when I checked the inputs and outputs I found 1131:that in numerous instances there was a value for mean temperature in the grid, 1132:with no corresponding dtr value. This led to tmn = tmx = tmp for thos cells. 1133:NOT GOOD. 1134: 1135:Actually, what was NOT GOOD was my grasp of context. Oh curse this poor 1136:memory! For the IDL gridding program produces ANOMALIES not ACTUALS. 1137: 1138:Wrote a program, 'glo2abs.for' does a file-for-file conversion of .glo 1139:files (as produced by quick_interp_tdm2.pro) to absolute-value files (also 1140:gridded and with headers). After some experiments realised that the .glo 1141:anomalies are in degrees, but the normals are in 10ths of a degree :-) 1142: 1143:Produced absolutes for TMP. Then wrote a program, 'cmpcruts.for', to 1144:compare the absolute grids with the published cru_ts_2.10 data. The 1145:comparison simply measures absolute differences between old and new, and 1146:categorises as either (1) identical, (2) within 0.5 degs, (3) within 1 deg, 1147:(4) over 1 deg apart. Results for temperature (TMP): 1148: 1149: Identical <0.5deg 0.5-1deg >1deg 1150: 30096176 48594200 2755281 1076423 1151: 1152:And for temperature range (DTR): 1153: 1154: 45361058 31267870 3893754 1999398 1155: 1156:These are very promising. The vast majority in both cases are within 0.5 1157:degrees of the published data. However, there are still plenty of values 1158:more than a degree out. 1159: 1160:The total number of comparisons is 67420*102*12 = 82,522,080 1161: 1162:It seems prudent to add percentage calculations.. 1163: 1164:TMP: 1165:Final Diff Totals: 30096176 48594200 2755281 1076423 1166:Percentages: 36.47 58.89 3.34 1.30 1167: 1168:TMP has a comforting 95%+ within half a degree, though one still wonders 1169:why it isn't 100% spot on.. 1170: 1171:DTR: 1172:Final Diff Totals: 45361058 31267870 3893754 1999398 1173:Percentages: 54.97 37.89 4.72 2.42 1174: 1175:DTR fares perhaps even better, over half are spot-on, though about 1176:7.5% are outside a half. 1177: 1178:However, it's not such good news for precip (PRE): 1179:Final Diff Totals: 11492331 21163924 9264554 40601271 1180:Percentages: 13.93 25.65 11.23 49.20 1181: 1182:21. A little experimentation goes a short way.. 1183: 1184:I tried using the 'stn' option of anomdtb.for. Not completely sure what 1185:it's supposed to do, but no matter as it didn't work: 1186: 1187:crua6[/cru/cruts/rerun1/data/cruts/rerun_pre] ./anomdtb 1188: 1189: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 1190: 1191: > Enter the suffix of the variable required: 1192:.pre 1193: > Will calculate percentage anomalies. 1194: > Select the .cts or .dtb file to load: 1195:pre.0312031600H.dtb 1196: 1197: > Specify the start,end of the normals period: 1198:1961,1990 1199: > Specify the missing percentage permitted: 1200:25 1201: > Data required for a normal: 23 1202: > Specify the no. of stdevs at which to reject data: 1203:5 1204: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 1205:4 1206: > Check for duplicate stns after anomalising? (0=no,>0=km range) 1207:8 1208: > Select the .stn file to save: 1209:pre.fromanomdtb.stn 1210: > Enter the correlation decay distance: 1211:450 1212: > Submit a grim that contains the appropriate grid. 1213: > Enter the grim filepath: 1214:cru_ts_2_10.1961-1970.pre 1215: 1216: > Grid dimensions and domain size: 720 360 67420 1217: > Select the first,last years AD to save: 1218:1901,2002 1219: > Operating... 1220: 1221: > NORMALS MEAN percent STDEV percent 1222: > .dtb 2635548 29.6 1223: > .cts 4711327 52.8 7325296 82.2 1224: > PROCESS DECISION percent %of-chk 1225: > no lat/lon 20761 0.2 0.2 1226: > no normal 1585342 17.8 17.8 1227: > out-of-range 20249 0.2 0.3 1228: > duplicated 317035 3.6 4.3 1229: > accepted 6972308 78.2 1230: > Calculating station coverages... 1231: > ##### WithinRange: Alloc: DataB ##### 1232:forrtl: severe (174): SIGSEGV, segmentation fault occurred 1233:crua6[/cru/cruts/rerun1/data/cruts/rerun_pre] 1234: 1235:..knowing how long it takes to debug this suite - the experiment 1236:endeth here. The option (like all the anomdtb options) is totally 1237:undocumented so we'll never know what we lost. 1238: 1239:22. Right, time to stop pussyfooting around the niceties of Tim's labyrinthine software 1240:suites - let's have a go at producing CRU TS 3.0! since failing to do that will be the 1241:definitive failure of the entire project.. 1242: 1243:Firstly, we need to identify the updated data files. I acquired the following: 1244: 1245:iran_asean_GHCN_WWR-CD_save50_CLIMAT_MCDW_updat_merged renamed to pre.0611301502.dat 1246:newbigfile0606.dat renamed to tmp.0611301507.dat 1247:glseries_tmn_final_merged renamed to tmn.0611301516.dat 1248:glseries_tmx_final_merged renamed to tmx.0611301516.dat 1249:anders9106m.dat renamed to tmp9106.0612011708.dat 1250: 1251:..and established a directory hierarchy under /cru/cruts/version_3_0 1252: 1253:Next step, convert the various db formats to the CRU TS one. Made a visual 1254:comparison which indicated that it would work. Unfortunately it will mean 1255:losing the 'extra' fields that have been tacked onto the headers willy-nilly 1256:as they are undocumented. Furthermore the two extra fields in the CRU TS 1257:format are undocumented, as far as I can see! So I wrote headergetter.for 1258:to produce stats on the CRU TS headers. It looks for violations of the 1259:mandatory blank spaces, and for variations in the two extra fields. Sample 1260:output for temperature and precip: 1261: 1262:Header report for tmp.0311051552.dtb 1263:Produced by headgetter.for 1264:Total Records Read: 12155 1265: 1266:BLANKS (expected at 8,14,21,26,47,61,66,71,78) 1267: position missed 1268: 8 0 1269: 14 0 1270: 21 0 1271: 26 0 1272: 47 0 1273: 61 0 1274: 66 0 1275: 71 0 1276: 78 2 1277: 1278:EXTRA FIELD 1 (72:77) 1279: type detected counted 1280:Missing Value Code 12155 1281:Possible F.P. Value 0 1282:Possible Exp. Value 0 1283:Integer Value Found 0 1284:Real Value Found 0 1285:Unidentifiable 0 1286: 1287:EXTRA FIELD 2 (79:86) 1288: type detected counted 1289:Missing Value Code 709 1290:Possible F.P. Value 697 1291:Possible Exp. Value 0 1292:Integer Value Found 10749 1293:Real Value Found 0 1294:Unidentifiable 0 1295: 1296:ENDS 1297: 1298:Header report for pre.0312031600.dtb 1299:Produced by headgetter.for 1300:Total Records Read: 12732 1301: 1302:BLANKS (expected at 8,14,21,26,47,61,66,71,78) 1303: position missed 1304: 8 0 1305: 14 0 1306: 21 0 1307: 26 0 1308: 47 0 1309: 61 0 1310: 66 0 1311: 71 0 1312: 78 154 1313: 1314:EXTRA FIELD 1 (72:77) 1315: type detected counted 1316:Missing Value Code 12732 1317:Possible F.P. Value 0 1318:Possible Exp. Value 0 1319:Integer Value Found 0 1320:Real Value Found 0 1321:Unidentifiable 0 1322: 1323:EXTRA FIELD 2 (79:86) 1324: type detected counted 1325:Missing Value Code 3635 1326:Possible F.P. Value 437 1327:Possible Exp. Value 0 1328:Integer Value Found 8660 1329:Real Value Found 0 1330:Unidentifiable 0 1331: 1332:ENDS 1333: 1334:As can be seen, there are no unidentifiable headers - hurrah! - but quite 1335:a few violations of the boundary between the two extra fields, particularly 1336:in the precip database. On examination, the culprits are all African 1337:stations. The two tmp exceptions: 1338: 1339: 641080 -330 1735 324 BANDUNDU DEM REP CONGO 1961 1990 -99908 1340: 642200 -436 1525 445 KINSHASA/BINZA DEM REP CONGO 1960 1990 -99920 1341: 1342:And samples of the pre exceptions: 1343: 1344:-656002 698 -958 150 SUAKOKO LIBERIA 1951 1970 -999123008050 1345:-655327 727 -723 350 KOUIBLY IVORY COAST 1977 1990 -999109001290 1346:-655001 1320 -235 332 GOURCY BURKINA FASO 1956 1980 -999120001240 1347:-618504 788 -1118 -999 KENEMA/FARM SIERRA LEONE 1951 1972 -999139003500 1348:-612067 1407 -307 253 KORO MALI 1958 1989 -999127002650 1349: 1350:So the first extra field is apparently unused! It would be a handy place for 1351:the 6-character data-code and valid-start-year from the temperature db. 1352: 1353:On to a more detailed look at the cru precip format; not sure whether there 1354:are two extra fields or one, and what the sizes are. A quick hack through 1355:the headers is not pleasing. There appears to be only one field, but it can 1356:have up to nine (9) digits in it, and at least three missing value codes: 1357: 1358:6785300-1863 2700 1080HWANGE/N.P.A. ZIMBABWE 19621996 40 1359:8100100 680 -5820 2GEORGETOWN GUYANA 18462006 -99 1360:6274000 1420 2460 1160KUTUM SUDAN 19291990 194 1361:6109200-9999-99999 -999UNKNOWN NIGER 19891989 -999 1362:6542000 945 -2 197YENDI GHANA 19071997 8010 1363:6544200 672 -160 293KUMASI GHANA 19062006 17009 1364:6122306 1670 -299 267KABARA MALI 19231989 270022 1365:6193128 32 672 -999SAO TOME SAO TOME 19391973 8888888 1366:6266000 1850 3180 249KARIMA SUDAN 19172006 18315801 1367:6109905 1208 -367 315OUARKOYE BURKINA FASO 19601980 120002470 1368: 1369:*unimpressed* 1370: 1371:This is irritating as it means precip has only 9 fields and I can't do a 1372:generic mapping from any cru format to cru ts. 1373: 1374:As a glutton for punishment I then looked at the tmin/tmax db format. Looks 1375:like two extra fields (i6,i7) with mvcs of 999999 and 8888888 respectively. 1376:However *sigh* inspection reveals the following two possibilities: 1377: 1378: 851300 3775 -2568 17PONTA DELGADA PORTUGAL 18652004 9999998888888 1379: 851500 3697 -2517 100SANTA MARIA A ACORES 19542006 -77777 8888888 1380: 1381:Isn't that marvellous? These can't even be read with a consistent header format! 1382: 1383:So, the approach will be to read exactly ONE extra field. For cru tmp that 1384:will be the i2+i4 anders/best-start codes as one. For cru pre it will be 1385:the amazing multipurpose, multilength field. For cru tmnx it will be the 1386:first field, which is at least stable at i6. 1387: 1388: 1389:Conversions/corrections performed: 1390: 1391:Temperature 1392: 1393:Converted tmp.0611301507.dat to tmp.0612081033.dat 1394: 1395:Found one corrupted station name: 1396:BEFORE 1397: 911900 209 1564 20 HI*KAHULUI WSO (PUU NENE) 1954 1990 101954 -999.00 1398:AFTER 1399: 911900 209 1564 20 KAHULUI ARPT/MAUI HAWAII 1954 1990 101954 -999.00 1400: 1401:Precipitation 1402: 1403:Converted pre.0611301502.dat to pre.0612081045.dat 1404: 1405:Found one corrupted station name: 1406:BEFORE 1407:4125600 2358 5828 15SEEB AP./=MUSCAT*0.9OMAN 18932006 301965 1408:AFTER 1409:4125600 2358 5828 15 SEEB INTL/MUSCAT OMAN 1893 2006 -999 -999.00 1410: 1411:(DL later reported that the name wasintended to signify that the data had been 1412:corrected by a factor of 0.9 when data from another station was incorporated 1413:to extend the series - this was Mike Hulme's work) 1414: 1415:Write db2dtb.for, which converts any of the CRU db formats to the CRU TS format. 1416: 1417:Started work on mergedb.for, which should merge a primary database with and incoming 1418:database of the same (CRU TS) format. Quite complicated. No operator interventions, 1419:just a log file of failed attempts - but hooks left in for op sections in case this 1420:turns out to be the main programmatic deliverable to BADC! 1421: 1422: 1423:23. Interrupted work on mergedb.for in order to trial a precip gridding for 3.0. This 1424:required another new proglet, addnormline.for, which adds a normals line below each 1425:header. It fills in the normals values if the condisions are met (75% of values, or 1426:23 for the 30 year period). 1427: 1428:Initial results promising.. ran it for precip, it added normals lines OK, a total of 1429:15942 with 6003 missing value lines. No errors, and no ops interventions because the 1430:file didn't have normals lines before! 1431: 1432:'Final' precip file: pre.0612151458.dtb 1433: 1434:Tried running anomdtb.f90.. failed because it couldn't find the .dts file! No matter 1435:that it doesn't need it - argh! 1436: 1437:Examined existing .dts files.. not sure what they're for. Headers are identical to 1438:the .dtb file, all missing values are retained, all other values are replaced with 1439:one of several code numbers, no idea what they mean. 1440: 1441:Wrote 'falsedts.for' to produce dummy .dts files with all zeros in place of real 1442:data values. Produced pre.0612151458.dts. 1443: 1444:Added normals line, producing: pre.0612181221.dtb 1445:Re-produced matching pre.0612181221.dts file. 1446: 1447:Tried running anomdtb.f90 again. This time it crashed at record #1096. Wrote a proglet 1448:'findstn.for' to find the n-th station in a dtb file, pulled out 1096: 1449: 1450: 0 486 10080 1036 BUKIT LARUT MALAYSIA 1951 1988 -999 -999.00 1451:6190 2094 2015 2874 3800 4619 3032 5604 3718 4626 5820 5035 3049 1452:1951 3330 2530 2790 5660 4420 4030 1700 2640 8000 5950 6250 2020 1453: 1454:(snipped normal years) 1455: 1456:1979 110 1920 1150 5490 3140 308067100 2500 4860 4280 4960 1600 1457: 1458:Uh-oh! That's 6.7m of rain in July 1979? Looks like a factor-of-10 problem. Confirmed 1459:with DL and changed to 6710. 1460: 1461:Next run, crashed at #4391, CHERRAPUNJI, the wettest place in the world. So here, the 1462:high values are realistic. However I did notice that the missing value code was -10 1463:instead of -9999! So modified db2dtb.for to fix that and re-produced the precip database 1464:as pre.0612181214.dat. This then had to have normals recalculated for it (after fixing 1465:#1096). 1466: 1467:Finally got it through anomdtb.for AND quick_interp_tdm2 - without crashing! IDL was even 1468:on the ball with the missing months at the end of 2006: 1469: 1470: 1471:IDL> quick_interp_tdm2,1901,2006,'preglo/pregrid.',450,gs=0.5,dumpglo='dumpglo',pts_prefix='preanoms/pre.' 1472:% Compiled module: QUICK_INTERP_TDM2. 1473:% Compiled module: GLIMIT. 1474:Defaults set 1475: 1901 1476:% Compiled module: MAP_SET. 1477:% Compiled module: CROSSP. 1478:% Compiled module: STRIP. 1479:% Compiled module: SAVEGLO. 1480:% Compiled module: SELECTMODEL. 1481: 1902 1482: 1903 1483:(etc) 1484: 2005 1485: 2006 1486:no stations found in: preanoms/pre.2006.09.txt 1487:no stations found in: preanoms/pre.2006.10.txt 1488:no stations found in: preanoms/pre.2006.11.txt 1489:no stations found in: preanoms/pre.2006.12.txt 1490: 1491:All good. Wrote mergegrids.for to create the more-familiar decadal and full-series 1492:files from the monthly *.glo.abs ones. 1493: 1494:Then.. like an idiot.. I had to test the data! Duh. 1495: 1496:Firstly, wrote mmeangrid.for and cmpmgrids.m to get a visual comparison of old and 1497:new precip grids (old being CRU TS 2.10). This showed variations in 'expected' areas 1498:where changes had been made, it the Southern tip of Greenland. 1499: 1500:Next, Phil requested some statistical plots of percentage change in annual totals, 1501:and long-term trends. Wrote 'anntots.for' to convert monthly gridded files into 1502:yearly totals files. Then tried to write precipchecker.m to do the rest in Matlab.. 1503:it wasn't having it, OUT OF MEMORY! Bah. So wrote 'prestats.for' to calculate the 1504:final stats, for printing with an emasculated precipchecker.m. BUT.. it wouldn't 1505:work, and on investigating I found 200-odd stations with zero precipitation for 1506:the entire 1901-2006 period! Modified anntots.for to dump a single grid with those 1507:cells that remained at zero marked, then plotted. 1508: 1509:Zero cells in North Africa and the Western coast of South America. None in the 1510:CRU TS 2.10 precip grids :-( 1511: 1512:Next step, produce a list of cell centres of the offending cells. wrote a quick 1513:proglet, 'idzerocells.for'. Then 'getcellstations.for', which, given a CRUTS DB 1514:file and a list of lat/lon values, extracts all stations lying inside the cells 1515:listed. 1516: 1517:Uh-oh. Looked in the new pre db and found 15 stations for 257 zero cells! They are: 1518: 1519:6061170 2810 670 381 FT FLATTERS ALGERIA 1925 1965 -999 -999.00 1520:6064000 2650 840 559 FORT POLIGNAC ALGERIA 1925 2006 -999 -999.00 1521:6262000 2080 3260 470 STATION NO. 6 SUDAN 1950 1988 -999 -999.00 1522:8450100 -810 -7900 26 TRUJILLO PERU 1961 2006 -999 -999.00 1523:8453100 -920 -7850 10 CHIMBOTE PERU 1961 2006 -999 -999.00 1524:8462800 -1200 -7710 13 LIMA-CALLAO/INTL.AP. PERU 1961 2006 -999 -999.00 1525:8463100 -1210 -7700 137 LIMATAMBO/C.DE MARTE PERU 1927 1980 -999 -999.00 1526:8469100 -1380 -7630 6 PISCO PERU 1942 2006 -999 -999.00 1527:8540600 -1850 -7030 29 ARICA/CHACALLUTA CHILE 1903 2006 -999 -999.00 1528:8541700 -2020 -7020 6 IQUIQUE/CAVANCHA CHILE 1886 1986 -999 -999.00 1529:8541800 -2053 -7018 52 IQUIQUE DIEGO ARACEN CHILE 1989 2006 -999 -999.00 1530:8700494 -707 -7957 150 CAYALTI PERU 1934 1959 -999 -999.00 1531:8700562 -1203 -7703 137 LIMA PERU 1929 1963 -999 -999.00 1532:8700581 -1207 -7717 13 LA PUNTA (NA PERU 1939 1963 -999 -999.00 1533:9932040 2810 670 381 FT FLATTER ALGERIA 1925 1965 -999 -999.00 1534: 1535:Looked for the same zero cell stations in the old pre db (pre.0312031600.dtb) and only 1536:found 10: 1537: 1538:-854031 -2021 -7015 5 IQUIQUE/CAVANCHA CHILE 1899 1986 -999 0.00 1539:-843002 -1210 -7700 135 LIMATAMBO PERU 1927 1980 -999 1540:-603550 2810 670 381 FT FLATTER ALGERIA 1925 1965 -999 -999.00 1541: 606400 2650 841 558 ILLIZI/ILLIRANE ALGERIA 1925 2002 -999 -999 1542: 626200 2075 3255 468 STATION NO. 6 SUDAN 1950 1988 -999 -999.00 1543: 845010 -810 -7903 30 TRUJILLO/MARTINEZ PERU 1961 2002 -999 -999 1544: 845310 -916 -7851 11 CHIMBOTE/TENIENTE PERU 1961 2001 -999 1545: 846280 -1200 -7711 13 LIMA/JORGE CHAVEZ PERU 1961 2002 -999 -999 1546: 846910 -1375 -7628 7 PISCO (CIV/MIL) PERU 1942 2002 -999 -999 1547: 854180 -2053 -7018 52 IQUIQUE/DIEGO ARAC CHILE 1989 2002 -999 -999.00 1548: 1549:So why does the old db result in no 'zero' cells, and the new db give us over 250? I 1550:wondered if normals might be the answer, but none of the 10 stations from the old db 1551:have in-db normals, wheras three of the new db have: 1552: 1553:8453100 -920 -7850 10 CHIMBOTE PERU 1961 2006 -999 -999.00 1554:6190 19 59 36 18 5 0 3 0 0 1 10 5 1555:8469100 -1380 -7630 6 PISCO PERU 1942 2006 -999 -999.00 1556:6190 3 0 3 0 0 1 1 3 1 4 0 0 1557:8540600 -1850 -7030 29 ARICA/CHACALLUTA CHILE 1903 2006 -999 -999.00 1558:6190 1 3 0 0 0 2 2 2 2 0 0 0 1559: 1560:So these alone ought to guarantee three of the cells being nonzero - they should have 1561:the bloody normals in! So the next check has to be the climatology, that which provides 1562:the cell-by-cell normals.. 1563: 1564:A check of the gridded climatology revealed that all 257 'zero cells' have their 1565:climatologies set to zero, too. This was partially checked in the GRIM-format climatology 1566:just in case! 1567: 1568:Next, a focus: on CHIMBOTE (see header line above). This has real data (not just zeros). 1569:It is in cell (162,203), or (-9.25,-78.75) [lat, lon in both cases]. So we extract the 1570:full timeseries for that cell from the published 2.10 (1901-2002) GRIM file: 1571: 1572:Grid-ref= 203, 162 1573: 2 0 0 0 2 0 2 0 0 5 0 3 1574: 2 0 0 0 2 0 2 0 0 5 0 3 1575: 2 0 0 0 2 0 2 0 0 5 0 3 1576: 2 0 0 0 2 0 2 0 0 5 0 3 1577: 2 0 0 0 2 0 2 0 0 5 0 3 1578: 2 0 0 0 2 0 2 0 0 5 0 3 1579: 2 0 0 0 2 0 2 0 0 5 0 3 1580: 2 0 0 0 2 0 2 0 0 5 0 3 1581: 2 0 0 0 2 0 2 0 0 5 0 3 1582: 2 0 0 0 2 0 2 0 0 5 0 3 1583: 2 0 0 0 2 0 2 0 0 5 0 3 1584: 2 0 0 0 2 0 2 0 0 5 0 3 1585: 2 0 0 0 2 0 2 0 0 5 0 3 1586: 2 0 0 0 2 0 2 0 0 5 0 3 1587: 2 0 0 0 2 0 2 0 0 5 0 3 1588: 2 0 0 0 2 0 2 0 0 5 0 3 1589: 2 0 0 0 2 0 2 0 0 5 0 3 1590: 2 0 0 0 2 0 2 0 0 5 0 3 1591: 2 0 0 0 2 0 2 0 0 5 0 3 1592: 2 0 0 0 2 0 2 0 0 5 0 3 1593: 2 0 0 0 2 0 2 0 0 5 0 3 1594: 2 0 0 0 2 0 0 0 0 5 0 3 1595: 2 0 0 0 2 0 2 0 0 5 0 3 1596: 2 0 0 0 2 0 2 0 0 5 0 3 1597: 2 0 0 0 2 0 2 0 0 5 0 3 1598: 2 0 0 0 2 0 2 0 0 5 0 3 1599: 2 0 0 0 2 0 2 0 0 5 0 3 1600: 2 0 0 0 2 0 2 0 0 5 0 3 1601: 2 0 0 0 2 0 2 0 0 5 0 3 1602: 2 0 0 0 2 0 2 0 0 5 0 3 1603: 2 0 0 0 2 0 2 0 0 5 0 3 1604: 2 0 0 0 2 0 2 0 0 5 0 3 1605: 2 0 0 0 2 0 2 0 0 7 0 3 1606: 2 0 0 0 2 0 0 0 0 5 0 3 1607: 2 0 0 0 2 0 2 0 0 5 0 3 1608: 2 0 0 0 2 0 2 0 0 5 0 3 1609: 2 0 0 0 2 0 2 0 0 5 0 3 1610: 2 0 0 0 2 0 2 0 0 5 0 3 1611: 2 0 0 0 2 0 2 0 0 5 0 3 1612: 2 0 0 0 2 0 0 0 0 5 0 3 1613: 2 0 0 0 2 0 2 0 0 5 0 3 1614: 2 0 0 0 2 0 0 0 0 5 0 3 1615: 0 0 0 0 2 0 0 0 0 0 0 0 1616: 0 0 0 0 2 0 0 0 0 0 0 2 1617: 0 0 0 0 2 5 6 0 0 0 0 3 1618: 2 3 0 0 0 0 17 0 0 4 0 3 1619: 2 0 0 0 3 0 2 0 0 2 0 3 1620: 0 0 0 0 0 0 14 0 0 9 0 0 1621: 0 0 0 0 0 0 0 0 0 2 0 2 1622: 0 0 0 0 0 0 12 0 0 0 0 5 1623: 0 0 0 0 0 0 0 0 0 3 0 2 1624: 0 0 0 0 0 0 10 0 0 0 0 2 1625: 0 0 0 0 3 0 11 0 0 2 0 3 1626: 0 0 0 0 2 0 0 0 0 0 0 2 1627: 0 0 0 0 0 0 0 0 0 4 0 0 1628: 3 0 0 0 0 0 15 0 0 0 0 2 1629: 0 0 0 0 0 0 0 0 0 4 3 2 1630: 5 0 0 0 0 0 0 0 0 12 0 3 1631: 0 0 2 2 4 2 0 0 2 3 0 3 1632: 0 0 0 0 3 0 0 2 0 2 2 3 1633: 0 0 0 0 0 0 0 3 0 0 0 0 1634: 0 0 0 0 0 0 0 0 0 0 0 0 1635: 0 0 0 7 0 3 0 0 0 0 0 0 1636: 0 0 2 3 0 0 0 4 0 0 12 0 1637: 0 0 9 0 0 0 0 0 0 0 0 0 1638: 0 0 0 0 0 0 0 0 0 0 0 0 1639: 0 6 2 0 0 0 6 0 0 0 0 0 1640: 0 0 0 0 0 0 0 2 2 0 0 0 1641: 0 0 0 0 0 3 0 0 0 0 0 0 1642: 0 0 0 0 2 2 0 0 0 0 0 0 1643: 0 2 0 0 0 0 0 2 0 0 0 0 1644: 0 0 0 7 0 0 0 2 0 0 0 3 1645: 2 0 7 0 0 2 0 0 2 0 0 0 1646: 0 0 0 7 0 0 2 2 2 0 0 0 1647: 8 0 2 0 0 0 0 2 0 0 0 0 1648: 0 7 0 0 0 0 0 0 0 0 0 0 1649: 0 0 2 0 0 0 0 0 0 0 0 0 1650: 0 0 0 0 0 0 0 0 0 2 0 0 1651: 0 0 0 0 2 0 0 0 0 0 0 0 1652: 2 0 0 0 0 2 0 0 0 0 10 0 1653: 3 0 0 0 0 0 9 0 0 0 0 3 1654: 0 0 0 0 0 0 0 0 0 3 0 5 1655: 4 0 0 2 10 2 0 0 0 0 0 4 1656: 0 0 0 0 0 0 0 0 2 5 0 0 1657: 0 0 0 0 0 0 9 0 0 0 0 0 1658: 0 0 0 0 0 0 0 3 0 0 0 0 1659: 0 0 0 0 0 0 0 0 0 5 0 0 1660: 3 0 0 0 0 0 0 0 0 0 0 0 1661: 2 0 0 0 0 0 0 0 3 0 0 0 1662: 0 0 0 0 8 2 0 0 0 0 0 3 1663: 0 0 2 0 2 0 0 0 0 0 2 3 1664: 0 0 0 0 2 0 2 0 0 5 0 2 1665: 0 0 0 0 2 0 2 0 0 0 0 0 1666: 0 0 0 0 0 0 0 0 0 5 0 0 1667: 0 0 0 0 2 0 0 2 0 0 0 0 1668: 2 0 2 0 0 0 0 0 0 5 0 0 1669: 0 0 0 0 11 0 2 0 0 4 0 3 1670: 2 3 2 0 13 0 0 0 0 0 0 0 1671: 2 6 0 3 0 0 0 0 2 3 0 7 1672: 2 0 0 0 2 0 0 0 0 0 0 3 1673: 0 0 0 0 0 0 2 0 0 0 0 2 1674: 0 0 0 0 0 0 0 0 0 0 0 3 1675: 1676:..yet in the 3.00 version, it's all zeros! 1677: 1678:Only one thing for it.. examine the attempt at regenerating 2.10. 1679:Unfortunately - well, interestingly then - this gave the same 1680:zero cells as the 3.00 generation! So it's something to do with 1681:the process, not the database (or the climatology, assuming that 1682:has remained constant, which I gather it has). 1683: 1684:Update: aha! Phil pointed out that for precip the climatology 1685:is used as a MULTIPLIER. So if the clim hasn't changed, the 1686:cells should always have been zero regardless of actual data. 1687: 1688:As I should have remembered: 1689: 1690:crua6[/cru/cruts/version_3_0/primaries/precip] ./glo2abs 1691:Welcome! This is the GLO2ABS program. 1692:I will create a set of absolute grids from 1693:a set of anomaly grids (in .glo format), also 1694:a gridded version of the climatology. 1695:Enter the path and name of the normals file: clim.6190.lan.pre 1696:Enter a name for the gridded climatology file: clim.6190.lan.pre.grid 1697:Enter the path and stem of the .glo files: preglo/pregrid. 1698:Enter the starting year: 1901 1699:Enter the ending year: 2006 1700:Enter the path (if any) for the output files: pregrid/ 1701:Now, CONCENTRATE. Addition or Percentage (A/P)? P 1702:Right, erm.. off I jolly well go! 1703:pregrid.01.1901.glo 1704:pregrid.02.1901.glo 1705:(etc) 1706: 1707:Decided to read Mitchell & Jones 2005 again. Noticed that the 1708:limit for SD when anomalising should be 4 for precip, not 3! So 1709:re-ran with that: 1710: 1711:crua6[/cru/cruts/version_3_0/primaries/precip] ./anomdtb 1712: 1713: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 1714: 1715: > Enter the suffix of the variable required: 1716:.pre 1717: > Will calculate percentage anomalies. 1718: > Select the .cts or .dtb file to load: 1719:pre.0612181221.dtb 1720: pre.0612181221.dtb 1721: 1722: 1723: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/precip/pre.0612181221.dtb 1724: 1725: 1726: > Specify the start,end of the normals period: 1727:1961,1990 1728: > Specify the missing percentage permitted: 1729:25 1730: > Data required for a normal: 23 1731: > Specify the no. of stdevs at which to reject data: 1732:4 1733: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 1734:3 1735: > Check for duplicate stns after anomalising? (0=no,>0=km range) 1736:8 1737: > Select the generic .txt file to save (yy.mm=auto): 1738:pre4sd.txt 1739: > Select the first,last years AD to save: 1740:1901,2006 1741: > Operating... 1742: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/precip/pre.0612181221.dtb 1743: 1744: 1745: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/precip/pre.0612181221.dtb 1746: 1747: 1748: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/precip/pre.0612181221.dts 1749: 1750: 1751: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/precip/pre.0612181221.dts 1752: 1753: 1754: > NORMALS MEAN percent STDEV percent 1755: > .dtb 7315040 73.8 1756: made it to here 1757: > .cts 299359 3.0 7613600 76.8 1758: > PROCESS DECISION percent %of-chk 1759: > no lat/lon 17527 0.2 0.2 1760: > no normal 2355659 23.8 23.8 1761: > out-of-range 13253 0.1 0.2 1762: > duplicated 586206 5.9 7.8 1763: > accepted 6934807 70.0 1764: > Dumping years 1901-2006 to .txt files... 1765: 1766: 1767:This is not as good a percentage as for 2.10: 1768: 1769: > NORMALS MEAN percent STDEV percent 1770: > .dtb 0 0.0 1771: > .cts 3375441 84.1 3375441 84.1 1772: > PROCESS DECISION percent %of-chk 1773: > no lat/lon 3088 0.1 0.1 1774: > no normal 638538 15.9 15.9 1775: > out-of-range 70225 1.7 2.1 1776: > duplicated 135457 3.4 4.1 1777: > accepted 3167636 78.9 1778: > Dumping years 1901-2002 to .txt files... 1779: 1780:But the actual number of accepted values is more than TWICE 2.10! 1781: 1782:Of course, the same 257 gridcells are zeros, because the multiplicative 1783:normals are still zero. 1784: 1785:For reference, these are the results for the 3 SD limit of 3.00: 1786: 1787: > NORMALS MEAN percent STDEV percent 1788: > .dtb 7315040 73.8 1789: made it to here 1790: > .cts 284160 2.9 7598401 76.7 1791: > PROCESS DECISION percent %of-chk 1792: > no lat/lon 17527 0.2 0.2 1793: > no normal 2370858 23.9 24.0 1794: > out-of-range 32379 0.3 0.4 1795: > duplicated 583193 5.9 7.8 1796: > accepted 6903495 69.7 1797: > Dumping years 1901-2006 to .txt files... 1798: 1799:So we've only gained 0.3% of values, a real figure of 31312 values. 1800: 1801:Conclusion: stick with a 3 Standard Deviation limit, like the 1802:Read_Me says. 1803: 1804: 1805:24. (cont of 22 really) 1806: 1807:Restarted work on mergedb.for. Decided I was taking the wrong approach, 1808:so the interruption was probably a GOOD THING. 1809: 1810:The process now is to read in the header lines AND line numbers from 1811:the main database, and to then process the incoming database one record 1812:at a time. It's more logical and haivng the line numbers will speed 1813:things up enormously (well it has done on previous occasions). 1814: 1815:The biggest immediate problem was the loss of an hour's edits to the 1816:program, when the network died.. no explanations from anyone, I hope 1817:it's not a return to last year's troubles. 1818: 1819:(some weeks later) 1820: 1821:well, it compiles OK, and even runs enthusiastically. However there are 1822:loads of bugs that I now have to fix. Eeeeek. Timesrunningouttimesrunningout. 1823: 1824:(even later) 1825: 1826:Getting there.. still ironing out glitches and poor programming. 1827: 1828:25. Wahey! It's halfway through April and I'm still working on it. This 1829:surely is the worst project I've ever attempted. Eeeek. I think the main 1830:problem is the rather nebulous concept of the automatic updater. If I 1831:hadn't had to write it to add the 1991-2006 temperature file to the 'main' 1832:one, it would probably have been a lot simpler. But that one operation has 1833:proved so costly in terms of time, etc that the program has had to bend 1834:over backwards to accommodate it. So yes, in retrospect it was not a 1835:brilliant idea to try and kill two birds with one stone - I should have 1836:realised that one of the birds was actually a pterodactyl with a temper 1837:problem. 1838: 1839:Success! 1840:crua6[/cru/cruts/version_3_0/db/testmergedb] ./mergedb 1841:************************************************** 1842:* MERGEDB * 1843:* * 1844:* Merging of two database files * 1845:* Ops ID: f098xxxx * 1846:* Date: 12:17 25/04/07 * 1847:* The Session ID is: 0704251217.f098xxxx * 1848:* (log file 'mergedb.0704251217.f098xxxx.log') * 1849:* * 1850:* Please choose the mode of working. * 1851:* This program can either run.. * 1852:* [1] Interactively, (in which case an operator * 1853:* must be present throughout to make decision), * 1854:* or [2] in Batch mode, (in which case it may * 1855:* be left unattended). If Batch mode is used, a * 1856:* file of outstanding issues will be saved for * 1857:* later [3] resolution by an operator. * 1858:* * 1859:* [1] Interactive (operator) processing * 1860:* [2] Batch (no operator) processing * 1861:* [3] Operator processing of saved batch * 1862:* [4] Run a previously-saved action file * 1863:* * 1864:* Please enter 1,2,3 or 4: 4 1865:* RUN ACTION FILE MODE * 1866:* * 1867:* Enter the ACTion filename, or 'x' for a list: x 1868:* The 1 most recent ACT files: 1869:* 1. mergedb.0704201343.f098xxxx.act * 1870:* Enter a number or 0 for none of the above: 1 1871:* Enter 'Y' to run this file or 'N' to abort: Y 1872:* * 1873:* Creation date/time: 13:43 20/04/07 * 1874:* Batch initiator was: f098 * 1875: 1876:* Number of actions/requests: 2586 1877:* This ACT file derived from original OPS file: * 1878:* mergedb.0704201210.f098xxxx.ops * 1879:* Main (existing) Database: tmp.0702091122.dtb 1880:* Secondary (incoming) Database: tmp.0612081519.dat 1881:* Parameter is 'tmp' - confirm (Y/N): Y 1882: 1883:* Actions Completed! * 1884:* Thank You for using MERGEDB! * 1885:************************************************** 1886: 1887:..well, 'success' in the sense that it ran and apparently all the data's 1888:in the right place, in tmp.0704251819.dtb. 1889: 1890:26. OK, now to merge in the US stations. First, wrote 'us2cru' to convert 1891:the marksusanonwmocru.dat file to the 'standard' format we're using. That 1892:worked OK. Then used 'addnormline' to, well - add a normals line. Only 17 1893:out of 1035 stations ended up with missing normals, which is pretty good! 1894: 1895:The with-normals US database file is tmp.0704251654.dat. 1896: 1897:Now, I knew that using mergedb as it stands would not work. It expects to 1898:be updating the existing records, and actions like 'addnew' require OPS 1899:to confirm each one. So I thought it best to add an OPS clause to auto- 1900:confirm additions where there's no WMO match and the data density is OK, 1901:say 50% or higher. Unfortunately, that didn't work either, and rather than 1902:spend even more time debugging mergedb.for, I knocked off simpleaddnew.for, 1903:which adds two non-overlapping databases. The resultant file, with all 1904:three partial databases, is tmp.0704271015.dtb. 1905: 1906:27. Well, enough excuses - time to remember how to do the anomalising and 1907:gridding things! Fisrtly, ran 'addnormline' just to ensure all normals are 1908:up to date. The result was 8 new sets of normals, so well worth doing. The 1909:database is now: 1910: 1911:tmp.0704292158.dtb 1912: 1913:Ran 'anomdtb' - got caught out by the requirement for a companion '.dts' 1914:file again, ran 'falsedts.for' and carried on.. would still be nice to be 1915:sure that it's not something meaningful **sigh**. 1916: 1917:Output: 1918: 1919:crua6[/cru/cruts/version_3_0/primaries/temp] ./anomdtb 1920: 1921: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 1922: 1923: > Enter the suffix of the variable required: 1924:.tmp 1925: > Select the .cts or .dtb file to load: 1926:tmp.0704292158.dtb 1927: tmp.0704292158.dtb 1928: 1929: 1930: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/temp/tmp.0704292158.dtb 1931: 1932: 1933: > Specify the start,end of the normals period: 1934:1961,1990 1935: > Specify the missing percentage permitted: 1936:25 1937: > Data required for a normal: 23 1938: > Specify the no. of stdevs at which to reject data: 1939:3 1940: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 1941:3 1942: > Check for duplicate stns after anomalising? (0=no,>0=km range) 1943:8 1944: > Select the generic .txt file to save (yy.mm=auto): 1945:tmp.txt 1946: > Select the first,last years AD to save: 1947:1901,2006 1948: > Operating... 1949: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/temp/tmp.0704292158.dtb 1950: 1951: 1952: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/temp/tmp.0704292158.dtb 1953: 1954: 1955: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/temp/tmp.0704292158.dts 1956: 1957: 1958: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/temp/tmp.0704292158.dts 1959: 1960: 1961: > Failed to find file. 1962: > Enter the file, with suffix: .dts 1963:tmp.0704292158.dts 1964: tmp.0704292158.dts 1965: 1966: 1967: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/temp/tmp.0704292158.dts 1968: 1969: 1970: > NORMALS MEAN percent STDEV percent 1971: > .dtb 3330007 81.3 1972: made it to here 1973: > .cts 92803 2.3 3422810 83.6 1974: > PROCESS DECISION percent %of-chk 1975: > no lat/lon 0 0.0 0.0 1976: > no normal 671592 16.4 16.4 1977: > out-of-range 744 0.0 0.0 1978: > duplicated 4102723 100.2 119.9 1979: > accepted -680657 -16.6 1980: > Dumping years 1901-2006 to .txt files... 1981: 1982:crua6[/cru/cruts/version_3_0/primaries/temp] 1983: 1984: 1985:.. which is a trifle worrying! And looking at the .txt files, they look 1986:rather odd as well - for instance, tmp.1953.03.txt starts like this: 1987: 1988: 7.09 0.87 10.0 0.10000 10010 1989: 7.83 -1.55 28.0 -4.80000 10080 1990: 6.97 -1.89 10.0 0.90000 -999 1991: 6.97 -1.89 100.0 0.50000 10260 1992: 7.45 -1.90 16.0 -3.10000 10280 1993: 6.95 -2.55 129.0 3.70000 10650 1994: 7.04 -3.11 14.0 0.00000 10980 1995: 6.60 -0.20 0.0 1.20000 11000 1996: 6.73 -1.44 13.0 1.60000 -999 1997: 6.68 -1.40 39.0 2.20000 11530 1998: 1999:Now, do those first two columns look like lat & lon to you? Me neither, 2000:here's what the old version of the same file looks like: 2001: 2002: 60.00 -20.00 -999.0 0.40000-990007 2003: 62.00 -33.00 -999.0 -0.40000-990002 2004: 56.50 -51.00 0.0 -0.50000-990000 2005: 6.90 122.06 6.0 -0.60000 -999 2006: 13.13 123.73 17.0 0.20000 -999 2007: 14.52 121.00 15.0 0.60000 -999 2008: 18.37 121.63 4.0 1.10000 -999 2009: 6.90 122.00 6.0 -0.60000 -999 2010: 10.70 122.50 14.0 -0.10000 -999 2011: 13.13 123.73 19.0 0.10000 -999 2012: 2013:In fact, the first two columns never get outside of +/- 30. Oh bugger. 2014:What the HELL is going on?! 2015: 2016:Decided to pursue that worrying (and impossible) 'duplicates' figure. 2017: 2018:The function 'sort' was used to sort the database so that any duplicate 2019:lines would be together - then 'uniq' was used to pull out duplicates. 2020:There were quite a few dupes, and one or two triples too, like these: 2021: 2022:crua6[/cru/cruts/version_3_0/primaries/temp] grep -n '1984 \-83 \-46 22 55 126 154 222 215 159 63 32 \-62' tmp.0704292158.dtb 2023:195789:1984 -83 -46 22 55 126 154 222 215 159 63 32 -62 2024:254265:1984 -83 -46 22 55 126 154 222 215 159 63 32 -62 2025:254380:1984 -83 -46 22 55 126 154 222 215 159 63 32 -62 2026: 2027:These are from the following stations: 2028: 720344 408 1158 1539 ELKO-FAA-AP---------USA--------- 1870 1996 301870 -999.00 2029: 725837 408 1158 1549 NV ELKO FAA AP 1930 1990 101930 -999.00 2030: 725910 401 1223 103 RED BLUFF USA 1878 2006 101878 -999.00 2031: 2032:The past two are consecutive stations. 2033: 2034:Looking at the last two.. it seems that 725910 has 725837's data! 2035: 2036:1977 71 124 118 184 167 275 283 280 230 190 126 99 2037:1978 107 114 149 144 208 248 289 282 232 220 118 72 2038:1979 85 99 139 150 218 256 282 258 253 189 117 94 2039:1980 99 121 119 156 192 216 275 262 241 196 128 102 2040:1981 14 19 49 90 123 196 233 227 164 71 47 11 2041:1982 -49 -14 32 57 114 164 206 214 148 74 11 -23 2042:1983 -9 -1 54 59 114 167 204 223 170 104 25 -19 2043:1984 -83 -46 22 55 126 154 222 215 159 63 32 -62 2044: 2045:Ascan be seen, 1981 sees a complete chance in range, especially for 2046:Autumn/Winter. In fact, from 1981 to 1990, 725910 is a copy of 2047:725837! It then reverts to the original range for the rest of the run. 2048:So.. did the merging program do this? Unfortunately, yes. Check dates: 2049: 2050:crua6[/cru/cruts/version_3_0/db/testmergedb] grep -n 'RED BLUFF' tmp.0*.* 2051:tmp.0612081519.dat:28595: 725910 401 1223 103 RED BLUFF USA 1991 2006 101991 -999.00 2052:tmp.0702091122.dtb:171674: 725910 401 1223 103 RED BLUFF USA 1878 1980 101878 -999.00 2053:tmp.0704251819.dtb:200331: 725910 401 1223 103 RED BLUFF USA 1878 2006 101878 -999.00 2054:tmp.0704271015.dtb:254272: 725910 401 1223 103 RED BLUFF USA 1878 2006 101878 -999.00 2055:tmp.0704292158.dtb:254272: 725910 401 1223 103 RED BLUFF USA 1878 2006 101878 -999.00 2056:crua6[/cru/cruts/version_3_0/db/testmergedb] 2057: 2058:The first file is the 1991-2006 update file. The second is the original 2059:temperature database - note that the station ends in 1980. 2060: 2061:It has *inherited* data from the previous station, where it had -9999 2062:before! I thought I'd fixed that?!!! 2063: 2064:/goes off muttering to fix mergedb.for for the five hundredth time 2065: 2066:Miraculously, despite being dog-tired at nearly midnight on a Sunday, I 2067:did find the problem. I was clearing the data array but not close enough 2068:to the action - when stations were being passed through (ie no data to 2069:add to them) they were not being cleaned off the array afterwards. Meh. 2070: 2071:Wrote a specific routine to clear halves of the data array, and back to 2072:square one. Re-ran the ACT file to merge the x-1990 and 1991-2006 files. 2073:Created an output file exactly the same size as the last time (phew!) 2074:but with.. 2075: 2076:crua6[/cru/cruts/version_3_0/db/testmergedb] comm -12 tmp.0704292355.dtb tmp.0704251819.dtb |wc -l 2077: 285516 2078:crua6[/cru/cruts/version_3_0/db/testmergedb] wc -l tmp.0704292355.dtb 2079: 285829 tmp.0704292355.dtb 2080: 2081:.. 313 lines different. Typically: 2082: 2083:14881,14886c14881,14886 2084:< 1965-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2085:< 1966-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2086:< 1967-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2087:< 1968-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2088:< 1969-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2089:< 1970-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2090:--- 2091:> 1965 -221 -177 -234 -182 -5 6 24 36 -15 -91 -100 -221 2092:> 1966 -272 -194 -248 -192 -66 10 27 45 -12 -75 -139 -228 2093:> 1967 -201 -243 -196 -158 -26 1 40 30 -18 -89 -183 -172 2094:> 1968 -253 -256 -253 -107 -42 10 46 33 -21 -64 -134 -195 2095:> 1969 -177 -202 -248 -165 -33 8 42 50 -1 -89 -157 -204 2096:> 1970 -237 -192 -217 -160 -87 6 30 25 -5 -55 -143 -222 2097: 2098:ie, what should have been missing data is now missing data again: 2099: 2100:200436,200445c200436,200445 2101:< 1981-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2102:< 1982-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2103:< 1983-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2104:< 1984-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2105:< 1985-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2106:< 1986-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2107:< 1987-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2108:< 1988-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2109:< 1989-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2110:< 1990-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 2111:--- 2112:> 1981 14 19 49 90 123 196 233 227 164 71 47 11 2113:> 1982 -49 -14 32 57 114 164 206 214 148 74 11 -23 2114:> 1983 -9 -1 54 59 114 167 204 223 170 104 25 -19 2115:> 1984 -83 -46 22 55 126 154 222 215 159 63 32 -62 2116:> 1985 -57 -29 17 89 122 181 244 188 121 79 -11 -50 2117:> 1986 2 31 66 72 113 187 194 214 116 78 11 -39 2118:> 1987 -59 -5 30 97 131 177 193 192 153 101 21 -35 2119:> 1988 -65 -15 29 80 108 184 222 198 138 116 8 -57 2120:> 1989 -113 -54 53 94 113 164 215 186 143 78 8 -24 2121:> 1990 -24 -30 49 100 100 166 214 194 177 77 9 -97 2122: 2123:Hurrah! 2124: 2125:So the interim database file is tmp.0704292355.dtb. Now to re-add 2126:the US station dataset with simpleaddnew.for. 2127: 2128:crua6[/cru/cruts/version_3_0/db/testmergedb] ./simpleaddnew 2129: 2130:SIMPLYADDNEW - add stations to a database 2131:This program assumes the two databases have 2132:NO COMMON STATIONS and will fail (stop) if 2133:any are found. 2134: 2135:Please enter the main database: tmp.0704292355.dtb 2136: 2137:Please enter the new database: tmp.0704251654.dat 2138:Please enter a 3-character parameter code: tmp 2139:Output database is: tmp.0704300053.dtb 2140:crua6[/cru/cruts/version_3_0/db/testmergedb] 2141: 2142:So now we have the combined database again, a bit quicker than 2143:last time: tmp.0704300053.dtb. Pity we slid into May: I was hoping 2144:to only be FIVE MONTHS late. 2145: 2146:What's worse - there are STILL duplicate non-missing lines, 210 of 2147:them. The first example is this: 2148: 2149:1835 92 73 141 187 260 279 281 288 241 195 183 106 2150: 2151:Which belongs to this in the original database (tmp.0702091122.dtb): 2152: 2153: 722080 329 800 15 CHARLESTON, S. CAROL UNITED STATES 1823 1990 101823 -999.00 2154:6190 84 100 142 180 224 257 274 270 245 191 145 104 2155: 2156:..and to this in the US database (tmp.0704251654.dat): 2157: 2158: 720467 328 799 3 CHARLESTON-CITY-----USA--------- 1835 1996 301835 -999.00 2159:6190 91 106 144 186 227 260 277 272 249 199 154 112 2160: 2161:These two stations obviously have a lot in common - though not 2162:everything, as their normals (shown) differ. In fact, on examination 2163:the US database record is a poor copy of the main database one, it 2164:has more missing data and so forth. By 1870 they have diverged, so 2165:in this case it's probably OK.. but what about the others? I just do 2166:not have the time to follow up everything. We'll have to take 210 2167:year repetitions as 'one of those things'. 2168: 2169:..actually, I decided in the end to follow up all 210 of them. The 2170:likelihood is that the number is far greater, since the filtering 2171:that gave the 210 figure excluded any lines with two or more 2172:consecutive missing values (to avoid hundreds of just-missing-value 2173:lines). Also I spotted some instances where data lines would be 2174:identical but for one or more missing values in one of the stations. 2175: 2176:After checking, I found that the majority of the duplications were 2177:between the original database and the US database, with just a couple 2178:of 'linked' stations within the original database, and half a dozen 2179:in the 1991-2006 update file. One surprise was that stations I'm sure 2180:I rejected ended up marked as 'addnew' in the .act file - quite 2181:unsettling! 2182: 2183:Rather foolishly, perhaps, I decided to have a go at interactively 2184:incorporating the US data rather than using 'simplyaddnew'. However, 2185:progress was so slow (because of the high number of 'near matches') 2186:that this approach was abandoned. 2187: 2188:Tried 'anomdtb' with the fixed final file (tmp.0704300053.dtb)... 2189:no better! The crucial bits: 2190: 2191: 2192: > NORMALS MEAN percent STDEV percent 2193: > .dtb 3323823 81.3 2194: made it to here 2195: > .cts 91963 2.2 3415786 83.5 2196: > PROCESS DECISION percent %of-chk 2197: > no lat/lon 0 0.0 0.0 2198: > no normal 675037 16.5 16.5 2199: > out-of-range 744 0.0 0.0 2200: > duplicated 4100117 100.2 120.1 2201: > accepted -685075 -16.7 2202: > Dumping years 1901-2006 to .txt files... 2203: > Failed to create file. Try again. 2204: > Enter the file, with suffix: .ann 2205:tmp.ann 2206: > Failed to create file. Try again. 2207: > Enter the file, with suffix: .ann 2208:h.ann 2209: 2210:crua6[/cru/cruts/version_3_0/primaries/temp] 2211: 2212: 2213:So the 'duplicated' figure is slightly lower.. but what's this 2214:error with the '.ann' file?! Never seen before. Oh GOD if I 2215:could start this project again and actually argue the case for 2216:junking the inherited program suite!! 2217: 2218:OK.. the .ann file was simply that it refuses to overwrite any 2219:existing one. Meh. It's happy to overwrite the log file of 2220:course - nice bit of logic there. 2221: 2222:and the duplicates? Well I inserted a debug line where the 2223:decision is made. Here's an example: 2224: 2225: 712600 vs. 727340: 4.7 8.4 4.7 8.4 -> 0.0km 2226: 2227:Here the two WMO codes look OK (though others are -999 which 2228:seems unlikely) but the two lat/lon pairs? Ooops. Here are the 2229:actual headers: 2230: 2231: 712600 465 845 187 Sault Ste Marie A CANADA 1945 2006 361945 -999.00 2232: 727340 465 844 220 SAULT-STE-MARIE----- USA--------- 1888 2006 101888 -999.00 2233: 2234:So, uhhhh.. what in tarnation is going on? Just how off-beam 2235:are these datasets?!! 2236: 2237:Not sure why the lats & lons are a factor of 10 too low - may 2238:be intentional though it wasn't happening before. 2239: 2240:Ran with the original database: 2241: 2242: 2243: > NORMALS MEAN percent STDEV percent 2244: > .dtb 2113609 81.7 2245: made it to here 2246: > .cts 0 0.0 2113608 81.7 2247: > PROCESS DECISION percent %of-chk 2248: > no lat/lon 0 0.0 0.0 2249: > no normal 474422 18.3 18.3 2250: > out-of-range 68179 2.6 3.2 2251: > duplicated 923258 35.7 45.1 2252: > accepted 1122172 43.4 2253: > Dumping years 1901-1990 to .txt files... 2254: 2255: 2256:The lats & lons look the same.. but a lot less duplicates! 2257: 2258:WHY? Well, it could just be those pesky US stations.. so 2259:why not compare the two bespoke log files (as excerpted above)? 2260: 2261:Immediately, another baffler: the log file from the run of 2262:the 'final' database has lots of 'DEBUG DETAIL' information, 2263:but the log file from the run of the original database does not! 2264:So cropping those away with a judicious 'tail'.. I ran comm: 2265: 2266:crua6[/cru/cruts/version_3_0/primaries/temp] comm -23 log_anomdtb_H.0702091122.dat barelog_anomdtb_H.0704300053.dat |wc -l 2267: 200 2268:crua6[/cru/cruts/version_3_0/primaries/temp] comm -13 log_anomdtb_H.0702091122.dat barelog_anomdtb_H.0704300053.dat | wc -l 2269: 2572 2270:crua6[/cru/cruts/version_3_0/primaries/temp] comm -12 log_anomdtb_H.0702091122.dat barelog_anomdtb_H.0704300053.dat | wc -l 2271: 1809 2272: 2273:So 200 duplication events are unique to the older database, 2274:and 2572 are unique to the new database - with 1809 common 2275:to both. A quick look at the 2572 'new' ones showed a majority 2276:of those with the first WMO as -999: this is the key. The 2277:databases do not have any records with WMO=-999 as far as I know, 2278:so something is going on.. 2279: 2280:28. With huge reluctance, I have dived into 'anomdtb' - and already I have 2281:that familiar Twilight Zone sensation. 2282: 2283:I have found that the WMO Code gets set to -999 if *both* lon and lat are 2284:missing. However, the following points are relevant: 2285: 2286:* LoadCTS multiplies non-missing lons by 0.1, so they range from -18 to +18 2287: with missing value codes passing through AS LONG AS THEY ARE -9999. If they 2288: are -999 they will be processed and become -99.9. It is not clear why lats 2289: are not treated in the same way! 2290: 2291:* The subroutine 'Anomalise' in anomdtb checks lon and lat against a simple 2292: 'MissVal', which is defined as -999. This will catch lats of -999 but not 2293: lons of -9999. 2294: 2295:* This does still not explain how we get so many -999 codes.. unless we don't 2296: and it's just one or two? 2297: 2298:And the real baffler: 2299: 2300:* If the code is -999 because lat and lon are both missing - how the bloody 2301: hell does it know there's a duplication within 8km?!!! 2302: 2303:.. ah, OK. well for a start, the last point above does not apply - not one 2304:case of the code being set to -999 because of lat/lon missing. In fact, I 2305:hate to admit it, bit it is *sort of* clever - the code is set to -999 to 2306:prevent it being used again, because the distance/duplication checker will 2307:not make a distance comparison if either code is -999. So HOW COME loads of 2308:the duplicates have a code of -999?!!! 2309: 2310:The plot thickens.. I changed the exclusion tests in the duplication loops 2311:from: 2312: if (AStn(XAStn).NE.MissVal) then 2313: to: 2314: if (int(AStn(XAStn)).NE.-999) then 2315: 2316:This made NO DIFFERENCE. So having tested to ensure that the first of the 2317:pair hasn't already been used - we then use it! What's more I've noticed 2318:that it's usually the one 'incorporated' in the previous iteration! 2319: 2320:Consider: 2321: 2322: 67700 vs. 160660: 4.6 -0.9 4.6 -0.9 -> 5.4km 2323: -999 vs. 160707: 4.6 -0.9 4.6 -0.9 -> 2.2km 2324: -999 vs. 160800: 4.6 -0.9 4.5 -0.9 -> 7.3km 2325: -999 vs. 160811: 4.6 -0.9 4.6 -0.9 -> 5.8km 2326: 2327:Here we can see (check the first set of lat/lons) that, after being 2328:incorporated into 160660, 67700 goes on to also be incorporated into 2329:160707, 160800 and 160811! So the same data could end up in three 2330:other stations. It gets worse!! Because later on, we find: 2331: 2332: 160660 vs. 160707: 4.6 -0.9 4.6 -0.9 -> 7.9km 2333: -999 vs. 160800: 4.6 -0.9 4.5 -0.9 -> 7.0km 2334: -999 vs. 160811: 4.6 -0.9 4.6 -0.9 -> 5.8km 2335: 160707 vs. 160800: 4.6 -0.9 4.5 -0.9 -> 7.9km 2336: -999 vs. 160811: 4.6 -0.9 4.6 -0.9 -> 6.6km 2337: 160800 vs. 160811: 4.5 -0.9 4.6 -0.9 -> 2.2km 2338: 2339:So three of those recipients have gone on to be incorporated into one 2340:of them (160811). But although in this case 67700 is within 8km of 2341:160811, there is no guarantee! Indeed, with this system, the 'chosen' 2342:station may hop all over the place in <8km steps, collecting data as 2343:it goes. In a densely-packed area this could drastically reduce the 2344:number of stations. Then there's these: 2345: 2346: 85997 vs. 390000: -10.0 -20.0 -10.0 -20.0 -> 0.0km 2347: -999 vs. 685807: -10.0 -20.0 -10.0 -20.0 -> 0.0km 2348: -999 vs. 688607: -10.0 -20.0 -10.0 -20.0 -> 0.0km 2349: -999 vs. 967811: -10.0 -20.0 -10.0 -20.0 -> 0.0km 2350: -999 vs. 968531: -10.0 -20.0 -10.0 -20.0 -> 0.0km 2351: 2352:as might be guessed, they all end up incorporated into 968531 - but 2353:no surprise seeing as their lats & lons are rubbish!!! Oh Tim what 2354:have you done, man? [actually - what he's done is to let missing 2355:lats & lons through. Missing lon code is -1999 not -9999 so these 2356:figures are the roundings] 2357: 2358:All that said, the biggest worry is still the lats & lons themselves. 2359:They just don't look realistic. Lats appear to have been reduced by 2360:a factor of 10 too, even though I can't find the code for that. And 2361:(from the top example) is 67700 really 5.4km from 160660? 2362: 2363: 67700 460 -90 273 LUGANO SWITZERLAND 1864 2006 101864 -999.00 2364: 160660 456 -87 -999 MILANO MALPENSA ITALY 1961 1970 101961 -999.00 2365: 2366:Of course not! It's just over 50km. I do not understand why the lats 2367:& lons have been scaled, when the stated distance threshold has not. 2368: 2369:At least I've found *where* they are scaled, in LoadCTS (crutsfiles.f90): 2370: 2371: if (StnInfo(XStn,2).NE.LatMissVal) Lat (XStn) = real(StnInfo(XStn,2)) / real(LatFactor) 2372: if (StnInfo(XStn,3).NE.LonMissVal) Lon (XStn) = real(StnInfo(XStn,3)) / real(LonFactor) 2373: 2374:Looking at how LoadCTS is called from anomdtb.. 2375: 2376:subroutine LoadCTS (StnInfo,StnLocal,StnName,StnCty,Code,Lat,Lon,Elv,OldCode,Data,YearAD,& 2377: NmlData,DtbNormals,CallFile,Hulme,Legacy,HeadOnly,HeadForm,LongType,Silent,Extra,PhilJ, & 2378: YearADMin,YearADMax,Source,SrcCode,SrcSuffix,SrcDate, & 2379: LatMV,LonMV,ElvMV,DataMV,LatF,LonF,ElvF,NmlYr0,NmlYr1,NmlSrc,NmlInc) 2380: 2381: call LoadCTS (StnInfoA,StnLocalA,StnNameA,StnCtyA,Code=AStn,OldCode=AStnOld, & 2382: Lat=ALat,Lon=ALon,Elv=AElv,DtbNormals=DtbNormalsA, & 2383: Data=DataA,YearAD=AYearAD,CallFile=LoadFileA,silent=1) ! get .dtb file 2384: 2385:.. we see that Legacy is not passed. This means that.. (from LoadCTS): 2386: 2387:LatFactor=100 ; LonFactor=100 ; ElvFactor=1 ! usual/hulme hdr factors 2388:if (present(Legacy)) then 2389: LatFactor=10 ; LonFactor=10 ; ElvFactor=1 ! legacy hdr factors 2390:end if 2391:if (present(LatF)) LatFactor = LatF ! custom hdr factors 2392:if (present(LonF)) LonFactor = LonF 2393:if (present(ElvF)) ElvFactor = ElvF 2394: 2395:..LatFactor and LonFactor are set to 100. 2396: 2397:So I added a specific pair of arguments, LatF=10,LonF=10, and got: 2398: 2399: > NORMALS MEAN percent STDEV percent 2400: > .dtb 3323823 81.3 2401: made it to here 2402: > .cts 91963 2.2 3415786 83.5 2403: > PROCESS DECISION percent %of-chk 2404: > no lat/lon 0 0.0 0.0 2405: > no normal 675037 16.5 16.5 2406: > out-of-range 744 0.0 0.0 2407: > duplicated 53553 1.3 1.6 2408: > accepted 3361489 82.2 2409: > Dumping years 1901-2006 to .txt files... 2410: 2411:Hurrah! Looking at the log it is still ignoring the -999 Code and re-intgrating stations.. 2412:but not to any extent worth worrying about. Not when duplications are down to 1.3% :-))) 2413: 2414:Then got a mail from PJ to say we shouldn't be excluding stations inside 8km anyway - yet 2415:that's in IJC - Mitchell & Jones 2005! So there you go. Ran again with 0km as the distance: 2416: 2417: > NORMALS MEAN percent STDEV percent 2418: > .dtb 3323823 81.3 2419: made it to here 2420: > .cts 91963 2.2 3415786 83.5 2421: > PROCESS DECISION percent %of-chk 2422: > no lat/lon 0 0.0 0.0 2423: > no normal 675037 16.5 16.5 2424: > out-of-range 744 0.0 0.0 2425: > accepted 3415042 83.5 2426: > Dumping years 1901-2006 to .txt files... 2427: 2428:Which hasn't saved much as it turns out. In fact, I must conclude that an inquiring mind is 2429:a very dangerous thing - I decided to see what difference it made, turning off the proximity 2430:duplicate detection and elimination: 2431: 2432:crua6[/cru/cruts/version_3_0/primaries/temp] wc -l */*1962.12.txt 2433: 2773 oldtxt/old.1962.12.txt 2434: 3269 tmptxt0km/tmp.1962.12.txt 2435: 3308 tmptxt8km/tmp.1962.12.txt 2436: 2437:So.. 'oldtxt' is before I fixed the lat/lon scaling problem. But look at the last two - I 2438:got MORE results when I used an elimination radius! Whaaaaaaaaat?!!! 2439: 2440:/goes home in a huff 2441: 2442:/gets out of huff and goes into house, checks things and thinks hard 2443: 2444:Okay, I guess if we don't do the roll-duplicates-together thing, then we could lose data 2445:because the 'rolled' station (ie the one subsumed into its neighbour) might have useful 2446:years but no normals, so that data would be lost? 2447: 2448:29. I suddenly thought - what about the Australian data? But luckily that's just tmax/tmin 2449:so I can roll that into the next database work. 2450: 2451:30. Being an idiot much experience I decided to go back to the 'perfectly-good' precip 2452:generation for v3.0 and re-do the anomalies with the new anomdtb. At 8km, we got the 2453:duplicates down from 5.9% to 2.1%: 2454: 2455: 2456: > NORMALS MEAN percent STDEV percent 2457: > .dtb 7315040 73.8 2458: made it to here 2459: > .cts 299359 3.0 7613600 76.8 2460: > PROCESS DECISION percent %of-chk 2461: > no lat/lon 17527 0.2 0.2 2462: > no normal 2355659 23.8 23.8 2463: > out-of-range 13253 0.1 0.2 2464: > duplicated 586206 5.9 7.8 2465: > accepted 6934807 70.0 2466: > Dumping years 1901-2006 to .txt files... 2467: 2468: 2469: > NORMALS MEAN percent STDEV percent 2470: > .dtb 7315040 73.8 2471: made it to here 2472: > .cts 299359 3.0 7613600 76.8 2473: > PROCESS DECISION percent %of-chk 2474: > no lat/lon 17527 0.2 0.2 2475: > no normal 2355659 23.8 23.8 2476: > out-of-range 13253 0.1 0.2 2477: > duplicated 207391 2.1 2.8 2478: > accepted 7313622 73.8 2479: > Dumping years 1901-2006 to .txt files... 2480: 2481:And, of course, all in with 0km range: 2482: 2483: > NORMALS MEAN percent STDEV percent 2484: > .dtb 7315040 73.8 2485: made it to here 2486: > .cts 299359 3.0 7613600 76.8 2487: > PROCESS DECISION percent %of-chk 2488: > no lat/lon 17527 0.2 0.2 2489: > no normal 2355659 23.8 23.8 2490: > out-of-range 13253 0.1 0.2 2491: > accepted 7521013 75.9 2492: > Dumping years 1901-2006 to .txt files... 2493: 2494:Happy? well.. no. Because something is happening for precip that does not happen for 2495:temp! But of course. Here are the first few lines from various 1962.12 text files.. 2496: 2497:tmptxt8km/tmp.1962.12.txt 2498: 70.90 8.70 10.0 2.10000 10010 2499: 78.30 -15.50 28.0 -3.30000 10080 2500: 69.70 -18.90 10.0 -1.40000 -999 2501: 69.70 -18.90 100.0 -1.50000 10260 2502: 74.50 -19.00 16.0 -1.20000 10280 2503: 69.50 -25.50 129.0 -3.10000 10650 2504: 70.40 -31.10 14.0 -0.20000 10980 2505: 66.00 -2.00 0.0 0.50000 11000 2506: 67.30 -14.40 13.0 -1.00000 11520 2507: 66.80 -14.00 39.0 -0.70000 11530 2508: 2509:tmptxt0km/tmp.1962.12.txt 2510: 70.90 8.70 10.0 2.10000 10010 2511: 78.30 -15.50 28.0 -3.30000 10080 2512: 69.70 -18.90 10.0 -1.40000 10250 2513: 69.70 -18.90 100.0 -1.50000 10260 2514: 74.50 -19.00 16.0 -1.20000 10280 2515: 69.50 -25.50 129.0 -3.10000 10650 2516: 70.40 -31.10 14.0 -0.20000 10980 2517: 66.00 -2.00 0.0 0.50000 11000 2518: 67.30 -14.40 13.0 -1.00000 11520 2519: 66.80 -14.00 39.0 -0.70000 11530 2520: 2521:preanoms/pre.1962.12.txt (old anomdtb output) 2522: 61.00 10.60 190.0 48.20000-511900 2523: 54.45 -6.07 116.0 -3.70000 -999 2524: 50.83 -4.55 15.0 -22.40000-389870 2525: 50.22 -5.30 76.0 39.70000 -999 2526: 50.63 -3.45 9.0 -28.10000-388730 2527: 51.43 -2.67 51.0 -36.90000 -999 2528: 51.05 -3.60 314.0 -27.80000-386030 2529: 51.72 -2.77 245.0 -37.70000-385850 2530: 51.62 -3.97 10.0 -46.10000-384130 2531: 52.35 -3.82 301.0 -4.40000-380860 2532: 2533:pretxt8km/pre.1962.12.txt 2534: 610.00 106.00 190.0 48.20000-511900 2535: 544.50 -60.70 116.0 -3.70000-392380 2536: 508.30 -45.50 15.0 -22.40000-389870 2537: 502.20 -53.00 76.0 39.70000-389280 2538: 506.30 -34.50 9.0 -28.10000-388730 2539: 514.30 -26.70 51.0 -36.90000-386780 2540: 510.50 -36.00 314.0 -27.80000-386030 2541: 517.20 -27.70 245.0 -37.70000-385850 2542: 516.20 -39.70 10.0 -46.10000-384130 2543: 523.50 -38.20 301.0 -4.40000-380860 2544: 2545:pretxt0km/pre.1962.12.txt 2546: 610.00 106.00 190.0 48.20000-511900 2547: 544.50 -60.70 116.0 -3.70000-392380 2548: 508.30 -45.50 15.0 -22.40000-389870 2549: 502.20 -53.00 76.0 39.70000-389280 2550: 506.30 -34.50 9.0 -28.10000-388730 2551: 514.30 -26.70 51.0 -36.90000-386780 2552: 510.50 -36.00 314.0 -27.80000-386030 2553: 517.20 -27.70 245.0 -37.70000-385850 2554: 516.20 -39.70 10.0 -46.10000-384130 2555: 523.50 -38.20 301.0 -4.40000-380860 2556: 2557:..As a result of fixing the lats and lons for temperature, and indeed 2558:precip it seems, we have buggered up the outputs!!! Obviously the 2559:correction factor is expecting 100 not 10, but why isn't this a problem 2560:for temperature?! Went back and ran exactly the same version of anomdtb 2561:on temperature - exactly the same as last time (2nd from top above). So 2562:it is precip specific (or, erm, .not.temp specific?). 2563: 2564:On the other hand, we've fixed the -999 WMO codes.. 2565: 2566:..and actually, those anomalies had better be percentage anomalies! 2567: 2568:(checks a few) - yes, they are :-) 2569: 2570:So oookay, LoadCTS reports the divisor is still 10 for lon/lat, so the 2571:stored values for the first station (-511900, BIRI) should be 61 and 10.6, 2572:sounds about right for Norway. The bit in anomdtb (actually the subroutine 2573:'Dumping', LOL) that writes the .txt files just writes directly from the 2574:arrays.. so they must have been modified somewhere in 'Anomalise' (there's 2575:nothing else in 'Dumping'). Modified anomdtb to dump the first station's 2576:lat & lon at key stages - they were too high throughout, so LoadCTS assumed 2577:to be the troublemaker. Modified LoadCTS in the same way, and it was 2578:holding them at x100 from their true values, ie 61.0 -> 6100. It was about 2579:now that I spotted something I'd not thought to examine before: precip 2580:headers use two decimal places for their coordinates! 2581: 2582:Temperature header: 2583: 10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00 2584: 2585:Precipitation header: 2586: 100100 7093 -867 10 JAN MAYEN NORWAY 1921 2006 -999 -999.00 2587: 2588:So.. this begs the question, how does the software suite know which it's got? 2589:By rights it should look at the most extreme values for each.. something tells 2590:me that's not the case. Decided to look at the ranges of values for different 2591:versions of the databases, starting with temperature: 2592: 2593:crua6[/cru/cruts] head -1 fromdpe1a/data/cruts/database/+norm/tmp.0311051552.dtb 2594:-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00 2595:crua6[/cru/cruts] head -1 fromdpe1a/data/cruts/database/+norm/_old/tmp.0310311715.dtb 2596:-176000 3520 3330 220 NICOSIA CYPRUS 1932 1974 -999 nocode 2597:crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/tmp.0311051552.dtb 2598:-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00 2599:crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/tmp.0311051552n.dtb 2600:-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00 2601:crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/database/+norm/_old/tmp.0310311715.dtb 2602:-176000 3520 3330 220 NICOSIA CYPRUS 1932 1974 -999 nocode 2603:crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/database/+norm/tmp.0311051552.dtb 2604:-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00 2605:crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/database/tmp.0311051552.dtb 2606:-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00 2607:crua6[/cru/cruts] head -1 version_3_0/primaries/temp/tmp.0702091122.dtb 2608: 10010 709 87 10 Jan Mayen NORWAY 1921 1990 341921 -999.00 2609:crua6[/cru/cruts] head -1 version_3_0/primaries/temp/tmp.0704300053.dtb 2610: 10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00 2611:crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/tmp.0702091122.dtb 2612: 10010 709 87 10 Jan Mayen NORWAY 1921 1990 341921 -999.00 2613:crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/tmp.0704292355.dtb 2614: 10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00 2615:crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/badtimeline/tmp.0704251819.dtb 2616: 10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00 2617:crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/badtimeline/tmp.0704271015.dtb 2618: 10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00 2619:crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/badtimeline/tmp.0704292158.dtb 2620: 10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00 2621:crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/tmp.0704300053.dtb 2622: 10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00 2623:crua6[/cru/cruts] head -1 version_3_0/db/tmp.0702091122.dtb 2624: 10010 709 87 10 Jan Mayen NORWAY 1921 1990 341921 -999.00 2625:crua6[/cru/cruts] head -1 version_3_0/db/tmp.0704300053.dtb 2626: 10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00 2627: 2628:Without going any further, it's obvious that LoadCTS is going to have to auto- 2629:sense the lat and lon ranges. Missing value codes can then be derived - if it 2630:always returns actual (unscaled) degrees (to one or two decimal places) then 2631:any value lower than -998 will suffice for both parameters. However, this does 2632:make me wonder why it wasn't done like that. Is there a likelihood of the 2633:programs being used on a spatial subset of stations? Say, English? Then lon 2634:would never get into double figures, though lat would.. well let's just hope 2635:not! *laughs hollowly* 2636: 2637:Okay.. so I wrote extra code into LoadCTS to detect Lat & Lon ranges. It excludes any 2638:values for which the modulus of 100 is -99, so hopefully missing value codes do not 2639:conribute. The factors are set accordingly (to 10 or 100). I had to default to 1 which 2640:is a pity. Once you've got the factors, detection of missing values can be a simple 2641:out-of-range test. 2642: 2643:However *sigh* this led me to examine the detection of 'non-standard longitudes' - a 2644:small section of code that converts PJ-style reversed longitudes, or 0-360 ones, to 2645:regular -180 (W) to +180 (E). This code is switched on by the presence of the 2646:'LongType' flag in the LoadCTS call - the trouble is, THAT FLAG IS NEVER SET BY 2647:ANOMDTB. There is a declaration 'integer :: QLongType' but that is never referred to 2648:again. Just another thing I cannot understand, and another reason why this should all 2649:have been rewritten from scratch a year ago! 2650: 2651:So, I wrote 'revlons.for' - a proglet to reverse all longitude values in a database 2652:file. Ran it on the temperature database (final): 2653: 2654: 2655:crua6[/cru/cruts/version_3_0/db/testmergedb] ./revlons 2656:REVLONS - Reverse All Longitudes! 2657: 2658:This nifty little proglet will fix all of your 2659:longitudes so that they point the right way, ie, 2660:positive = East of Greenwich, negative = West. 2661: 2662:..of course, if they are already fixed, this will 2663:UNfix them. I am not that smart! So be careful!! 2664: 2665:Please enter the database to be fixed: tmp.0704300053.dtb 2666: 2667:Output file will be: tmp.0705101334.dtb 2668:Confirm this filename (Y/N): Y 2669: 2670:Log file will be: tmp.0705101334.log 2671: 2672: 5065 stations written to tmp.0705101334.dtb 2673: 2674: 2675: 2676:Thus the 'final' temperature database is now tmp.0705101334.dtb. 2677: 2678:Re-ran anomdtb - with working lat/lon detection and missing lat/lon value 2679:detection - for both precip and temperature. This should ensure that all 2680:WMO codes are present and all lats and lons are correct. 2681: 2682:Temp: 2683: 2684: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 2685: 2686: > Enter the suffix of the variable required: 2687:.tmp 2688: > Select the .cts or .dtb file to load: 2689:tmp.0705101334.dtb 2690: 2691: > Specify the start,end of the normals period: 2692:1961,1990 2693: > Specify the missing percentage permitted: 2694:25 2695: > Data required for a normal: 23 2696: > Specify the no. of stdevs at which to reject data: 2697:3 2698: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 2699:3 2700: > Check for duplicate stns after anomalising? (0=no,>0=km range) 2701:0 2702: > Select the generic .txt file to save (yy.mm=auto): 2703:tmp.txt 2704: > Select the first,last years AD to save: 2705:1901,2006 2706: > Operating... 2707: 2708: > NORMALS MEAN percent STDEV percent 2709: > .dtb 3323823 81.3 2710: > .cts 91963 2.2 3415786 83.5 2711: > PROCESS DECISION percent %of-chk 2712: > no lat/lon 1993 0.0 0.0 2713: > no normal 673044 16.5 16.5 2714: > out-of-range 744 0.0 0.0 2715: > accepted 3415042 83.5 2716: > Dumping years 1901-2006 to .txt files... 2717: 2718: 2719:Precip: 2720: 2721:crua6[/cru/cruts/version_3_0/primaries/precip] ./anomdtb 2722: 2723: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 2724: 2725: > Enter the suffix of the variable required: 2726:.pre 2727: > Will calculate percentage anomalies. 2728: > Select the .cts or .dtb file to load: 2729:pre.0612181221.dtb 2730: 2731: > Specify the start,end of the normals period: 2732:1961,1990 2733: > Specify the missing percentage permitted: 2734:25 2735: > Data required for a normal: 23 2736: > Specify the no. of stdevs at which to reject data: 2737:4 2738: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 2739:3 2740: > Check for duplicate stns after anomalising? (0=no,>0=km range) 2741:0 2742: > Select the generic .txt file to save (yy.mm=auto): 2743:pre.txt 2744: > Select the first,last years AD to save: 2745:1901,2006 2746: > Operating... 2747: 2748: > NORMALS MEAN percent STDEV percent 2749: > .dtb 7315040 73.8 2750: > .cts 299359 3.0 7613600 76.8 2751: > PROCESS DECISION percent %of-chk 2752: > no lat/lon 17911 0.2 0.2 2753: > no normal 2355275 23.8 23.8 2754: > out-of-range 13253 0.1 0.2 2755: > accepted 7521013 75.9 2756: > Dumping years 1901-2006 to .txt files... 2757: 2758: 2759:Note that precip accepted values is up to 75.9%, I honestly don't 2760:think we'll get higher. 2761: 2762:Decided to process temperature all the way. Ran IDL: 2763: 2764:IDL> quick_interp_tdm2,1901,2006,'tmpglo/tmpgrid.',1200,gs=0.5,dumpglo='dumpglo',pts_prefix='tmp0km0705101334txt/tmp.' 2765: 2766:then glo2abs, then mergegrids, to produce monthly output grids. It apparently worked: 2767: 2768:-rw------- 1 f098 cru 138964083 May 13 20:42 cru_ts_3_00.1901.2006.tmp.dat.gz 2769:-rw------- 1 f098 cru 7852589 May 13 20:42 cru_ts_3_00.2001.2006.tmp.dat.gz 2770:-rw------- 1 f098 cru 13108065 May 13 20:39 cru_ts_3_00.1991.2000.tmp.dat.gz 2771:-rw------- 1 f098 cru 13106515 May 13 20:36 cru_ts_3_00.1981.1990.tmp.dat.gz 2772:-rw------- 1 f098 cru 13106963 May 13 20:33 cru_ts_3_00.1971.1980.tmp.dat.gz 2773:-rw------- 1 f098 cru 13123939 May 13 20:30 cru_ts_3_00.1961.1970.tmp.dat.gz 2774:-rw------- 1 f098 cru 13120586 May 13 20:26 cru_ts_3_00.1951.1960.tmp.dat.gz 2775:-rw------- 1 f098 cru 13120691 May 13 20:23 cru_ts_3_00.1941.1950.tmp.dat.gz 2776:-rw------- 1 f098 cru 13130077 May 13 20:20 cru_ts_3_00.1931.1940.tmp.dat.gz 2777:-rw------- 1 f098 cru 13104881 May 13 20:16 cru_ts_3_00.1921.1930.tmp.dat.gz 2778:-rw------- 1 f098 cru 13094948 May 13 20:13 cru_ts_3_00.1911.1920.tmp.dat.gz 2779:-rw------- 1 f098 cru 13085509 May 13 17:08 cru_ts_3_00.1901.1910.tmp.dat.gz 2780: 2781:As a reminder, these output grids are based on the tmp.0705101334.dtb database, with no 2782:merging of neighbourly stations and a limit of 3 standard deviations on anomalies. 2783: 2784:Decided to (re-) process precip all the way, in the hope that I was in the zone or 2785:something. Started with IDL: 2786: 2787:IDL> quick_interp_tdm2,1901,2006,'preglo/pregrid.',450,gs=0.5,dumpglo='dumpglo',pts_prefix='pre0km0612181221txt/pre.' 2788: 2789:Then glo2abs, then mergegrids.. all went fine, apparently. 2790: 2791:31. And so.. to DTR! First time for generation I think. 2792: 2793:Wrote 'makedtr.for' to tackle the thorny problem of the tmin and tmax databases not 2794:being kept in step. Sounds familiar, if worrying. am I the first person to attempt 2795:to get the CRU databases in working order?!! The program pulls no punches. I had 2796:already found that tmx.0702091313.dtb had seven more stations than tmn.0702091313.dtb, 2797:but that hadn't prepared me for the grisly truth: 2798: 2799: 2800:crua6[/cru/cruts/version_3_0/db/dtr] ./makedtr 2801: 2802: MAKEDTR - Produce a DTR database 2803: 2804: This program takes as its input a database of 2805: of minimum temperatures and another of maximum 2806: temperatures, and produces a database of diurnal 2807: temperatures. If the input databases are found 2808: to be out of synchronisation, the option is also 2809: given to save synchronised versions. 2810: 2811:So, may I please have the tmin database? tmn.0702091139.dtb 2812: 2813:May I please now have the tmax database? tmx.0702091313.dtb 2814: 2815:The output database will now be called: dtr.0705152339.dtb 2816: 2817: IMPORTANT: PLEASE READ! (it's good for you) 2818: 2819: The databases you gave are NOT synchronised! 2820: 2821:tmn.0702091139.dtb has 42 'extra' stations 2822: 2823:tmx.0702091313.dtb has 49 'extra' stations 2824:You have the choice of quitting, or of allowing 2825:me to create two new synchronised databases, 2826:which will be saved and used to create the dtr db 2827: 2828: Enter Q to Quit, S to Synchronise: S 2829: New tmin database is: tmn.0705152339.dtb 2830: Discarded tmin stations here: tmn.0702091139.dtb.del 2831: New tmax database is: tmx.0705152339.dtb 2832: Discarded tmax stations here: tmx.0702091313.dtb.del 2833:Number of stations to process: 14267 2834: 2835: 2836:Yes, the difference is a lot more than seven! And the program helpfully dumps a listing 2837:of the surplus stations to the log file. Not a pretty sight. 2838: 2839:Unfortunately, it hadn't worked either. It turns out that there are 3518 stations in 2840:each database with a WMO Code of ' 0'. So, as the makedtr program indexes on the 2841:WMO Code.. you get the picture. *cries* 2842: 2843:Rewrote as makedtr2, which uses the first 20 characters of the header to match: 2844: 2845: 2846: MAKEDTR2 - Produce a DTR database 2847: 2848: This program takes as its input a database of 2849: of minimum temperatures and another of maximum 2850: temperatures, and produces a database of diurnal 2851: temperatures. If the input databases are found 2852: to be out of synchronisation, the option is also 2853: given to save synchronised versions. 2854: 2855:So, may I please have the tmin database? tmn.0702091139.dtb 2856: 2857:May I please now have the tmax database? tmx.0702091313.dtb 2858: 2859:The output database will now be called: dtr.0705162028.dtb 2860: 2861: IMPORTANT: PLEASE READ! (it's good for you) 2862: 2863: The databases you gave are NOT synchronised! 2864: 2865:tmn.0702091139.dtb has 203 'extra' stations 2866: 2867:tmx.0702091313.dtb has 209 'extra' stations 2868:You have the choice of quitting, or of allowing 2869:me to create two new synchronised databases, 2870:which will be saved and used to create the dtr db 2871: 2872: Enter Q to Quit, S to Synchronise: S 2873: New tmin database is: tmn.0705162028.dtb 2874: Discarded tmin stations here: tmn.0702091139.dtb.del 2875: New tmax database is: tmx.0705162028.dtb 2876: Discarded tmax stations here: tmx.0702091313.dtb.del 2877: 2878: 2879:The big jump in the number of 'surplus' stations is because we are no longer automatically 2880:matching stations with WMO=0. 2881: 2882:Here's what happened to the tmin and tmax databases, and the new dtr database: 2883: 2884:Old tmin: tmn.0702091139.dtb Total Records Read: 14309 2885:New tmin: tmn.0705162028.dtb Total Records Read: 14106 2886:Del tmin: tmn.0702091139.dtb.del Total Records Read: 203 2887: 2888:Old tmax: tmx.0702091313.dtb Total Records Read: 14315 2889:New tmax: tmx.0705162028.dtb Total Records Read: 14106 2890:Del tmax: tmx.0702091313.dtb.del Total Records Read: 209 2891: 2892:New dtr: dtr.0705162028.dtb Total Records Read: 14107 2893: 2894:*sigh* - one record out! Also three header problems: 2895: 2896:BLANKS (expected at 8,14,21,26,47,61,66,71,78) 2897: position missed 2898: 8 1 2899: 14 1 2900: 21 0 2901: 26 0 2902: 47 1 2903: 61 0 2904: 66 0 2905: 71 0 2906: 78 0 2907: 2908:Why?!! Well the sad answer is.. because we've got a date wrong. All three 'header' problems 2909:relate to this line: 2910: 2911:6190 94 95 98 100 101 101 102 103 102 97 94 94 2912: 2913:..and as we know, this is not a conventional header. Oh bum. But, but.. how? I know we do 2914:muck around with the header and start/end years, but still.. 2915: 2916:Wrote filtertmm.for, which simply steps through one database (usually tmin) and 2917:looks for a 'perfect' match in another database (usually tmax). 'Perfect' here 2918:means a match of WMO Code, Lat, Lon, Start-Year and End-Year. If a match is 2919:found, both stations are copied to new databases: 2920: 2921: 2922:crua6[/cru/cruts/version_3_0/db/dtr] ./filtertmm 2923: 2924: FILTERTMM - Create GOOD tmin/max databases 2925: 2926:Please enter the tmin database: tmn.0702091139.dtb 2927: 2928:Please enter the tmax database: tmx.0702091313.dtb 2929: 2930: working.. 2931: 2932:Old tmin database: tmn.0702091139.dtb had 14309 stations 2933:New tmin database: tmn.0705182204.dtb has 13016 stations 2934:Old tmax database: tmx.0702091313.dtb had 14315 stations 2935:New tmax database: tmx.0705182204.dtb has 13016 stations 2936: 2937: 2938:I am going to *assume* that worked! So now.. to incorporate the Australian 2939:monthly data packs. Ow. Most future-proof strategy is probably to write a 2940:converter that takes one or more of the packs and creates CRU-format databases 2941:of them. Edit: nope, thought some more and the *best* strategy is a program 2942:that takes *pairs* of Aus packs and updates the actual databases. Bearing in 2943:mind that these are trusted updates and won't be used in any other context. 2944: 2945:From Dave L - who incorporated the initial Australian dump - for the tmin/tmax bulletins, 2946:he used a threshold of 26 days/month or greater for inclusion. 2947: 2948:Obtained two files from Dave - an email that explains some of the Australian 2949:bulletin data/formatting, and a list of Austraian headers matched with their 2950:internal codes (the latter being generated by Dave). 2951: 2952:Actually.. although I was going to assume that filtertmm had done the synching job OK, a 2953:brief look at the Australian stations in the databases showed me otherwise. For instance, 2954:I pulled all the headers with 'AUSTRALIA' out of the two 0705182204 databases. Now because 2955:these were produced by filtertmm, we know that the codes (if present), lats, lons and dates 2956:will all match. Any differences will be in altitude and/or name. And so they were: 2957: 2958:crua6[/cru/cruts/version_3_0/db/dtr] diff tmn.0705182204.dtb.oz tmx.0705182204.dtb.oz | wc -l 2959: 336 2960: 2961:..so roughly 100 don't match. They are mostly altitude discrepancies, though there are an 2962:alarming number of name mismatches too. Examples of both: 2963: 2964:74c74 2965:< 0 -3800 14450 11 AVALON AIRPORT AUSTRALIA 2000 2006 -999 -999.00 2966:--- 2967:> 0 -3800 14450 8 AVALON AIRPORT AUSTRALIA 2000 2006 -999 -999.00 2968: 2969:16c16 2970:< 0 -4230 14650 585 TARRALEAH VILLAGE AUSTRALIA 2000 2006 -999 -999.00 2971:--- 2972:> 0 -4230 14650 595 TARRALEAH CHALET AUSTRALIA 2000 2006 -999 -999.00 2973: 2974:Examples of the second kind (name mismatch) are most concerning as they may well be 2975:different stations. Looked for all occurences in all tmin/tmax databases: 2976: 2977:crua6[/cru/cruts/version_3_0/db/dtr] grep 'TARRALEAH' *dtb 2978:tmn.0702091139.dtb: 0 -4230 14650 585 TARRALEAH VILLAGE AUSTRALIA 2000 2006 -999 -999.00 2979:tmn.0702091139.dtb:9597000 -4230 14645 595 TARRALEAH CHALET AUSTRALIA 1991 2000 -999 -999.00 2980:tmn.0705182204.dtb: 0 -4230 14650 585 TARRALEAH VILLAGE AUSTRALIA 2000 2006 -999 -999.00 2981:tmn.0705182204.dtb:9597000 -4230 14645 595 TARRALEAH CHALET AUSTRALIA 1991 2000 -999 -999.00 2982:tmx.0702091313.dtb: 0 -4230 14650 595 TARRALEAH CHALET AUSTRALIA 2000 2006 -999 -999.00 2983:tmx.0702091313.dtb:9597000 -4230 14645 595 TARRALEAH CHALET AUSTRALIA 1991 2000 -999 -999.00 2984:tmx.0705182204.dtb: 0 -4230 14650 595 TARRALEAH CHALET AUSTRALIA 2000 2006 -999 -999.00 2985:tmx.0705182204.dtb:9597000 -4230 14645 595 TARRALEAH CHALET AUSTRALIA 1991 2000 -999 -999.00 2986: 2987:This takes a little sorting out. Well first, recognise that we are dealing with four files: tmin 2988:and tmax, early and late (before and after filtertmm.for). We see there are two TARRALEAH entries 2989:in each of the four files. We see that 'TARRALEAH VILLAGE' only appears in the tmin file. We see, 2990:most importantly perhaps, that they are temporally contiguous - that is, each pair could join with 2991:minimal overlap, as one is 1991-2000 and the other 2000-2006. Also, we note that the 'early' one 2992:of each pair has a slightly different longitude and altitude (the former being the thing that 2993:distinguished the stations in filtertmm.for). 2994: 2995:Finally, this, from the tmax.2005120120051231.txt bulletin: 2996: 2997: 95018, 051201051231, -42.30, 146.45, 18.0, 00, 31, 31, 585, TARRALEAH VILLAGE 2998: 2999:So we can resolve this case - a single station called TARRALEAH VILLAGE, running from 1991 to 2006. 3000: 3001:But what about the others?! There are close to 1000 incoming stations in the bulletins, must 3002:every one be identified in this way?!! Oh God. There's nothing for it - I'll have to write a prog 3003:to find matches for the incoming Australian bulletin stations in the main databases. I'll have to 3004:use the databases from before the filtertmm application, so *0705182204.dtb. And it will only 3005:need the Australian headers, so I used grep to create *0705182204.dtb.auhead files. The other 3006:input is the list of stations taken from the monthly bulletins. Now these have a different number 3007:of stations each month, so the prog will build an array of all possible stations based on the 3008:files we have. Oh boy. And the program shall be called, 'auminmaxmatch.for'. 3009: 3010:Assembled some information: 3011: 3012:crua6[/cru/cruts/version_3_0/db] wc -l *auhead 3013: 1518 glseries_tmn_final_merged.auhead 3014: 1518 tmn.0611301516.dat.auhead 3015: 1518 tmn.0612081255.dat.auhead 3016: 1518 tmn.0702091139.dtb.auhead 3017: 1518 tmn.0705152339.dtb.auhead 3018: 1426 tmn.0705182204.dtb.auhead 3019: 3020:(the 'auhead' files were created with ) 3021: 3022:Actually, stopped work on that. Trying to match over 800 'bulletin' stations against over 3,000 3023:database stations *in two unsynchronised files* was just hurting my brain. The files have to be 3024:properly synchronised first, with a more lenient and interactive version of filtertmm. Or... 3025:could I use mergedb?! Pretend to merge tmin into tmax and see what pairings it managed? No 3026:roll through obviously. Well it's worth a play. 3027: 3028:..unfortunately, not. Because when I tried, I got a lot of odd errors followed by a crash. The 3029:reason, I eventually deduced, was that I didn't build mergedb with the idea that WMO codes might 3030:be zero (many of the australian stations have wmo=0). This means that primary matching on WMO 3031:code is impossible. This just gets worse and worse: now it looks as though I'll have to find WMO 3032:Codes (or pseudo-codes) for the *3521* stations in the tmin file that don't have one!!! 3033: 3034:OK.. let's break the problem down. Firstly, a lot of stations are going to need WMO codes, if 3035:available. It shouldn't be too hard to find any matches with the existing WMO coded stations in 3036:the other databases (precip, temperature). Secondly, we need to exclude stations that aren't 3037:synchronised between the two databases (tmin/tmax). So can mergedb be modified to treat WMO codes 3038:of 0 as 'missing'? Had a look, and it does check that the code isn't -999 OR 0.. but not when 3039:preallocating flags in subroutine 'countscnd'. Fixed that and tried running it again.. exactly 3040:the same result (crash). I can't see anything odd about the station it crashes on: 3041: 3042: 0 -2810 11790 407 MOUNT MAGNET AERO AUSTRALIA 2000 2006 -999 -999.00 3043:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3044:2000 339 344 280 252 214 202 189 196 262 291 316 377 3045:2001 371 311 310 300 235 212 201 217 249 262 314 333 3046:2002-9999-9999 339 297 258 209 205 212 246 299 341 358 3047:2003 365 367 336 296 249 195 193 200 238 287 325 368 3048:2004 395 374 321 284 219 214 173 188 239 309 305 370 3049:2005 389 396 358 315 251 182 189 201 233 267 332 341 3050:2006 366 331 314 246 240-9999-9999-9999-9999-9999-9999-9999 3051: 3052:.. it's very similar to preceding (and following) stations, and the station before has even 3053:less real data (the one before that has none at all and is auto-deleted). The nature of the 3054:crash is 'forrtl: error (65): floating invalid' - so a type mismatch possibly. The station has 3055:a match in the tmin database (tmn.0702091139.dtb) but the longitude is different: 3056: 3057:tmn.0702091139.dtb: 3058: 0 -2810 11780 407 MOUNT MAGNET AERO AUSTRALIA 2000 2006 -999 -999.00 3059:tmx.0702091313.dtb: 3060: 0 -2810 11790 407 MOUNT MAGNET AERO AUSTRALIA 2000 2006 -999 -999.00 3061: 3062:It also appears in the tmin/tmax bulletins, eg: 3063: 7600, 070401070430, -28.12, 117.84, 16.0, 00, 30, 30, 407, MOUNT MAGNET AERO 3064: 3065:Note that the altitude matches (as distinct from the station below). 3066: 3067:Naturally, there is a further 'MOUNT MAGNET' station, but it's probably distinct: 3068: 3069:tmn.0702091139.dtb: 3070:9442800 -2807 11785 427 MOUNT MAGNET (MOUNT AUSTRALIA 1956 1992 -999 -999.00 3071:tmx.0702091313.dtb: 3072:9442800 -2807 11785 427 MOUNT MAGNET (MOUNT AUSTRALIA 1957 1992 -999 -999.00 3073: 3074:I am at a bit of a loss. It will take a very long time to resolve each of these 'rogue' 3075:stations. Time I do not have. The only pragmatic thing to do is to dump any stations that are 3076:too recent to have normals. They will not, after all, be contributing to the output. So I 3077:knocked out 'goodnorm.for', which simply uses the presence of a valid normals line to sort. 3078:The results were pretty scary: 3079: 3080: 3081:crua6[/cru/cruts/version_3_0/db/dtr] ./goodnorm 3082: 3083: GOODNORM: Extract stations with non-missing normals 3084: 3085:Please enter the input database name: tmn.0702091139.dtb 3086:The output database will be called: tmn.0705281724.dtb 3087: 3088:(removed stations will be placed in: tmn.0705281724.del) 3089: 3090:FINISHED. 3091: 3092:Stations retained: 5026 3093:Stations removed: 9283 3094: 3095:crua6[/cru/cruts/version_3_0/db/dtr] ./goodnorm 3096: 3097: GOODNORM: Extract stations with non-missing normals 3098: 3099:Please enter the input database name: tmx.0702091313.dtb 3100:The output database will be called: tmx.0705281724.dtb 3101: 3102:(removed stations will be placed in: tmx.0705281724.del) 3103: 3104:FINISHED. 3105: 3106:Stations retained: 4997 3107:Stations removed: 9318 3108: 3109: 3110: 3111:Essentially, two thirds of the stations have no normals! Of course, this still leaves us with 3112:a lot more stations than we had for tmean (goodnorm reported 3316 saved, 1749 deleted) though 3113:still far behind precipitation (goodnorm reported 7910 saved, 8027 deleted). 3114: 3115:I suspect the high percentage lost reflects the influx of modern Australian data. Indeed, nearly 3116:3,000 of the 3,500-odd stations with missing WMO codes were excluded by this operation. This means 3117:that, for tmn.0702091139.dtb, 1240 Australian stations were lost, leaving only 278. 3118: 3119:This is just silly. I can't dump these stations, they are needed to potentially match with the 3120:bulletin stations. I am now going to try the following: 3121: 3122:1. Attempt to pair bulletin stations with existing in the tmin database. Mark pairings in the 3123: database headers and in a new 'Australian Mappings' file. Program auminmatch.for. 3124: 3125:2. Run an enhanced filtertmm to synchronise the tmin and tmax databases, but prioritising the 3126: 'paired' stations from step 1 (so they are not lost). Mark the same pairings in the tmax 3127: headers too, and update the 'Australian Mappings' file. 3128: 3129:3. Add the bulletins to the databases. 3130: 3131: 3132:OK.. step 1. Modified auminmaxmatch.for to produce auminmatch.for. Hit a semi-philosophical 3133:problem: what to do with a positive match between a bulletin station and a zero-wmo database 3134:station? The station must have a real WMO code or it'll be rather hard to describe the match! 3135: 3136:Got a list of around 12,000 wmo codes and stations from Dave L; unfortunately there was a problem 3137:with its formatting that I just couldn't resolve. 3138: 3139:So.. current thinking is that, if I find a pairing between a bulletin station and a zero-coded 3140:Australain station in the CRU database, I'll give the CRU database station the Australian local 3141:(bulletin) code twice: once at the end of the header, and once as the WMO code *multiplied by -1* 3142:to avoid implying that it's legitimate. Then if a 'proper' code is found or allocated later, the 3143:mapping to the bulletin code will still be there at the end of the header. Of course, an initial 3144:check will ensure that a match can't be found, within the CRU database, between the zero-coded 3145:station and a properly-coded one. 3146: 3147:Debated header formats with David. I think we're going to go with (i8,a8) at the end of the header, 3148:though really it's (2x,i6,a8) as I remember the Anders code being i2 and the real start year being 3149:i4 (both from the tmean database). This will mean post-processing existing databases of course, 3150:but that's not a priority. 3151: 3152:A brief (hopefully) diversion to get station counts sorted. David needs them so might as well sort 3153:the procedure. In the upside-down world of Mark and Tim, the numbers of stations contributing to 3154:each cell during the gridding operation are calculated not in the IDL gridding program - oh, no! - 3155:but in anomdtb! Yes, the program which reads station data and writes station data has a second, 3156:almost-entirely unrelated function of assessing gridcell contributions. So, to begin with it runs 3157:in the usual way: 3158: 3159:crua6[/cru/cruts/version_3_0/primaries/precip] ./anomdtb 3160: 3161: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 3162: 3163: > Enter the suffix of the variable required: 3164:.pre 3165: > Will calculate percentage anomalies. 3166: > Select the .cts or .dtb file to load: 3167:pre.0612181221.dtb 3168: 3169: > Specify the start,end of the normals period: 3170:1961,1990 3171: > Specify the missing percentage permitted: 3172:25 3173: > Data required for a normal: 23 3174: > Specify the no. of stdevs at which to reject data: 3175:4 3176: 3177:But then, we choose a different output, and it all shifts focus and has to ask all the IDL 3178:questions!! 3179: 3180: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 3181:4 3182: > Check for duplicate stns after anomalising? (0=no,>0=km range) 3183:0 3184: > Select the .stn file to save: 3185:pre.stn 3186: > Enter the correlation decay distance: 3187:450 3188: > Submit a grim that contains the appropriate grid. 3189: > Enter the grim filepath: 3190:clim.6190.lan.pre 3191: 3192: > Grid dimensions and domain size: 720 360 67420 3193: > Select the first,last years AD to save: 3194:1901,2006 3195: > Operating... 3196: 3197: > NORMALS MEAN percent STDEV percent 3198: > .dtb 7315040 73.8 3199: > .cts 299359 3.0 7613600 76.8 3200: > PROCESS DECISION percent %of-chk 3201: > no lat/lon 17911 0.2 0.2 3202: > no normal 2355275 23.8 23.8 3203: > out-of-range 13253 0.1 0.2 3204: > accepted 7521013 75.9 3205: > Calculating station coverages... 3206: 3207:And then.. it unhelpfully crashes: 3208: 3209: > ##### WithinRange: Alloc: DataB ##### 3210:forrtl: severe (174): SIGSEGV, segmentation fault occurred 3211: 3212:Ho hum. I did try this last year which is why I'm not tearing my hair out. The plan is to use the 3213:outputs from the regular anomdtb runs - ie, the monthly files of valid stations. After all we need 3214:to know the station counts on a per month basis. We can use the lat and lon, along with the 3215:correlation decay distance.. shouldn't be too awful. Just even more programming and work. So before 3216:I commit to that, a quick look at the IDL gridding prog to see if it can dump the figures instead: 3217:after all, this is where the actual 'station count' information is assembled and used!! 3218: 3219:..well that was, erhhh.. 'interesting'. The IDL gridding program calculates whether or not a 3220:station contributes to a cell, using.. graphics. Yes, it plots the station sphere of influence then 3221:checks for the colour white in the output. So there is no guarantee that the station number files, 3222:which are produced *independently* by anomdtb, will reflect what actually happened!! 3223: 3224:Well I've just spent 24 hours trying to get Great Circle Distance calculations working in Fortran, 3225:with precisely no success. I've tried the simple method (as used in Tim O's geodist.pro, and the 3226:more complex and accurate method found elsewhere (wiki and other places). Neither give me results 3227:that are anything near reality. FFS. 3228: 3229:Worked out an algorithm from scratch. It seems to give better answers than the others, so we'll go 3230:with that. Also decided that the approach I was taking (pick a gridline of latitude and reverse- 3231:engineer the GCD algorithm so the unknown is the second lon) was overcomplicated, when we don't 3232:need to know where it hits, just that it does. Since for any cell the nearest point to the station 3233:will be a vertex, we can test candidate cells for the distance from the appropriate vertex to the 3234:station. Program is stncounts.for, but is causing immense problems. 3235: 3236:The problem is, really, the huge numbers of cells potentially involved in one station, particularly 3237:at high latitudes. Working out the possible bounding box when you're within cdd of a pole (ie, for 3238:tmean with a cdd of 1200, the N-S extent is over 20 cells (10 degs) in each direction. Maybe not a 3239:serious problem for the current datasets but an example of the complexity. Also, deciding on the 3240:potential bounding box is nontrivial, because of cell 'width' changes at high latitudes (at 61 degs 3241:North, the half-degree cells are only 27km wide! With a precip cdd of 450 km this means the 3242:bounding box is dozens of cells wide - and will be wider at the Northern edge! 3243: 3244:Clearly a large number of cells are being marked as covered by each station. So in densely-stationed 3245:areas there will be considerable smoothing, and in sparsely-stationed (or empty) areas, there will be 3246:possibly untypical data. I might suggest two station counts - one of actual stations contributing from 3247:within the cell, one for stations contributing from within the cdd. The former being a subset of the 3248:latter, so the latter could be used as the previous release was used. 3249: 3250:Well, got stncounts.for working, finally. And, out of malicious interest, I dumped the first station's 3251:coverage to a text file and counted up how many cells it 'influenced'. The station was at 10.6E, 61.0N. 3252:The total number of cells covered was a staggering 476! Or, if you prefer, 475 indirect and one direct. 3253: 3254:Ran for the first month (01/1901). Compared the resulting grid with that from CRU TS 2.1. Seems to 3255:compare fine, some higher, some lower. Example: 3256: 3257:2.10: 139 142 146 154 156 157 165 170 3258:3.00: 141 148 154 153 153 159 163 168 3259: 3260:(data are on latitude #265 and longitudes #163-170) 3261: 3262:Wrote 'makelsmask.for' to, well, make a land-sea mask. It'll work with any gridded 3263:data file that uses -999 for sea. The mask is called 'lsmask.halfdeg.dat'. Adapted 3264:stncounts.for to read it and use it to mask the output files. 3265: 3266:Still a bit disturbed by the large number of cells marked as 'influenced' by a single station. IDL 3267:seems to use the inbuilt 'TRIGRID' function to interpolate the grid, so there's no way of getting 3268:the station count for a particular cell that way anyway. Not that it would mean much, since there 3269:is bound to be some kind of weighting (it's not clear what that weighting is, though, from the IDL 3270:website). So the figures in the station count files are really rather loose. What might be useful 3271:as a companion dataset would be the ACTUAL station counts. Counts for cells with stations actually 3272:INSIDE them. Of course, that might be rather sensitive information.. 3273: 3274:Managed a full run of stncounts. It took over five and a half hours, which is a bit much! 3275: 3276:Back to the gridding. I am seriously worried that our flagship gridded data product is produced by 3277:Delaunay triangulation - apparently linear as well. As far as I can see, this renders the station 3278:counts totally meaningless. It also means that we cannot say exactly how the gridded data is arrived 3279:at from a statistical perspective - since we're using an off-the-shelf product that isn't documented 3280:sufficiently to say that. Why this wasn't coded up in Fortran I don't know - time pressures perhaps? 3281:Was too much effort expended on homogenisation, that there wasn't enough time to write a gridding 3282:procedure? Of course, it's too late for me to fix it too. Meh. 3283: 3284:Well, it's been a real day of revelations, never mind the week. This morning I 3285:discovered that proper angular weighted interpolation was coded into the IDL 3286:routine, but that its use was discouraged because it was slow! Aaarrrgghh. 3287:There is even an option to tri-grid at 0.1 degree resolution and then 'rebin' 3288:to 720x360 - also deprecated! And now, just before midnight (so it counts!), 3289:having gone back to the tmin/tmax work, I've found that most if not all of the 3290:Australian bulletin stations have been unceremoniously dumped into the files 3291:without the briefest check for existing stations. A classic example would be 3292:these 'two' stations: 3293: 3294: 0 -1570 12870 31 KIMBERLEY RES.STATIO AUSTRALIA 2000 2006 -999 -999.00 3295:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3296:2000 245 243 243 232 184 143 138 155 193 231 249 249 3297:2001 245 247 241 216 156 167 163 129 201 238 246 247 3298:2002 244 246 230 208 167 122 92 119 202 217 248 259 3299:2003 253 249 222 220 169 151 144 158 203 216 248 250 3300:2004 252 247 244 209 202 135 129 140 176 230 248 257 3301:2005 245 246 237-9999-9999-9999-9999-9999-9999-9999-9999-9999 3302:2006-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3303: 3304: 0 -1565 12871 31 KIMBERLEY RES.STATIO AUSTRALIA 1971 2000 -999 -999.00 3305:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3306:1971 254 249 239 218 166 147 142 169 214 246 253 241 3307:1972 246 244 226 198 175 158 126 182 200 222 244 259 3308:1973 255 259 252 232 215 186 171 189 216 240 256 246 3309:1974 247 243 240 217 183 144 134 171 216 247 248 246 3310:1975 239 239 237 216 180 157 168 171 223 233 243 246 3311:1976 235 244 227 190 148 142 142 144 177 236 252 250 3312:1977 253 249 245 218 177 135 130 137 187 226 250 248 3313:1978 247 244 239 199 218 174 162 186 195 233 245 253 3314:1979 247 246 238 217 205 166 147 178 216 234 248 254 3315:1980 249 245 240 221 186 161 141 171 192 241 249 252 3316:1981-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3317:1982-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3318:1983-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3319:1984-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3320:1985-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3321:1986-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3322:1987-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3323:1988-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3324:1989-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3325:1990-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3326:1991 248 244 234 224 169 160 160 140 210 225 252 260 3327:1992 253 251 247 239 206-9999 141 173 218 237 246 260 3328:1993 247-9999 242 225 207 172 149 170 204 237 249 258 3329:1994 253-9999 214 196 171 140 130 141 171 222 248 247 3330:1995 245 249 234 205 186 155 148 151 198 217 245 244 3331:1996 245 238 220 208 159 166 136 161 179 225 233 247 3332:1997 245 243 217 195 186 149 138 156 195 230 242 247 3333:1998 248 250 245 229 188 167 177 158 200 247 253 250 3334:1999 250 245 242 216 144 150 123-9999 188 239 240 251 3335:2000 245 243 243 232 184 143 138 154 194 231 249 249 3336: 3337:Now, I admit the lats and lons aren't spot on. But c'mon, what are the chances 3338:of them being different? The two year 2000s are almost identical. What about: 3339: 3340: 0 -1550 12450 12 KURI BAY AUSTRALIA 2000 2006 -999 -999.00 3341:9420800 -1548 12452 29 KURI BAY AUSTRALIA 1965 1992 -999 -999.00 3342: 3343:Or: 3344: 3345: 0 -1550 12810 11 WYNDHAM AUSTRALIA 2000 2006 -999 -999.00 3346: 0 -1550 12820 4 WYNDHAM AERO AUSTRALIA 2000 2006 -999 -999.00 3347:9421400 -1549 12812 11 WYNDHAM POST OFFICE AUSTRALIA 1968 2000 -999 -999.00 3348:9421401 -1547 12810 20 WYNDHAM (WYNDHAM POR AUSTRALIA 1898 1966 -999 -999.00 3349: 3350:Come On!! This is one station isn't it. 3351: 3352:I'd be content to leave it - but I have to match the bulletins! And I can match 3353:to the long, stable series or to the loose, flapping ones put in for the 3354:purpose! Meh II. 3355: 3356:So.. in the end I matched to the 2000-2006 stations, where they actually did match. 3357:Unfortunately the huge bulk of the bulletins still had to have new entries created for 3358:them, which is a shame, and begs the question of why the Australian update bulletins 3359:don't match the original 'catch-up' block they sent us. 3360: 3361:For some reason, the auminmatch program is causing no end of grief. I thought I'd 3362:managed a complete run, and it did produce a good-looking tmin database with lots of 3363:new station stubs tacked on the end: 3364: 3365: -1009 -6628 11054 12 KURI BAY AUSTRALIA 2007 2007 -999 1009 3366:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3367:2006-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3368: -1019 -6628 11054 23 KALUMBURU AUSTRALIA 2007 2007 -999 1019 3369:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3370:2006-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3371: -1020 -6628 11054 51 TRUSCOTT AUSTRALIA 2007 2007 -999 1020 3372:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3373:2006-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 3374: 3375:However, it doesn't seem to have put the bulletin codes on the (a8) header field, for 3376:some of the matches only! 3377: 3378:Not sure why this is yet.. but have found also that there are cases of duplicated lat/lon pairs, 3379:so multiple matches are being made.. argh.. will have to further augment auminmatch. Not happy. 3380: 3381:An interesting aside.. David was looking at the v3.00 precip to help National Geographic with 3382:an enquiry. I produced a second 'station' file with the 'honest' counts (see above) and he used 3383:that to mask out cells with a 0 count (ie that only had indirect data from 'nearby' stations). 3384:There were some odd results.. with certain months havign data, and others being missing. After 3385:considerable debate and investigation, it was understood that anomdtb calculates normals on a 3386:monthly basis. So, where there are 7 or 8 missing values in each month (1961-1990), a station 3387:may end up contributing only in certain months of the year, throughout its entire run! This was 3388:noticed in the Seychelles, where only October has real data (the remaining months being relaxed 3389:to the climatology but excluded by David using the 'tight' station mask). There is no easy 3390:solution, because essentially it's an honest result: only October has sufficient values to form 3391:a normal, so only October gets anomalised. It's an unfortunate concidence that it's the only 3392:station in the cell, but it's not the only one. A 'solution' could be for anomdtb to get a bit 3393:more involved in the gridding, to check that if a cell only has one station (for one or more 3394:years) then it's all-or-nothing. Maybe if only one month has a normal then it's dumped and the 3395:whole reverts to climatology. Maybe if 4 or more months have normals.. maybe if >0 months have 3396:normals and the rest can be brought in with a minor relaxation of the '75% rule'.. who knows. 3397: 3398:Back to auminmatch.for, and a (philosophical) breakthrough. Built a loop to find 'fuzzy' 3399:matches and group them together. The user then processes one group at a time, pairing up 3400:matches until the potential for further matches is zero (or the user decides it is). Uses a 3401:FSM to work out each chain (all db matches for a bulletin, then all bulletins that match 3402:each of those db stations, then.. etc). To understand it, either read the code (especially 3403:the comments) or just look at this mind-boggling example from the first run of it: 3404: 3405:-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 3406:User Match Decision(s) Please! 3407:Bulletin stations: 8 3408: 1. 9021 -3193 11598 15 PERTH AIRPORT 3409: 2. 9225 -3192 11587 25 PERTH METRO 3410: 3. 9106 -3205 11598 10 GOSNELLS CITY 3411: 4. 9240 -3201 11614 384 BICKLEY 3412: 5. 9172 -3210 11588 30 JANDAKOT AERO 3413: 6. 9215 -3196 11576 41 SWANBOURNE 3414: 7. 9194 -3222 11581 14 MEDINA RESEARCH CENT 3415: 8. 9256 -3224 11568 6 GARDEN ISLAND HSF 3416:Database stations: 18 3417: 1. 0 -3190 11590 25 PERTH METRO 2000 2006 3418: 2. 0 -3190 11600 15 PERTH AIRPORT 2000 2006 3419: 3. 9461000 -3190 11600 20 PERTH AIRPORT COMPAR 1944 2006 3420: 4. 9461001 -3190 11600 18 PERTH AIRPORT 1944 2004 3421: 5. 9461501 -3198 11607 210 KALAMUNDA (KALAMUNDA 1908 1992 3422: 6. 0 -3200 11580 41 SWANBOURNE 2000 2006 3423: 7. 0 -3200 11580 20 SUBIACO TREATMENT PL 2000 2006 3424: 8. 0 -3196 11579 20 SUBIACO TREATMENT PL 1991 1999 3425: 9. 9460800 -3195 11587 19 PERTH (PERTH REGIONA 1897 1992 3426: 10. 9460801 -3195 11587 19 PERTH-(PERTH-REGIONA 1897 1992 3427: 11. 0 -3210 11590 30 JANDAKOT AERO 2000 2006 3428: 12. 0 -3210 11600 10 GOSNELLS CITY 2000 2006 3429: 13. 0 -3200 11610 384 BICKLEY 2000 2006 3430: 14. 0 -3220 11580 14 MEDINA RESEARCH CENT 2000 2006 3431: 15. 0 -3220 11580 4 KWINANA BP REFINERY 2000 2006 3432: 16. 0 -3223 11576 4 KWINANA BP REFINERY 1961 2000 3433: 17. 9560800 -3222 11581 14 MEDINA RESEARCH CENT 1991 2000 3434: 18. 0 -3220 11570 6 GARDEN ISLAND HSF 2000 2006 3435: 3436:Enter a matching pair, (bulletin,database) or 'n' to end: 1,2 3437:Bulletin stations: 7 3438: 2. 9225 -3192 11587 25 PERTH METRO 3439: 3. 9106 -3205 11598 10 GOSNELLS CITY 3440: 4. 9240 -3201 11614 384 BICKLEY 3441: 5. 9172 -3210 11588 30 JANDAKOT AERO 3442: 6. 9215 -3196 11576 41 SWANBOURNE 3443: 7. 9194 -3222 11581 14 MEDINA RESEARCH CENT 3444: 8. 9256 -3224 11568 6 GARDEN ISLAND HSF 3445:Database stations: 17 3446: 1. 0 -3190 11590 25 PERTH METRO 2000 2006 3447: 3. 9461000 -3190 11600 20 PERTH AIRPORT COMPAR 1944 2006 3448: 4. 9461001 -3190 11600 18 PERTH AIRPORT 1944 2004 3449: 5. 9461501 -3198 11607 210 KALAMUNDA (KALAMUNDA 1908 1992 3450: 6. 0 -3200 11580 41 SWANBOURNE 2000 2006 3451: 7. 0 -3200 11580 20 SUBIACO TREATMENT PL 2000 2006 3452: 8. 0 -3196 11579 20 SUBIACO TREATMENT PL 1991 1999 3453: 9. 9460800 -3195 11587 19 PERTH (PERTH REGIONA 1897 1992 3454: 10. 9460801 -3195 11587 19 PERTH-(PERTH-REGIONA 1897 1992 3455: 11. 0 -3210 11590 30 JANDAKOT AERO 2000 2006 3456: 12. 0 -3210 11600 10 GOSNELLS CITY 2000 2006 3457: 13. 0 -3200 11610 384 BICKLEY 2000 2006 3458: 14. 0 -3220 11580 14 MEDINA RESEARCH CENT 2000 2006 3459: 15. 0 -3220 11580 4 KWINANA BP REFINERY 2000 2006 3460: 16. 0 -3223 11576 4 KWINANA BP REFINERY 1961 2000 3461: 17. 9560800 -3222 11581 14 MEDINA RESEARCH CENT 1991 2000 3462: 18. 0 -3220 11570 6 GARDEN ISLAND HSF 2000 2006 3463: 3464:Enter a matching pair, (bulletin,database) or 'n' to end: 2,1 3465:Bulletin stations: 6 3466: 3. 9106 -3205 11598 10 GOSNELLS CITY 3467: 4. 9240 -3201 11614 384 BICKLEY 3468: 5. 9172 -3210 11588 30 JANDAKOT AERO 3469: 6. 9215 -3196 11576 41 SWANBOURNE 3470: 7. 9194 -3222 11581 14 MEDINA RESEARCH CENT 3471: 8. 9256 -3224 11568 6 GARDEN ISLAND HSF 3472:Database stations: 16 3473: 3. 9461000 -3190 11600 20 PERTH AIRPORT COMPAR 1944 2006 3474: 4. 9461001 -3190 11600 18 PERTH AIRPORT 1944 2004 3475: 5. 9461501 -3198 11607 210 KALAMUNDA (KALAMUNDA 1908 1992 3476: 6. 0 -3200 11580 41 SWANBOURNE 2000 2006 3477: 7. 0 -3200 11580 20 SUBIACO TREATMENT PL 2000 2006 3478: 8. 0 -3196 11579 20 SUBIACO TREATMENT PL 1991 1999 3479: 9. 9460800 -3195 11587 19 PERTH (PERTH REGIONA 1897 1992 3480: 10. 9460801 -3195 11587 19 PERTH-(PERTH-REGIONA 1897 1992 3481: 11. 0 -3210 11590 30 JANDAKOT AERO 2000 2006 3482: 12. 0 -3210 11600 10 GOSNELLS CITY 2000 2006 3483: 13. 0 -3200 11610 384 BICKLEY 2000 2006 3484: 14. 0 -3220 11580 14 MEDINA RESEARCH CENT 2000 2006 3485: 15. 0 -3220 11580 4 KWINANA BP REFINERY 2000 2006 3486: 16. 0 -3223 11576 4 KWINANA BP REFINERY 1961 2000 3487: 17. 9560800 -3222 11581 14 MEDINA RESEARCH CENT 1991 2000 3488: 18. 0 -3220 11570 6 GARDEN ISLAND HSF 2000 2006 3489: 3490:Enter a matching pair, (bulletin,database) or 'n' to end: 3,12 3491:Bulletin stations: 5 3492: 4. 9240 -3201 11614 384 BICKLEY 3493: 5. 9172 -3210 11588 30 JANDAKOT AERO 3494: 6. 9215 -3196 11576 41 SWANBOURNE 3495: 7. 9194 -3222 11581 14 MEDINA RESEARCH CENT 3496: 8. 9256 -3224 11568 6 GARDEN ISLAND HSF 3497:Database stations: 15 3498: 3. 9461000 -3190 11600 20 PERTH AIRPORT COMPAR 1944 2006 3499: 4. 9461001 -3190 11600 18 PERTH AIRPORT 1944 2004 3500: 5. 9461501 -3198 11607 210 KALAMUNDA (KALAMUNDA 1908 1992 3501: 6. 0 -3200 11580 41 SWANBOURNE 2000 2006 3502: 7. 0 -3200 11580 20 SUBIACO TREATMENT PL 2000 2006 3503: 8. 0 -3196 11579 20 SUBIACO TREATMENT PL 1991 1999 3504: 9. 9460800 -3195 11587 19 PERTH (PERTH REGIONA 1897 1992 3505: 10. 9460801 -3195 11587 19 PERTH-(PERTH-REGIONA 1897 1992 3506: 11. 0 -3210 11590 30 JANDAKOT AERO 2000 2006 3507: 13. 0 -3200 11610 384 BICKLEY 2000 2006 3508: 14. 0 -3220 11580 14 MEDINA RESEARCH CENT 2000 2006 3509: 15. 0 -3220 11580 4 KWINANA BP REFINERY 2000 2006 3510: 16. 0 -3223 11576 4 KWINANA BP REFINERY 1961 2000 3511: 17. 9560800 -3222 11581 14 MEDINA RESEARCH CENT 1991 2000 3512: 18. 0 -3220 11570 6 GARDEN ISLAND HSF 2000 2006 3513: 3514:Enter a matching pair, (bulletin,database) or 'n' to end: 4,13 3515:Bulletin stations: 4 3516: 5. 9172 -3210 11588 30 JANDAKOT AERO 3517: 6. 9215 -3196 11576 41 SWANBOURNE 3518: 7. 9194 -3222 11581 14 MEDINA RESEARCH CENT 3519: 8. 9256 -3224 11568 6 GARDEN ISLAND HSF 3520:Database stations: 14 3521: 3. 9461000 -3190 11600 20 PERTH AIRPORT COMPAR 1944 2006 3522: 4. 9461001 -3190 11600 18 PERTH AIRPORT 1944 2004 3523: 5. 9461501 -3198 11607 210 KALAMUNDA (KALAMUNDA 1908 1992 3524: 6. 0 -3200 11580 41 SWANBOURNE 2000 2006 3525: 7. 0 -3200 11580 20 SUBIACO TREATMENT PL 2000 2006 3526: 8. 0 -3196 11579 20 SUBIACO TREATMENT PL 1991 1999 3527: 9. 9460800 -3195 11587 19 PERTH (PERTH REGIONA 1897 1992 3528: 10. 9460801 -3195 11587 19 PERTH-(PERTH-REGIONA 1897 1992 3529: 11. 0 -3210 11590 30 JANDAKOT AERO 2000 2006 3530: 14. 0 -3220 11580 14 MEDINA RESEARCH CENT 2000 2006 3531: 15. 0 -3220 11580 4 KWINANA BP REFINERY 2000 2006 3532: 16. 0 -3223 11576 4 KWINANA BP REFINERY 1961 2000 3533: 17. 9560800 -3222 11581 14 MEDINA RESEARCH CENT 1991 2000 3534: 18. 0 -3220 11570 6 GARDEN ISLAND HSF 2000 2006 3535: 3536:Enter a matching pair, (bulletin,database) or 'n' to end: 5,11 3537:Bulletin stations: 3 3538: 6. 9215 -3196 11576 41 SWANBOURNE 3539: 7. 9194 -3222 11581 14 MEDINA RESEARCH CENT 3540: 8. 9256 -3224 11568 6 GARDEN ISLAND HSF 3541:Database stations: 13 3542: 3. 9461000 -3190 11600 20 PERTH AIRPORT COMPAR 1944 2006 3543: 4. 9461001 -3190 11600 18 PERTH AIRPORT 1944 2004 3544: 5. 9461501 -3198 11607 210 KALAMUNDA (KALAMUNDA 1908 1992 3545: 6. 0 -3200 11580 41 SWANBOURNE 2000 2006 3546: 7. 0 -3200 11580 20 SUBIACO TREATMENT PL 2000 2006 3547: 8. 0 -3196 11579 20 SUBIACO TREATMENT PL 1991 1999 3548: 9. 9460800 -3195 11587 19 PERTH (PERTH REGIONA 1897 1992 3549: 10. 9460801 -3195 11587 19 PERTH-(PERTH-REGIONA 1897 1992 3550: 14. 0 -3220 11580 14 MEDINA RESEARCH CENT 2000 2006 3551: 15. 0 -3220 11580 4 KWINANA BP REFINERY 2000 2006 3552: 16. 0 -3223 11576 4 KWINANA BP REFINERY 1961 2000 3553: 17. 9560800 -3222 11581 14 MEDINA RESEARCH CENT 1991 2000 3554: 18. 0 -3220 11570 6 GARDEN ISLAND HSF 2000 2006 3555: 3556:Enter a matching pair, (bulletin,database) or 'n' to end: 6,6 3557:Bulletin stations: 2 3558: 7. 9194 -3222 11581 14 MEDINA RESEARCH CENT 3559: 8. 9256 -3224 11568 6 GARDEN ISLAND HSF 3560:Database stations: 12 3561: 3. 9461000 -3190 11600 20 PERTH AIRPORT COMPAR 1944 2006 3562: 4. 9461001 -3190 11600 18 PERTH AIRPORT 1944 2004 3563: 5. 9461501 -3198 11607 210 KALAMUNDA (KALAMUNDA 1908 1992 3564: 7. 0 -3200 11580 20 SUBIACO TREATMENT PL 2000 2006 3565: 8. 0 -3196 11579 20 SUBIACO TREATMENT PL 1991 1999 3566: 9. 9460800 -3195 11587 19 PERTH (PERTH REGIONA 1897 1992 3567: 10. 9460801 -3195 11587 19 PERTH-(PERTH-REGIONA 1897 1992 3568: 14. 0 -3220 11580 14 MEDINA RESEARCH CENT 2000 2006 3569: 15. 0 -3220 11580 4 KWINANA BP REFINERY 2000 2006 3570: 16. 0 -3223 11576 4 KWINANA BP REFINERY 1961 2000 3571: 17. 9560800 -3222 11581 14 MEDINA RESEARCH CENT 1991 2000 3572: 18. 0 -3220 11570 6 GARDEN ISLAND HSF 2000 2006 3573: 3574:Enter a matching pair, (bulletin,database) or 'n' to end: 7,14 3575:Bulletin stations: 1 3576: 8. 9256 -3224 11568 6 GARDEN ISLAND HSF 3577:Database stations: 11 3578: 3. 9461000 -3190 11600 20 PERTH AIRPORT COMPAR 1944 2006 3579: 4. 9461001 -3190 11600 18 PERTH AIRPORT 1944 2004 3580: 5. 9461501 -3198 11607 210 KALAMUNDA (KALAMUNDA 1908 1992 3581: 7. 0 -3200 11580 20 SUBIACO TREATMENT PL 2000 2006 3582: 8. 0 -3196 11579 20 SUBIACO TREATMENT PL 1991 1999 3583: 9. 9460800 -3195 11587 19 PERTH (PERTH REGIONA 1897 1992 3584: 10. 9460801 -3195 11587 19 PERTH-(PERTH-REGIONA 1897 1992 3585: 15. 0 -3220 11580 4 KWINANA BP REFINERY 2000 2006 3586: 16. 0 -3223 11576 4 KWINANA BP REFINERY 1961 2000 3587: 17. 9560800 -3222 11581 14 MEDINA RESEARCH CENT 1991 2000 3588: 18. 0 -3220 11570 6 GARDEN ISLAND HSF 2000 2006 3589: 3590:Enter a matching pair, (bulletin,database) or 'n' to end: 8,18 3591:-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 3592: 3593:Amazing, huh? Most are actually more like this: 3594: 3595:-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 3596:User Match Decision(s) Please! 3597:Bulletin stations: 1 3598: 1. 9053 -3167 11602 40 PEARCE RAAF 3599:Database stations: 2 3600: 1. 0 -3170 11600 40 PEARCE RAAF 2000 2006 3601: 2. 9461200 -3167 11602 49 BULLSBROOK (PEARCE A 1940 1992 3602: 3603:Enter a matching pair, (bulletin,database) or 'n' to end: 1,1 3604:-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 3605: 3606:However.. still teething troubles, with previously-paired stations reappearing 3607:for a second chance sometimes! So more debugging.. fixed. Also added a test 3608:before the user gets a chain, to anticipate what the user (er, I) would do. For 3609:instance, I generally match to a 200-2006, WMO=0 database station if the names 3610:match, as they're the ones David L put in from the Aus update files. I, er, the 3611:user then gets ambiguities and nearby but unconnected stations. Fine, until you 3612:get a nasty surprise like this one: 3613: 3614:User Match Decision(s) Please! 3615:Bulletin stations: 2 3616: 1. 58009 -2864 15364 95 BYRON BAY (CAPE BYRO 3617: 2. 58216 -2864 15364 95 BYRON BAY (CAPE BYRO 3618:Database stations: 3 3619: 1. 0 -2860 15360 95 BYRON BAY (CAPE BYRO 2000 2006 3620: 2. 0 -2860 15360 95 BYRON BAY (CAPE BYRO 2000 2006 3621: 3. 9459500 -2863 15363 98 CAPE BYRON 1974 1992 3622: 3623:Looking in the files I see that Bulletin 58009 is 'BYRON BAY (CAPE BYRON LIGHTHOUSE)', 3624:and 58216 is 'BYRON BAY (CAPE BYRON AWS)'. But the database stubs that have been 3625:entered have not been intelligently named, just truncated - so I have no way of 3626:knowing which is which! CRU NEEDS A DATA MANAGER. In this case I had to assume that 3627:the updates were processed in .au code order, so 1-1 and 2-2. Argh. A few doubles 3628:found, too: 3629: 3630:Bulletin stations: 1 3631: 1. 33106 -2037 14895 59 HAMILTON ISLAND AIRP 3632:Database stations: 3 3633: 1. 0 -2040 14900 23 HAMILTON ISLAND AIRP 2000 2006 3634: 2. 0 -2040 14900 59 HAMILTON ISLAND AIRP 2000 2006 3635: 3. 9436800 -2035 14895 23 HAMILTON ISLAND AIRP 1991 2000 3636: 3637:Bulletin stations: 1 3638: 1. 90186 -3829 14245 71 WARRNAMBOOL AIRPORT 3639:Database stations: 4 3640: 1. 0 -3830 14250 71 WARRNAMBOOL AIRPORT 2000 2006 3641: 2. 0 -3830 14240 75 WARRNAMBOOL AIRPORT 2000 2006 3642: 3. 0 -3840 14248 21 WARRNAMBOOL (POST OF 1961 1980 3643: 4. 0 -3828 14243 76 WARRNAMBOOL A 1983 1999 3644: 3645:And the results? Strictly average, I thought.. but I'd forgotten to count the extra 3646:'anticipated match' routine achievements! So I grepped the match-by-match file, 3647:matches.0706281447.dat, and got: 3648: 3649:crua6[/cru/cruts/version_3_0/db/dtr] grep 'AUTO\:' matches.0706281447.dat |wc -l 3650: 232 3651:crua6[/cru/cruts/version_3_0/db/dtr] grep 'AUTO FROM CHAIN' matches.0706281447.dat | wc -l 3652: 514 3653:crua6[/cru/cruts/version_3_0/db/dtr] grep 'MANUAL' matches.0706281447.dat | wc -l 3654: 12 3655: 3656:In other words, all that sweat was worth it - 746 stations matched automatically, and 3657:a further 12 manually! Only (797-758=) 39 bulletins unmatched! Wheeee! And here they are: 3658: 3659: -6072 -2303 11504 111 EMU CREEK STATION AUSTRALIA 2007 2007 -999 6072 3660: -12044 -3355 12070 220 MUNGLINUP WEST AUSTRALIA 2007 2007 -999 12044 3661: -12241 -2888 12132 370 LEONORA AERO AUSTRALIA 2007 2007 -999 12241 3662: -17031 -2965 13806 50 MARREE COMPARISON AUSTRALIA 2007 2007 -999 17031 3663: -21118 -3323 13800 10 PORT PIRIE AERODROME AUSTRALIA 2007 2007 -999 21118 3664: -22801 -3575 13659 143 CAPE BORDA COMPARISO AUSTRALIA 2007 2007 -999 22801 3665: -23122 -3451 13868 65 ROSEWORTHY AWS AUSTRALIA 2007 2007 -999 23122 3666: -24521 -3512 13926 33 MURRAY BRIDGE COMPAR AUSTRALIA 2007 2007 -999 24521 3667: -25509 -3533 14052 99 LAMEROO COMPARISON AUSTRALIA 2007 2007 -999 25509 3668: -26026 -3716 13976 3 ROBE COMPARISON AUSTRALIA 2007 2007 -999 26026 3669: -32004 -1826 14602 5 CARDWELL MARINE PDE AUSTRALIA 2007 2007 -999 32004 3670: -35019 -2282 14764 260 CLERMONT SIRIUS ST AUSTRALIA 2007 2007 -999 35019 3671: -48243 -2943 14797 154 LIGHTNING RIDGE VISI AUSTRALIA 2007 2007 -999 48243 3672: -55024 -3103 15027 307 GUNNEDAH RESOURCE CE AUSTRALIA 2007 2007 -999 55024 3673: -56037 -3053 15167 987 ARMIDALE (TREE GROUP AUSTRALIA 2007 2007 -999 56037 3674: -60013 -3218 15251 4 FORSTER - TUNCURRY R AUSTRALIA 2007 2007 -999 60013 3675: -63039 -3371 15031 1015 KATOOMBA (MURRI ST) AUSTRALIA 2007 2007 -999 63039 3676: -63226 -3348 15013 900 LITHGOW (COOERWULL) AUSTRALIA 2007 2007 -999 63226 3677: -68257 -3406 15077 112 CAMPBELLTOWN (MOUNT AUSTRALIA 2007 2007 -999 68257 3678: -70263 -3475 14970 670 GOULBURN TAFE AUSTRALIA 2007 2007 -999 70263 3679: -82170 -3655 14600 171 BENALLA AIRPORT AUSTRALIA 2007 2007 -999 82170 3680: -84150 -3787 14801 4 LAKES ENTRANCE (EAST AUSTRALIA 2007 2007 -999 84150 3681: -85099 -3863 14581 3 POUND CREEK AUSTRALIA 2007 2007 -999 85099 3682: -88023 -3723 14591 230 LAKE EILDON AUSTRALIA 2007 2007 -999 88023 3683:-200001 -2166 15027 209 MIDDLE PERCY ISLAND AUSTRALIA 2007 2007 -999 200001 3684:-200100 -2066 11558 24 VARANUS ISLAND AUSTRALIA 2007 2007 -999 200100 3685:-200212 -1061 12598 -999 NORTHERN ENDEAVOUR AUSTRALIA 2007 2007 -999 200212 3686:-200283 -1629 14997 8 WILLIS ISLAND AUSTRALIA 2007 2007 -999 200283 3687:-200288 -2904 16794 112 NORFOLK ISLAND AERO AUSTRALIA 2007 2007 -999 200288 3688:-200731 -1176 13003 7 POINT FAWCETT AUSTRALIA 2007 2007 -999 200731 3689:-200783 -1772 14845 3 FLINDERS REEF AUSTRALIA 2007 2007 -999 200783 3690:-200790 -1045 10569 261 CHRISTMAS ISLAND AER AUSTRALIA 2007 2007 -999 200790 3691:-200824 -1753 21040 2 PAPEETE AUSTRALIA 2007 2007 -999 200824 3692:-200838 -3922 14698 116 HOGAN ISLAND AUSTRALIA 2007 2007 -999 200838 3693:-200851 -52 16692 7 NAURU ARCS-2 AUSTRALIA 2007 2007 -999 200851 3694:-200852 -206 14743 4 MANUS ARCS-1 AUSTRALIA 2007 2007 -999 200852 3695:-300000 -6858 7797 18 DAVIS AUSTRALIA 2007 2007 -999 300000 3696:-300001 -6760 6287 10 MAWSON AUSTRALIA 2007 2007 -999 300001 3697:-300017 -6628 11054 40 CASEY AUSTRALIA 2007 2007 -999 300017 3698: 3699:Resultant database: tmn.0707021605.dtb 3700: 3701:[edit: found another fault, had to re-run. Headers weren't being modded if the WMO code was 3702: already there] 3703: 3704:32. The next stage *heart falls* will be to synchronise tmax *against* tmin, sweeping 3705:up duplicates in the process. How long's THIS gonna take? Well actually, it might be fairly easy, 3706:if we use a similar approach. We can base it all around the user being given a 'cloud' of 3707:related stations to pick pairs from, only they will be uniquely numbered so that two from the 3708:same database can be selected. The user can in this way 'pair up' stations in groups. 3709: 3710:Of course, this comes with the downside of complexity (and therefore bugs). And both databases 3711:will almost certainly have to be preloaded in their entirety because of the need for the user to 3712:be able to confirm header and data precedence info when stations within a database are merged. 3713: 3714:Oh - and I'll have to move bloody quick. So more bugs. 3715: 3716:Well.. it's written, and debugging. Around 1500 lines of code, or 1000 without all the comments ;-) 3717:It does indeed read in all the data, so has to be compiled on uealogin1 (as crua6 doesn't have 3718:enough memory!). Reusing code from auminmatch.for did speed things up a bit, though two new 3719:subroutines had to be written to carry out checking for merges (within a database) and for 3720:matches (between the databases). Also introduced a user decision at the start to allow the TMin 3721:database to take precedence in terms of station metadata. Here's the current state of play: 3722: 3723: 3724:uealogin1[/cru/cruts/version_3_0/db/dtr] ./auminmaxsync 3725: 3726:WELCOME TO THE TMIN/TMAX SYNCHRONISER 3727: 3728:Before we get started, an important question: Should TMin header info take precedence over TMax? 3729: 3730:This will significantly reduce user decisions later, but is a big step as TMax settings may be silently overridden! 3731: 3732:To let TMin header values take precedence over those of TMax, enter 'YES': YES 3733:Please enter the tmin database name: tmn.0707021605.dtb 3734:Please enter the tmax database name: tmx.0702091313.dtb 3735: 3736:Reading in both databases.. 3737:TMin database stations: 14349 3738:TMax database stations: 14315 3739: 3740:Processing one-to-one matches.. 3741: 3742:Initial scan found: 3743: one-to-one matches: 7875 3744: of which confirmed: 7691 3745: in a station cloud: 6411 (tmin) 3746: in a station cloud: 6392 (tmax) 3747: unmatchable: 63 (tmin) 3748: unmatchable: 48 (tmax) 3749:Processing match clouds.. 3750:-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 3751:User Match Decision(s) Please! 3752:TMin stations: 2 3753: 1. -401000 3178 3522 783 JERUSALEM 1863 2000 -999 0 3754: 2. 4018400 3178 3522 809 JERUSALEM 1977 1995 -999 0 3755:TMax stations: 2 3756: 3. -401000 3178 3522 783 JERUSALEM 1863 2000 -999 0 3757: 4. 4018400 3178 3522 809 JERUSALEM 1977 1995 -999 0 3758: 3759:*** Remember: Merge first, Match second! *** 3760:Enter ANY pair to match or merge, or 'n' to end: 3761: 3762: 3763:So stats pretty much as expected/hoped. The one-to-one matches should, of course, be 100%.. but as 3764:the databases aren't synchronised, and as there are hundreds of 'duplicate' entries.. only around 3765:50% match straight away. The situation isn't as bleak as it looks, though - there is further 3766:automatching at the beginning of each cloud, so the user can still be spared the obvious. If the 3767:merging gets too onerous, though, I might have to automate that - with associated risks. 3768: 3769:And of course - if you look closely - things are still a little offbeam :-/ 3770: 3771:Found another database bug by chance.. a instead of a space after 'CRANWELL': 3772: 3773:-324320 5303 -50 62 CRANWELL UK 1961 1995 -999 -999.00 3774: 3775:Doesn't show up in reads as it's a white space character. Argh. Fixed in tmin & tmax. Now to find 3776:out why some matched stations STILL don't have the backref in the last header field!! ..found it, 3777:not my problem, it's the ones that *pre-existed* in the databases, there's 84 in total I think. So 3778:I can write a proglet to check that any with negative WMO codes have the positive version in that 3779:last field. And I did - 'fixtnxrefs.for'. Fixed: 3780:tmn.0702091139.dtb (84 fixed) 3781:tmn.0707021605.dtb (651 'fixed' - includes all with negative WMOs regardless of end field) 3782:tmx.0702091313.dtb (84 fixed) 3783: 3784:So why, when we matched 758 bulletins in the first place, did this program only 'fix' 651, of which 3785:84 were preexisting? Because, of course, the matches only get a negative WMO code if the original 3786:WMO code is missing (zero). The 'missing' stations would be ones that already had a WMO code. 3787: 3788:So, try again, and it's looking good! 3789: 3790: 3791:uealogin1[/cru/cruts/version_3_0/db/dtr] ./auminmaxsync 3792: 3793:WELCOME TO THE TMIN/TMAX SYNCHRONISER 3794: 3795:Before we get started, an important question: Should TMin header info take precedence over TMax? 3796: 3797:This will significantly reduce user decisions later, but is a big step as TMax settings may be silently overridden! 3798: 3799:To let TMin header values take precedence over those of TMax, enter 'YES': YES 3800:Please enter the tmin database name: tmn.0702091139.dtb 3801:Please enter the tmax database name: tmx.0702091313.dtb 3802: 3803:Reading in both databases.. 3804:TMin database stations: 14309 3805:TMax database stations: 14315 3806: 3807:Processing one-to-one matches.. 3808: 3809:Initial scan found: 3810: one-to-one matches: 7889 3811: of which confirmed: 7702 3812: in a station cloud: 6365 (tmin) 3813: in a station cloud: 6378 (tmax) 3814: unmatchable: 55 (tmin) 3815: unmatchable: 48 (tmax) 3816:Processing match clouds.. 3817:-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 3818:User Match Decision(s) Please! 3819:TMin stations: 2 3820: 1. -401000 3178 3522 783 JERUSALEM 1863 2000 -999 401000 3821: 2. 4018400 3178 3522 809 JERUSALEM 1977 1995 -999 0 3822:TMax stations: 2 3823: 3. -401000 3178 3522 783 JERUSALEM 1863 2000 -999 401000 3824: 4. 4018400 3178 3522 809 JERUSALEM 1977 1995 -999 0 3825: 3826:*** Remember: Merge first, Match second! *** 3827:Enter ANY pair to match or merge, or 'n' to end: 1,2 3828:Merging two stations from the TMin database: 3829:Stn 1: -401000 3178 3522 783 JERUSALEM ISRAEL 1863 2000 -999 401000 3830:Stn 2: -401000 3178 3522 783 JERUSALEM ISRAEL 1863 2000 -999 401000 3831:Please resolve the following inconsistencies: 3832: Overlap: Station A) -401000 3178 3522 783 JERUSALEM ISRAEL 1863 2000 -999 401000 3833: Station B) 4018400 3178 3522 809 JERUSALEM ISRAEL 1977 1995 -999 -999.00 3834: 3835:You must decide which station's data takes precedence. 3836:The intercorrelation for the period is: 0.99 3837:Enter A or B, or undo pair(X): 3838: 3839: 3840: 3841:Well.. it's kinda working. I found some idiotic bugs, though it is a fearsomely complicated program with 3842:lots of indirect pointers (though I do try and resolve them at the first opportunity). One thing that's 3843:making debugging frustratingly difficult is something that must be a uealogin1 feature, and I haven't seen 3844:it before: the program doesn't actually flush the output channels whenever you write! For example, as I 3845:write this the program has dispensed with auto-matching: 3846: 3847:Initial scan found: 3848: one-to-one matches: 7875 3849: of which confirmed: 7691 3850: in a station cloud: 6411 (tmin) 3851: in a station cloud: 6392 (tmax) 3852: unmatchable: 63 (tmin) 3853: unmatchable: 48 (tmax) 3854: 3855:(yes, it's a little tighter now) 3856: 3857:Anyway, since then I've merged two pair (JERUSALEM) then paired the remainder. That activity has generated 3858:match reports on channel 31 BUT THEY ARE NOT IN THE FILE YET. Here is the tail of channel 31: 3859: 3860:crua6[/cru/cruts/version_3_0/db/dtr] tail mat.0707121500.dat 3861:TMax: 9929470 4330 1340 342 MACERATA ITALY 1953 1975 -999 -999.00 3862:AUTO PAIRING FROM ONE-TO-ONE SCAN: 3863:TMin: 9929480 4030 880 585 MACOMER ITALY 1952 1978 -999 -999.00 3864:TMax: 9929480 4030 880 585 MACOMER ITALY 1952 1978 -999 -999.00 3865:AUTO PAIRING FROM ONE-TO-ONE SCAN: 3866:TMin: 9929500 4010 1850 86 PALASCIA AERO ITALY 1952 1978 -999 -999.00 3867:TMax: 9929500 4010 1850 86 PALASCIA AERO ITALY 1952 1978 -999 -999.00 3868:AUTO PAIRING FROM ONE-TO-ONE SCAN: 3869:TMin: 9929520 4060 1490 30 PONTECAGNANO ITALY 1951 1978 -999 -999.00 3870:TMax: 9929520 4060 1490 30 PONTECAGNANO ITALY 1951 1978 -999 -999.00 3871: 3872:In addition, the log file is EMPTY, yet at least 416 bytes have been written to it. How the hell can I 3873:debug if I can't monitor what's being written to the log files?!! Of course, once I force-quit the program, 3874:and wait a bit.. the missing info appears. Similarly if I carry on using the program, the files get more 3875:info. It's as if there's a write buffer that runs FIFO. Must look at the 'help'.. why is it that whenever I 3876:crack the programming, the systems themselves step in the screw it up? And computer support is away of course. 3877: 3878:Looked at f77 -help.. nothing. well nothing obvious. Anyway, more debugging and.. 3879: 3880:Seems to be working. But it's going to take ages. Here is an example of the problem: 3881: 3882: 3883:-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 3884:User Match Decision(s) Please! 3885:TMin stations: 2 3886: 1. -315770 5638 -287 10 LEUCHARS UK 1959 1995 -999 315770 3887: 2. 317100 5640 -287 12 LEUCHARS UNITED KINGDO 1997 2006 -999 0 3888:TMax stations: 2 3889: 3. -315770 5638 -287 10 LEUCHARS UK 1959 1995 -999 315770 3890: 4. 317100 5638 -287 12 LEUCHARS RAF UK UK 1973 2006 -999 0 3891: 3892:*** Remember: Merge first, Match second! *** 3893:Enter ANY pair to match or merge, or 'n' to end: 3894: 3895: 3896:Not only do both databases have unnecessary duplicates, introduced for external mapping purposes 3897:by the look of it, but the 'main' stations (2 and 4) have different station name & country. In fact 3898:one of the country names is illegal! Dealing with things like this cannot be automated as they're 3899:the results of non-automatic decisions. 3900: 3901:Something new - a listing of 147 Australian 'bulletin' stations, most of which have mappings to 3902:WMO codes. Decided to xref against the (mapped) TMin database, for a laugh. Then decided to take it 3903:more seriously. Wrote a prog to IMPOSE the mappings onto tmn.0707021605.dtb, overriding existing 3904:mappings as necessary. What a bloody mess. 3905: 3906:Decided to be vaguely sensible and let the program, auwmoxref.for, evolve. so to begin with it just 3907:did a scan between the mappings file (au_mapping_to_wmo.dat) and the tmin database with my mappings 3908:in (tmn.0707021605.dtb). Results: 3909: 3910:crua6[/cru/cruts/version_3_0/db/dtr] ./auwmoxref 3911: 3912: 3913:AUWMOXREF: Check Australian cross-references 3914: 3915: Enter the file of WMO mappings: au_mapping_to_wmo.dat 3916: 115 mappings read 3917: 3918: Enter the mapped TMin database: tmn.0707021605.dtb 3919: 14349 database headers read 3920: 3921: 3922:RESULTS: 3923: 3924:WMO Matches: 92 3925: (multiples) ( 0) 3926:> Ref matches: 60 3927:> Ref empty: 31 3928:> Ref WRONG: 1 3929: 3930:Ref Matches: 114 3931: (multiples) ( 0) 3932:> WMO matches: 60 3933:> WMO -1*Ref: 41 3934:> WMO WRONG: 13 3935: 3936: 3937:So first the good news - no duplicates. Well there shouldn't have been any anyway of course, but the 3938:way things are going I'm taking nothing for granted. See, I count something turning out as expected 3939:as 'good news'. So anyway.. I also extracted the statistic that 26 mappings matched both Ref and WMO, 3940:but to separate database entries. Thus the 115 mappings are allocated as follows: 3941: 3942:60 Mapping found to be correctly implemented (over half, excellent) 3943:41 WMO Missing, of which: 3944: 26 WMO found elsewhere (one of which has an unmapped ref attached to it) 3945: 15 WMO not in database (can add wmo codes for these) 3946:13 WMO wrong, of which: 3947: 5 Can be merged with real WMO (effectively same station) 3948: 8 WMO not in database 3949: 1 Completely unmatched (96003 -> 949500) 3950: 3951:For the purposes of actions to take, the 13 'WMO Wrong' refs can simply be unmapped from their incorrect 3952:mappings and be rolled into the 41 'WMO Missing' refs. 3953: 3954:So: 3955: 3956:60 Mapping found to be correctly implemented (over half, excellent) 3957:54 WMO Missing or wrong, of which: 3958: 31 WMO found elsewhere (one of which has an unmapped ref attached to it) 3959: 23 WMO not in database but pairing made (can add wmo codes for these) 3960: 8 WMO not in database and no pairing (can add new stations for these) 3961: 1 Completely unmatched (96003 -> 949500) 3962: 3963:So, actions to take: 3964: 3965:1. For the first 60, no action required. 3966:2. For the 13 with incorrectly-assigned WMOs, disengage and roll into the rest below 3967:3. For the 1 WMO with an unmapped ref attached, disengage and roll into the rest below 3968:3. For the 31 with dislocated WMOs, print a list and ref when doing the tmin/tmax syncing 3969:4. For the 23 with WMO-less stations, add the WMO codes.. 3970:5. For the 8 with no WMO found and no pairing found, create new stations. 3971: 3972:For the disengagements, decided to work directly with an editor rather than craft another program! So 3973:changes made to tmn.0707021605.dtb (after a suitable backup was made of course!). 3974: 3975:The following assignments were disengaged (and replaced with -999.00). Where a WMO code follows in 3976:brackets, the ref was reassigned there. 3977: 3978: 1. 9460300 -3200 11550 43 ROTTNEST ISLAND AUSTRALIA 1898 2006 -999 9193 (9460200) 3979: 2. 9464600 -3090 12810 159 FORREST AUSTRALIA 1946 2006 -999 11052 (no) 3980: 3. 9432200 -2020 13000 340 RABBIT FLAT AUSTRALIA 1969 2006 -999 15666 (no) 3981: 4. 9557400 -2640 15300 6 TEWANTIN RSL PARK AUSTRALIA 1949 2006 -999 40908 (no) 3982: 5. 9451600 -2810 14860 199 ST GEORGE AIRPORT AUSTRALIA 1938 2006 -999 43109 (9451700) 3983: 6. 9452700 -2950 14990 213 MOREE AERO AUSTRALIA 1964 2006 -999 53115 (9552700) 3984: 7. 9454100 -2980 15110 582 INVERELL (RAGLAN ST) AUSTRALIA 1907 2006 -999 56242 (no) 3985: 8. 9478700 -3140 15290 4 PORT MACQUARIE AIRPO AUSTRALIA 1907 2006 -999 60139 (no) 3986: 9. 9475800 -3210 15090 216 SCONE SCS AUSTRALIA 2000 2006 -999 61089 (9473800) 3987: 10. 9494000 -3510 15080 85 JERVIS BAY (POINT PE AUSTRALIA 1907 2006 -999 68151 (no) 3988: 11. 9491600 -3590 14840 1482 CABRAMURRA SMHEA AWS AUSTRALIA 1962 2006 -999 72161 (no) 3989: 12. 9482700 -3630 14160 133 NHILL AUSTRALIA 1897 2006 -999 78031 (9582900) 3990: 13. 9597900 -4300 14710 63 GROVE (COMPARISON) AUSTRALIA 1961 2006 -999 94069 (no) 3991: 3992:The 'mismatched WMO code' station was disengaged from it's reference and given 48027 instead: 3993: 1. 9471100 -3150 14580 218 COBAR AIRPORT AWS AUSTRALIA 1962 2006 -999 48237 -> 48027 3994: 3995:I mailed BOM as we have 94711 = COBAR AWS but they have *94710* for AWS and 94711 for COBAR MO. The 3996:reply was as follows: 3997: 3998: 3999:On 18 Jul 2007, at 8:51, Matthew Bastin wrote: 4000: 4001:Hi Ian, 4002: 4003:I hope this table helps 4004: 4005:Name BoM No. WMO No. Opened Closed 4006:Cobar Comparison 48244 94711 1/11/1997 15/11/2000 4007:Cobar MO 48027 94711 1/01/1962 4008:Cobar Airport AWS 48237 94710 11/06/1993 4009:Cobar PO 48030 1/1/1881 31/12/1965 4010: 4011:The blank in the Closed column means that the site is still open 4012:When Cobar Comparison site closed it transferred its WMO number to Cobar MO 4013:A blank in the WMO No. column means that the site never had a WMO number. 4014: 4015:I am not sure of the overlap between the assignment of 94711 between 48244 and 48027. I will find 4016:out and get back to you. 4017: 4018: 4019:Here are our current 'COBAR' headers: 4020: 4021: 0 -3150 14580 260 COBAR COMPARISON AUSTRALIA 2000 2006 -999 -999.00 4022: 0 -3150 14580 260 COBAR MO AUSTRALIA 2000 2006 -999 -999.00 4023: 0 -3148 14582 265 COBAR AUSTRALIA 1962 2004 -999 -999.00 4024: 0 -3150 14580 251 COBAR POST OFFICE AUSTRALIA 1902 1960 -999 -999.00 4025:9471100 -3150 14580 218 COBAR AIRPORT AWS AUSTRALIA 1962 2006 -999 48027 4026: 4027:Now looking at the dates.. something bad has happened, hasn't it. COBAR AIRPORT AWS cannot start 4028:in 1962, it didn't open until 1993! Looking at the data - the COBAR station 1962-2004 seems to be 4029:an exact copy of the COBAR AIRPORT AWS station 1962-2004, except that the latter has more missing 4030:values. Now, COBAR AIRPORT AWS has 15 months of missing value codes beginning Oct 1993.. coincidence? 4031:No. I think that that series should start there. Furthermore, the overlap between COBAR and COBAR MO 4032:(2000-2004) is *almost* identical: 4033: 4034: 0 -3148 14582 265 COBAR AUSTRALIA 1962 2004 -999 -999.00 4035:2000 177 209 183 135 80 51 45 52 105 122 166 186 4036:2001 223 214 159 126 72 61 43 52 105 110 148 181 4037:2002 195 185 168 148 88 58 49 63 101 128 186 192 4038:2003 222 216 161 137 97 71 56 61 92 113 159 208 4039:2004 207 226 175 141 74 69 46 69 90 136 160 186 4040: 4041: 4042: 0 -3150 14580 260 COBAR MO AUSTRALIA 2000 2006 -999 -999.00 4043:2000 178 209 184 136 80 52 45 55 105 122 166 186 (7/12) 4044:2001 223 214 159 126 72 61 43 52 105 110 148 181 (12/12) 4045:2002 195 185 168 148 88 58 49 63 101 128 187 192 (11/12) 4046:2003 222 216 161 137 97 71 56 61 92 113 159 208 (12/12) 4047:2004 207 226 175 141 74 69 46 69 90 136 160 186 (12/12) 4048: 4049:I therefore propose to extend COBAR MO using COBAR, and to truncate COBAR AIRPORT AWS at 1993. 4050:All BOM codes will be appended for completeness. So the new headers (with lat/lon from BOM too) are: 4051: 4052: 0 -3149 14583 260 COBAR COMPARISON AUSTRALIA 2000 2006 -999 48244 (closed) 4053:9471100 -3149 14583 260 COBAR MO AUSTRALIA 1962 2006 -999 48027 4054: 0 -3150 14583 251 COBAR POST OFFICE AUSTRALIA 1902 1960 -999 48030 (closed) 4055:9471000 -3154 14580 218 COBAR AIRPORT AWS AUSTRALIA 1995 2006 -999 48237 4056: 4057:Deleted: 4058: 0 -3148 14582 265 COBAR AUSTRALIA 1962 2004 -999 -999.00 4059: 4060:The remaining 26 dislocated references were reassigned as for the 13 above. Legitimate mappings: 4061: 4062: 1. 3003 9420300 4063: 2. 4032 9431200 4064: 3. 5007 9430200 4065: 4. 7176 9431700 4066: 5. 9021 9461000 4067: 6. 14508 9415000 4068: 7. 14932 9413100 4069: 8. 17031 9448000 4070: 9. 22801 9480500 4071: 10. 26026 9481200 4072: 11. 27045 9417000 4073: 12. 32040 9429400 4074: 13. 40842 9457800 4075: 14. 50052 9470700 4076: 15. 55024 9474000 4077: 16. 67105 9575300 4078: 17. 68072 9475000 4079: 18. 71041 9590800 4080: 19. 86282 9486600 4081: 20. 200283 9429900 4082: 21. 200288 9499600 4083: 22. 200790 9699500 4084: 23. 200839 9499500 4085: 24. 300000 8957100 4086: 25. 300001 8956400 4087: 26. 300017 8961100 4088: 4089:WMO codes were added to these uncoded sites as shown: 4090: 4091: 1. 9410000 -1430 12670 23 KALUMBURU AUSTRALIA 2000 2006 -999 1019 4092: 2. 9562500 -3160 11720 217 CUNDERDIN AIRFIELD AUSTRALIA 2000 2006 -999 10286 4093: 3. 9564000 -3270 11670 275 WANDERING AUSTRALIA 2000 2006 -999 10917 4094: 4. 9567000 -3380 13820 109 SNOWTOWN (RAYVILLE P AUSTRALIA 2000 2006 -999 21133 4095: 5. 9481400 -3530 13890 58 STRATHALBYN RACECOUR AUSTRALIA 2000 2006 -999 24580 4096: 6. 9548200 -2590 13940 47 BIRDSVILLE AIRPORT AUSTRALIA 2000 2006 -999 38026 4097: 7. 9552900 -2670 15020 305 MILES CONSTANCE STRE AUSTRALIA 2000 2006 -999 42112 4098: 8. 9549200 -2800 14380 132 THARGOMINDAH AIRPORT AUSTRALIA 2000 2006 -999 45025 4099: 9. 9578400 -3190 15250 8 TAREE AIRPORT AWS AUSTRALIA 2000 2006 -999 60141 4100: 10. 9571900 -3220 14860 284 DUBBO AIRPORT AWS AUSTRALIA 2000 2006 -999 65070 4101: 11. 9586900 -3560 14500 94 DENILIQUIN AIRPORT A AUSTRALIA 2000 2006 -999 74258 4102: 12. 9495400 -4070 14470 94 CAPE GRIM BAPS AUSTRALIA 2000 2006 -999 91245 4103: 13. 9596400 -4110 14680 3 LOW HEAD AUSTRALIA 2000 2006 -999 91293 4104: 14. 9595900 -4190 14670 1055 LIAWENEE AUSTRALIA 2000 2006 -999 96033 4105: 4106:The following was corrected (ref had been mistyped as 78013): 4107: 1. 9582900 -3783 14206 200 HAMILTON RESEARCH ST AUSTRALIA 1971 1998 -999 78031 4108: 4109:Now the results look like this: 4110: 4111:WMO Matches: 106 4112:> Ref matches: 106 4113:> Ref empty: 0 4114:> Ref WRONG: 0 4115:Ref Matches: 106 4116:> WMO matches: 106 4117:> WMO -1*Ref: 0 4118:> WMO WRONG: 0 4119: 4120:In other words, there are (115-106=) 9 mappings unfulfilled. The ref hasn't been matched and 4121:WMO code isn't in the database. However, that didn't mean they weren't in the database with a 4122:missing WMO code, did it? The following were found and augmented with both WMO code and ref. 4123: 4124:9457000 -2639 15304 6 TEWANTIN RSL PARK AUSTRALIA 2000 2004 -999 40908 4125:9594000 -3509 15080 85 JERVIS BAY (PT PERP AWS) AUSTRALIA 2000 2006 -999 68151 4126: 4127:The following were added as new station stubs: 4128:9532200 -2018 13001 340 RABBIT FLAT AUSTRALIA 2007 2007 -999 15666 4129:9554100 -2978 15111 582 INVERELL (RAGLAN ST) AUSTRALIA 2007 2007 -999 56242 4130:9478600 -3143 15287 4 PORT MACQUARIE AIRPT AUSTRALIA 2007 2007 -999 60139 4131:9591600 -3594 14838 1482 CABRAMURRA SMHEA AWS AUSTRALIA 2007 2007 -999 72161 4132:9597100 -4298 14708 63 GROVE (COMPARISON) AUSTRALIA 2007 2007 -999 94069 4133: 4134:The following was complicated by the fact that two versions of the station appear to have been 4135:concatenated. This is the station as it already exists in the TMin database: 4136:9464600 -3085 12811 159 FORREST AUSTRALIA 1946 2006 -999 -999.00 4137:However, the current 'live' FORREST station (11052) started in 1993, according to bom.au 4138:records. And wouldn't you know it, the data for this station has missing data between 12/92 4139:and 12/99 inclusive. So I reckon it's the old FORREST AERO station (WMO 9464600, .au ID 11004), 4140:with the new Australian bulletin updates tacked on (hence starting in 2000). Especially as the 4141:old station started in 1946 (http://www.bom.gov.au/climate/averages/tables/cw_011004.shtml). 4142:The trouble is that the bom.au mappings all agree that FORREST is now WMO=9564600. So.. do I 4143:split off the 2000-present data to a new station with the new number, or accept that whoever 4144:joined them (Dave?) looked into it and decided it would be OK? The BOM website says they're 4145:800m apart. Decided to be brave and split the data back into two stations, with both codes 4146:attached (in case we ever get replacement data for the closed station, the site says it went to 4147:1995 after all). So there are now two FORREST stations: 4148: 4149:9464600 -3085 12811 159 FORREST AERO AUSTRALIA 1946 1992 -999 11004 4150:9564600 -3085 12811 159 FORREST AUSTRALIA 2000 2006 -999 11052 4151: 4152:Hope that's right.. 4153: 4154:The following mapping was added, though the station does not currently feature in the bulletins. 4155:9495900 -4228 14628 -999 BUTLERS GORGE AUSTRALIA 2007 2007 -999 96003 4156:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 4157:2007-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 4158: 4159:Also ran a risky search&replace to left-justify the 'AUSTRALIA' in its field, provided the 4160:field wasn't touched by an extended station name. Seems to have been 100% successful. 4161: 4162:All 115 refs now matched in the TMin database. Confidence in the fidelity of the Australian 4163:station in the database drastically reduced. Likelihood of invalid merging of Australian 4164:stations high. Let's go.. 4165: 4166:Well OK, made some final 'improvements' to the syncing program. Now, after it forms a cloud, it 4167:should automatically merge stations provided the criteria are met and no others are possibles. 4168:It also records, in a separate 'action' file (act.*), every relevant action performed during 4169:the run, so that if interrupted I should be able to hack in something to enable a 'resume'. It's 4170:been done a bit hastily so no guarantees that enough information's been saved! 4171: 4172:Debugging is still a big issue, unfortunately. It's a complicated program to sort out and the 4173:possibilities for indexing errors are many. In fact, for the first time ever, it's just locked up! 4174:That's a first (it was due to getmos not defaulting to months 1 & 12 if the data was all missing). 4175: 4176:Another problem solved - spent ages wondering how the start & end years for a particular station 4177:(WARATAH) were being corrupted. Turns out they weren't - I'd written 'getmos' to trim empty years, 4178:but forgot to check the return flag! Duh. 4179: 4180:So.. perhaps a debugged run through? I'm quickly realising that the Australian stations are in 4181:such a state that I'm having to constantly refer to the station descriptions on the BOM website, 4182:which are individual PDFs: 4183: 4184:http://www.bom.gov.au/climate/cdo/metadata/pdf/metadata088110.pdf 4185: 4186:It takes time.. time I don't have! Though I'm pleased to see that the second FSM is helpfully 4187:chipping in to pair things up when possible. 4188: 4189:getting seriously fed up with the state of the Australian data. so many new stations have been 4190:introduced, so many false references.. so many changes that aren't documented. Every time a 4191:cloud forms I'm presented with a bewildering selection of similar-sounding sites, some with 4192:references, some with WMO codes, and some with both. And if I look up the station metadata with 4193:one of the local references, chances are the WMO code will be wrong (another station will have 4194:it) and the lat/lon will be wrong too. I've been at it for well over an hour, and I've reached 4195:the 294th station in the tmin database. Out of over 14,000. Now even accepting that it will get 4196:easier (as clouds can only be formed of what's ahead of you), it is still very daunting. I go 4197:on leave for 10 days after tomorrow, and if I leave it running it isn't likely to be there when 4198:I return! As to whether my 'action dump' will work (to save repetition).. who knows? 4199: 4200:Yay! Two-and-a-half hours into the exercise and I'm in Argentina! 4201: 4202:Pfft.. and back to Australia almost immediately :-( .. and then Chile. Getting there. 4203: 4204:Unfortunately, after around 160 minutes of uninterrupted decision making, my screen has started 4205:to black out for half a second at a time. More video cable problems - but why now?!! The count is 4206:up to 1007 though. 4207: 4208:I am very sorry to report that the rest of the databases seem to be in nearly as poor a state as 4209:Australia was. There are hundreds if not thousands of pairs of dummy stations, one with no WMO 4210:and one with, usually overlapping and with the same station name and very similar coordinates. I 4211:know it could be old and new stations, but why such large overlaps if that's the case? Aarrggghhh! 4212:There truly is no end in sight. Look at this: 4213: 4214: 4215:-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 4216:User Match Decision(s) Please! 4217:TMin stations: 4 4218: 1. 0 153 12492 80 MENADO/DR. SA INDONESIA 1960 1975 -999 0 4219: 2. 0 153 12492 80 MENADO/ SAM RATULANG INDONESIA 1986 2004 -999 0 4220: 4. 9701400 153 12492 80 MENADO/DR. SAM RATUL INDONESIA 1995 2006 -999 0 4221: 5. 9997418 153 12492 81 SAMRATULANGI MENADO INDONESIA 1973 1989 -999 0 4222:TMax stations: 4 4223: 6. 0 153 12492 80 MAPANGET/MANADO INDONESIA 1960 1975 -999 0 4224: 7. 0 153 12492 80 MENADO/ SAM RATULANG ID ID 1957 2004 -999 0 4225: 9. 9701400 153 12492 80 MENADO/DR. SAM RATUL INDONESIA 1995 2006 -999 0 4226: 10. 9997418 153 12492 81 SAMRATULANGI MENADO INDONESIA 1972 1989 -999 0 4227: 4228:*** Remember: Merge first, then Match *** 4229:Enter ANY pair to match or merge, 'a' to auto-match (no merges), or 'x' to end: 4230: 4231:I honestly have no idea what to do here. and there are countless others of equal bafflingness. 4232: 4233:I'll have to go home soon, leaving it running and hoping none of the systems die overnight :-((( 4234: 4235:.. it survived, thank $deity. And a long run of duplicate stations, each requiring multiple 4236:decisions concerning spatial info, exact names, and data precedence for overlaps. If for any reason 4237:this has to be re-run, it can certainly be speeded up! Some large clouds, too - this one started 4238:with 59 members from each database: 4239: 4240:-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 4241:User Match Decision(s) Please! 4242:TMin stations: 7 4243: 11. 7101965 4362 -7940 78 TORONTO ISLAND 1905 1959 -999 0 4244: 14. 7163427 4363 -7940 77 TORONTO ISLAND A CANADA 1957 1994 -999 0 4245: 23. 7101987 4380 -7955 194 TORONTO MET RES STN 1965 1988 -999 0 4246: 24. 7163434 4380 -7955 194 TORONTO MET RES STN CANADA 1965 1988 -999 0 4247: 36. 0 4388 -7944 233 RICHMOND HILL 1959 2003 -999 0 4248: 39. 7163408 4388 -7945 233 RICHMOND HILL CANADA 1959 1990 -999 0 4249: 40. 7163409 4387 -7943 218 RICHMOND HILL WPCP 1960 1981 -999 0 4250:TMax stations: 8 4251: 70. 7101965 4362 -7940 78 TORONTO ISLAND 1905 1959 -999 0 4252: 71. 7126500 4363 -7940 77 TORONTO ISLAND A 1957 1994 -999 0 4253: 73. 7163427 4363 -7940 77 TORONTO ISLAND A CANADA 1957 1990 -999 0 4254: 82. 7101987 4380 -7955 194 TORONTO MET RES STN 1965 1988 -999 0 4255: 83. 7163434 4380 -7955 194 TORONTO MET RES STN CANADA 1965 1988 -999 0 4256: 95. 0 4388 -7944 233 RICHMOND HILL 1959 2003 -999 0 4257: 98. 7163408 4388 -7945 233 RICHMOND HILL CANADA 1959 1990 -999 0 4258: 99. 7163409 4387 -7943 218 RICHMOND HILL WPCP 1960 1981 -999 0 4259: 4260:There were even larger clouds later. 4261: 4262:One thing that's unsettling is that many of the assigned WMo codes for Canadian stations do 4263:not return any hits with a web search. Usually the country's met office, or at least the 4264:Weather Underground, show up - but for these stations, nothing at all. Makes me wonder if 4265:these are long-discontinued, or were even invented somewhere other than Canada! Examples: 4266: 4267:7162040 brockville 4268:7163231 brockville 4269:7163229 brockville 4270:7187742 forestburg 4271:7100165 forestburg 4272: 4273:Here's a heartwarming example of a cloud which self-paired completely (debug ines included): 4274: 4275: 4276:DBG: cloud formed with ( 6, 6) members 4277:DBG: automerging done, leaving ( 6, 6) 4278:DBG: pot.auto i,j: 1 1 4279:DBG: i,ncs2m,cs2m(1-5): 1 1 1 8578 8582 8596 0 4280:DBG: paired: 1 1 108 MILE HOUSE ABEL 4281: 4282:Attempting to pair stations: 4283:From TMin: 0 5170 -12140 994 108 MILE HOUSE ABEL 1987 2002 -999 -999.00 4284:From TMax: 0 5170 -12140 994 108 MILE HOUSE ABEL 1987 2002 -999 -999.00 4285:DBG: AUTOPAIRED: 1 1 4286:DBG: pot.auto i,j: 2 2 4287:DBG: i,ncs2m,cs2m(1-5): 2 1 2 8578 8582 8596 0 4288:DBG: paired: 2 2 100 MILE HOUSE 4289: 4290:Attempting to pair stations: 4291:From TMin: 7194273 5165 -12130 1059 100 MILE HOUSE CANADA 1970 1999 -999 -999.00 4292:From TMax: 7194273 5165 -12130 1059 100 MILE HOUSE CANADA 1970 1999 -999 -999.00 4293:DBG: AUTOPAIRED: 2 2 4294:DBG: pot.auto i,j: 3 3 4295:DBG: i,ncs2m,cs2m(1-5): 3 1 3 8578 8582 8596 0 4296:DBG: paired: 3 3 HORSE LAKE 4297: 4298:Attempting to pair stations: 4299:From TMin: 7103611 5160 -12120 994 HORSE LAKE 1983 1994 -999 -999.00 4300:From TMax: 7103611 5160 -12120 994 HORSE LAKE 1983 1994 -999 -999.00 4301:DBG: AUTOPAIRED: 3 3 4302:DBG: pot.auto i,j: 4 4 4303:DBG: i,ncs2m,cs2m(1-5): 4 1 4 8578 8582 8596 0 4304:DBG: paired: 4 4 LONE BUTTE 2 4305: 4306:Attempting to pair stations: 4307:From TMin: 7103629 5155 -12120 1145 LONE BUTTE 2 1981 1991 -999 -999.00 4308:From TMax: 7103629 5155 -12120 1145 LONE BUTTE 2 1981 1991 -999 -999.00 4309:DBG: AUTOPAIRED: 4 4 4310:DBG: pot.auto i,j: 5 5 4311:DBG: i,ncs2m,cs2m(1-5): 5 1 5 8578 8582 8596 0 4312:DBG: paired: 5 5 100 MILE HOUSE 6NE 4313: 4314:Attempting to pair stations: 4315:From TMin: 7103637 5168 -12122 928 100 MILE HOUSE 6NE 1987 2002 -999 -999.00 4316:From TMax: 7103637 5168 -12122 928 100 MILE HOUSE 6NE 1987 2002 -999 -999.00 4317:DBG: AUTOPAIRED: 5 5 4318:DBG: pot.auto i,j: 6 6 4319:DBG: i,ncs2m,cs2m(1-5): 6 1 6 8578 8582 8596 0 4320:DBG: paired: 6 6 WATCH LAKE NORTH 4321: 4322:Attempting to pair stations: 4323:From TMin: 7103660 5147 -12112 1069 WATCH LAKE NORTH 1987 1996 -999 -999.00 4324:From TMax: 7103660 5147 -12112 1069 WATCH LAKE NORTH 1987 1996 -999 -999.00 4325:DBG: AUTOPAIRED: 6 6 4326: 4327: 4328:Now arguably, the MILE HOUSE ABEL stations should have rolled into one of the other MILE HOUSE ones with 4329:a WMO code.. but the lat/lon/alt aren't close enough. Which is as intended. 4330: 4331:* 4332: 4333:* 4334: 4335:Well, it *kind of* worked. Thought the resultant files aren't exactly what I'd expected: 4336: 4337:-rw------- 1 f098 cru 12715138 Jul 25 15:25 act.0707241721.dat 4338:-rw------- 1 f098 cru 435839 Jul 25 15:25 log.0707241721.dat 4339:-rw------- 1 f098 cru 4126850 Jul 25 15:25 mat.0707241721.dat 4340:-rw------- 1 f098 cru 6221390 Jul 25 15:25 tmn.0707021605.dtb.lost 4341:-rw------- 1 f098 cru 2962918 Jul 25 15:25 tmn.0707241721.dat 4342:-rw------- 1 f098 cru 0 Jul 25 15:25 tmx.0702091313.dtb.lost 4343:-rw------- 1 f098 cru 2962918 Jul 25 15:25 tmx.0707241721.dat 4344: 4345: 4346:act.0707241721.dat: hopefully-complete record of all activities 4347: 4348:log.0707241721.dat: hopefully-useful log of odd happenings (and mergeinfo() trails) 4349: 4350:mat.0707241721.dat: hopefully-complete list of all merges and pairings 4351: 4352:tmn.0707021605.dtb.lost: too-small collection of unpaired stations 4353: 4354:tmn.0707241721.dat: too-small output database 4355: 4356:tmx.0702091313.dtb.lost: MUCH too-small collection of unpaired stations!!! 4357: 4358:tmx.0707241721.dat: too-small (but hey, the same size as the twin) output database 4359: 4360:ANALYSIS 4361: 4362:Well, LOL, the reason the output databases are so small is that every station looks like this: 4363: 4364:9999810 -748 10932 114 SEMPOR INDONESIA 1971 2000 -999 -999.00 4365:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 4366:1971 229 225 225 229 229-9999 223 221 222 225 224-9999 4367: 4368:Yes - just one line of data. The write loops went from start year to start year. Ho hum :-/ 4369: 4370:Not as easy to fix as you might think, seeing as the data may well be the result of a merge and 4371:so can't just be pasted in from the source database. 4372: 4373:As for the 'unbalanced' 'lost' files: well for a start, the same error as above (just one line of data), 4374:then on top of that, both sets written to the same file. what time did I write that bit, 3am?!! Ecch. 4375: 4376:33. So, as expected.. I'm gonna have to write in clauses to make use of the log, act and mat files. I so do 4377:not want to do this.. but not as much as I don't want to do a day's interacting again!! 4378: 4379:Got it to work.. sort of. Turns out I had included enough information in the ACT file, and so was able to 4380:write auminmaxresync.for. A few teething troubles, but two new databases ('tm[n|x].0707301343.dtb') 4381:created with 13654 stations in each. And yes - the headers are identical :-) 4382: 4383:[edit: see below - the 'final' databases are tm*.0708071548.dtb] 4384: 4385:Here are the header counts, demonstrating that something's still not quite right.. 4386: 4387:Original: 4388: 14355 tmn.0707021605.dtb.heads 4389: 4390:New: 4391: 13654 tmn.0707301343.dtb.heads 4392: 4393:Lost/merged: 4394: 14318 tmn.0707021605.dtb.lost.heads (should be 14355-13654-37 = 664?) 4395: 37 tmn.0707021605.dtb.merg.heads (seems low) 4396: 4397:Original: 4398: 14315 tmx.0702091313.dtb.heads 4399: 4400:New: 4401: 13654 tmx.0707301343.dtb.heads 4402: 4403:Lost/merged: 4404: 14269 tmx.0702091313.dtb.lost.heads (should be 14315-13654-46 = 615?) 4405: 46 tmx.0702091313.dtb.merg.heads (seems low) 4406: 4407:In fact, looking at the original ACT file that we used: 4408: 4409:crua6[/cru/cruts/version_3_0/db/dtr] grep 'usermerg' act.0707241721.dat | wc -l 4410: 258 4411:crua6[/cru/cruts/version_3_0/db/dtr] grep 'automerg' act.0707241721.dat | wc -l 4412: 889 4413: 4414:..so will have to look at how the db1/2xref arrays are prepped and set in the program. Nonetheless the 4415:construction of the new databases looks pretty good. There's aminor problem where the external reference 4416:field is sometimes -999.00 and sometimes 0. Not sure which is best, probably 0, as the field will usually 4417:be used for reference numbers/characters rather than real data values. Used an inline perl command to fix. 4418: 4419:..after some rudimentary corrections: 4420: 4421:uealogin1[/cru/cruts/version_3_0/db/dtr] wc -l *.heads 4422: 14355 tmn.0707021605.dtb.heads 4423: 122 tmn.0707021605.dtb.lost.heads 4424: 579 tmn.0707021605.dtb.merg.heads 4425: 13654 tmn.0708062250.dtb.heads 4426: 14315 tmx.0702091313.dtb.heads 4427: 93 tmx.0702091313.dtb.lost.heads 4428: 570 tmx.0702091313.dtb.merg.heads 4429: 13654 tmx.0708062250.dtb.heads 4430: 4431:Almost perfect! But unfortunately, there is a slight discrepancy, and they have a habit of being tips of 4432:icebergs. If you add up the header/station counts of the new tmin database, merg and lost files, you get 4433:13654 + 579 + 122 = 14355, the original station count. If you try the same check for tmax, however, you get 4434:13654 + 570 + 93 = 14317, two more than the original count! I suspected a couple of stations were being 4435:counted twice, so using 'comm' I looked for identical headers. Unfortunately there weren't any!! So I have 4436:invented two stations, hmm. Got the program to investigate, and found two stations in the cross-reference 4437:array which had cross refs *and* merge flags: 4438: 4439:ERROR: db2xref( 126) = 127 -14010 : 4440: 126> 9596400 -4110 14680 3 LOW HEAD AUSTRALIA 2000 2006 -999 91293 4441:14010> 9596900 -4170 14710 150 CRESSY RESEARCH STAT AUSTRALIA 1971 2006 -999 91306 4442: 4443:and 4444: 4445:ERROR: db2xref(13948) = 227 -226 : 4446:13948> 9570600 -3470 14650 145 NARRANDERA AIRPORT AUSTRALIA 1971 2006 -999 0 4447: 226> 0 -3570 14560 110 FINLEY (CSIRO) AUSTRALIA 2000 2001 -999 0 4448: 4449:So in the first case, LOW HEAD has been merged with another station (#14010) AND paired with #127. 4450:Similarly, NARRANDERA AIRPORT has been mreged with #226 and paired with #227. However, these apparent 4451:merges are false! As we see in the first case, 14010 is not LOW HEAD. Similarly for the second case. 4452: 4453:Looking in the relevant match file from the process (mat.0707241721.dat) we find: 4454: 4455:AUTO MERGE FROM CHAIN: 4456:TMax Stn 1: 0 -4110 14680 3 LOW HEAD AUSTRALIA 2000 2006 -999 -999.00 4457:TMax Stn 2: 0 -4105 14678 4 LOW HEAD AUSTRALIA 2000 2004 -999 -999.00 4458:New Header: 0 -4110 14680 3 LOW HEAD AUSTRALIA 2000 2006 -999 0 4459:Note: Stn 1 data overwrote Stn 2 data 4460: 4461:MANUAL PAIRING FROM CHAIN: 4462:TMin: 9596400 -4110 14680 3 LOW HEAD AUSTRALIA 2000 2006 -999 91293 4463:TMax: 0 -4110 14680 3 LOW HEAD AUSTRALIA 2000 2006 -999 0 4464:New Header: 9596400 -4110 14680 3 LOW HEAD AUSTRALIA 2000 2006 -999 91293 4465: 4466:and 4467: 4468:AUTO MERGE FROM CHAIN: 4469:TMax Stn 1: 0 -3470 14650 145 NARRANDERA AIRPORT AUSTRALIA 2000 2006 -999 -999.00 4470:TMax Stn 2: 9570600 -3471 14651 145 NARRANDERA AIRPORT AUSTRALIA 1972 1980 -999 -999.00 4471:New Header: 9570600 -3470 14650 145 NARRANDERA AIRPORT AUSTRALIA 1972 2006 -999 0 4472:Note: Stn 2 data overwrote Stn 1 data 4473: 4474:MANUAL PAIRING FROM CHAIN: 4475:TMin: 9570600 -3470 14650 145 NARRANDERA AIRPORT AUSTRALIA 1971 2003 -999 0 4476:TMax: 9570600 -3470 14650 145 NARRANDERA AIRPORT AUSTRALIA 1972 2006 -999 0 4477:New Header: 9570600 -3470 14650 145 NARRANDERA AIRPORT AUSTRALIA 1971 2006 -999 0 4478: 4479:Found the problem - mistyping of an assignment.. and so: 4480: 4481:crua6[/cru/cruts/version_3_0/db/dtr] wc -l *.heads 4482: 4483: 14355 tmn.0707021605.dtb.heads 4484: 122 tmn.0707021605.dtb.lost.heads 4485: 579 tmn.0707021605.dtb.merg.heads 4486: 13654 tmn.0708071548.dtb.heads 4487: 4488: 14315 tmx.0702091313.dtb.heads 4489: 93 tmx.0702091313.dtb.lost.heads 4490: 568 tmx.0702091313.dtb.merg.heads 4491: 13654 tmx.0708071548.dtb.heads 4492: 4493:Phew! Well the headers are identical for the two new databases: 4494: 4495:crua6[/cru/cruts/version_3_0/db/dtr] cmp tmn.0708071548.dtb.heads tmx.0708071548.dtb.heads |wc -l 4496: 0 4497: 4498:34. So the to the real test - converting to DTR! Wrote tmnx2dtr.for, which does exactly that. It reported 4499:233 instances where tmin > tmax (all set to missing values) and a handful where tmin == tmax (no prob). 4500:Looking at the 233 illogicals, most of the stations look as though considerable work is needed on them. 4501:This highlights the fact that all I've done is to synchronise the tmin and tmax databases with each 4502:other, and with the Australian stations - there is still a lot of data cleansing to perform at some 4503:stage! But not right now :-) 4504: 4505:Input Files 4506: TMin: tmn.0708071548.dtb 4507: TMax: tmx.0708071548.dtb 4508: 4509:Output file 4510: DTR: dtr.0708071924.dtb 4511: 4512:Cases of identical values: 39 4513:Cases of min > max (BAD!): 233 4514:All illegals written to: illdtr.0708071924.dat 4515: 4516:Example of 'illegal' values to demonstrate quality of station data: 4517: 4518:station: 9600100 587 9532 126 SABANG/CUT BAU ID ID 1984 2006 -999 0 4519:min data: 2006 203 -197 200-9999 -211 207 233-9999-9999-9999-9999-9999 4520:max data: 2006 290 -299 307-9999 -315 309 308-9999-9999-9999-9999-9999 4521: 4522:Doesn't look very likely! 4523: 4524:Normals added: 4525: 4526:crua6[/cru/cruts/version_3_0/db/dtr] ./addnormline 4527: 4528: **** ADDNORMLINE **** 4529: 4530: Calculates monthly normals 4531: for 1961-1990, provided at 4532: least 75% of values are 4533: present. Results go into a 4534: normals line coming after 4535: the header. Operator called 4536: if different normals exist! 4537: 4538:Please enter the input database: dtr.0708071924.dtb 4539: 4540:Proposed output database name: dtr.0708081052.dat 4541: 4542: ACCEPT/REJECT (A/R): A 4543:Output database name: dtr.0708081052.dat 4544:Derived logfile name: dtr.0708081052.log 4545: 4546:So the final DTR database is dtr.0708081052.dtb. 4547: 4548:And so to the main process: 4549: 4550: 4551:crua6[/cru/cruts/version_3_0/primaries/dtr] ./anomdtb 4552: 4553: > ***** AnomDTB: converts .dtb to anom .txt for gridding ***** 4554: 4555: > Enter the suffix of the variable required: 4556:.dtr 4557: > Select the .cts or .dtb file to load: 4558:dtr.0708081052.dtb 4559: 4560: > Specify the start,end of the normals period: 4561:1961,1990 4562: > Specify the missing percentage permitted: 4563:25 4564: > Data required for a normal: 23 4565: > Specify the no. of stdevs at which to reject data: 4566:3 4567: > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 4568:3 4569: > Check for duplicate stns after anomalising? (0=no,>0=km range) 4570:0 4571: > Select the generic .txt file to save (yy.mm=auto): 4572:dtr.txt 4573: > Select the first,last years AD to save: 4574:1901,2006 4575: > Operating... 4576: 4577: > NORMALS MEAN percent STDEV percent 4578: > .dtb 3746373 65.9 4579: > .cts 178161 3.1 3924534 69.0 4580: > PROCESS DECISION percent %of-chk 4581: > no lat/lon 650 0.0 0.0 4582: > no normal 1763302 31.0 31.0 4583: > out-of-range 24 0.0 0.0 4584: > accepted 3924510 69.0 4585: > Dumping years 1901-2006 to .txt files... 4586: 4587: 4588: 4589:So a lower pewrcentage than last time (69.0 vs. 78.9), but then, more data overall so a better 4590:result (3924510 vs. 3167636). 4591: 4592:Gridding: 4593:IDL> quick_interp_tdm2,1901,2006,'dtrglo/dtr.',750,gs=0.5,pts_prefix='dtrtxt/dtr.',dumpglo='dumpglo' 4594: 4595:Convert from .glo: 4596:crua6[/cru/cruts/version_3_0/primaries/dtr] ./glo2abs 4597:Welcome! This is the GLO2ABS program. 4598:I will create a set of absolute grids from 4599:a set of anomaly grids (in .glo format), also 4600:a gridded version of the climatology. 4601:Enter the path and name of the normals file: clim.6190.lan.dtr 4602:Enter a name for the gridded climatology file: clim.6190.lan.dtr.grid 4603:Enter the path and stem of the .glo files: dtrglo/dtr. 4604:Enter the starting year: 1901 4605:Enter the ending year: 2006 4606:Enter the path (if any) for the output files: 4607:Now, CONCENTRATE. Addition or Percentage (A/P)? A 4608:Right, erm.. off I jolly well go! 4609:dtr.01.1901.glo 4610:(etc) 4611:dtr.12.2006.glo 4612: 4613:Finally, gridding: 4614:Writing cru_ts_3_00.1901.1910.dtr.dat 4615:Writing cru_ts_3_00.1911.1920.dtr.dat 4616:Writing cru_ts_3_00.1921.1930.dtr.dat 4617:Writing cru_ts_3_00.1931.1940.dtr.dat 4618:Writing cru_ts_3_00.1941.1950.dtr.dat 4619:Writing cru_ts_3_00.1951.1960.dtr.dat 4620:Writing cru_ts_3_00.1961.1970.dtr.dat 4621:Writing cru_ts_3_00.1971.1980.dtr.dat 4622:Writing cru_ts_3_00.1981.1990.dtr.dat 4623:Writing cru_ts_3_00.1991.2000.dtr.dat 4624:Writing cru_ts_3_00.2001.2006.dtr.dat 4625:Writing cru_ts_3_00.1901.2006.dtr.dat 4626: 4627: 4628:35. Onto the secondaries, working from the rerun methodology (see section 20 above). 4629: 4630:Began with temperature, using the anomaly txt files from the half-degree generation: 4631: 4632:IDL> quick_interp_tdm2,1901,2006,'tmpbin/tmpbin',1200,gs=2.5,dumpbin='dumpbin',pts_prefix='tmp0km0705101334txt/tmp.' 4633: 4634:This produced binaries such as 'tmpbin1901'. 4635: 4636:Then precipitation: 4637: 4638:IDL> quick_interp_tdm2,1901,2006,'prebin/prebin',450,gs=2.5,dumpbin='dumpbin',pts_prefix='pre0km0612181221txt/pre.' 4639: 4640:Finally, dtr: 4641: 4642:IDL> quick_interp_tdm2,1901,2006,'dtrbin/dtrbin',50,gs=2.5,dumpbin='dumpbin',pts_prefix='dtrtxt/dtr.' 4643: 4644:*** EEEK! Is that '50' a mistype? Meaning that anything using binary DTR will need re-doing? (RAL, Dec 07) *** 4645: 4646:And so to the synthetics. 4647: 4648:FRS: 4649: 4650:IDL> .compile /cru/cruts/fromdpe1a/code/idl/pro/rdbin.pro 4651:% Compiled module: RDBIN. 4652:IDL> .compile /cru/cruts/fromdpe1a/code/idl/pro/frs_gts_tdm.pro 4653:% Compiled module: FRS_GTS. 4654:IDL> frs_gts,dtr_prefix='dtrbin/dtrbin',tmp_prefix='tmpbin/tmpbin',1901,2006,outprefix='frssyn/frssyn' 4655:IDL> quick_interp_tdm2,1901,2006,'frsgrid/frsgrid',750,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='frssyn/frssyn' 4656: 4657:crua6[/cru/cruts/version_3_0/secondaries/frs] ../glo2abs 4658:Welcome! This is the GLO2ABS program. 4659:I will create a set of absolute grids from 4660:a set of anomaly grids (in .glo format), also 4661:a gridded version of the climatology. 4662:Enter the path and name of the normals file: clim.6190.lan.frs 4663:Enter a name for the gridded climatology file: clim.6190.lan.frs.grid 4664:Enter the path and stem of the .glo files: frsgrid/frsgrid. 4665:Enter the starting year: 1901 4666:Enter the ending year: 2006 4667:Enter the path (if any) for the output files: 4668:Now, CONCENTRATE. Addition or Percentage (A/P)? A 4669:Right, erm.. off I jolly well go! 4670:frsgrid.01.1901.glo 4671:(etc) 4672:frsgrid.12.2006.glo 4673: 4674:crua6[/cru/cruts/version_3_0/secondaries/frs] ../mergegrids 4675:Welcome! This is the MERGEGRIDS program. 4676:I will create decadal and full gridded files 4677:from the output files of (eg) glo2abs.for. 4678: 4679:Enter a gridfile with YYYY for year and MM for month: frsgridabs/frsgrid.MM.YYYY.glo.abs 4680:Enter Start Year: 1901 4681:Enter Start Month: 01 4682:Enter End Year: 2006 4683:Enter End Month: 12 4684: 4685:Please enter a sample OUTPUT filename, replacing 4686:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.frs.dat 4687:Writing cru_ts_3_00.1901.1910.frs.dat 4688:Writing cru_ts_3_00.1911.1920.frs.dat 4689:Writing cru_ts_3_00.1921.1930.frs.dat 4690:Writing cru_ts_3_00.1931.1940.frs.dat 4691:Writing cru_ts_3_00.1941.1950.frs.dat 4692:Writing cru_ts_3_00.1951.1960.frs.dat 4693:Writing cru_ts_3_00.1961.1970.frs.dat 4694:Writing cru_ts_3_00.1971.1980.frs.dat 4695:Writing cru_ts_3_00.1981.1990.frs.dat 4696:Writing cru_ts_3_00.1991.2000.frs.dat 4697:Writing cru_ts_3_00.2001.2006.frs.dat 4698: 4699:RD0: 4700: 4701:IDL> .compile /cru/cruts/fromdpe1a/code/idl/pro/rdbin.pro 4702:% Compiled module: RDBIN. 4703:IDL> .compile /cru/cruts/fromdpe1a/code/idl/pro/rd0_gts_tdm.pro 4704:% Compiled module: RD0_GTS. 4705:IDL> rd0_gts,1901,2006,1961,1990,outprefix='rd0syn/rd0syn',pre_prefix='prebin/prebin' 4706:Reading precip and rd0 normals 4707:% Compiled module: STRIP. 4708:yes 4709:filesize= 6220800 4710:gridsize= 0.500000 4711:% Compiled module: DEFXYZ. 4712:yes 4713:filesize= 6220800 4714:gridsize= 0.500000 4715:% Compiled module: DAYS. 4716:Calculating synthetic Rd0 normal 4717: 1961 4718:yes 4719:filesize= 248832 4720:gridsize= 2.50000 4721:% Compiled module: RD0CAL. 4722: 1962 4723:yes 4724: 4725:(etc) 4726: 4727: 2006 4728:yes 4729:filesize= 248832 4730:gridsize= 2.50000 4731:% Program caused arithmetic error: Floating divide by 0 4732:% Program caused arithmetic error: Floating illegal operand 4733:IDL> 4734: 4735:(as before, see section 20.) 4736: 4737:IDL> quick_interp_tdm2,1901,2006,'rd0grid/rd0grid',450,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='rd0syn/rd0syn' 4738: 4739:crua6[/cru/cruts/version_3_0/secondaries/rd0] ../glo2abs 4740:Welcome! This is the GLO2ABS program. 4741:I will create a set of absolute grids from 4742:a set of anomaly grids (in .glo format), also 4743:a gridded version of the climatology. 4744:Enter the path and name of the normals file: forrtl: error (69): process interrupted (SIGINT) 4745:crua6[/cru/cruts/version_3_0/secondaries/rd0] mkdir rd0gridabs 4746:crua6[/cru/cruts/version_3_0/secondaries/rd0] ../glo2abs 4747:Welcome! This is the GLO2ABS program. 4748:I will create a set of absolute grids from 4749:a set of anomaly grids (in .glo format), also 4750:a gridded version of the climatology. 4751:Enter the path and name of the normals file: clim.6190.lan.wet 4752:Enter a name for the gridded climatology file: clim.6190.lan.wet.grid 4753:Enter the path and stem of the .glo files: rd0grid/rd0grid. 4754:Enter the starting year: 1901 4755:Enter the ending year: 2006 4756:Enter the path (if any) for the output files: rd0gridabs/ 4757:Now, CONCENTRATE. Addition or Percentage (A/P)? A 4758:Right, erm.. off I jolly well go! 4759:rd0grid.01.1901.glo 4760:(etc) 4761:rd0grid.12.2006.glo 4762: 4763:crua6[/cru/cruts/version_3_0/secondaries/rd0] ../mergegrids 4764:Welcome! This is the MERGEGRIDS program. 4765:I will create decadal and full gridded files 4766:from the output files of (eg) glo2abs.for. 4767: 4768:Enter a gridfile with YYYY for year and MM for month: rd0gridabs/rd0grid.MM.YYYY.glo.abs 4769:Enter Start Year: 1901 4770:Enter Start Month: 01 4771:Enter End Year: 2006 4772:Enter End Month: 12 4773: 4774:Please enter a sample OUTPUT filename, replacing 4775:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.rd0.dat 4776:Writing cru_ts_3_00.1901.1910.rd0.dat 4777:(etc) 4778: 4779: 4780:I have to admit, I still don't understand secondary parameter generation. I've read the papers, and the 4781:miniscule amount of 'Read Me' documentation, and it just doesn't make sense. In particular, why use 2.5 4782:degree grids of the primaries instead of 0.5? Why deliberately lose spatial resolution, only to have to 4783:reinterpolate later? 4784: 4785:No matter; on to Vapour Pressure. Here's the complete output from the initial binary gridding,using dtr and tmp: 4786: 4787:IDL> vap_gts_anom,dtr_prefix='dtrbin/dtrbin',tmp_prefix='tmpbin/tmpbin',1901,2006,outprefix='vapsyn/vapsyn',dumpbin=1 4788:% Compiled module: VAP_GTS_ANOM. 4789:% Compiled module: RDBIN. 4790:% Compiled module: STRIP. 4791:% Compiled module: DEFXYZ. 4792:Land,sea: 56016 68400 4793:Calculating tmn normal 4794:% Compiled module: TVAP. 4795:Calculating synthetic vap normal 4796:% Compiled module: ESAT. 4797:Calculating synthetic anomalies 4798:% Compiled module: MOMENT. 4799:1901 vap (x,s2,<<,>>): 1.61250e-05 6.15570e-06 -0.160607 0.222689 4800:% Compiled module: WRBIN. 4801:1902 vap (x,s2,<<,>>): -0.000123188 3.46116e-05 -0.268891 0.0261283 4802:1903 vap (x,s2,<<,>>): 6.86689e-05 4.52675e-06 -0.121429 0.123995 4803:1904 vap (x,s2,<<,>>): -1.30788e-05 1.83887e-05 -0.454975 0.0919596 4804:1905 vap (x,s2,<<,>>): 1.94645e-05 1.32224e-05 -0.408679 0.0498396 4805:1906 vap (x,s2,<<,>>): 3.22279e-05 3.74796e-06 -0.178658 0.0261283 4806:1907 vap (x,s2,<<,>>): -2.56545e-05 1.68228e-05 -0.268768 0.0498040 4807:1908 vap (x,s2,<<,>>): 6.39573e-05 3.49149e-06 -0.173230 0.354836 4808:1909 vap (x,s2,<<,>>): 3.50080e-05 3.21530e-06 -0.201157 0.0261283 4809:1910 vap (x,s2,<<,>>): 3.45249e-05 6.15026e-06 -0.130285 0.144744 4810:1911 vap (x,s2,<<,>>): 3.99470e-05 5.85673e-06 -0.360082 0.0261283 4811:1912 vap (x,s2,<<,>>): -7.91931e-06 1.06891e-05 -0.279282 0.0261283 4812:1913 vap (x,s2,<<,>>): 6.07153e-05 7.10663e-07 -0.0148902 0.0261283 4813:1914 vap (x,s2,<<,>>): 7.22507e-05 2.52354e-06 -0.130205 0.124774 4814:1915 vap (x,s2,<<,>>): -2.11176e-05 1.59592e-05 -0.308456 0.0579963 4815:1916 vap (x,s2,<<,>>): -8.95735e-05 2.41852e-05 -0.247123 0.140438 4816:1917 vap (x,s2,<<,>>): -0.000105104 2.43058e-05 -0.229282 0.282290 4817:1918 vap (x,s2,<<,>>): 1.14711e-05 7.76188e-06 -0.248782 0.0261283 4818:1919 vap (x,s2,<<,>>): 2.51597e-05 5.75406e-06 -0.295303 0.215085 4819:1920 vap (x,s2,<<,>>): -2.78549e-06 1.81183e-05 -0.373193 0.0261283 4820:1921 vap (x,s2,<<,>>): 6.07153e-05 7.10663e-07 -0.0148902 0.0261283 4821:1922 vap (x,s2,<<,>>): -1.86602e-05 1.22345e-05 -0.275667 0.0261283 4822:1923 vap (x,s2,<<,>>): 5.76800e-05 1.22728e-06 -0.170021 0.0261283 4823:1924 vap (x,s2,<<,>>): 6.07153e-05 7.10663e-07 -0.0148902 0.0261283 4824:1925 vap (x,s2,<<,>>): 8.32519e-05 5.55618e-06 -0.109315 0.186182 4825:1926 vap (x,s2,<<,>>): 0.000106602 5.15263e-06 -0.105764 0.206929 4826:1927 vap (x,s2,<<,>>): 5.23023e-05 2.64333e-06 -0.194649 0.0498040 4827:1928 vap (x,s2,<<,>>): 5.50934e-05 2.47944e-06 -0.314917 0.0261283 4828:1929 vap (x,s2,<<,>>): -0.000524952 0.000155755 -0.417342 0.215959 4829:1930 vap (x,s2,<<,>>): 8.28323e-05 1.87314e-05 -0.328074 0.193805 4830:1931 vap (x,s2,<<,>>): -7.80687e-05 3.63543e-05 -0.315060 0.215417 4831:1932 vap (x,s2,<<,>>): 5.62579e-05 3.81547e-06 -0.249130 0.120583 4832:1933 vap (x,s2,<<,>>): -3.47433e-05 1.69009e-05 -0.218800 0.148224 4833:1934 vap (x,s2,<<,>>): 0.000156604 1.56121e-05 -0.173230 0.152809 4834:1935 vap (x,s2,<<,>>): 6.69520e-05 4.91451e-06 -0.160529 0.120391 4835:1936 vap (x,s2,<<,>>): -0.000255663 6.63373e-05 -0.398866 0.0261283 4836:1937 vap (x,s2,<<,>>): 6.99402e-05 2.70766e-05 -0.328074 0.201202 4837:1938 vap (x,s2,<<,>>): 5.91796e-05 6.70722e-06 -0.215017 0.155977 4838:1939 vap (x,s2,<<,>>): 4.88266e-05 5.25789e-06 -0.173294 0.0893239 4839:1940 vap (x,s2,<<,>>): 9.63896e-06 7.45103e-06 -0.214763 0.0758103 4840:1941 vap (x,s2,<<,>>): 4.11127e-05 4.15525e-06 -0.234030 0.0261283 4841:1942 vap (x,s2,<<,>>): -9.97969e-05 3.88466e-05 -0.288682 0.148893 4842:1943 vap (x,s2,<<,>>): 8.38607e-05 3.48416e-06 -0.0148902 0.163562 4843:1944 vap (x,s2,<<,>>): 7.96681e-05 7.91305e-06 -0.227413 0.104055 4844:1945 vap (x,s2,<<,>>): 3.37215e-05 3.99524e-06 -0.248782 0.0261283 4845:1946 vap (x,s2,<<,>>): 5.31976e-05 2.63755e-06 -0.128263 0.163584 4846:1947 vap (x,s2,<<,>>): 0.000131113 1.66296e-05 -0.353903 0.193758 4847:1948 vap (x,s2,<<,>>): 6.80941e-05 1.62353e-06 -0.0148902 0.163624 4848:1949 vap (x,s2,<<,>>): 2.47925e-05 2.45819e-05 -0.328074 0.237848 4849:1950 vap (x,s2,<<,>>): -9.57348e-05 7.78468e-05 -0.366764 0.726541 4850:1951 vap (x,s2,<<,>>): -6.54446e-06 1.35656e-05 -0.446058 0.0261283 4851:1952 vap (x,s2,<<,>>): -0.000158974 5.02732e-05 -0.262313 0.193617 4852:1953 vap (x,s2,<<,>>): 1.18525e-05 4.22691e-05 -0.282204 0.230629 4853:1954 vap (x,s2,<<,>>): -0.000151975 6.78713e-05 -0.373235 0.230602 4854:1955 vap (x,s2,<<,>>): -0.000134153 5.23124e-05 -0.298578 0.0841820 4855:1956 vap (x,s2,<<,>>): -9.61671e-05 5.20484e-05 -0.492004 0.0888951 4856:1957 vap (x,s2,<<,>>): -1.18048e-05 1.31769e-05 -0.220902 0.0261283 4857:1958 vap (x,s2,<<,>>): -8.61762e-06 1.12079e-05 -0.207799 0.148170 4858:1959 vap (x,s2,<<,>>): 8.27399e-05 4.88857e-06 -0.0929929 0.170919 4859:1960 vap (x,s2,<<,>>): 3.38773e-05 1.53901e-05 -0.207944 0.155940 4860:1961 vap (x,s2,<<,>>): 5.72571e-05 9.01807e-07 -0.0653905 0.0261283 4861:1962 vap (x,s2,<<,>>): 8.20891e-05 3.78016e-06 -0.240435 0.126662 4862:1963 vap (x,s2,<<,>>): -0.000108489 3.85148e-05 -0.266356 0.0836364 4863:1964 vap (x,s2,<<,>>): 3.02043e-05 6.37207e-06 -0.240547 0.150816 4864:1965 vap (x,s2,<<,>>): 5.76898e-05 2.48022e-06 -0.279282 0.143283 4865:1966 vap (x,s2,<<,>>): -0.000300312 5.32054e-05 -0.622719 0.0261283 4866:1967 vap (x,s2,<<,>>): 6.43500e-05 8.58218e-07 -0.0148902 0.0496181 4867:1968 vap (x,s2,<<,>>): -0.000241750 4.22773e-05 -0.214442 0.271730 4868:1969 vap (x,s2,<<,>>): -0.000568502 9.92260e-05 -0.385322 0.0732047 4869:1970 vap (x,s2,<<,>>): 6.07153e-05 7.10663e-07 -0.0148902 0.0261283 4870:1971 vap (x,s2,<<,>>): 2.15333e-05 4.77100e-06 -0.188071 0.0261283 4871:1972 vap (x,s2,<<,>>): -7.14160e-05 3.56948e-05 -0.365803 0.201611 4872:1973 vap (x,s2,<<,>>): 5.77503e-05 1.17079e-06 -0.160550 0.0261283 4873:1974 vap (x,s2,<<,>>): 3.49354e-05 4.93069e-06 -0.149678 0.144313 4874:1975 vap (x,s2,<<,>>): 6.14429e-05 7.36204e-07 -0.0148902 0.0380432 4875:1976 vap (x,s2,<<,>>): 6.49657e-05 3.25410e-06 -0.266356 0.165472 4876:1977 vap (x,s2,<<,>>): 0.000107180 1.92804e-05 -0.304625 0.208459 4877:1978 vap (x,s2,<<,>>): -4.80106e-05 3.28909e-05 -0.285492 0.105108 4878:1979 vap (x,s2,<<,>>): -0.000102001 2.35900e-05 -0.214390 0.112952 4879:1980 vap (x,s2,<<,>>): 4.16963e-05 2.70211e-06 -0.144913 0.0864268 4880:1981 vap (x,s2,<<,>>): 0.000274196 1.86668e-05 -0.0148902 0.222522 4881:1982 vap (x,s2,<<,>>): 8.57426e-07 7.08135e-06 -0.161781 0.0831981 4882:1983 vap (x,s2,<<,>>): -5.84499e-06 1.76470e-05 -0.234194 0.128289 4883:1984 vap (x,s2,<<,>>): -0.000106476 2.97454e-05 -0.335850 0.150833 4884:1985 vap (x,s2,<<,>>): 9.32757e-06 4.35533e-05 -0.323331 0.222522 4885:1986 vap (x,s2,<<,>>): 7.22110e-05 4.76179e-06 -0.141725 0.185658 4886:1987 vap (x,s2,<<,>>): -2.27107e-05 2.09631e-05 -0.291446 0.103599 4887:1988 vap (x,s2,<<,>>): 6.58090e-05 9.21014e-07 -0.0148902 0.0670816 4888:1989 vap (x,s2,<<,>>): 9.54406e-05 1.72599e-05 -0.266297 0.160293 4889:1990 vap (x,s2,<<,>>): 0.000218826 3.56583e-05 -0.174187 0.236204 4890:1991 vap (x,s2,<<,>>): 5.93288e-05 8.18618e-07 -0.0776650 0.0261283 4891:1992 vap (x,s2,<<,>>): 7.57687e-05 4.27091e-06 -0.174292 0.215085 4892:1993 vap (x,s2,<<,>>): -1.69378e-05 2.36942e-05 -0.314882 0.0420169 4893:1994 vap (x,s2,<<,>>): 6.36348e-05 1.18760e-06 -0.0148902 0.163543 4894:1995 vap (x,s2,<<,>>): 0.000281573 6.09912e-05 -0.463574 0.259426 4895:1996 vap (x,s2,<<,>>): 5.03362e-05 5.47691e-06 -0.224751 0.124774 4896:1997 vap (x,s2,<<,>>): 0.000132649 2.97693e-05 -0.446455 0.281070 4897:1998 vap (x,s2,<<,>>): 5.96544e-07 3.39098e-05 -0.359037 0.201228 4898:1999 vap (x,s2,<<,>>): 5.91499e-05 2.37232e-06 -0.166206 0.215985 4899:2000 vap (x,s2,<<,>>): 4.06034e-05 4.61604e-06 -0.0898572 0.191977 4900:2001 vap (x,s2,<<,>>): 0.000138230 8.53512e-06 -0.0512625 0.206929 4901:2002 vap (x,s2,<<,>>): 0.000218003 4.36873e-05 -0.760830 0.282290 4902:2003 vap (x,s2,<<,>>): 7.00864e-05 7.67472e-06 -0.301868 0.237875 4903:2004 vap (x,s2,<<,>>): 5.49200e-06 2.13246e-05 -0.500544 0.112129 4904:2005 vap (x,s2,<<,>>): 6.05939e-06 5.83817e-05 -0.885566 0.199814 4905:2006 vap (x,s2,<<,>>): 9.02885e-05 3.60834e-05 -0.455230 0.607388 4906: 4907:How very useful! No idea what any of that means. although it's heartwarming to see that it's 4908:nothing like the results of the 2.10 rerun, where 1991 looked like this: 4909: 4910:1991 vap (x,s2,<<,>>): 0.000493031 0.000742087 -0.0595093 1.86497 4911: 4912:Now, of course, it looks like this: 4913: 4914:1991 vap (x,s2,<<,>>): 5.93288e-05 8.18618e-07 -0.0776650 0.0261283 4915: 4916:From this I can deduce.. err.. umm.. 4917: 4918:Anyway now I need to use whatever VAP station data we have. And here I'm a little flaky (again), 4919:the vap database hasn't been updated, is it going to be? Asked Dave L and he supplied summaries 4920:he'd produced of CLIMAT bulletins from 2000-2006. Slightly odd format but very useful all the 4921:same. 4922: 4923:And now, a brief interlude. As we've reached the stage of thinking about secondary variables, I 4924:wondered about the CLIMAT updates, as one of the outstanding work items is to write routines to 4925:convert CLIMAT and MCDW bulletins to CRU format (so that mergedb.for can read them). So I look at 4926:a CLIMAT bulletin, and what's the first thing I notice? It's that there is absolutely no station 4927:identification information apart from the WMO code. None. No lat/lon, no name, no country. Which 4928:means that all the bells and whistles I built into mergedb, (though they were needed for the db 4929:merging of course) are surplus to requirements. The data must simply be added to whichever station 4930:has the same number at the start, and there's no way to check it's right. I don't appear to have a 4931:copy of a MCDW bulletin yet, only a PDF.. I wonder if that's the same? Anyway, back to the main job. 4932: 4933:As I was examining the vap database, I noticed there was a 'wet' database. Could I not use that to 4934:assist with rd0 generation? well.. it's not documented, but then, none of the process is so I might 4935:as well bluff my way into it! Units seem to vary: 4936: 4937:CLIMAT bulletins have day counts: 4938: 4939: SURFACE LAND 'CLIMAT' DATA FOR 2006/10. MISSING DATA=-32768 4940: MET OFFICE, HADLEY CENTRE CROWN COPYRIGHT 4941:WMO BLK WMO STN STNLP MSLP TEMP VAP P DAYS RN RAIN R QUINT SUN HRS SUN % MIN_T MAX_T 4942: 01 001 10152 10164 5 52 9 63 2 -32768 -32768 -12 20 4943: 4944:Dave L's CLIMAT update has days x 10: 4945: 4946: 100100 7093 -867 9JAN MAYEN(NOR-NAVY) NORWAY 20002006 -7777777 4947: 2000 150 120 180 60 150 20 30 130 120 150 70 70 4948: 4949:The existing 'wet' database (wet.0311061611.dtb) has days x 100: 4950: 4951: 10010 7093 -866 9 JAN MAYEN(NOR NAVY) NORWAY 1990 2003 -999 -999 4952:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 4953:1990-9999-9999-9999-9999 400 600 600 1800 1500 1100 800 1800 4954: 4955:The published climatology has days x 100 as well: 4956: 4957:Tyndall Centre grim file created on 13.01.2004 at 15:22 by Dr. Tim Mitchell 4958:.wet = wet day frequency (days) 4959:0.5deg lan clim:1961-90 MarkNew but adj so that wet=

4975:crua6[/cru/cruts/version_3_0/db] ./dave2cru

4976:

4977:DAVE2CRU - convert Dave L CLIMAT composites to dtb files

4978:Enter the CLIMAT composite to be converted: CLIMAT_MCDW_MCDW_rdy_updat_merged

4979:

4980:Example data line from that file:

4981:2000  150  120  180   60  150   20   30  130  120  150   70   70

4982:

4983:Please enter a factor to apply (or 1): 10

4984:Please enter the 3-ch parameter code: rd0

4985:

4986:The output file will be: rd0.0708151122.dtb

4987:

4988: 3411 stations written.

4989:

4990:

4991:Then tried to merge that into wet.0311061611.dtb, and immediately hot formatting issues - that pesky last

4992:field has been badly abused here, taking values including:

4993:

4994: -999.00

4995:    0.00

4996:  nocode     (yes, really!)

4997:  

4998:Had a quick review of mergedb; it won't be trivial to update it to treat that field as a8. So reluctantly,

4999:changed all the 'nocode' entries to '0':

5000:

5001:crua6[/cru/cruts/version_3_0/db/rd0] perl -pi -e 's/nocode/     0/g' wet.0311061611.dt*

5002:

5003:

5004:Unfortunately, that didn't solve the problems.. as there are alphanumerics in that field later on:

5005:

5006:-712356  5492 -11782  665 SPRING CRK WOLVERINE CANADA        1969 1988   -999  307F0P9                                                          

5007:

5008:So.. ***sigh***.. will have to alter mergedb.for to treat that field as alpha. Aaarrgghhh.

5009:

5010:Did that. Next problem is best summarised with an example:

5011:

5012:**************************************************

5013:*                                                *

5014:*  OPERATOR DECISION REQUIRED:                   *

5015:*                                                *

5016: 100100  7093   -867    9 JAN MAYEN(NOR-NAVY)  NORWAY        2000 2006    -999       0

5017:*                                                *

5018:*  This incoming station has a possible match in *

5019:*  the current database, but either the WMO code *

5020:*  or the lat/lon values differ.                 *

5021:*                                                *

5022:*  Incoming:                                     *

5023: 100100  7093   -867    9 JAN MAYEN(NOR-NAVY)  NORWAY        2000 2006    -999       0

5024:*  Potential match:                              *

5025:  10010  7093   -866    9 JAN MAYEN(NOR NAVY)  NORWAY        1990 2003    -999    -999

5026:

5027:Yes, the 'wet' database features old-style 5-digit WMO codes. The best approach is probably to alter

5028:mergedb again, to multiply any 5-digit codes by 10. Not sure if there is a similar problem with 7-digit

5029:codes, hopefully not.

5030:

5031:Oh, more bloody delays. Modified mergedb to 'adjust' the WMO codes, fine. But then a proper run of it

5032:just demonstrated that it's far too picky. Even a 0.01-degree difference in coordinates required ops

5033:intervention. What we need for updates is an absolute priority for WMO codes, and only a shout if the

5034:name or the spatial coordinates are waaay off. I am seriously considering scrapping mergedb in favour of

5035:a version of auminmaxresync - its cloud-based approach and 'intelligent' matching is far more efficient

5036:than mergedb's brute-force attack, as you'd expect from a program built on top of that knowledge. And it

5037:does save all its actions. But I don't know that I have the wherewithal.. okay, I do.

5038:

5039:Derived newmergedb.for from auminmaxresync.for. Should be fairly robust. Doesn't offer as many bells

5040:and whistles as mergedb.for, but should be faster and more helpful all the same.

5041:

5042:Well.. it works.. but the data doesn't. It's that old devil called WMO numbering again:

5043:

5044:Comparing Update:  718000  4868    622  217 NANCY/ESSEY          FRANCE        2001 2002    -999       0

5045:   ..with Master:  718000  4665  -5306   28 CAPE RACE (MARS)     CANADA        1920 1969   -999     -999

5046:

5047:Now what's happened here? Well the CLIMAT numbering only gives five digits (71 800) and so an extra zero

5048:has been added to bring it up to six. Unfortunately, that's the wrong thing to do, because that's the code

5049:of CAPE RACE. The six-digit code for NANCY/ESSEY is 071800. Mailed Phil and DL as this could be a big

5050:problem - many of the Update stations have no other metadata!

5051:

5052:Also noticed that some of the CLIMAT data seemed to be missing, eg for NANCY/ESSEY:

5053:

5054: 718000 4868   622  217NANCY/ESSEY         FRANCE       20002006        -7777777

5055: 2000-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

5056: 2001-9999  110-9999-9999-9999-9999-9999  120  150  110  130   90

5057: 2002   80  160   70   70   80   30   60  120  100  130  180  140

5058: 2003-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

5059: 2004-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

5060: 2005-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

5061: 2006-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

5062:

5063:I have the CLIMAT bulletin for 10/2006, which gives data for Rain Days (12 in this case). It doesn't seem

5064:likely that nothing was reported after 2002.

5065:

5066:I am now wondering whether it would be best to go back to the MCDW and CLIMAT bulletins themselves and work

5067:directly from those.

5068:

5069:--

5070:

5071:Well, information is always useful. And I probably did know this once.. long ago. All official WMO codes

5072:are five digits, countrycountrystationstationstation. However, we use seven-digit codes, because when no

5073:official code is available we improvise with two extra digits. Now I can't see why we didn't leave the rest

5074:at five digits, that would have been clear. I also can't see why, if we had to make them all seven digits,

5075:we extended the 'legitimate' five-digit codes by multiplying by 100, instead of adding two numerically-

5076:meaningless zeros at the most significant (left) end. But, that's what happened, and like everything else

5077:that's the way it's staying.

5078:

5079:So - incoming stations with WMO codes can only match stations with codes ending '00'. Put another way, for

5080:comparison purposes any 7-digit codes ending '00' should be truncated to five digits.

5081:

5082:Also got the locations of the original CLIMAT and MCDW bulletins.

5083:

5084:CLIMAT are here:

5085:http://hadobs.metoffice.com/crutem3/data/station_updates/

5086:

5087:MCDW are here:

5088:ftp://ftp1.ncdc.noaa.gov/pub/data/mcdw

5089:http://www1.ncdc.noaa.gov/pub/data/mcdw/

5090:

5091:Downloaded all CLIMAT and MCDW bulletins (CLIMAT 01/2003 to 07/2007; MCDW 01/2003 to 06/2007 (with a

5092:mysterious extra called 'ssm0302.Apr211542' - which turns out to be identical to ssm0302.fin)).

5093:

5094:Wrote mcdw2cru.for and climat2cru.for, just guess what they do, go on..

5095:

5096:

5097:uealogin1[/cru/cruts/version_3_0/incoming/MCDW] ./mcdw2cru 

5098:

5099:MCDW2CRU: Convert MCDW Bulletins to CRU Format

5100:

5101:Enter the earliest MCDW file: ssm0301.fin

5102:Enter the latest MCDW file (or  for single files): ssm0706.fin

5103:

5104:All Files Processed

5105:tmp.0709071541.dtb: 2407 stations written

5106:vap.0709071541.dtb: 2398 stations written

5107:pre.0709071541.dtb: 2407 stations written

5108:sun.0709071541.dtb: 1693 stations written

5109:

5110:Thanks for playing! Byeee!

5111:

5112:

5113:

5114:uealogin1[/cru/cruts/version_3_0/incoming/CLIMAT] ./climat2cru

5115:

5116:CLIMAT2CRU: Convert MCDW Bulletins to CRU Format

5117:

5118:Enter the earliest CLIMAT file: climat_data_200301.txt

5119:Enter the latest CLIMAT file (or  for single file): climat_data_200707.txt

5120:

5121:All Files Processed

5122:tmp.0709071547.dtb: 2881 stations written

5123:vap.0709071547.dtb: 2870 stations written

5124:pre.0709071547.dtb: 2878 stations written

5125:sun.0709071547.dtb: 2020 stations written

5126:tmn.0709071547.dtb: 2800 stations written

5127:tmx.0709071547.dtb: 2800 stations written

5128:

5129:Thanks for playing! Byeee!

5130:

5131:

5132:Of course, it wasn't quite that simple. MCDW has an inexplicably complex format, which I'm sure will vary

5133:over time and eventually break the converter. For instance, most text is left-justified, except the month

5134:names for the overdue data, which are right-justified. Also, there is no missing value code, just blank

5135:space if a value is absent. This necessitates reading everything as strings and then testing for content.

5136:Oh, and a small amount of rain is marked 'T'.. as are small departures from the mean!!

5137:

5138:So moan over, now we have a set of updates for the secondary databases. And, indeed for the primary ones -

5139:except that I've already processed those, as updated by Dave L.. er.. ah well. So as I'm running stupidly

5140:late anyway - why not find out? It's that Imp of the Perverse on my shoulder again.

5141:

5142:Actually as I examined all the databases in the tree to work out what was wheat and what chaff, I had my

5143:awful memory jogged quite nastily: WE NEED RAIN DAYS. So both conversion progs will need adjusting and

5144:re-running!! Waaaaah! And frankly at 18:45 on a Friday evening.. it's not gonna happen right now.

5145:

5146:..okay, a another week, another razorblade to slide down. Modified mcdw2cru to include rain days:

5147:

5148:

5149:uealogin1[/cru/cruts/version_3_0/incoming/MCDW] ./mcdw2cru 

5150:

5151:MCDW2CRU: Convert MCDW Bulletins to CRU Format

5152:

5153:Enter the earliest MCDW file: ssm0301.fin

5154:Enter the latest MCDW file (or  for single files): ssm0706.fin

5155:

5156:All Files Processed

5157:tmp.0709111032.dtb: 2407 stations written

5158:vap.0709111032.dtb: 2398 stations written

5159:rdy.0709111032.dtb: 2407 stations written

5160:pre.0709111032.dtb: 2407 stations written

5161:sun.0709111032.dtb: 1693 stations written

5162:

5163:Thanks for playing! Byeee!

5164:

5165:

5166:Checked, and the four preexisting databases match perfectly with their counterparts, so I didn't break

5167:anything in the adjustments. and the rdy file looks good too (actually the above is the *final* run;

5168:there were numerous bugs as per).

5169:

5170:

5171:uealogin1[/cru/cruts/version_3_0/incoming/CLIMAT] ./climat2cru 

5172:

5173:CLIMAT2CRU: Convert MCDW Bulletins to CRU Format

5174:

5175:Enter the earliest CLIMAT file: climat_data_200301.txt

5176:Enter the latest CLIMAT file (or  for single file): climat_data_200707.txt

5177:

5178:All Files Processed

5179:tmp.0709101706.dtb: 2881 stations written

5180:vap.0709101706.dtb: 2870 stations written

5181:rdy.0709101706.dtb: 2876 stations written

5182:pre.0709101706.dtb: 2878 stations written

5183:sun.0709101706.dtb: 2020 stations written

5184:tmn.0709101706.dtb: 2800 stations written

5185:tmx.0709101706.dtb: 2800 stations written

5186:

5187:Thanks for playing! Byeee!

5188:

5189:

5190:Again, existing outputs are unchanged and the new rdy file looks OK (though see bracketed note above for MCDW).

5191:

5192:So.. to the incorporation of these updates into the secondary databases. Oh, my.

5193:

5194:Beginning with Rain Days, known variously as rd0, rdy, pdy.. this allowed me to modify newmergedb.for to cope

5195:with various 'freedoms' enjoyed by the existing databases (such as six-digit WMO codes). And then, when run,

5196:an unexpected side-effect of my flash correlation display thingy: it shows up existing problems with the data!

5197:

5198:Here is the first 'issue' encountered by newmergedb, taken from the top and with my comments in :

5199:

5200:

5201:uealogin1[/cru/cruts/version_3_0/db/rd0] ./newmergedb

5202:

5203:WELCOME TO THE DATABASE UPDATER

5204:

5205:Before we get started, an important question:

5206:Should the incoming 'update' header info and data take precedence over the existing database?

5207:Or even vice-versa? This will significantly reduce user decisions later, but is a big step!

5208:

5209:Enter 'U' to give Updates precedence, 'M' to give Masters precedence, 'X' for equality: U

5210:Please enter the Master Database name: wet.0311061611.dtb

5211:Please enter the Update Database name: rdy.0709111032.dtb

5212:

5213:Reading in both databases..

5214:Master database stations:     4988

5215:Update database stations:     2407

5216:

5217:Looking for WMO code matches..

5218:

5219:***** OPERATOR ADJUDICATION REQUIRED *****

5220:

5221:In attempting to pair two stations, possible data incompatibilities have been found.

5222:

5223:MASTER:  221130  6896   3305   51 MURMANSK             EX USSR       1936 2003   -999     -999

5224:UPDATE: 2211300  6858   3303   51 MURMANSK             RUSSIAN FEDER 2003 2007    -999       0

5225:

5226:CORRELATION STATISTICS (enter 'C' for more information):

5227:> -0.60 is minimum correlation coeff.

5228:>  0.65 is maximum correlation coeff.

5229:> -0.01 is mean correlation coeff.

5230:

5231:Enter 'Y' to allow, 'N' to deny, or an information code letter: C

5232:

5233:

5234:

5235:Master Data: Correlation with Update first year aligned to this year -v

5236: 1936  900  600 1000  800 1000  900 1300 1700 2100 1800  900 1000    0.27

5237: 1937  300 1400 1300  800 1400 1800  500 1200 1600 1000 1100 1500    0.15

5238: 1938  900 1000 1500 1800 1200 1500 1200 1700  500  700 1600  700   -0.13

5239: 1939 1500 1300 1100 1400 1200 1200 1000 1300 1800 1600 1100 1300    0.24

5240: 1940 1000 1500 1000 1200 1100 1700 2600 1500 1500 1400 1700 1100    0.15

5241: 1941 1800 1200 1000 1200  900 1100  900 1200 1900 1500 1000 1400    0.48

5242: 1942  900  900 1700  900 1600 1000  600 1100 1400 1300  700  700    0.51

5243: 1943  800 1000 1000 1300  900  800 1500 1600 1400 1500 1300 1200    0.44

5244: 1944 1000  400  900  800 1200  600  900 2000  900 1100 1000  900    0.32

5245: 1945  500  400  700  700  800 1800  900 1100 1200 1100 1300  700    0.19

5246: 1946 1200 1200  100  700  900 1200  400  900  800 1900 1300 1400    0.16

5247: 1947  900 1300 1300 1100 1600 1000  800 1400 1400 1700 2100 1900    0.09

5248: 1948 1100 1400 1400 1200 1300 1800 1200 1700 1500 2200 2100 1900    0.10

5249: 1949 1100 1100  500 1500 1600 1100 1500 1200 2200 2500  900 1600    0.04

5250: 1950 1300  800 1000 1100 1700 1200 1500  800 1100 1300 1500 1400   -0.04

5251: 1951 1100  600 1400 1400 1500 1600 2100 1300 1500 1700 2000 1700   -0.13

5252: 1952 2100  800 1100 1800 1300 1200 2400 2200 1600 1000 1000 2300   -0.23

5253: 1953 2100 1400 2100 1500  900  300 1300 1700 1500  800 1200  800   -0.24

5254: 1954 2100  600 1300 1000 1300 1700 1600 2000 1800 1300 1400 1200   -0.40

5255: 1955 2200 1300  900 1000 1600 2000 1100 1400 1000 2100 2300 1600   -0.20

5256: 1956 1300 1100 1300  400 1600 1300  900 1500 2000 1300 2000 1400   -0.30

5257: 1957 1700 1600 1100 1100 1900 1900 1400 1600 1400 1700 2300 2600   -0.27

5258: 1958 1300 2200 1900  700 1500 1200 2100 1000 1900 1700 1600 1000   -0.21

5259: 1959 2500 1800 1300  900  900 1600 1600 1500 2200 1700 1000  900   -0.33

5260: 1960 1800 1700 1500  400 1300 1500  400 1000 1300 1500 1000 1400   -0.21

5261: 1961 2100 1800 2200 1500  800 1400 1600 1100 1900 1200 1200 2100   -0.59

5262: 1962 2100 1100 1000 1500 1300 1100 1300 1700 1200 2000 1600 2300   -0.37

5263: 1963 2100 2100 2000 1000  700 2000 1400 1800 1400 1600 2000 2400   -0.56

5264: 1964 2400 1100 1000 1700 1100 1400 1400 1400 2000 1200 2100 1800   -0.42

5265: 1965 1400 2100 1300 1000 1700 1700 1400 2400 1300 2100 1900 2100   -0.41

5266: 1966 1600 1600 2000 2000 1700 1200 2000 2500 2500 2700 1600  600   -0.34

5267: 1967 2200 1700 1600 1200 1000 1400 1600 1300 1700 1500 1200 2100   -0.21

5268: 1968 1600 1800 1800 1800 1500 1800 1400 2100 1000 2000 2100 2000   -0.28

5269: 1969 1100  300 1900 1200 1000 1300 1500 1200 1200 2000 1700  800   -0.25

5270: 1970 1900 1400 1200  900  600 1200 1500  700 2300 1700 1700 2100   -0.23

5271: 1971 2000 1300 1600 1600 1200 1100 1400 1800 2000 1600 1700 1500   -0.39

5272: 1972 1300 1200 1300 1200 1700  800 1400 1800 1900 2000 1700 1600   -0.26

5273: 1973 1800 1100 1700  900 1200 1500  500 1800 1200 2000 2100 2100   -0.36

5274: 1974 1100 2400  700 1600 1300 1300 1800 2000 1900 1200 1400 2400   -0.29

5275: 1975 1500 2200 1400 1700 2500 2200 2300 1600 1700 2300 1800 2600   -0.47

5276: 1976 1900  800 1100 1500 1000  900 1300 1800 2200 1600 1400 1600   -0.33

5277: 1977 1800 1400 2200 1200 1600 1900 1300 1500 1500 1900 1500 2000   -0.40

5278: 1978 1500 1800 1400 2100  700 1000 1100 1900 1700 2300 1500 2200   -0.24

5279: 1979 1700 1700 1700 1200 1500 1800  900 1200 1800 1600 1500 2300   -0.39

5280: 1980 1900 1300 1300 1000 1400  900  700 1100 1300 1600 2200 1700   -0.36

5281: 1981 2600  500 1900 2000  800 1900 1500 2000 1400 1500 1800 1600   -0.46

5282: 1982 2200 1800 1100 1600 1500 2200 1800 1400 1700 1700 1900 1400   -0.60

5283: 1983 2400 1900 1700 1200  800 1500 1200 2000 1400 2100 2000 2500   -0.23

5284: 1984 1900  800 1500 2000 1100 1600 2000 1700 1100 1400 1000 1200        

5285: 1985-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999        

5286: 1986-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999        

5287: 1987-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999        

5288: 1988-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999        

5289: 1989-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999    0.65

5290: 1990-9999-9999-9999-9999-9999  500 1300  900  700  900 1300  700    0.62

5291: 1991-9999  900  500  300  700 1000 1500  700 1700 1000 1300 1300    0.54

5292: 1992  800 1000  600  500  700  900-9999 1300-9999  700  900 1200    0.60

5293: 1993  600  900  400  500  900 1500 1000  800  800 1000  400 1000    0.55

5294: 1994 1300 1000  300  600  700 1000  900  600 1200    0 1400  600    0.43

5295: 1995  900  900  600  700  700  900 1100 1300  600 1800 1300  500    0.61

5296: 1996  500 1100  400  700  700 1200 1200 1100 1100  900 1000 1400    0.54

5297: 1997 1200  800 1300  600  600  100  500 1100  900-9999 1000  900    0.61

5298: 1998 1200 1300  800 1100 1100 1100  800  600 1200 1100  600 1200    0.52

5299: 1999  600  400  600 1000  700  700 1800 1400  700 1600  800 1200    0.62

5300: 2000 1100  600 1500 1700  900 1500  800  800 1000 1000  600  600    0.40

5301: 2001  600  500  700  700  600  500 1200 1200  700 1300  900 1000    0.63

5302: 2002 1000  800 1300  200  900 1100 1400 1200 1400 1800 1100  700        

5303: 2003 1100-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999        

5304:Update Data:

5305: 2003 1100  700  700  500 1000  400  700 1100 1200 2100  800 1900

5306: 2004  900  700  600  600 1300 1200 1000 1200 1400  900 1000 1000

5307: 2005 1000  400  800 1100  900  600 1200 1000 1600 1000 1300 1200

5308: 2006  700  500 1300  400  600 1200 1600  700 1000-9999  600 1500

5309: 2007 1400  400  400 1300 1200 1200-9999-9999-9999-9999-9999-9999

5310:

5311:

5315:

5316:MASTER:  221130  6896   3305   51 MURMANSK             EX USSR       1936 2003   -999     -999

5317:UPDATE: 2211300  6858   3303   51 MURMANSK             RUSSIAN FEDER 2003 2007    -999       0

5318:

5319:CORRELATION STATISTICS (enter 'C' for more information):

5320:> -0.60 is minimum correlation coeff.

5321:>  0.65 is maximum correlation coeff.

5322:> -0.01 is mean correlation coeff.

5323:

5324:Enter 'Y' to allow, 'N' to deny, or an information code letter: 

5325:

5326:

5327:So.. should I really go to town (again) and allow the Master database to be 'fixed' by this

5328:program? Quite honestly I don't have time - but it just shows the state our data holdings

5329:have drifted into. Who added those two series together? When? Why? Untraceable, except

5330:anecdotally.

5331:

5332:It's the same story for many other Russian stations, unfortunately - meaning that (probably)

5333:there was a full Russian update that did no data integrity checking at all. I just hope it's

5334:restricted to Russia!!

5335:

5336:There are, of course, metadata issues too. Take:

5337:

5338:

5339:MASTER:  206740  7353   8040   47 DIKSON ISLAND        EX USSR       1936 2003   -999     -999

5340:UPDATE: 2067400  7330   8024   47 OSTROV DIKSON        RUSSIAN FEDER 2003 2007    -999       0

5341:

5342:CORRELATION STATISTICS (enter 'C' for more information):

5343:> -0.70 is minimum correlation coeff.

5344:>  0.81 is maximum correlation coeff.

5345:> -0.01 is mean correlation coeff.

5346:

5347:

5348:This is pretty obviously the same station (well OK.. apart from the duff early period, but I've

5349:got used to that now). But look at the longitude! That's probably 20km! LUckily I selected

5350:'Update wins' and so the metadata aren't compared. This is still going to take ages, because although

5351:I can match WMO codes (or should be able to), I must check that the data correlate adequately - and

5352:for all these stations there will be questions. I don't think it would be a good idea to take the

5353:usual approach of coding to avoid the situation, because (a) it will be non-trivial to code for, and

5354:(b) not all of the situations are the same. But I am beginning to wish I could just blindly merge

5355:based on WMO code.. the trouble is that then I'm continuing the approach that created these broken

5356:databases. Look at this one:

5357:

5358:

5359:***** OPERATOR ADJUDICATION REQUIRED *****

5360:

5361:In attempting to pair two stations, possible data incompatibilities have been found.

5362:

5363:MASTER:  239330  6096   6906   40 HANTY MANSIJSK       EX USSR       1936 1984   -999     -999

5364:UPDATE: 2393300  6101   6902   46 HANTY-MANSIJSK       RUSSIAN FEDER 2003 2007    -999       0

5365:

5366:CORRELATION STATISTICS (enter 'C' for more information):

5367:> -0.42 is minimum correlation coeff.

5368:>  0.39 is maximum correlation coeff.

5369:> -0.02 is mean correlation coeff.

5370:

5371:Enter 'Y' to allow, 'N' to deny, or an information code letter: C

5372:Master Data: Correlation with Update first year aligned to this year -v

5373: 1936 1400  800 1700  900 1200  800  700  800 1800-9999-9999-9999    0.33

5374: 1937 1400  800  500 1700 1500  800 1200 1000 1700 1300  700 1200    0.32

5375: 1938 1000 1700 1200 1100 1100  800  800 1300 1400 1900 1800 1300    0.04

5376: 1939 1100 1700 1600 1800 1500  800 1500 1900 1700 1800 1300 1300    0.09

5377: 1940 1300  700  900  900 1800 1200  900 1300 1200 2200 1900 1800    0.08

5378: 1941 1400 1100 1800 1000 1400 1900 1400  700 1300 1200 1900 2000    0.02

5379: 1942 1700  900 1600  900 1200 1500 1300 1500 1200 1900 1500 1500   -0.06

5380: 1943 1400 1300 1300  800 1400 1600 1300 1500 1900 2000  700 1900   -0.17

5381: 1944 1900 1500 2000 1100 1200 1300 1500 1700 1800 1200 1500 1900   -0.32

5382: 1945 1300 1000 1400 2100 2000 1100 1700  700 1600 1800 2300 1700   -0.42

5383: 1946 2300 1900 1500 1100 1100 2000 1800 1000 1200 2100 2000 1800   -0.35

5384: 1947 1900 1400 1600 1000 2100 1900 2100 1000 1200 2000 2100 1500   -0.35

5385: 1948 1700 1500 1800  800 1300 1800 1700 1300 1800 2200 2000 2100   -0.15

5386: 1949 2300 2100 1000  700 1600 1400 1200  800 2100 2000 1100 1400   -0.07

5387: 1950 2100 2300 1000 1100 1500 1600 1600 2300 1900 1200 1100 1500    0.00

5388: 1951 1600 1000 1500  800 1500 1400 1200  600 1800 1800 1400 2400   -0.07

5389: 1952 1600  400 1100 1300 1100 1400  800 2000 1500 2300 1300 1600   -0.04

5390: 1953 2000 1200 1500  500 1300 1500 1100 1200 2300 2200 1600 2100   -0.02

5391: 1954 1700 1800  700  700 1000 1300 1200 1600 2000 1800 1800  600    0.01

5392: 1955 2400 1400 1000 1100 1700 1200 1000 1300 1500 1300 2300 1600   -0.08

5393: 1956 1300  800 1000 1100 1000 1000 1400 1800 1900 1900 2600 2000   -0.29

5394: 1957 1900 1200 1700 1000 1100 1100 1100  700  800 2300 1900 2200   -0.18

5395: 1958 1300 1600 1500  400 1500 1100 1300 1400 1900 2400 2000 1600   -0.28

5396: 1959 1700 1600  700 1300 1700 1100 1100 1600 2000 2100 1900 1600   -0.04

5397: 1960 1800 1600-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999    0.24

5398: 1961-9999-9999-9999-9999-9999-9999-9999 1600 1600 1700 1900 1600    0.33

5399: 1962 1700  800 1200  600  400 1100  900 2000 1100 1900 1700 1500    0.25

5400: 1963 1200 1300 1700  700 1100 1600  900 1000 1100 1400 1800 2000   -0.04

5401: 1964 1900  500 1300 1300 1200 1200 1100 1100 1700 1500 2000 1800    0.13

5402: 1965 1200 1400  700  900 1200 1100 1300 1400 1800 2500 1000 1700    0.23

5403: 1966 1800 1600 2100 1300 1500 2100  900 1800 1500 2400 1900  800    0.11

5404: 1967 1600 1200 1100  600  800 1100 1100  700 1300 1200 1300 1900    0.39

5405: 1968 1600 1400 1600 1200  900 1300 1400 1000 1700 1300 1400 1200    0.24

5406: 1969  900 1000 1100 1500 1700 1700 1000 1800 1200 1400 1900 1300    0.04

5407: 1970 1500 1200 1600 1400  700 1600  700 1600 1000 1500 1900 1600   -0.02

5408: 1971 1700  400 1100 1700 1300 1700  700 2000  900 2100 2000 1900   -0.11

5409: 1972 1200 1500 1400  800 1700 1300 1700 2000 2100 1700 2500 1900   -0.08

5410: 1973 1200 1100 1100  700  800 1300 2100 1000 2400 1900 1800 2300   -0.11

5411: 1974  700 1200 1800 1800 1400 1200 1000 1300 1100 1600 1900  700   -0.14

5412: 1975 2200 1800 1400 1300 1500 1500 1400 1500 1400 2300 1900 2100   -0.15

5413: 1976 2000 1500  600  700 1100 1600 1300 1100 1500 1800 1600 1200   -0.11

5414: 1977 1900 1700 1800 1400 1000 1100 1000 1300 1500 1800 1700 2100   -0.15

5415: 1978 1600 1000  800 1400 1400  800 1600 1600 2300 2200 2200 1800    0.03

5416: 1979 1600 1600 1600  900  900 1900 1200 1700 1200 2100 1600 2000    0.00

5417: 1980 1600 1200  500  800 1500 1100  800 1700 1200  600 2200 2200   -0.05

5418: 1981 2000 1000 1700 1300 1500 1100  800  400 1500  800 1500 1900    0.06

5419: 1982 2400 1800 1100 1200 1200 1100 1000 1700 1200 2100 1800 2000    0.03

5420: 1983 2500 2100 1800 1300 1400 1200 1200 1300 1300 1900 2300 1900    0.10

5421: 1984 1200  700  500 1300  900  800 1100 1000 1700 1600 1600 1300        

5422:Update Data:

5423: 2003 1500  900  600  400  900 1200  500  700 1100  600  700 1500

5424: 2004  700  600  700  400  600 1100  500  900  900 1400 1500  600

5425: 2005  700  400  800 1400  300  900  800  800  900  500 1200  600

5426: 2006  800  700  900 1000  800  500 1000  500 1300 1100  700 1600

5427: 2007 1100 1100  900  700 1300 1500-9999-9999-9999-9999-9999-9999

5428:

5429:

5430:Here, the expected 1990-2003 period is MISSING - so the correlations aren't so hot! Yet

5431:the WMO codes and station names /locations are identical (or close). What the hell is

5432:supposed to happen here? Oh yeah - there is no 'supposed', I can make it up. So I have :-)

5433:

5434:If an update station matches a 'master' station by WMO code, but the data is unpalatably

5435:inconsistent, the operator is given three choices:

5436:

5437:

5438:You have failed a match despite the WMO codes matching.

5439:This must be resolved!! Please choose one:

5440:

5441:1. Match them after all.

5442:2. Leave the existing station alone, and discard the update.

5443:3. Give existing station a false code, and make the update the new WMO station.

5444:

5445:Enter 1,2 or 3: 

5446:

5447:

5448:You can't imagine what this has cost me - to actually allow the operator to assign false

5449:WMO codes!! But what else is there in such situations? Especially when dealing with a 'Master'

5450:database of dubious provenance (which, er, they all are and always will be).

5451:

5452:False codes will be obtained by multiplying the legitimate code (5 digits) by 100, then adding

5453:1 at a time until a number is found with no matches in the database. THIS IS NOT PERFECT but as

5454:there is no central repository for WMO codes - especially made-up ones - we'll have to chance

5455:duplicating one that's present in one of the other databases. In any case, anyone comparing WMO

5456:codes between databases - something I've studiously avoided doing except for tmin/tmax where I

5457:had to - will be treating the false codes with suspicion anyway. Hopefully.

5458:

5459:Of course, option 3 cannot be offered for CLIMAT bulletins, there being no metadata with which

5460:to form a new station.

5461:

5462:This still meant an awful lot of encounters with naughty Master stations, when really I suspect

5463:nobody else gives a hoot about. So with a somewhat cynical shrug, I added the nuclear option - 

5464:to match every WMO possible, and turn the rest into new stations (er, CLIMAT excepted). In other

5465:words, what CRU usually do. It will allow bad databases to pass unnoticed, and good databases to

5466:become bad, but I really don't think people care enough to fix 'em, and it's the main reason the

5467:project is nearly a year late.

5468:

5469:And there are STILL WMO code problems!!! Let's try again with the issue. Let's look at the first

5470:station in most of the databases, JAN MAYEN. Here it is in various recent databases:

5471:

5472:

5473:dtr.0705152339.dtb: 100100  7093   -867    9 JAN MAYEN            NORWAY        1998 2006   -999  -999.00

5474:pre.0709111032.dtb:0100100  7056   -840    9 JAN MAYEN            NORWAY        2003 2007    -999       0

5475:sun.0709111032.dtb:0100100  7056   -840    9 JAN MAYEN            NORWAY        2003 2007    -999       0

5476:tmn.0702091139.dtb: 100100  7093   -867    9 JAN MAYEN            NORWAY        1998 2006   -999  -999.00

5477:tmn.0705152339.dtb: 100100  7093   -867    9 JAN MAYEN            NORWAY        1998 2006   -999  -999.00

5478:tmp.0709111032.dtb:0100100  7056   -840    9 JAN MAYEN            NORWAY        2003 2007    -999       0

5479:tmx.0702091313.dtb: 100100  7093   -867    9 JAN MAYEN            NORWAY        1998 2006   -999  -999.00

5480:tmx.0705152339.dtb: 100100  7093   -867    9 JAN MAYEN            NORWAY        1998 2006   -999  -999.00

5481:vap.0709111032.dtb:0100100  7056   -840    9 JAN MAYEN            NORWAY        2003 2007    -999       0

5482:

5483:As we can see, even I'm cocking it up! Though recoverably. DTR, TMN and TMX need to be written as (i7.7).

5484:

5485:Anyway, here it is in the problem database:

5486:

5487:wet.0311061611.dtb:  10010  7093   -866    9 JAN MAYEN(NOR NAVY)  NORWAY        1990 2003   -999     -999

5488:

5489:You see? The leading zero's been lost (presumably through writing as i7) and then a zero has been added at

5490:the trailing end. So it's a 5-digi WMO code BUT NOT THE RIGHT ONE. Aaaarrrgghhhhhh!!!!!!

5491:

5492:I think this can only be fixed in one of two ways:

5493:

5494:1. By hand.

5495:

5496:2. By automatic comparison with other (more reliable) databases.

5497:

5498:As usual - I'm going with 2. Hold onto your hats.

5499:

5500:Actually, a brief interlude to churn out the tmin & tmax primaries, which got sort-of

5501:forgotten after dtr was done:

5502:

5503:

5504:   > ***** AnomDTB: converts .dtb to anom .txt for gridding *****

5505:   > Enter the suffix of the variable required:

5506:.tmn

5507:   > Select the .cts or .dtb file to load:

5508:tmn.0708071548.dtb

5509:   > Specify the start,end of the normals period: 

5510:1961,1990

5511:   > Specify the missing percentage permitted: 

5512:25

5513:   > Data required for a normal:           23

5514:   > Specify the no. of stdevs at which to reject data: 

5515:3

5516:   > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 

5517:3

5518:   > Check for duplicate stns after anomalising? (0=no,>0=km range)

5519:0

5520:   > Select the generic .txt file to save (yy.mm=auto):

5521:tmn.txt

5522:   > Select the first,last years AD to save: 

5523:1901,2006

5524:   > Operating...

5525:   > NORMALS            MEAN percent      STDEV percent

5526:   >         .dtb    3814210    65.5

5527:   >         .cts     210801     3.6    4025011    69.2

5528:   > PROCESS        DECISION percent %of-chk

5529:   > no lat/lon          650     0.0     0.0

5530:   > no normal       1793923    30.8    30.8

5531:   > out-of-range        976     0.0     0.0

5532:   > accepted        4024035    69.1

5533:   > Dumping years 1901-2006 to .txt files...

5534:#####

5535:IDL> quick_interp_tdm2,1901,2006,'tmnglo/tmn.',750,gs=0.5,pts_prefix='tmntxt/tmn.',dumpglo='dumpglo'

5536:#####

5537:Welcome! This is the GLO2ABS program.

5538:I will create a set of absolute grids from

5539:a set of anomaly grids (in .glo format), also

5540:a gridded version of the climatology.

5541:Enter the path and name of the normals file: gunzip clim.6190.lan.tmn

5542:FILE NOT FOUND - PLEASE TRY AGAIN: clim.6190.lan.tmn

5543:Enter a name for the gridded climatology file: clim.6190.lan.tmn.grid

5544:Enter the path and stem of the .glo files: tmnglo/tmn.

5545:Enter the starting year: 1901

5546:Enter the ending year:   2006

5547:Enter the path (if any) for the output files: tmnabs

5548:Now, CONCENTRATE. Addition or Percentage (A/P)? A

5549:Right, erm.. off I jolly well go!

5550:tmn.01.1901.glo

5551:(etc)

5552:tmn.12.2006.glo

5553:#####

5554:Welcome! This is the MERGEGRIDS program.

5555:I will create decadal and full gridded files

5556:from the output files of (eg) glo2abs.for.

5557:Enter a gridfile with YYYY for year and MM for month: tmnabs/tmn.MM.YYYY.glo.abs

5558:Enter Start Year:  1901

5559:Enter Start Month: 01

5560:Enter End Year:    2006

5561:Enter End Month:   12

5562:Please enter a sample OUTPUT filename, replacing

5563:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.tmn.dat  

5564:Writing cru_ts_3_00.1901.1910.tmn.dat

5565:(etc)

5566:#####

5567:   > ***** AnomDTB: converts .dtb to anom .txt for gridding *****

5568:   > Enter the suffix of the variable required:

5569:.tmx

5570:   > Select the .cts or .dtb file to load:

5571:tmx.0708071548.dtb

5572:   > Specify the start,end of the normals period: 

5573:1961,1990

5574:   > Specify the missing percentage permitted: 

5575:25

5576:   > Data required for a normal:           23

5577:   > Specify the no. of stdevs at which to reject data: 

5578:3

5579:   > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 

5580:3

5581:   > Check for duplicate stns after anomalising? (0=no,>0=km range)

5582:0

5583:   > Select the generic .txt file to save (yy.mm=auto):

5584:tmx.txt

5585:   > Select the first,last years AD to save: 

5586:1901,2006

5587:   > Operating...

5588:   > NORMALS            MEAN percent      STDEV percent

5589:   >         .dtb    3795470    65.4

5590:   >         .cts     205607     3.5    4001077    68.9

5591:   > PROCESS        DECISION percent %of-chk

5592:   > no lat/lon          652     0.0     0.0

5593:   > no normal       1805313    31.1    31.1

5594:   > out-of-range        471     0.0     0.0

5595:   > accepted        4000606    68.9

5596:   > Dumping years 1901-2006 to .txt files...

5597:#####

5598:IDL> quick_interp_tdm2,1901,2006,'tmxglo/tmx.',750,gs=0.5,pts_prefix='tmxtxt/tmx.',dumpglo='dumpglo'

5599:#####

5600:Welcome! This is the GLO2ABS program.

5601:I will create a set of absolute grids from

5602:a set of anomaly grids (in .glo format), also

5603:a gridded version of the climatology.

5604:Enter the path and name of the normals file: clim.6190.lan.tmx

5605:Enter a name for the gridded climatology file: clim.6190.lan.tmx.grid

5606:Enter the path and stem of the .glo files: tmxglo/tmx.

5607:Enter the starting year: 1901

5608:Enter the ending year:   2006

5609:Enter the path (if any) for the output files: tmxabs

5610:Now, CONCENTRATE. Addition or Percentage (A/P)? A

5611:Right, erm.. off I jolly well go!

5612:tmx.01.1901.glo

5613:(etc)

5614:tmx.12.2006.glo

5615:#####

5616:Welcome! This is the MERGEGRIDS program.

5617:I will create decadal and full gridded files

5618:from the output files of (eg) glo2abs.for.

5619:Enter a gridfile with YYYY for year and MM for month: tmxabs/tmx.MM.YYYY.glo.abs

5620:Enter Start Year:  1901

5621:Enter Start Month: 01

5622:Enter End Year:    2006

5623:Enter End Month:   12

5624:Please enter a sample OUTPUT filename, replacing

5625:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.tmx.dat

5626:Writing cru_ts_3_00.1901.1910.tmx.dat

5627:(etc)

5628:

5629:

5630:This took longer than hoped.. running out of disk space again. This is why Tim didn't save more of

5631:the intermediate products - which would have made my detective work easier. The ridiculous process

5632:he adopted - and which we have dutifully followed - creates hundreds of intermediate files at every

5633:stage, none of which are automatically zipped/unzipped. Crazy. I've filled a 100gb disk!

5634:

5635:So, anyway, back on Earth I wrote wmocmp.for, a program to - you guessed it - compare WMO codes from

5636:a given set of databases.  Results were, ah.. 'interesting':

5637:

5638:

5639:REPORT:

5640:

5641:Database Title                Exact Match  Close Match  Vague Match  Awful Match  Codes Added      WMO = 0

5642:../db/pre/pre.0612181221.dtb          n/a          n/a          n/a          n/a        14397         1540

5643:../db/dtr/tmn.0708071548.dtb         1865         3389           57           77         5747         2519

5644:../db/tmp/tmp.0705101334.dtb            0            4           28          106         4927            0

5645:

5646:

5647:So the largest database, precip, contained 14397 stations with usable WMO codes (and 1540 without).

5648:The TMin, (and TMax and DTR, which were tested then excluded as they matched TMin 100%) database only agreed

5649:perfectly with precip for 1865 stations, nearby 3389, believable 57, worrying 77. TMean fared worse, with NO

5650:exact matches (WMO misformatting again) and over 100 worrying ones.

5651:

5652:The big story is the need to fix the tmean WMO codes. For instance:

5653:

5654:  10010   709    -87   10 Jan Mayen            NORWAY        1921 2006 341921  -999.00

5655:

5656:is illegal, and needs to become one of:

5657:  01001   709    -87   10 Jan Mayen            NORWAY        1921 2006 341921  -999.00

5658:0001001   709    -87   10 Jan Mayen            NORWAY        1921 2006 341921  -999.00

5659:0100100   709    -87   10 Jan Mayen            NORWAY        1921 2006 341921  -999.00

5660:

5661:I favour the first as it's technically accurate. Alternatively we seem to have widely adopted the third, which

5662:at least has the virtue of being consistent. Of course it's the only one that will match the precip:

5663:

5664: 100100  7093   -867   10 JAN MAYEN            NORWAY        1921 2006   -999  -999.00

5665: 

5666:..which itself should be either:

5667:

5668:0100100  7093   -867   10 JAN MAYEN            NORWAY        1921 2006   -999  -999.00

5669:

5670:or:

5671:

5672:  01001  7093   -867   10 JAN MAYEN            NORWAY        1921 2006   -999  -999.00

5673:

5674:Aaaaarrrggghhhh!!!!

5675:

5676:And the reason this is so important is that the incoming updates will rely PRIMARILY on matching the WMO codes!

5677:In fact CLIMAT bulletins carry no other identification, of course. Clearly I am going to need a reference set

5678:of 'qenuine WMO codes'.. and wouldn't you know it, I've found four!

5679:

5680:Location                                                N. Stations      Notes

5681:http://weather.noaa.gov/data/nsd_bbsss.txt              11548            Full country names, ';' delim

5682:http://www.htw-dresden.de/~kleist/wx_stations_ct.html   13000+           *10, leading zeros kept, fmt probs

5683:From Dave Lister                                        13080            *10 and leading zeros lost, country codes

5684:From Philip Brohan                                      11894            2+3, No countries

5685:

5686:The strategy is to use Dave Lister's list, grabbing country names from the Dresden list. Wrote

5687:getcountrycodes.for and extracted an imperfect but useful-as-a-reference list. Hopefully in the main the country

5688:will not need fixing or referring to!!

5689:

5690:Wrote 'fixwmos.for' - probably not for the first time, but it's the first prog of that name in my repository so I'll

5691:have to hope for the best. After an unreasonable amount of teething troubles (due to my forgetting that the tmp

5692:database stores lats & lons in degs*100 not degs*10, and also to the presence of a '-99999' as the lon for GUATEMALA

5693:in the reference set) I managed to sort-of fix the tmp database:

5694:

5695:

5696:crua6[/cru/cruts/version_3_0/db/tmp] ./fixwmos

5697:

5698:FIXWMOS - Fix WMO Codes in a Database

5699:

5700:Enter the database to be fixed: tmp.0705101334.dtb

5701:

5702:The operation completed successfully.

5703:

5704: 2263 WMO Codes were 'fixed' and all were rewritten as (i7.7)

5705:

5706:The output database is tmp.0709281456.dtb

5707:

5708:crua6[/cru/cruts/version_3_0/db/tmp] 

5709:

5710:

5711:The first records have changed as follows:

5712:

5713:crua6[/cru/cruts/version_3_0/db/tmp] diff tmp.0705101334.dtb tmp.0709281456.dtb |head -30

5714:1c1

5715:<   10010   709    -87   10 Jan Mayen            NORWAY        1921 2006 341921  -999.00

5716:---

5717:> 0100100   709    -87   10 Jan Mayen            NORWAY        1921 2006 341921  -999.00

5718:

5719:So far so good.. but records that weren't matched with the reference set didn't fare so well:

5720:

5721:89c89

5722:<   10050   780    142    9 ISFJORD RADIO        NORWAY        1912 1979 101912  -999.00

5723:---

5724:> 0010050   780    142    9 ISFJORD RADIO        NORWAY        1912 1979 101912  -999.00

5725:

5726:This is misleading because, although there probably won't BE any incoming updates for ISFJORD RADIO, we can't say for

5727:certain that there will never be updates for any station outside the current reference set. In fact, we can say with

5728:confidence that there will be!

5729:

5730:So, what to do? Do we assume a particular factor to adjust ALL codes by, based on the matches? Or do we attempt (note

5731:careful use of verb) to use the country codes database to work out the most significant 'real' digits of these codes?

5732:

5733:Well, I fancy the first one. We'll make two passes through the data, the first pass changes nothing but saves counts of

5734:the successful factors in bins: *0.01, *0.1, *1, *10, *100 should do it. I sure hope all the results are in one bin!

5735:

5736:It worked. An initial 'verbose' run showed a consistent choice of factor, though it'll exit with an error code if multiple

5737:factors are registered in one database.

5738:

5739:

5740:crua6[/cru/cruts/version_3_0/db/tmp] ./fixwmos 

5741:

5742:FIXWMOS - Fix WMO Codes in a Database

5743:

5744:Enter the database to be fixed: tmp.0705101334.dtb

5745:locfac set to: 10

5746:First ref: 0100100

5747:

5748:The operation completed successfully.

5749:

5750: 2263 WMO Codes were 'matched'

5751:All codes were modified with a factor of  10   

5752:Lons/lats were modified with a factor of  10

5753:

5754:The output database is tmp.0710011359.dtb

5755:

5756:crua6[/cru/cruts/version_3_0/db/tmp] 

5757:

5758:

5759:Example results:

5760:

5761:crua6[/cru/cruts/version_3_0/db/tmp] diff tmp.0705101334.dtb tmp.0710011359.dtb | head -12

5762:1c1

5763:<   10010   709    -87   10 Jan Mayen            NORWAY        1921 2006 341921  -999.00

5764:---

5765:> 0100100  7090   -870   10 Jan Mayen            NORWAY        1921 2006 341921  -999.00

5766:89c89

5767:<   10050   780    142    9 ISFJORD RADIO        NORWAY        1912 1979 101912  -999.00

5768:---

5769:> 0100500  7800   1420    9 ISFJORD RADIO        NORWAY        1912 1979 101912  -999.00

5770:159c159

5771:<   10080   783    155   28 Svalbard Lufthavn    NORWAY        1911 2006 341911  -999.00

5772:---

5773:> 0100800  7830   1550   28 Svalbard Lufthavn    NORWAY        1911 2006 341911  -999.00

5774:

5775:

5776:Then.. attacked the wet database! And immediately found this beauty:

5777:

5778:      0 -9999 -99999 -999 UNKNOWN              UNKNOWN       1994 2003   -999        0

5779:6190-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

5780:1994  500  800  600  400  600  100    0  100  200  400 1000 1300

5781:1995  400  100 1100  900 1200  800  200  100  200  400  800  500

5782:1996  500 1100 1500  600  900-9999    0  300  400  700    0 1100

5783:1997  800 1000  700 1000 1000 1000  200  200  400  700  200 1000

5784:1998  700  700 1000 1000-9999  800  100  100    0  200  400  700

5785:1999  300 1000  800-9999  700  800    0  200-9999  600  400  200

5786:2000 1100  600  900  900 1000  400-9999  100  200  300    0  400

5787:2001    0  800  300  500 1200    0    0    0  200  200  500  800

5788:2002  800  300  600 1300  800  500  400  100  300  400  400  600

5789:2003  300-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

5790:

5791:Gotta love the system! Like this is ever going to be a blind bit of use. Modified the code to

5792:leave such stations unmolested, but identified in a separate file so they can be 'cleansed', it

5793:being a little too risky to auto-cleanse such things.

5794:

5795:Hopefully the final attack on 'wet':

5796:

5797:

5798:crua6[/cru/cruts/version_3_0/db/rd0] ./fixwmos

5799:

5800:FIXWMOS - Fix WMO Codes in a Database

5801:

5802:Enter the database to be fixed: wet.0311061611.dtb

5803:

5804:The operation completed successfully.

5805:

5806: 1920 WMO Codes were 'matched'

5807:All codes were modified with a factor of  10   

5808:Lons/lats were modified with a factor of   1

5809:

5810:The output database is wet.0710021341.dtb

5811:

5812:

5813:IMPORTANT: the following WMO codes were not altered:

5814:False codes (wmo<0):          2917

5815:Illegal codes (0<=wmo<1000):     1

5816:  (illegals written to wet.0311061611.bad)

5817:crua6[/cru/cruts/version_3_0/db/rd0] 

5818:

5819:

5820:I then removed the sole illegal (see above) from wet.0710021341.dtb, which becomes the 'new old'

5821:wet/rd0 database.

5822:

5823:So.. to incorporate the updates! Finally. First, the MCDW, metadata-rich ones:

5824:

5825:

5826:uealogin1[/cru/cruts/version_3_0/db/rd0] ./newmergedb 

5827:

5828:WELCOME TO THE DATABASE UPDATER

5829:

5830:Before we get started, an important question:

5831:If you are merging an update - CLIMAT, MCDW, 

5832:ian - do

5833:you want the quick and dirty approach? This will blindly match

5834:on WMO codes alone, ignoring data/metadata checks, and making any

5835:unmatched updates into new stations (metadata permitting)?

5836:

5837:Enter 'B' for blind merging, or : B

5838:Please enter the Master Database name: wet.0710021341.dtb

5839:Please enter the Update Database name: rdy.0709111032.dtb

5840:

5841:Reading in both databases..

5842:Master database stations:     4987

5843:Update database stations:     2407

5844:

5845:Looking for WMO code matches..

5846:* new header 0100100  7056   -840    9 JAN MAYEN            NORWAY        1990 2007    -999    -999 *

5847:    2 reject(s) from update process 0710041559

5848:

5849:Writing wet.0710041559.dtb

5850:

5851:OUTPUT(S) WRITTEN

5852:

5853:New master database: wet.0710041559.dtb

5854:

5855:Update database stations:         2407

5856: > Matched with Master stations:  1556

5857:                 (automatically:  1556)

5858:                   (by operator:     0)

5859: > Added as new Master stations:     0

5860: > Rejected:                         2

5861:   Rejects file:                 rdy.0709111032.dtb.rejected

5862: Note: IEEE floating-point exception flags raised: 

5863:    Inexact;  Invalid Operation; 

5864: See the Numerical Computation Guide, ieee_flags(3M) 

5865:uealogin1[/cru/cruts/version_3_0/db/rd0] 

5866:

5867:

5868:(also knocked up rrstats.for at this stage, to analyse replication rates by

5869: latitude band for a given database - needs a Matlab prog to drive really)

5870:

5871:[a bit of debugging here as the last records weren't being written properly,

5872: filenames adjusted above accordingly]

5873:

5874:

5875:Then, the CLIMAT, nothing-but-the-code ones:

5876:

5877:*WARNING: ignore this, the CLIMAT bulletins were later improved with metadata and newmergedb rerun*

5878:

5879:

5880:uealogin1[/cru/cruts/version_3_0/db/rd0] ./newmergedb 

5881:

5882:WELCOME TO THE DATABASE UPDATER

5883:

5884:Before we get started, an important question:

5885:If you are merging an update - CLIMAT, MCDW, Australian - do

5886:you want the quick and dirty approach? This will blindly match

5887:on WMO codes alone, ignoring data/metadata checks, and making any

5888:unmatched updates into new stations (metadata permitting)?

5889:

5890:Enter 'B' for blind merging, or : B

5891:Please enter the Master Database name: wet.0710041559.dtb

5892:Please enter the Update Database name: rdy.0709101706.dtb

5893:

5894:Reading in both databases..

5895:Master database stations:     5836

5896:Update database stations:     2876

5897:

5898:Looking for WMO code matches..

5899:  378 reject(s) from update process 0710081508

5900:

5901:Writing wet.0710081508.dtb

5902:

5903:OUTPUT(S) WRITTEN

5904:

5905:New master database: wet.0710081508.dtb

5906:

5907:Update database stations:         2876

5908: > Matched with Master stations:  2498

5909:                 (automatically:  2498)

5910:                   (by operator:     0)

5911: > Added as new Master stations:     0

5912: > Rejected:                       378

5913:   Rejects file:                 rdy.0709101706.dtb.rejected

5914: Note: IEEE floating-point exception flags raised: 

5915:    Inexact;  Invalid Operation; 

5916: See the Numerical Computation Guide, ieee_flags(3M) 

5917:uealogin1[/cru/cruts/version_3_0/db/rd0] 

5918:

5919:

5920:Now of course, we can't add any of the CLIMAT bulletin stations as 'new' stations

5921:because we don't have any metadata! so.. is it worth using the lookup table? Because

5922:although I'm thrilled at the high match rate (87%!), it does seem worse when you

5923:realise that you lost the rest..

5924:

5925:* see below, CLIMAT metadata fixed! *

5926:

5927:At this stage I knocked up rrstats.for and the visualisation companion tool, cmprr.m. A simple process

5928:to show station counts against time for each 10-degree latitude band (with 20-degree bands at the

5929:North and South extremities). A bit basic and needs more work - but good for a quick & dirty check.

5930:

5931:Wrote dllist2headers.for to convert the 'Dave Lister' WMO list to CRU header format - the main difficulty

5932:being the accurate conversion of the two-character 'country codes' - especially since many are actually

5933:state codes for the US! Ended up with wmo.0710151633.dat as our reference WMO set.

5934:

5935:Incorporated the reference WMO set into climat2cru.for. Successfully reprocessed the CLIMAT bulletins

5936:into databases with at least SOME metadata:

5937:

5938:pre.0710151817.dtb

5939:rdy.0710151817.dtb

5940:sun.0710151817.dtb

5941:tmn.0710151817.dtb

5942:tmp.0710151817.dtb

5943:tmx.0710151817.dtb

5944:vap.0710151817.dtb

5945:

5946:In fact, it was far more successful than I expected - only 11 stations out of 2878 without metadata!

5947:

5948:Re-ran newmergedb:

5949:

5950:

5951:uealogin1[/cru/cruts/version_3_0/db/rd0] ./newmergedb

5952:

5953:WELCOME TO THE DATABASE UPDATER

5954:

5955:Before we get started, an important question:

5956:If you are merging an update - CLIMAT, MCDW, Australian - do

5957:you want the quick and dirty approach? This will blindly match

5958:on WMO codes alone, ignoring data/metadata checks, and making any

5959:unmatched updates into new stations (metadata permitting)?

5960:

5961:Enter 'B' for blind merging, or : B

5962:Please enter the Master Database name: wet.0710041559.dtb

5963:Please enter the Update Database name: rdy.0710151817.dtb

5964:

5965:Reading in both databases..

5966:Master database stations:     5836

5967:Update database stations:     2876

5968:

5969:Looking for WMO code matches..

5970:   71 reject(s) from update process 0710161148

5971:

5972:Writing wet.0710161148.dtb

5973:

5974:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

5975:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

5976:

5977:OUTPUT(S) WRITTEN

5978:

5979:New master database: wet.0710161148.dtb

5980:

5981:Update database stations:         2876

5982: > Matched with Master stations:  2498

5983:                 (automatically:  2498)

5984:                   (by operator:     0)

5985: > Added as new Master stations:   307

5986: > Rejected:                        71

5987:   Rejects file:                 rdy.0710151817.dtb.rejected

5988: Note: IEEE floating-point exception flags raised: 

5989:    Inexact;  Invalid Operation; 

5990: See the Numerical Computation Guide, ieee_flags(3M) 

5991:uealogin1[/cru/cruts/version_3_0/db/rd0] 

5992:

5993:

5994:307 stations rescued! and they'll be there in future of course, for metadata-free CLIMAT bulletins

5995:to match with.

5996:

5997:So where were we.. Rain Days. Family tree:

5998:

5999:wet.0311061611.dtb

6000:        +

6001:rdy.0709111032.dtb  (MCDW composite)

6002:        +

6003:rdy.0710151817.dtb  (CLIMAT composite with metadata added)

6004:        V

6005:        V

6006:wet.0710161148.dtb

6007:

6008:

6009:Now it gets tough. The current model for a secondary is that it is derived from one or more primaries,

6010:plus their normals, plus the normals for the secondary.

6011:

6012:The IDL secondary generators do not allow 'genuine' secondary data to be incorporated. This would have

6013:been ideal, as the gradual increase in observations would have gradually taken precedence over the

6014:primary-derived synthetics.

6015:

6016:The current stats for the wet database were derived from the new proglet, dtbstats.for:

6017:

6018:

6019:crua6[/cru/cruts/version_3_0/secondaries/rd0] ./dtbstat

6020:

6021:DTBSTAT: Database Stats Report

6022:

6023:Please enter the (18ch.) database name: wet.0710161148.dtb

6024:

6025:Report for: wet.0710161148.dtb

6026:

6027:Stations in Northern Hemisphere:     5365

6028:Stations in Southern Hemisphere:      778

6029:                          Total:     6143

6030:

6031:Maximum Timespan in Northern Hemisphere: 1840 to 2007

6032:Maximum Timespan in Southern Hemisphere: 1943 to 2007

6033:                        Global Timespan: 1840 to 2007

6034:

6035:crua6[/cru/cruts/version_3_0/secondaries/rd0] 

6036:

6037:

6038:So, without further ado, I treated RD0 as a Primary and derived gridded output from the database:

6039:

6040:

6041:crua6[/cru/cruts/version_3_0/secondaries/rd0] ./anomdtb

6042: 

6043:   > ***** AnomDTB: converts .dtb to anom .txt for gridding *****

6044: 

6045:   > Enter the suffix of the variable required:

6046:.rd0

6047:   > Select the .cts or .dtb file to load:

6048:wet.0710161148.dtb   

6049:   > Specify the start,end of the normals period: 

6050:1961,1990

6051:   > Specify the missing percentage permitted: 

6052:25

6053:   > Data required for a normal:           23

6054:   > Specify the no. of stdevs at which to reject data: 

6055:3

6056:   > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 

6057:3

6058:   > Check for duplicate stns after anomalising? (0=no,>0=km range)

6059:0

6060:   > Select the generic .txt file to save (yy.mm=auto):

6061:rd0.txt

6062:   > Select the first,last years AD to save: 

6063:1901,2007

6064:   > Operating...

6065:

6066:   > NORMALS            MEAN percent      STDEV percent

6067:   >         .dtb          0     0.0

6068:   >         .cts     731118    45.4     730956    45.4

6069:   > PROCESS        DECISION percent %of-chk

6070:   > no lat/lon            0     0.0     0.0

6071:   > no normal        878015    54.6    54.6

6072:   > out-of-range         56     0.0     0.0

6073:   > accepted         731062    45.4

6074:   > Dumping years 1901-2007 to .txt files...

6075: 

6076:crua6[/cru/cruts/version_3_0/secondaries/rd0]

6077:

6078:

6079:Not particularly good - the bulk of the data being recent, less than half had valid normals (anomdtb

6080:calculates normals on the fly, on a per-month basis). However, this isn't so much of a problem as the

6081:plan is to screen it for valid station contributions anyway.

6082:

6083:

6084:IDL> quick_interp_tdm2,1901,2007,'rd0glo/rd0.',450,gs=0.5,dumpglo='dumpglo',pts_prefix='rd0txt/rd0.'

6085:% Compiled module: QUICK_INTERP_TDM2.

6086:% Compiled module: GLIMIT.

6087:Defaults set

6088:    1901

6089:% Compiled module: MAP_SET.

6090:% Compiled module: CROSSP.

6091:% Compiled module: STRIP.

6092:% Compiled module: SAVEGLO.

6093:% Compiled module: SELECTMODEL.

6094:    1902

6095:(etc)

6096:    2007

6097:no stations found in: rd0txt/rd0.2007.08.txt

6098:no stations found in: rd0txt/rd0.2007.09.txt

6099:no stations found in: rd0txt/rd0.2007.10.txt

6100:no stations found in: rd0txt/rd0.2007.11.txt

6101:no stations found in: rd0txt/rd0.2007.12.txt

6102:IDL> 

6103:

6104:

6105:

6106:crua6[/cru/cruts/version_3_0/secondaries/rd0] ./glo2abs

6107:Welcome! This is the GLO2ABS program.

6108:I will create a set of absolute grids from

6109:a set of anomaly grids (in .glo format), also

6110:a gridded version of the climatology.

6111:Enter the path and name of the normals file: clim.6190.lan.wet

6112:Enter a name for the gridded climatology file: clim.6190.lan.wet.grid2

6113:Enter the path and stem of the .glo files: rd0glo/rd0.

6114:Enter the starting year: 1901

6115:Enter the ending year:   2007

6116:Enter the path (if any) for the output files: rd0abs/

6117:Now, CONCENTRATE. Addition or Percentage (A/P)? A         ! this was a guess! We'll see how the results look

6118:Right, erm.. off I jolly well go!

6119:rd0.01.1901.glo

6120:(etc)

6121:

6122:

6123:Then.. wait a minute! I checked back, and sure enough, quick_interp_tdm.pro DOES allow both synthetic and 'real' data

6124:to be included in the gridding. From the program description:

6125:

6126:

6127:; TDM: the dummy grid points default to zero, but if the synth_prefix files are present in call,

6128:;  the synthetic data from these grids are read in and used instead

6129:

6130:

6131:And so.. (after some confusion, and renaming so that anomdtb selects percentage anomalies)..

6132:

6133:IDL> quick_interp_tdm2,1901,2006,'rd0pcglo/rd0pc',450,gs=0.5,dumpglo='dumpglo',synth_prefix='rd0syn/rd0syn',pts_prefix='rd0pctxt/rd0pc.'  

6134:

6135:The trouble is, we won't be able to produce reliable station count files this way. Or can we use the same strategy,

6136:producing station counts from the wet database route, and filling in 'gaps' with the precip station counts? Err.

6137:

6138:

6139:crua6[/cru/cruts/version_3_0/secondaries/rd0] ./glo2abs

6140:Welcome! This is the GLO2ABS program.

6141:I will create a set of absolute grids from

6142:a set of anomaly grids (in .glo format), also

6143:a gridded version of the climatology.

6144:Enter the path and name of the normals file: clim.6190.lan.wet

6145:Enter a name for the gridded climatology file: clim.grid

6146:Enter the path and stem of the .glo files: rd0pcglo/rd0pc.

6147:Enter the starting year: 1901

6148:Enter the ending year:   2006

6149:Enter the path (if any) for the output files: rd0pcgloabs/

6150:Now, CONCENTRATE. Addition or Percentage (A/P)? P

6151:Right, erm.. off I jolly well go!

6152:rd0pc.01.1901.glo

6153:(etc)

6154:

6155:

6156:

6157:crua6[/cru/cruts/version_3_0/secondaries/rd0] ./mergegrids 

6158:Welcome! This is the MERGEGRIDS program.

6159:I will create decadal and full gridded files

6160:from the output files of (eg) glo2abs.for.

6161:

6162:Enter a gridfile with YYYY for year and MM for month: rd0pcgloabs/rd0pc.MM.YYYY.glo.abs

6163:Enter Start Year:  1901

6164:Enter Start Month: 01

6165:Enter End Year:    2006

6166:Enter End Month:   12

6167:

6168:Please enter a sample OUTPUT filename, replacing

6169:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.rd0.dat

6170:Writing cru_ts_3_00.1901.1910.rd0.dat

6171:Writing cru_ts_3_00.1911.1920.rd0.dat

6172:Writing cru_ts_3_00.1921.1930.rd0.dat

6173:Writing cru_ts_3_00.1931.1940.rd0.dat

6174:Writing cru_ts_3_00.1941.1950.rd0.dat

6175:Writing cru_ts_3_00.1951.1960.rd0.dat

6176:Writing cru_ts_3_00.1961.1970.rd0.dat

6177:Writing cru_ts_3_00.1971.1980.rd0.dat

6178:Writing cru_ts_3_00.1981.1990.rd0.dat

6179:Writing cru_ts_3_00.1991.2000.rd0.dat

6180:Writing cru_ts_3_00.2001.2006.rd0.dat

6181:crua6[/cru/cruts/version_3_0/secondaries/rd0]

6182:

6183:

6184:All according to plan.. except the values themselves!

6185:

6186:For January, 2001:

6187:

6188:Minimum      =      0

6189:Maximum      =  32630

6190:Vals >31000  =      1

6191:

6192:For the whole of 2001:

6193:

6194:Minimum      =      0

6195:Maximum      =  56763

6196:Vals >31000  =      5

6197:

6198:Not good. We're out by a factor of at least 10, though the extremes are few enough to just cap at DiM. So where has

6199:this factor come from?

6200:

6201:Well here's the January 2001 climatology:

6202:

6203:Minimum      =      0

6204:Maximum      =   3050

6205:Vals >3100   =      0

6206:

6207:That all seems fine for a percentage normals set. Not entirly sure about 0 though.

6208:

6209:so let's look at the January 2001 gridded anomalies file:

6210:

6211:Minimum      =    -48.046

6212:Maximum      =      0.0129

6213:

6214:This leads to a show-stopper, I'm afraid. It looks as though the calculation I'm using for percentage anomalies is,

6215:not to put too fine a point on it, cobblers.

6216:

6217:This is what I use to build actuals from anomalies in glo2abs.for:

6218:

6219:              absgrid(ilon(i),ilat(i)) = nint(normals(i,imo) +

6220:     *                  anoms(ilon(i),ilat(i)) * normals(i,imo) / 100)

6221:

6222:or, to put it another way, V = N(A+N)/100

6223:

6224:This is what anomdtb.f90 uses to build anomalies from actuals:

6225:

6226:                DataA(XAYear,XMonth,XAStn) = nint(1000.0*((real(DataA(XAYear,XMonth,XAStn)) / &

6227:                				real(NormMean(XMonth,XAStn)))-1.0))

6228:

6229:or, in the same terms, A = 1000((V/N)-1)

6230:

6231:which reverses to: V = N(A+1000)/1000

6232:

6233:This could well explain things. It could also mean that I have to reproduce v3.00 precip AFTER it's been used (against

6234:my wishes) by Dave L and Dimitrious.

6235:

6236:Well to start with, I'll try the new calculation in glo2abs to reproduce the rd0 data.

6237:

6238:

6239:crua6[/cru/cruts/version_3_0/secondaries/rd0] ./glo2abs

6240:Welcome! This is the GLO2ABS program.

6241:I will create a set of absolute grids from

6242:a set of anomaly grids (in .glo format), also

6243:a gridded version of the climatology.

6244:Enter the path and name of the normals file: clim.6190.lan.wet

6245:Enter a name for the gridded climatology file: c.grid

6246:Enter the path and stem of the .glo files: rd0pcglo/rd0pc.

6247:Enter the starting year: 1901

6248:Enter the ending year:   2006

6249:Enter the path (if any) for the output files: rd0pcgloabs

6250:Now, CONCENTRATE. Addition or Percentage (A/P)? P

6251:Right, erm.. off I jolly well go!

6252:rd0pc.01.1901.glo

6253:(etc)

6254:

6255:

6256:This *does* improve matters considerably. Now, for January 2001:

6257:

6258:Minimum      =      0

6259:Maximum      =   5090  (a little high but not fatal)

6260:Vals >3100   =    556

6261:Vals >3500   =    110

6262:Vals >4000   =      2  (so the bulk of the excessions are only a few days over)

6263:

6264:In fact the 2nd highest Max is 4369, well below 5090.

6265:

6266:So, good news - but only in the sense that I've found the error. Bad news in that it's a further confirmation that my

6267:abilities are short of what's required here.

6268:

6269:Rushed back to precip. Found the .glo files in /cru/cruts/version_3_0/primaries/precip/pre0km0612181221glo/, and

6270:re-ran glo2abs with the revised percentage anomaly equation:

6271:

6272:

6273:crua6[/cru/cruts/version_3_0/primaries/precip] ./glo2abs

6274:Welcome! This is the GLO2ABS program.

6275:I will create a set of absolute grids from

6276:a set of anomaly grids (in .glo format), also

6277:a gridded version of the climatology.

6278:Enter the path and name of the normals file: clim.6190.lan.pre

6279:Enter a name for the gridded climatology file: clim.6190.lan.pre.gridded2

6280:Enter the path and stem of the .glo files: pre0km0612181221glo/pregrid.

6281:Enter the starting year: 1901

6282:Enter the ending year:   2006

6283:Enter the path (if any) for the output files: pre0km0612181221abs/

6284:Now, CONCENTRATE. Addition or Percentage (A/P)? P

6285:Right, erm.. off I jolly well go!

6286:pregrid.01.1901.glo

6287:(etc)

6288:

6289:

6290:

6291:crua6[/cru/cruts/version_3_0/primaries/precip] ./mergegrids 

6292:Welcome! This is the MERGEGRIDS program.

6293:I will create decadal and full gridded files

6294:from the output files of (eg) glo2abs.for.

6295:

6296:Enter a gridfile with YYYY for year and MM for month: pre0km0612181221abs/pregrid.MM.YYYY.glo.abs

6297:Enter Start Year:  1901

6298:Enter Start Month: 01

6299:Enter End Year:    2006

6300:Enter End Month:   12

6301:

6302:Please enter a sample OUTPUT filename, replacing

6303:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.pre.dat

6304:Writing cru_ts_3_00.1901.1910.pre.dat

6305:Writing cru_ts_3_00.1911.1920.pre.dat

6306:Writing cru_ts_3_00.1921.1930.pre.dat

6307:Writing cru_ts_3_00.1931.1940.pre.dat

6308:Writing cru_ts_3_00.1941.1950.pre.dat

6309:Writing cru_ts_3_00.1951.1960.pre.dat

6310:Writing cru_ts_3_00.1961.1970.pre.dat

6311:Writing cru_ts_3_00.1971.1980.pre.dat

6312:Writing cru_ts_3_00.1981.1990.pre.dat

6313:Writing cru_ts_3_00.1991.2000.pre.dat

6314:Writing cru_ts_3_00.2001.2006.pre.dat

6315:crua6[/cru/cruts/version_3_0/primaries/precip]

6316:

6317:

6318:Then back to finish off rd0. Modified glo2abs to allow the operator to set minima and maxima, with a

6319:specific option to set wet day limits (DiM*100):

6320:

6321:

6322:crua6[/cru/cruts/version_3_0/secondaries/rd0] ./glo2abs

6323:Welcome! This is the GLO2ABS program.

6324:I will create a set of absolute grids from

6325:a set of anomaly grids (in .glo format), also

6326:a gridded version of the climatology.

6327:Enter the path and name of the normals file: clim.6190.lan.wet

6328:Enter a name for the gridded climatology file: clim...grid

6329:Enter the path and stem of the .glo files: rd0pcglo/rd0pc.

6330:Enter the starting year: 1901

6331:Enter the ending year:   2006

6332:Enter the path (if any) for the output files: rd0pcgloabs/

6333:Now, CONCENTRATE. Addition or Percentage (A/P)? P

6334:Do you wish to limit the output values? (Y/N): Y

6335:1. Set minimum to zero

6336:2. Set a single minimum and maximum

6337:3. Set monthly minima and maxima (for wet/rd0)

6338:

6339:Choose: 3

6340:Right, erm.. off I jolly well go!

6341:rd0pc.01.1901.glo

6342:(etc)

6343:

6344:

6345:Output was checked.. and as expected, January 2001 had 556 values of 3100 :-)

6346:

6347:

6348:crua6[/cru/cruts/version_3_0/secondaries/rd0] ./mergegrids 

6349:Welcome! This is the MERGEGRIDS program.

6350:I will create decadal and full gridded files

6351:from the output files of (eg) glo2abs.for.

6352:

6353:Enter a gridfile with YYYY for year and MM for month: rd0pcgloabs/rd0pc.MM.YYYY.glo.abs

6354:Enter Start Year:  1901

6355:Enter Start Month: 01

6356:Enter End Year:    2006

6357:Enter End Month:   12

6358:

6359:Please enter a sample OUTPUT filename, replacing

6360:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.rd0.dat

6361:Writing cru_ts_3_00.1901.1910.rd0.dat

6362:Writing cru_ts_3_00.1911.1920.rd0.dat

6363:Writing cru_ts_3_00.1921.1930.rd0.dat

6364:Writing cru_ts_3_00.1931.1940.rd0.dat

6365:Writing cru_ts_3_00.1941.1950.rd0.dat

6366:Writing cru_ts_3_00.1951.1960.rd0.dat

6367:Writing cru_ts_3_00.1961.1970.rd0.dat

6368:Writing cru_ts_3_00.1971.1980.rd0.dat

6369:Writing cru_ts_3_00.1981.1990.rd0.dat

6370:Writing cru_ts_3_00.1991.2000.rd0.dat

6371:Writing cru_ts_3_00.2001.2006.rd0.dat

6372:crua6[/cru/cruts/version_3_0/secondaries/rd0]

6373:

6374:

6375:Back to where this all started - Vapour Pressure.

6376:

6377:We have:

6378:

6379:1. 'Master' (ie original) database                    vap.0311181410.dtb

6380:2. MCDW updates database                              vap.0709111032.dtb

6381:3. CLIMAT updates database *with added metadata*      vap.0710151817.dtb

6382:

6383:so first we incorporate the MCDW updates..

6384:

6385:

6386:uealogin1[/cru/cruts/version_3_0/db/vap] ./newmergedb 

6387:

6388:WELCOME TO THE DATABASE UPDATER

6389:

6390:Before we get started, an important question:

6391:If you are merging an update - CLIMAT, MCDW, Australian - do

6392:you want the quick and dirty approach? This will blindly match

6393:on WMO codes alone, ignoring data/metadata checks, and making any

6394:unmatched updates into new stations (metadata permitting)?

6395:

6396:Enter 'B' for blind merging, or : B

6397:Please enter the Master Database name: vap.0311181410.dtb

6398:Please enter the Update Database name: vap.0709111032.dtb

6399:

6400:Reading in both databases..

6401:Master database stations:     7691

6402:Update database stations:     2398

6403:

6404:Looking for WMO code matches..

6405:    2 reject(s) from update process 0710241541

6406:

6407:Writing vap.0710241541.dtb

6408:

6409:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

6410:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

6411:

6412:OUTPUT(S) WRITTEN

6413:

6414:New master database: vap.0710241541.dtb

6415:

6416:Update database stations:         2398

6417: > Matched with Master stations:  1847

6418:                 (automatically:  1847)

6419:                   (by operator:     0)

6420: > Added as new Master stations:   549

6421: > Rejected:                         2

6422:   Rejects file:                 vap.0709111032.dtb.rejected

6423:uealogin1[/cru/cruts/version_3_0/db/vap] 

6424:

6425:

6426:Then, the CLIMAT ones:

6427:

6428:

6429:uealogin1[/cru/cruts/version_3_0/db/vap] ./newmergedb

6430:

6431:WELCOME TO THE DATABASE UPDATER

6432:

6433:Before we get started, an important question:

6434:If you are merging an update - CLIMAT, MCDW, Australian - do

6435:you want the quick and dirty approach? This will blindly match

6436:on WMO codes alone, ignoring data/metadata checks, and making any

6437:unmatched updates into new stations (metadata permitting)?

6438:

6439:Enter 'B' for blind merging, or : B

6440:Please enter the Master Database name: vap.0710241541.dtb

6441:Please enter the Update Database name: vap.0710151817.dtb

6442:

6443:Reading in both databases..

6444:Master database stations:     8240

6445:Update database stations:     2870

6446:

6447:Looking for WMO code matches..

6448:   68 reject(s) from update process 0710241549

6449:

6450:Writing vap.0710241549.dtb

6451:

6452:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

6453:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

6454:

6455:OUTPUT(S) WRITTEN

6456:

6457:New master database: vap.0710241549.dtb

6458:

6459:Update database stations:         2870

6460: > Matched with Master stations:  2599

6461:                 (automatically:  2599)

6462:                   (by operator:     0)

6463: > Added as new Master stations:   203

6464: > Rejected:                        68

6465:   Rejects file:                 vap.0710151817.dtb.rejected

6466:uealogin1[/cru/cruts/version_3_0/db/vap] 

6467:

6468:

6469:So, not as good as the MCDW update.. lost 68.. but then of course we are talking about station data that

6470:arrived with NO metadata AT ALL.

6471:

6472:So we will try the unaltered rd0 process on vap. It should be the same; a mix of synthetic and observed.

6473:

6474:*************************************************************************************

6475:* PRIORITY INTERRUPT * PRIORITY INTERRUPT * PRIORITY INTERRUPT * PRIORITY INTERRUPT * 

6476:*************************************************************************************

6477:

6478:After an email enquiry from Wladimir J. Alonso (alonsow@mail.nih.gov), in which unusual behaviour of CRU TS 2.10

6479:Vapour Pressure data was observed, I discovered that some of the Wet Days and Vepour Pressure datasets had been

6480:swapped!!

6481:

6482:The files I was looking at were decadal, 1981-1990.

6483:

6484:Vapour Pressure, January:     Min 0       Max 310

6485:Vapour Pressure, February:    Min 0       Max 280

6486:

6487:Wet Days, January:            Min 0       Max 3220

6488:Wet days, February:           Min 0       Max 3240

6489:

6490:So I wrote crutsstats.for, whioch returns monthly and annual minima, maxima and means for any gridded output file.

6491:

6492:Tried it on the full runs, and they look OK:

6493:

6494:crua6[/cru/cruts/vap_wet_investigation] head -90 cru_ts_2_10.1901-2002.vap.grid.stats |tail -10

6495:1981       0     322      82       0     324      84       0     320      90       0     335      99       0     352     111       0     356     130       0     349     144       0     344     143       0     360     124       0     323     105       0     320      90       0     321      83       0     360     107

6496:1982       0     312      80       0     323      83       0     318      88       0     329      98       0     348     111       0     357     126       0     365     143       0     364     140       0     355     124       0     318     105       0     321      90       0     318      84       0     365     106

6497:1983       0     348      82       0     340      85       0     330      90       0     505      99       0     348     112       0     364     130       0     360     145       0     362     143       0     368     126       0     323     105       0     318      91       0     317      82       0     505     108

6498:1984       0     312      80       0     320      82       0     315      89       0     329      97       0     347     112       0     359     130       0     353     144       0     343     140       0     353     122       0     324     105       0     318      89       0     316      81       0     359     106

6499:1985       0     314      80       0     320      81       0     319      88       0     359      98       0     352     111       0     367     128       0     358     141       0     355     141       0     353     123       0     323     105       0     322      90       0     323      82       0     367     106

6500:1986       0     312      81       0     330      83       0     316      89       0     321      99       0     366     112       0     394     129       0     371     143       0     342     139       0     354     122       0     323     104       0     318      90       0     316      82       0     394     106

6501:1987       0     320      81       0     318      85       0     318      88       0     335      98       0     363     112       0     366     130       0     397     147       0     356     142       0     354     126       0     345     105       0     325      91       0     365      84       0     397     107

6502:1988       0     413      83       0     324      84       0     352      90       0     323      99       0     346     113       0     363     131       0     367     148       0     358     144       0     387     126       0     342     105       0     320      89       0     315      83       0     413     108

6503:1989       0     336      80       0     320      83       0     327      90       0     324      98       0     343     112       0     366     130       0     365     145       0     349     142       0     353     124       0     323     105       0     324      90       0     332      84       0     366     107

6504:1990       0     320      83       0     323      85       0     476      92       0     413     101       0     361     113       0     363     132       0     371     146       0     353     143       0     371     124       0     327     106       0     318      93       0     317      84       0     476     108

6505:

6506:crua6[/cru/cruts/vap_wet_investigation] head -90 cru_ts_2_10.1901-2002.wet.grid.stats | tail -10

6507:1981       0    3100    1018       0    2800     919       0    3100     980       0    3000     911       0    3100     945       0    3000    1010       0    3100    1051       0    3100    1040       0    3000     981       0    3100    1017       0    3000    1021       0    3100    1003       0    3100     992

6508:1982       0    3100     983       0    2800     894       0    3100     967       0    3000     925       0    3100     927       0    3000     941       0    3100     979       0    3100    1054       0    3000    1007       0    3100    1055       0    3000     996       0    3100    1044       0    3100     981

6509:1983       0    3100    1035       0    2800     863       0    3100     941       0    3000     919       0    3100     929       0    3000     949       0    3100     990       0    3100    1039       0    3000     996       0    3100    1026       0    3000    1034       0    3100    1057       0    3100     982

6510:1984       0    3100     981       0    2900     848       0    3100     920       0    3000     841       0    3100     932       0    3000     973       0    3100    1048       0    3100    1057       0    3000    1023       0    3100    1057       0    3000     992       0    3100    1016       0    3100     974

6511:1985       0    3100     969       0    2800     896       0    3100     952       0    3000     896       0    3100     928       0    3000     938       0    3100    1057       0    3100    1043       0    3000     993       0    3100    1043       0    3000    1066       0    3100    1029       0    3100     984

6512:1986       0    3100     988       0    2800     908       0    3100     950       0    3000     895       0    3100     922       0    3000     962       0    3100    1022       0    3100    1052       0    3000    1037       0    3100    1052       0    3000    1048       0    3100     986       0    3100     985

6513:1987       0    3100    1011       0    2800     909       0    3100     930       0    3000     856       0    3100     954       0    3000     972       0    3100    1021       0    3100    1064       0    3000     978       0    3100     991       0    3000    1002       0    3100    1047       0    3100     978

6514:1988       0    3100    1033       0    2900     924       0    3100     971       0    3000     903       0    3100     938       0    3000     980       0    3100    1039       0    3100    1101       0    3000    1014       0    3100    1017       0    3000    1007       0    3100    1054       0    3100     998

6515:1989       0    3100    1019       0    2800     936       0    3100    1015       0    3000     892       0    3100     978       0    3000    1020       0    3100    1054       0    3100    1075       0    3000    1023       0    3100    1070       0    3000    1046       0    3100    1053       0    3100    1015

6516:1990       0    3100     996       0    2800     959       0    3100    1011       0    3000     953       0    3100     928       0    3000     907       0    3100     983       0    3100     986       0    3000     915       0    3100     968       0    3000     949       0    3100     959       0    3100     960

6517:

6518:So the monthly maxima are fine here. But for the decadal files?

6519:

6520:crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.1981-1990.vap.grid.stats0

6521:1981       0     310     102       0     280      92       0     310      98       0     300      91       0     310      95       0     300     101       0     310     105       0     310     104       0     300      98       0     310     102       0     300     102       0     310     100       0     310      99

6522:1982       0     310      98       0     280      89       0     310      97       0     300      93       0     310      93       0     300      94       0     310      98       0     310     105       0     300     101       0     310     106       0     300     100       0     310     104       0     310      98

6523:1983       0     310     104       0     280      86       0     310      94       0     300      92       0     310      93       0     300      95       0     310      99       0     310     104       0     300     100       0     310     103       0     300     103       0     310     106       0     310      98

6524:1984       0     310      98       0     290      85       0     310      92       0     300      84       0     310      93       0     300      97       0     310     105       0     310     106       0     300     102       0     310     106       0     300      99       0     310     102       0     310      97

6525:1985       0     310      97       0     280      90       0     310      95       0     300      90       0     310      93       0     300      94       0     310     106       0     310     104       0     300      99       0     310     104       0     300     107       0     310     103       0     310      98

6526:1986       0     310      99       0     280      91       0     310      95       0     300      90       0     310      92       0     300      96       0     310     102       0     310     105       0     300     104       0     310     105       0     300     105       0     310      99       0     310      99

6527:1987       0     310     101       0     280      91       0     310      93       0     300      86       0     310      95       0     300      97       0     310     102       0     310     106       0     300      98       0     310      99       0     300     100       0     310     105       0     310      98

6528:1988       0     310     103       0     290      92       0     310      97       0     300      90       0     310      94       0     300      98       0     310     104       0     310     110       0     300     101       0     310     102       0     300     101       0     310     105       0     310     100

6529:1989       0     310     102       0     280      94       0     310     101       0     300      89       0     310      98       0     300     102       0     310     105       0     310     107       0     300     102       0     310     107       0     300     105       0     310     105       0     310     101

6530:1990       0     310     100       0     280      96       0     310     101       0     300      95       0     310      93       0     300      91       0     310      98       0     310      99       0     300      91       0     310      97       0     300      95       0     310      96       0     310      96

6531:

6532:crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.1981-1990.wet.grid.stats

6533:1981       0    3220     819       0    3240     842       0    3200     903       0    3350     992       0    3520    1113       0    3560    1304       0    3490    1440       0    3440    1427       0    3600    1236       0    3230    1048       0    3200     898       0    3210     833       0    3600    1071

6534:1982       0    3120     801       0    3230     827       0    3180     881       0    3290     982       0    3480    1108       0    3570    1264       0    3650    1432       0    3640    1405       0    3550    1239       0    3180    1048       0    3210     901       0    3180     835       0    3650    1060

6535:1983       0    3480     820       0    3400     850       0    3300     898       0    5050     993       0    3480    1125       0    3640    1295       0    3600    1451       0    3620    1428       0    3680    1259       0    3230    1050       0    3180     912       0    3170     822       0    5050    1075

6536:1984       0    3120     803       0    3200     823       0    3150     887       0    3290     971       0    3470    1124       0    3590    1299       0    3530    1437       0    3430    1404       0    3530    1218       0    3240    1053       0    3180     894       0    3160     812       0    3590    1060

6537:1985       0    3140     803       0    3200     815       0    3190     882       0    3590     978       0    3520    1113       0    3670    1277       0    3580    1405       0    3550    1411       0    3530    1233       0    3230    1048       0    3220     900       0    3230     821       0    3670    1057

6538:1986       0    3120     809       0    3300     827       0    3160     889       0    3210     990       0    3660    1120       0    3940    1294       0    3710    1428       0    3420    1393       0    3540    1220       0    3230    1041       0    3180     895       0    3160     821       0    3940    1061

6539:1987       0    3200     810       0    3180     849       0    3180     880       0    3350     980       0    3630    1124       0    3660    1296       0    3970    1466       0    3560    1423       0    3540    1260       0    3450    1054       0    3250     910       0    3650     844       0    3970    1075

6540:1988       0    4130     829       0    3240     835       0    3520     902       0    3230     989       0    3460    1133       0    3630    1311       0    3670    1475       0    3580    1441       0    3870    1264       0    3420    1054       0    3200     889       0    3150     832       0    4130    1079

6541:1989       0    3360     804       0    3200     825       0    3270     898       0    3240     978       0    3430    1120       0    3660    1301       0    3650    1447       0    3490    1421       0    3530    1240       0    3230    1052       0    3240     900       0    3320     836       0    3660    1069

6542:1990       0    3200     827       0    3230     853       0    4760     918       0    4130    1005       0    3610    1127       0    3630    1322       0    3710    1462       0    3530    1428       0    3710    1236       0    3270    1062       0    3180     930       0    3170     844       0    4760    1084

6543:

6544:Much confusion! The orders of magnitude have changed to reflect the expected ranges - but the data have clearly been swapped!

6545:

6546:Another decade:

6547:

6548:crua6[/cru/cruts/vap_wet_investigation]cat cru_ts_2_10.1921-1930.vap.grid.stats

6549:1921       0     310     102       0     280      89       0     310     100       0     300      88       0     310      95       0     300      97       0     310     101       0     310     104       0     300     102       0     310     104       0     300      97       0     310     101       0     310      98

6550:1922       0     310      95       0     280      93       0     310      97       0     300      89       0     310      95       0     300      98       0     310     105       0     310     107       0     300      98       0     310     104       0     300     102       0     310     103       0     310      99

6551:1923       0     310     100       0     280      88       0     310      97       0     300      90       0     310      97       0     300      98       0     310     101       0     310     101       0     300     100       0     310     104       0     300     101       0     310     103       0     310      98

6552:1924       0     310      97       0     290      89       0     310      95       0     300      90       0     310      91       0     300      97       0     310     100       0     310     102       0     300     101       0     310     102       0     300     102       0     310     100       0     310      97

6553:1925       0     310      98       0     280      89       0     310      98       0     300      87       0     310      90       0     300      96       0     310     101       0     310     103       0     300     103       0     310     101       0     300     103       0     310     100       0     310      97

6554:1926       0     310      99       0     280      87       0     310      95       0     300      87       0     310      95       0     300      93       0     310     103       0     310     104       0     300      99       0     310     102       0     300     102       0     310     101       0     310      97

6555:1927       0     310      96       0     280      87       0     310      96       0     300      89       0     310      94       0     300      97       0     310     103       0     310     104       0     300     102       0     310     103       0     300     102       0     310      99       0     310      98

6556:1928       0     310      97       0     290      89       0     310      91       0     300      88       0     310      90       0     300      96       0     310     101       0     310     104       0     300      97       0     310      99       0     300      99       0     310      96       0     310      96

6557:1929       0     310      95       0     280      84       0     310      95       0     300      86       0     310      91       0     300      95       0     310     100       0     310     102       0     300      98       0     310     102       0     300      98       0     310      98       0     310      95

6558:1930       0     310      98       0     280      88       0     310      97       0     300      88       0     310      93       0     300      93       0     310      99       0     310     103       0     300      99       0     310     105       0     300     101       0     310      97       0     310      97

6559:crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.1921-1930.wet.grid.stats

6560:1921       0    3120     805       0    3190     814       0    3140     874       0    3210     969       0    3800    1106       0    3590    1289       0    3600    1439       0    3440    1390       0    3530    1220       0    3230    1032       0    3180     877       0    3160     824       0    3800    1053

6561:1922       0    3120     794       0    3220     813       0    3140     874       0    3210     971       0    3470    1104       0    3590    1280       0    3560    1420       0    3440    1387       0    3530    1211       0    3230    1025       0    3180     896       0    3140     812       0    3590    1049

6562:1923       0    3070     799       0    3140     808       0    3140     871       0    3210     947       0    3460    1082       0    3660    1276       0    3560    1410       0    3440    1392       0    3530    1222       0    3230    1048       0    3180     907       0    3160     826       0    3660    1049

6563:1924       0    3270     792       0    3230     817       0    3160     879       0    3340     955       0    3460    1094       0    3710    1264       0    3560    1415       0    3440    1386       0    3530    1228       0    3160    1034       0    3180     892       0    3140     806       0    3710    1047

6564:1925       0    3110     786       0    3190     815       0    3140     873       0    3210     966       0    3470    1084       0    3590    1253       0    3560    1408       0    3460    1397       0    3530    1231       0    3230    1025       0    3160     896       0    3220     828       0    3590    1047

6565:1926       0    3260     815       0    3290     842       0    3310     889       0    3310     957       0    3460    1085       0    3950    1266       0    3560    1406       0    3450    1402       0    3530    1237       0    3230    1042       0    3250     899       0    3150     811       0    3950    1054

6566:1927       0    3120     795       0    3300     822       0    3170     873       0    3360     959       0    3540    1096       0    3610    1271       0    3550    1424       0    3450    1390       0    3530    1233       0    3230    1053       0    3180     897       0    3280     814       0    3610    1052

6567:1928       0    3200     809       0    3240     823       0    3140     875       0    3400     963       0    3470    1095       0    3590    1263       0    3560    1425       0    3450    1397       0    3530    1228       0    3230    1039       0    3180     902       0    3160     824       0    3590    1054

6568:1929       0    3150     794       0    3190     802       0    3160     867       0    3310     950       0    3600    1084       0    3580    1250       0    3550    1399       0    3440    1385       0    3530    1218       0    3230    1049       0    3180     897       0    3160     806       0    3600    1042

6569:1930       0    3190     798       0    3190     824       0    3150     881       0    3210     965       0    3470    1099       0    3590    1276       0    3530    1424       0    3440    1409       0    3540    1220       0    3200    1042       0    3300     907       0    3280     829       0    3590    1056

6570:

6571:The same story. And the final two years:

6572:

6573:crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.2001-2002.vap.grid.stats 

6574:2001       0     310      87       0     280      84       0     310      90       0     300      81       0     310      87       0     300      93       0     310      95       0     310      95       0     300      89       0     310      95       0     300      95       0     310      87       0     310      90

6575:2002       0     310      91       0     280      85       0     310      92       0     300      83       0     310      88       0     300      89       0     310      93       0     310      94       0     300      92       0     310      93       0     300      88       0     310      86       0     310      90

6576:crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.2001-2002.wet.grid.stats

6577:2001       0    3320     834       0    3250     841       0    3180     913       0    3490    1010       0    3490    1147       0    4380    1323       0    3660    1487       0    5120    1466       0    3530    1266       0    3460    1088       0    3620     932       0    3410     843       0    5120    1096

6578:2002       0    3310     837       0    3390     863       0    3270     918       0    3370    1012       0    3930    1151       0    4140    1339       0    3750    1503       0    5110    1453       0    3530    1261       0    3310    1067       0    3470     922       0    3300     833       0    5110    1096

6579:

6580:It looks like a consistent problem: all the decadal VAp and WET files should be discarded, and only the 'full run' 1901-2002

6581:files used. But my theory that the error occurred when the 1901-2002 files were converted to decadal doesn't sound true now,

6582:because why would the precision levels change? Surely, if the decadal files are derived from the 1901-2002 files, it's just

6583:a case of copying data across?

6584:

6585:Let's look at *just* 1981, to try and assess this issue:

6586:

6587:FULL 1901-2002 FILE

6588:VAP:

6589:1981       0     322      82       0     324      84       0     320      90       0     335      99       0     352     111       0     356     130       0     349     144       0     344     143       0     360     124       0     323     105       0     320      90       0     321      83       0     360     107

6590:WET:

6591:1981       0    3100    1018       0    2800     919       0    3100     980       0    3000     911       0    3100     945       0    3000    1010       0    3100    1051       0    3100    1040       0    3000     981       0    3100    1017       0    3000    1021       0    3100    1003       0    3100     992

6592:

6593:DECADAL 1981-1990 FILE

6594:VAP:

6595:1981       0     310     102       0     280      92       0     310      98       0     300      91       0     310      95       0     300     101       0     310     105       0     310     104       0     300      98       0     310     102       0     300     102       0     310     100       0     310      99

6596:WET:

6597:1981       0    3220     819       0    3240     842       0    3200     903       0    3350     992       0    3520    1113       0    3560    1304       0    3490    1440       0    3440    1427       0    3600    1236       0    3230    1048       0    3200     898       0    3210     833       0    3600    1071

6598:

6599:It's evident that the data have not only been swapped - they've been scaled too. Aaaarrrgghhhhhh!!!!!

6600:

6601:*******************************************************************************

6602:* PRIORITY INTERRUPT ENDS * PRIORITY INTERRUPT ENDS * PRIORITY INTERRUPT ENDS * 

6603:*******************************************************************************

6604:

6605:Now, where were we.. ah yes, Vapour Pressure. So far:

6606:

6607:Original:          vap.0311181410.dtb

6608:                            +

6609:MCDW:              vap.0709111032.dtb

6610:                            v

6611:                            v

6612:Intermediate:      vap.0710241541.dtb

6613:                            +

6614:CLIMAT:            vap.0710151817.dtb

6615:                            v

6616:                            v

6617:Final:             vap.0710241549.dtb

6618:

6619:Produce anomalies:

6620:

6621:

6622:crua6[/cru/cruts/version_3_0/secondaries/vap] ./anomdtb

6623: 

6624:   > ***** AnomDTB: converts .dtb to anom .txt for gridding *****

6625: 

6626:   > Enter the suffix of the variable required:

6627:.vap

6628:   > Select the .cts or .dtb file to load:

6629:vap.0710241549.dtb

6630:   > Specify the start,end of the normals period: 

6631:1961,1990

6632:   > Specify the missing percentage permitted: 

6633:25

6634:   > Data required for a normal:           23

6635:   > Specify the no. of stdevs at which to reject data: 

6636:3

6637:   > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 

6638:3

6639:   > Check for duplicate stns after anomalising? (0=no,>0=km range)

6640:0

6641:   > Select the generic .txt file to save (yy.mm=auto):

6642:vap.txt

6643:   > Select the first,last years AD to save: 

6644:1901,2006

6645:   > Operating...

6646:

6647:   > NORMALS            MEAN percent      STDEV percent

6648:   >         .dtb     908812    45.2

6649:   >         .cts      35390     1.8     944202    47.0

6650:   > PROCESS        DECISION percent %of-chk

6651:   > no lat/lon          105     0.0     0.0

6652:   > no normal       1064261    53.0    53.0

6653:   > out-of-range         49     0.0     0.0

6654:   > accepted         944153    47.0

6655:   > Dumping years 1901-2006 to .txt files...

6656: 

6657:crua6[/cru/cruts/version_3_0/secondaries/vap] 

6658:

6659:

6660:Well.. 47% accepted, 53% no normals.. pretty much as expected, and unlikely to improve no matter how many new CLIMAT

6661:and MCDW updates there are. We need back data for 1961-1990.

6662:

6663:Synthetic production:

6664:

6665:

6666:IDL> vap_gts_anom,dtr_prefix='../dtrbin/dtrbin',tmp_prefix='../tmpbin/tmpbin',1901,2006,outprefix='vapsyn/vapsyn',dumpbin=1             

6667:% Compiled module: VAP_GTS_ANOM.

6668:% Compiled module: RDBIN.

6669:% Compiled module: STRIP.

6670:% Compiled module: DEFXYZ.

6671:Land,sea:       56016       68400

6672:Calculating tmn normal

6673:% Compiled module: TVAP.

6674:Calculating synthetic vap normal

6675:% Compiled module: ESAT.

6676:Calculating synthetic anomalies

6677:% Compiled module: MOMENT.

6678:1901 vap (x,s2,<<,>>):  1.61250e-05  6.15570e-06    -0.160607     0.222689

6679:% Compiled module: WRBIN.

6680:1902 vap (x,s2,<<,>>): -0.000123188  3.46116e-05    -0.268891    0.0261283

6681:1903 vap (x,s2,<<,>>):  6.86689e-05  4.52675e-06    -0.121429     0.123995

6682:(etc)

6683:

6684:

6685:(also produced, vapsyn/vapsyn1901 .. vapsyn/vapsyn2006)

6686:

6687:Gridding with both observed and synthetic data:

6688:

6689:IDL> quick_interp_tdm2,1901,2006,'vapglo/vap.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='vapsyn/vapsyn',pts_prefix='vaptxt/vap.'

6690:

6691:Create absolute grids from anomaly grids:

6692:

6693:

6694:crua6[/cru/cruts/version_3_0/secondaries/vap] ./glo2abs

6695:Welcome! This is the GLO2ABS program.

6696:I will create a set of absolute grids from

6697:a set of anomaly grids (in .glo format), also

6698:a gridded version of the climatology.

6699:Enter the path and name of the normals file: clim.6190.lan.vap

6700:Enter a name for the gridded climatology file: clim.6190.lan.vap.grid

6701:Enter the path and stem of the .glo files: vapglo/vap.

6702:Enter the starting year: 1901

6703:Enter the ending year:   2006

6704:Enter the path (if any) for the output files: vapabs/

6705:Now, CONCENTRATE. Addition or Percentage (A/P)? A

6706:Do you wish to limit the output values? (Y/N): Y

6707:1. Set minimum to zero

6708:2. Set a single minimum and maximum

6709:3. Set monthly minima and maxima (for wet/rd0)

6710:

6711:Choose: 1

6712:Right, erm.. off I jolly well go!

6713:vap.01.1901.glo

6714:vap.02.1901.glo

6715:(etc)

6716:

6717:

6718:and finally, create the output files:

6719:

6720:

6721:crua6[/cru/cruts/version_3_0/secondaries/vap] ./mergegrids 

6722:Welcome! This is the MERGEGRIDS program.

6723:I will create decadal and full gridded files

6724:from the output files of (eg) glo2abs.for.

6725:

6726:Enter a gridfile with YYYY for year and MM for month: vapabs/vap.MM.YYYY.glo.abs

6727:Enter Start Year:  1901

6728:Enter Start Month: 01

6729:Enter End Year:    2006

6730:Enter End Month:   12

6731:

6732:Please enter a sample OUTPUT filename, replacing

6733:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.YYYY.vap.dat

6734:Try again.. read instructions this time?

6735:

6736:Please enter a sample OUTPUT filename, replacing

6737:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.dat

6738:Writing cru_ts_3_00.1901.1910.vap.dat

6739:Writing cru_ts_3_00.1911.1920.vap.dat

6740:Writing cru_ts_3_00.1921.1930.vap.dat

6741:Writing cru_ts_3_00.1931.1940.vap.dat

6742:Writing cru_ts_3_00.1941.1950.vap.dat

6743:Writing cru_ts_3_00.1951.1960.vap.dat

6744:Writing cru_ts_3_00.1961.1970.vap.dat

6745:Writing cru_ts_3_00.1971.1980.vap.dat

6746:Writing cru_ts_3_00.1981.1990.vap.dat

6747:Writing cru_ts_3_00.1991.2000.vap.dat

6748:Writing cru_ts_3_00.2001.2006.vap.dat

6749:

6750:

6751:Ah - and I was really hoping this time that it would just WORK. But of course not - nothing works first

6752:time in this project. I ran crutsstats on cru_ts_3_00.1901.2006.vap.dat, and:

6753:

6754:

6755:crua6[/cru/cruts/version_3_0/secondaries/vap] ./crutsstats 

6756:

6757:CRUTSSTATS: Stats for CRU TS gridded files

6758:

6759:Enter the monthly gridded data file: cru_ts_3_00.1901.2006.vap.dat

6760:

6761:Please enter the start year: 1901

6762:

6763: 106 years from 1901 to 2006

6764:

6765:Output file is cru_ts_3_00.1901.2006.vap.dat.stats

6766:1901       1     358     106

6767:1902       1     358     106

6768:1903       1     358     106

6769:1904       1     358     106

6770:1905       1     358     106

6771:(etc)

6772:2002       1     358     106

6773:2003       1     358     106

6774:2004       1     358     106

6775:2005       1     358     106

6776:2006       1     358     106

6777:

6778:

6779:What?! Every year has the same min (fine, VAP of 0 is probably impossible), max (I can just about believe,

6780:if there's a cell with no stations inside the cdd and the normal for it happens to be the highest value, and

6781:MEAN (oh no, NO WAY!). What's odder - the .glo files are different:

6782:

6783:crua6[/cru/cruts/version_3_0/secondaries/vap/vapabs] diff vap.06.1974.glo.abs.nh vap.06.1975.glo.abs.nh |wc -l

6784:        56

6785:

6786:Admittedly, 56 lines different out of 360 isn't hugely different. And looking, they are only slight and

6787:infrequent differences. But the monthly stats are all cloned as well:

6788:

6789:1901       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

6790:1902       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

6791:1903       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

6792:1904       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

6793:1905       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

6794:1906       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

6795:1907       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

6796:

6797:

6798:Well the first thing to do, after the inevitable wailing and gnashing of teeth, is to re-run glo2abs

6799:without the 'zero minimum' flag (just in case I coded that badly, I was in a hurry):

6800:

6801:

6802:crua6[/cru/cruts/version_3_0/secondaries/vap] ./glo2abs

6803:Welcome! This is the GLO2ABS program.

6804:I will create a set of absolute grids from

6805:a set of anomaly grids (in .glo format), also

6806:a gridded version of the climatology.

6807:Enter the path and name of the normals file: clim.6190.lan.vap

6808:Enter a name for the gridded climatology file: clim.6190.lan.vap.grid2

6809:Enter the path and stem of the .glo files: vapglo/vap.

6810:Enter the starting year: 1901

6811:Enter the ending year:   2006

6812:Enter the path (if any) for the output files: vapabs/

6813:Now, CONCENTRATE. Addition or Percentage (A/P)? A

6814:Do you wish to limit the output values? (Y/N): N

6815:Right, erm.. off I jolly well go!

6816:vap.01.1901.glo

6817:vap.02.1901.glo

6818:(etc)

6819:

6820:

6821:

6822:crua6[/cru/cruts/version_3_0/secondaries/vap] ./mergegrids 

6823:Welcome! This is the MERGEGRIDS program.

6824:I will create decadal and full gridded files

6825:from the output files of (eg) glo2abs.for.

6826:

6827:Enter a gridfile with YYYY for year and MM for month: vapabs/vap.MM.YYYY.glo.abs

6828:Enter Start Year:  1901

6829:Enter Start Month: 01

6830:Enter End Year:    2006

6831:Enter End Month:   12

6832:

6833:Please enter a sample OUTPUT filename, replacing

6834:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.dat

6835:Writing cru_ts_3_00.1901.1910.vap.dat

6836:Writing cru_ts_3_00.1911.1920.vap.dat

6837:Writing cru_ts_3_00.1921.1930.vap.dat

6838:Writing cru_ts_3_00.1931.1940.vap.dat

6839:Writing cru_ts_3_00.1941.1950.vap.dat

6840:Writing cru_ts_3_00.1951.1960.vap.dat

6841:Writing cru_ts_3_00.1961.1970.vap.dat

6842:Writing cru_ts_3_00.1971.1980.vap.dat

6843:Writing cru_ts_3_00.1981.1990.vap.dat

6844:Writing cru_ts_3_00.1991.2000.vap.dat

6845:Writing cru_ts_3_00.2001.2006.vap.dat

6846:

6847:

6848:Sadly, that gave the same result. So what of the published (v2.10) VAP dataset? That looks ~ok:

6849:

6850:

6851:crua6[/cru/cruts/version_3_0/secondaries/vap] ./crutsstats 

6852:

6853:CRUTSSTATS: Stats for CRU TS gridded files

6854:

6855:Enter the monthly gridded data file: cru_ts_2_10.1901-2002.vap.grid

6856:

6857:Please enter the start year: 1901

6858:

6859: 102 years from 1901 to 2002

6860:

6861:Output file is cru_ts_2_10.1901-2002.vap.grid.stats

6862:1901       0     411     105

6863:1902       0     413     104

6864:1903       0     465     104

6865:1904       0     359     104

6866:1905       0     383     104

6867:1906       0     376     105

6868:1907       0     387     104

6869:(etc)

6870:

6871:

6872:Not good at all. Or, rather, good that it must be a solvable problem. Except that it's 10 to 5 on a Sunday

6873:afternoon and it's me that's got to solve it.

6874:

6875:Where to start? Well, retrace your steps, that's how you get out of a minefield. So first up, to compare

6876:similar months in the anomaly files. Though I already know what I'm going to find, don't I? Because glo2abs

6877:isn't going to do anything unusual, it just adds the normal and there you go. So if the absolutes are very

6878:similar, the anomalies will be, too.. hmm. Well, I *suppose* I could try producing two more copies of the

6879:output files - one with just synthetic data and one with just observed data? It's only a couple of re-runs

6880:of the quick_interp_tdm2.pro IDL routine..

6881:

6882:Started with the synthetic-only run:

6883:

6884:

6885:IDL> quick_interp_tdm2,1901,2006,'vapsynglo/vapsyn.',1000,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='vapsyn/vapsyn'

6886:

6887:crua6[/cru/cruts/version_3_0/secondaries/vap/syn_only] ./glo2abs 

6888:Welcome! This is the GLO2ABS program.

6889:I will create a set of absolute grids from

6890:a set of anomaly grids (in .glo format), also

6891:a gridded version of the climatology.

6892:Enter the path and name of the normals file: ../clim.6190.lan.vap

6893:Enter a name for the gridded climatology file: clim.6190.lan.vap.grid

6894:Enter the path and stem of the .glo files: vapsynglo/vapsyn.

6895:Enter the starting year: 1901

6896:Enter the ending year:   2006

6897:Enter the path (if any) for the output files: vapsynabs/

6898:Now, CONCENTRATE. Addition or Percentage (A/P)? A

6899:Do you wish to limit the output values? (Y/N): N

6900:Right, erm.. off I jolly well go!

6901:vapsyn.01.1901.glo

6902:vapsyn.02.1901.glo

6903:(etc)

6904:

6905:crua6[/cru/cruts/version_3_0/secondaries/vap/syn_only] ./mergegrids 

6906:Welcome! This is the MERGEGRIDS program.

6907:I will create decadal and full gridded files

6908:from the output files of (eg) glo2abs.for.

6909:

6910:Enter a gridfile with YYYY for year and MM for month: vapsynabs/vapsyn.MM.YYYY.glo.abs

6911:Enter Start Year:  1901

6912:Enter Start Month: 01

6913:Enter End Year:    2006

6914:Enter End Month:   12

6915:

6916:Please enter a sample OUTPUT filename, replacing

6917:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.syn.dat

6918:Writing cru_ts_3_00.1901.1910.vap.syn.dat

6919:Writing cru_ts_3_00.1911.1920.vap.syn.dat

6920:Writing cru_ts_3_00.1921.1930.vap.syn.dat

6921:Writing cru_ts_3_00.1931.1940.vap.syn.dat

6922:Writing cru_ts_3_00.1941.1950.vap.syn.dat

6923:Writing cru_ts_3_00.1951.1960.vap.syn.dat

6924:Writing cru_ts_3_00.1961.1970.vap.syn.dat

6925:Writing cru_ts_3_00.1971.1980.vap.syn.dat

6926:Writing cru_ts_3_00.1981.1990.vap.syn.dat

6927:Writing cru_ts_3_00.1991.2000.vap.syn.dat

6928:Writing cru_ts_3_00.2001.2006.vap.syn.dat

6929:

6930:

6931:And then the observed-only:

6932:

6933:

6934:IDL> quick_interp_tdm2,1901,2006,'vapobsglo/vapobs.',1000,gs=0.5,dumpglo='dumpglo',pts_prefix='vaptxt/vap.'

6935:

6936:crua6[/cru/cruts/version_3_0/secondaries/vap/obs_only] ./glo2abs 

6937:Welcome! This is the GLO2ABS program.

6938:I will create a set of absolute grids from

6939:a set of anomaly grids (in .glo format), also

6940:a gridded version of the climatology.

6941:Enter the path and name of the normals file: ../clim.6190.lan.vap

6942:Enter a name for the gridded climatology file: clim.6190.lan.vap.grid

6943:Enter the path and stem of the .glo files: vapobsglo/vapobs.

6944:Enter the starting year: 1901

6945:Enter the ending year:   2006

6946:Enter the path (if any) for the output files: vapobsabs/

6947:Now, CONCENTRATE. Addition or Percentage (A/P)? A

6948:Do you wish to limit the output values? (Y/N): N

6949:Right, erm.. off I jolly well go!

6950:vapobs.01.1901.glo

6951:vapobs.02.1901.glo

6952:(etc)

6953:

6954:crua6[/cru/cruts/version_3_0/secondaries/vap/obs_only] ./mergegrids 

6955:Welcome! This is the MERGEGRIDS program.

6956:I will create decadal and full gridded files

6957:from the output files of (eg) glo2abs.for.

6958:

6959:Enter a gridfile with YYYY for year and MM for month: vapobsabs/vapobs.MM.YYYY.glo.abs

6960:Enter Start Year:  1901

6961:Enter Start Month: 01

6962:Enter End Year:    2006

6963:Enter End Month:   12

6964:

6965:Please enter a sample OUTPUT filename, replacing

6966:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.obs.dat

6967:Writing cru_ts_3_00.1901.1910.vap.obs.dat

6968:Writing cru_ts_3_00.1911.1920.vap.obs.dat

6969:Writing cru_ts_3_00.1921.1930.vap.obs.dat

6970:Writing cru_ts_3_00.1931.1940.vap.obs.dat

6971:Writing cru_ts_3_00.1941.1950.vap.obs.dat

6972:Writing cru_ts_3_00.1951.1960.vap.obs.dat

6973:Writing cru_ts_3_00.1961.1970.vap.obs.dat

6974:Writing cru_ts_3_00.1971.1980.vap.obs.dat

6975:Writing cru_ts_3_00.1981.1990.vap.obs.dat

6976:Writing cru_ts_3_00.1991.2000.vap.obs.dat

6977:Writing cru_ts_3_00.2001.2006.vap.obs.dat

6978:

6979:

6980:So.. how do the stats look for these two datasets?

6981:

6982:Synthetic-only:

6983:

6984:

6985:crua6[/cru/cruts/version_3_0/secondaries/vap/syn_only] ./crutsstats 

6986:

6987:CRUTSSTATS: Stats for CRU TS gridded files

6988:

6989:Enter the monthly gridded data file: cru_ts_3_00.1901.2006.vap.syn.dat

6990:

6991:Please enter the start year: 1901

6992:

6993: 106 years from 1901 to 2006

6994:

6995:Output file is cru_ts_3_00.1901.2006.vap.syn.dat.stats

6996:1901       1     358     106

6997:1902       1     358     106

6998:1903       1     358     106

6999:1904       1     358     106

7000:1905       1     358     106

7001:1906       1     358     106

7002:(etc)

7003:

7004:

7005:Observed-only:

7006:

7007:

7008:crua6[/cru/cruts/version_3_0/secondaries/vap/obs_only] ./crutsstats 

7009:

7010:CRUTSSTATS: Stats for CRU TS gridded files

7011:

7012:Enter the monthly gridded data file: cru_ts_3_00.1901.2006.vap.obs.dat

7013:

7014:Please enter the start year: 1901

7015:

7016: 106 years from 1901 to 2006

7017:

7018:Output file is cru_ts_3_00.1901.2006.vap.obs.dat.stats

7019:1901       1     358     106

7020:1902       1     358     106

7021:1903       1     358     106

7022:1904       1     358     106

7023:1905       1     358     106

7024:1906       1     358     106

7025:(etc)

7026:

7027:

7028:Oh, GOD. What is going on? Are we data sparse and just looking at the climatology? How can a synthetic

7029:dataset derived from tmp and dtr produce the same statistics as an 'real' dataset derived from observations?

7030:

7031:Let's be logical. Here are the two 'separated' gridding runs:

7032:

7033:IDL> quick_interp_tdm2,1901,2006,'vapsynglo/vapsyn.',1000,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='vapsyn/vapsyn'

7034:IDL> quick_interp_tdm2,1901,2006,'vapobsglo/vapobs.',1000,gs=0.5,dumpglo='dumpglo',pts_prefix='vaptxt/vap.'

7035:

7036:Well they look fine. The synthetic run has no other data inputs ('nostn=1'), and the observed run has no references to

7037:the synthetic data. So.. either quick_interp_tdm2.pro is doing something 'unusual', or, or.. hang on, let's try the

7038:climatology for stats:

7039:

7040:1961       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7041:

7042:Ah, Bingo was his name-o! as I was hoping (well OK it's a bad kind of hope), the reason it's all the same is that it is

7043:by and large defaulting to the climatology. Which means that not much (any?) data is getting through, no matter if we

7044:use synthetic, observed, or both together. What's odd about that conclusion is that the synthetic data is derived from

7045:TMP and DTR - two very well-populated datasets! So synthetics alone should pretty much fill the.. hang on, just though

7046:of something horrendous.. oh, okay, probably not that. I was wondering if glo2abs.for was factoring the normals so that

7047:the anomalies were insignificant, but the equation is:

7048:

7049:              absgrid(ilon(i),ilat(i)) = 

7050:     *          nint(anoms(ilon(i),ilat(i))*10) + normals(i,imo)

7051:

7052:..so the anomaly is getting the weight! But still - - not a wise thing to leave to automatics. So glo2abs should prompt

7053:the user.. but with what? Just one anomaly and normal? Several? The same one from different timesteps? Eeek. Let's look

7054:at this actual case.

7055:

7056:January 1961, lines 11103, 11104 in the glo file (11099, 11100 without header, putting it on about 33.5 degs N)

7057:  0.0000E+00  0.0000E+00  0.0000E+00  0.0000E+00  0.0000E+00  0.0000E+00  4.7173E-04  4.7224E-03

7058:  5.4273E-03  6.1323E-03  6.8372E-03  7.5422E-03  8.2472E-03  1.9677E-03  0.0000E+00  0.0000E+00

7059:

7060:Those anomalies are mighty tiny, given that the absolutes are three-digit integers! Hardly surprising they're not really

7061:appearing on the radar when added to normals typically two orders of magnitude higher! Even with the *10 in the glo2abs

7062:prog, we're still looking at values around 0.06.

7063:

7064:Looked at the observed anomalies (output from anomdtb.f90) - here the anomalies are larger! Between -5 and +5, roughly,

7065:which is what I'm used to seeing in .txt files.

7066:

7067:To investigate the synthetics, I needed to look at re-run vap_gts_tdm.pro. It says,

7068:

7069:; Note that anomalies are in hPa*10 (bin) or hPa (glo)

7070:

7071:So the binary file anomaly units - the ones we're using - are in hPa*10. Let's get one o' them synthetic glo files:

7072:

7073:IDL> vap_gts_anom,dtr_prefix='../dtrbin/dtrbin',tmp_prefix='../tmpbin/tmpbin',1961,1961,outprefix='vapsynglo/vapsyn.',dumpglo=1

7074:Land,sea:       56016       68400

7075:Calculating tmn normal

7076:% Compiled module: TVAP.

7077:Calculating synthetic vap normal

7078:% Compiled module: ESAT.

7079:Calculating synthetic anomalies

7080:% Compiled module: MOMENT.

7081:1961 vap (x,s2,<<,>>):  5.72571e-05  9.01807e-07   -0.0653905    0.0261283

7082:% Compiled module: SAVEGLO.

7083:% Compiled module: SELECTMODEL.

7084:

7085:For Jan 1961 (may as well stick with it), -999 is the missing value code. The range is -0.0149 to +0.0222 (remember this is

7086:an anomaly in hPa according to the program comment). So if it's telling the truth, the binary anomalies presented to

7087:quick_interp_tdm2.pro will range from roughly -0.3 to +0.3. still nt going to impinge on normals between 1 and 358, is it?

7088:

7089:So, what are the normals in? Well according to clim.6190.lan.vap:

7090:

7091:crua6[/cru/cruts/version_3_0/secondaries/vap] head -11 clim.6190.lan.vap

7092:Tyndall Centre grim file created on 12.01.2004 at 11:47 by Dr. Tim Mitchell

7093:.vap = vapour pressure (hPa)

7094:0.5deg lan clim:1961-90 MarkNew

7095:[Long=-180.00, 180.00] [Lati= -90.00,  90.00] [Grid X,Y= 720, 360]

7096:[Boxes=   67420] [Years=1975-1975] [Multi=    0.1000] [Missing=-999]

7097:Grid-ref=   1, 148

7098:  291  294  296  293  287  279  265  262  271  279  286  287

7099:Grid-ref=   1, 311

7100:   14   11   13   21   44   69   92   90   65   37   22   14

7101:Grid-ref=   1, 312

7102:   13   10   12   20   43   67   90   87   63   35   21   13

7103:

7104:That's what I've been missing! D'oh. That '[Multi=    0.1000]'. That would still only give a range of 0.1 to 35.8 hPa, and

7105:my anomalies are still around 0.006 (or 0.3 for synthetics).

7106:

7107:Two things, then. Firstly to get glo2abs to read the multiplicative factor from the climatology header and impose it on the

7108:output. Secondly to work out why all the anomalies have different magnitudes! Or is vapour pressure really so teeny?

7109:

7110:Working on glo2abs. Well my theory for additive anomalies is this: I read in the normals, and apply the multiplicative factor

7111:in the header (for VAP it's 0.1). I assume the anomalies are already in the relevant units (ie require no factoring). This

7112:looks to be the case for .txt files anyway. So I can add the anomaly to the adjusted normal. Then (because I need integer

7113:output) I can DIVIDE by the factor (because that got us from integer to real before). Fine in theory but it all depends on

7114:the anomalies being in regular 'units' (why wouldn't they be? They're reals!). OK, check from the beginning, obs first:

7115:

7116:Database:       hPa*10 (typically 3-digit integers)

7117:

7118:anomdtb.for calls subroutine CheckVariSuffix, which contains:

7119:

7120:

7121:else if (Suffix.EQ.".vap") then

7122:  	Variable="vapour pressure (hPa)"

7123:  	Factor = 0.1

7124:

7125:

7126:And how does anomdtb.f90 use the Factor? well in the original version:

7127:

7128:

7129:crua6[/cru/cruts/untouched/code/linux/cruts] grep 'Factor' anomdtb.f90 

7130:real :: MissThresh,StdevThresh,DistanceThresh,Factor, ExeSpace,WyeSpace

7131:                call CheckVariSuffix (LoadSuffix,Variable,Factor)

7132:                OpTot = OpTot + (real(DataA(XAYear,XMonth,XAStn))/Factor)

7133:                OpTotSq = OpTotSq + ((real(DataA(XAYear,XMonth,XAStn))/Factor) ** 2)

7134:      NormMean  (XMonth,XAStn) = Factor*OpTot/OpEn

7135:      if (OpTotSq.GT.0) NormStdev (XMonth,XAStn) = Factor*sqrt((OpTotSq/OpEn)-((OpTot/OpEn)**2))

7136:                OpTot = OpTot + (real(DataA(XAYear,XMonth,XAStn))/Factor)

7137:                OpTotSq = OpTotSq + ((real(DataA(XAYear,XMonth,XAStn))/Factor) ** 2)

7138:        NormMean  (XMonth,XAStn) = Factor*OpTot/OpEn

7139:          NormStdev (XMonth,XAStn) = Factor*sqrt((OpEn/(OpEn-1))*((OpTotSq/OpEn)-((OpTot/OpEn)**2)))

7140:                             OpTot = OpTot + (DataA(XAYear,XMonth,XAStn)/Factor)

7141:                             OpTotSq = OpTotSq + (DataA(XAYear,XMonth,XAStn)/Factor) ** 2

7142:                             OpStDev = Factor*sqrt((OpEn/(OpEn-1))*((OpTotSq/OpEn)-((OpTot/OpEn)**2)))

7143:                             OpMean  = Factor*(OpTot/OpEn)

7144:           ALat(XAStn),ALon(XAStn),AElv(XAStn),real(DataA(XAYear,XMonth,XAStn))*Factor,AStn(XAStn)

7145:

7146:

7147:I *think* the factor is being used multiplicatively. I don't understand why it's being used as a divisor though.. I must

7148:have understood last December because I managed to rewrite the 'standard deviation' section, also using it as a divisor!

7149:

7150:One obvious thing to try is to use the revised glo2abs. That should now be working in 'units' (but saving in whatever

7151:range the normals are in). After that I could try comparing the old and 'new' (ie modded by me) versions of anomdtb.f90

7152:to ensure I didn't break something (sure I didn't, but still..)

7153:

7154:So, I revised glo2abs. It now reads the 'Multi' factor from the climatology header, and applies it to the normals before

7155:they're used.

7156:

7157:So, re-ran quick_interp+tdm2.pro:

7158:

7159:IDL> quick_interp_tdm2,1901,2006,'vapglo/vap.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='vapsyn/vapsyn',pts_prefix='vaptxt/vap.'

7160:

7161:A sample of the outputs, vap.12.1962.glo, had a range of values from -2.3006 to +1.8388, with the majority being 0. A total

7162:of 56387 cells were nonzero, which given that there are 67420 land cells, isn't too bad. It's a pretty gaussian distribution,

7163:too. It still seems like a small variation (typically +/- 0.5). For the cell where I live (Norwich, 363,286), the normals are:

7164:

7165:Grid-ref= 363, 286

7166:   71   69   76   86  107  129  147  149  135  115   88   77

7167:

7168:Or in hPa:

7169:

7170:Grid-ref= 363, 286

7171:    7.1   6.9   7.6   8.6   10.7  12.9  14.7  14.9  13.5  11.5  8.8   7.7

7172:

7173:The nearest station (well based on a quick search) is LOWESTOFT. Taking 1962 and 1963 and scaling:

7174:

7175:62  7.6   6.9   6.5   9.2  10.9  12.6  14.4  15.0  13.6  12.3   8.9   6.5

7176:63  5.4   5.5   7.9   9.9  11.1  14.8  15.8  15.1  14.6  11.7  10.3   6.9

7177:

7178:The ranges:

7179:    2.2   1.4   1.4   0.7   0.2   2.2   1.4   0.1   1.0   0.6   1.4   0.4

7180:    

7181:Well our sample December 1962 range of anomalies was -2.3006 to +1.8388, and the January range is -3.3640 to +2.1250. So, I

7182:have to admit, that's the same order of magnitude for our particular cell, year and month(s).

7183:

7184:So, assuming these .glo files are OK, we'll try glo2abs again:

7185:

7186:

7187:crua6[/cru/cruts/version_3_0/secondaries/vap] ./glo2abs

7188:Welcome! This is the GLO2ABS program.

7189:I will create a set of absolute grids from

7190:a set of anomaly grids (in .glo format), also

7191:a gridded version of the climatology.

7192:Enter the path and name of the normals file: clim.6190.lan.vap

7193:Enter a name for the gridded climatology file: deleteme1

7194:Enter the path and stem of the .glo files: vapglo/vap.

7195:Enter the starting year: 1901

7196:Enter the ending year:   2006

7197:Enter the path (if any) for the output files: vapabs/

7198:Now, CONCENTRATE. Addition or Percentage (A/P)? A

7199:Do you wish to limit the output values? (Y/N): Y

7200:1. Set minimum to zero

7201:2. Set a single minimum and maximum

7202:3. Set monthly minima and maxima (for wet/rd0)

7203:Choose: 1

7204:Right, erm.. off I jolly well go!

7205:vap.01.1901.glo

7206:vap.02.1901.glo

7207:(etc)

7208:

7209:

7210:..and the result.. look good! For (again) December 1962:

7211:

7212:Min   0 (well I did set that, see above)

7213:Max 315

7214:

7215:Number of zeros: 1078, perfectly respectable although I do wonder if VAP=0 is illegal.. hmm.. OK, added an option in glo2abs:

7216:

7217:

7218:crua6[/cru/cruts/version_3_0/secondaries/vap] ./glo2abs

7219:Welcome! This is the GLO2ABS program.

7220:I will create a set of absolute grids from

7221:a set of anomaly grids (in .glo format), also

7222:a gridded version of the climatology.

7223:Enter the path and name of the normals file: clim.6190.lan.vap

7224:Enter a name for the gridded climatology file: deleteme3

7225:Enter the path and stem of the .glo files: vapglo/vap.

7226:Enter the starting year: 1901

7227:Enter the ending year:   2006

7228:Enter the path (if any) for the output files: vapabs/

7229:Now, CONCENTRATE. Addition or Percentage (A/P)? A

7230:Do you wish to limit the output values? (Y/N): Y

7231:1. Set minimum to zero

7232:2. Set a single minimum and maximum

7233:3. Set monthly minima and maxima (for wet/rd0)

7234:4. Set all values >0, (ie, positive)

7235:Choose: 4

7236:Right, erm.. off I jolly well go!

7237:vap.01.1901.glo

7238:vap.02.1901.glo

7239:(etc)

7240:

7241:

7242:Result for December 1962: Min 1, Max 315. A good spread of values, without a disproportionate number of '1's, I'm please

7243:to say.

7244:

7245:So, to generate the output files. Again.

7246:

7247:

7248:crua6[/cru/cruts/version_3_0/secondaries/vap] ./mergegrids 

7249:Welcome! This is the MERGEGRIDS program.

7250:I will create decadal and full gridded files

7251:from the output files of (eg) glo2abs.for.

7252:

7253:Enter a gridfile with YYYY for year and MM for month: vapabs/vap.MM.YYYY.glo.abs

7254:Enter Start Year:  1901

7255:Enter Start Month: 01

7256:Enter End Year:    2006

7257:Enter End Month:   12

7258:

7259:Please enter a sample OUTPUT filename, replacing

7260:start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.dat

7261:Writing cru_ts_3_00.1901.1910.vap.dat

7262:Writing cru_ts_3_00.1911.1920.vap.dat

7263:Writing cru_ts_3_00.1921.1930.vap.dat

7264:Writing cru_ts_3_00.1931.1940.vap.dat

7265:Writing cru_ts_3_00.1941.1950.vap.dat

7266:Writing cru_ts_3_00.1951.1960.vap.dat

7267:Writing cru_ts_3_00.1961.1970.vap.dat

7268:Writing cru_ts_3_00.1971.1980.vap.dat

7269:Writing cru_ts_3_00.1981.1990.vap.dat

7270:Writing cru_ts_3_00.1991.2000.vap.dat

7271:Writing cru_ts_3_00.2001.2006.vap.dat

7272:

7273:

7274:And what of the statistics. Well by now I've realised that we don't have complete coverage! So the normals are

7275:bound to poke through quite a bit. In fact, the story is as it was in the beginning! *cries*

7276:

7277:

7278:crua6[/cru/cruts/version_3_0/secondaries/vap] ./crutsstats 

7279:

7280:CRUTSSTATS: Stats for CRU TS gridded files

7281:

7282:Enter the monthly gridded data file: cru_ts_3_00.1901.2006.vap.dat

7283:

7284:Please enter the start year: 1901

7285:

7286: 106 years from 1901 to 2006

7287:

7288:Output file is cru_ts_3_00.1901.2006.vap.dat.stats

7289:1901       1     358     106

7290:1902       1     358     106

7291:1903       1     358     106

7292:1904       1     358     106

7293:1905       1     358     106

7294:1906       1     358     106

7295:1907       1     358     106

7296:1908       1     358     106

7297:(etc)

7298:

7299:

7300:Now admittedly, the 106 mean does vary.. it hioits the dizzying heights of 107 on occasion! With a couple of 105s

7301:thrown in to balance the books. Had a look at the stats in detail, compared to those for CRU TS 2.10. And guess

7302:what? Yes.. the old stats are better! Here's the first decade:

7303:

7304:CRU TS 2.10

7305:1901       0     324      79       0     338      82       0     314      88       0     321      97       0     411     110       0     378     128       0     358     143       0     343     140       0     353     122       0     332     103       0     318      88       0     314      81       0     411     105

7306:1902       0     312      80       0     319      82       0     314      87       0     321      96       0     413     109       0     366     125       0     356     141       0     343     138       0     353     122       0     323     102       0     318      88       0     315      80       0     413     104

7307:1903       0     314      79       0     331      82       0     315      88       0     334      95       0     465     109       0     359     125       0     371     141       0     359     139       0     353     122       0     323     102       0     318      88       0     315      80       0     465     104

7308:1904       0     310      78       0     319      81       0     312      86       0     321      95       0     347     109       0     359     126       0     355     140       0     344     138       0     354     121       0     323     103       0     318      89       0     316      81       0     359     104

7309:1905       0     314      79       0     319      79       0     321      86       0     326      95       0     346     109       0     383     127       0     356     142       0     344     139       0     353     122       0     330     103       0     318      90       0     321      82       0     383     104

7310:1906       0     328      80       0     330      81       0     323      87       0     335      98       0     376     111       0     359     128       0     356     142       0     343     140       0     353     122       0     323     103       0     318      89       0     316      82       0     376     105

7311:1907       0     312      79       0     327      80       0     314      87       0     321      94       0     387     106       0     359     125       0     379     140       0     343     139       0     353     122       0     323     104       0     318      87       0     316      81       0     387     104

7312:1908       0     312      79       0     323      81       0     330      86       0     338      95       0     346     109       0     359     127       0     353     142       0     343     138       0     353     122       0     316     102       0     318      87       0     316      81       0     359     104

7313:1909       0     312      79       0     319      81       0     323      87       0     321      94       0     346     107       0     359     125       0     355     141       0     343     140       0     354     122       0     320     103       0     318      90       0     316      81       0     359     104

7314:1910       0     312      80       0     319      82       0     315      86       0     321      95       0     347     109       0     359     126       0     383     142       0     343     139       0     353     122       0     318     102       0     318      87       0     316      80       0     383     104

7315:

7316:CRU TS 3.00

7317:1901       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7318:1902       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7319:1903       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7320:1904       1     311      80       1     320      82       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7321:1905       1     311      80       1     320      83       1     315      88       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7322:1906       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7323:1907       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     141       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7324:1908       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     129       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7325:1909       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7326:1910       1     311      80       1     320      83       1     315      89       1     320      98       1     346     111       1     358     128       1     356     143       1     342     140       1     354     123       1     323     104       1     318      90       1     315      82       1     358     106

7327:

7328:..and here's a more recent decade:

7329:

7330:CRU TS 2.10

7331:1991       0     314      82       0     322      84       0     331      90       0     672     100       0     523     113       0     540     134       0     607     147       0     424     143       0     353     125       0     328     106       0     386      91       0     350      83       0     672     108

7332:1992       0     337      82       0     383      84       0     450      90       0     613      98       0     347     112       0     359     128       0     373     140       0     345     140       0     353     122       0     347     103       0     414      89       0     384      83       0     613     106

7333:1993       0     324      81       0     403      83       0     449      90       0     622      98       0     518     113       0     534     131       0     652     147       0     398     143       0     353     122       0     333     105       0     408      89       0     339      84       0     652     107

7334:1994       0     346      82       0     396      82       0     457      90       0     626     100       0     524     113       0     507     132       0     605     146       0     416     143       0     349     125       0     332     107       0     397      93       0     341      84       0     626     108

7335:1995       0     369      83       0     406      86       0     461      90       0     686     100       0     505     114       0     565     134       0     673     146       0     492     147       0     364     127       0     342     108       0     427      91       0     339      82       0     686     109

7336:1996       0     334      81       0     431      83       0     548      88       0     634      97       0     524     113       0     530     131       0     645     147       0     422     142       0     366     124       0     337     106       0     413      91       0     344      84       0     645     107

7337:1997       0     367      82       0     322      84       0     348      90       0     323      99       0     344     113       0     484     133       0     426     147       0     523     145       0     353     126       0     348     108       0     345      93       0     370      86       0     523     109

7338:1998       0     339      84       0     345      89       0     338      92       0     355     104       0     361     116       1     531     137       1     356     152       0     560     149       0     370     128       0     347     108       0     369      92       0     334      85       0     560     111

7339:1999       0     323      83       0     334      86       0     324      90       0     336     100       0     362     113       0     487     132       0     362     148       0     357     143       1     353     127       0     331     107       0     337      91       0     316      85       0     487     109

7340:2000       0     319      82       0     319      85       0     319      91       0     328     102       0     356     114       0     476     133       0     358     146       0     520     146       0     353     124       0     333     107       0     335      91       0     334      84       0     520     109

7341:

7342:CRU TS 3.00

7343:1991       1     311      81       1     320      83       1     320      90       1     320     100       1     346     113       1     358     132       1     356     146       1     342     143       1     354     125       1     323     105       1     318      91       1     315      82       1     358     108

7344:1992       1     311      82       1     319      84       1     315      90       1     320      97       1     346     111       1     358     127       1     356     141       1     342     140       1     354     122       1     323     102       1     317      89       1     315      83       1     358     106

7345:1993       1     313      81       1     315      83       1     315      89       1     320      98       1     346     112       1     358     131       1     356     146       1     342     142       1     354     122       1     323     103       1     323      88       1     317      83       1     358     106

7346:1994       1     311      82       1     322      82       1     315      89       1     320      99       1     346     112       1     358     131       1     356     146       1     346     142       1     354     125       1     323     106       1     318      92       1     315      83       1     358     107

7347:1995       1     311      82       1     318      85       1     320      90       1     324      99       1     346     112       1     358     131       1     356     146       1     345     144       1     354     124       1     323     107       1     321      90       1     315      81       1     358     108

7348:1996       1     311      80       1     321      82       1     320      87       1     320      96       1     346     111       1     358     130       1     356     145       1     343     141       1     354     122       1     323     105       1     318      90       1     319      82       1     358     106

7349:1997       1     311      81       1     320      84       1     315      90       1     320      99       1     346     113       1     358     131       1     356     145       1     342     143       1     354     123       1     323     106       1     318      90       1     315      83       1     358     107

7350:1998       1     311      81       1     334      85       1     326      89       1     338     100       1     346     114       1     358     134       1     356     148       1     342     145       1     354     125       1     323     105       1     318      89       1     315      84       1     358     108

7351:1999       1     316      82       1     320      85       1     322      88       1     320      99       1     346     112       1     358     131       1     356     148       1     342     142       1     354     125       1     323     106       1     318      91       1     315      84       1     358     108

7352:2000       1     317      82       1     320      84       1     315      90       1     320     100       1     346     113       1     358     131       1     356     146       1     342     144       1     354     123       1     323     105       1     318      90       1     315      83       1     358     108

7353:

7354:I DON'T UNDERSTAND!!!!!

7355:

7356:Well, OK - I see that a VAP of zero is acceptable. Though as it's a pressure, I don't believe it! I'll stick with 1.

7357:

7358:The issue is that the earlier dataset has a variability (in the maximum) that we just don't have in the new one. And

7359:I feel that I've been through every bloody phase of the process and checked we're doing it right!!!

7360:

7361:~~~

7362:

7363:Right. Let's look at the distributions of values in each dataset. We'll take Jan 1910 and Jun 2000. And as this is

7364:a textual document, I'll have to describe the results.

7365:

7366:Offsets. Well each month has 360 lines, so each year has 4320 lines. So for Jan 1910 we need to skip nine years,

7367:or 38880 lines, then take the next 360. For Jun 2000 we need to skip 99 years, or 427680 lines, then another five

7368:months, or 1800 lines, then take the next 360. So:

7369:

7370:head -39240 cru_ts_2.10.1901-2002.vap.dat |tail -360 > cru_ts_2.10.Jan.1910.vap.dat

7371:head -39240 cru_ts_3.00.1901.2006.vap.dat |tail -360 > cru_ts_3.00.Jan.1910.vap.dat

7372:

7373:head -428040 cru_ts_2.10.1901-2002.vap.dat |tail -360 > cru_ts_2.10.Jun.2000.vap.dat

7374:head -428040 cru_ts_3_00.1901.2006.vap.dat |tail -360 > cru_ts_3_00.Jun.2000.vap.dat

7375:

7376:I loaded the resultant monthly files into Matlab, and played with them mercilessly.

7377:

7378:Well to start with, they all look the same. Truly. I've got a 4-plot page with TS 2.10 in the left-hand column,

7379:and TS 3.00 on the right. January 1910 on the top, June 2000 on the bottom. and they look pretty much inseparable,

7380:though if I had to Spot The Difference, the TS 2.10 June 2000 distribution is a little flatter (that is, the

7381:massive spike at the low end is a little shorter, and the rest of the entourage are a little taller.

7382:

7383:What are particularly worthy of note are the maximums. Because they don't match those produced by crutsstats.for.

7384:

7385:Month        Model    Max (Matlab)   Max (crutsstats)

7386:Jan 1910   TS 2.10             312                312

7387:Jan 1910   TS 3.00             311                311

7388:Jun 2000   TS 2.10             319                476

7389:Jun 2000   TS 3.00             317                358

7390:

7391:Not entirely sure why the latter ones would be wrong. But I suspect crutsstats - because otherwise I miscounted

7392:the line numbers to extract June 2000 with! Actually, OK, that does seem more likely.

7393:

7394:Let's try it from the 1991-2000 files. The offset will be 9*4320 + 5*360 + 360 = 41040.

7395:

7396:gunzip -c /cru/cruts/fromtyn1/data/cru_ts_2.10/newly_gridded/data_dec/cru_ts_2_10.1991-2000.vap.grid.gz | head -41040 | tail -360 > cru_ts_2_10.Jun.2000.vap.dat

7397:gunzip -c cru_ts_3_00.1991.2000.vap.dat.gz | head -41040 | tail -360 > cru_ts_3_00.Jun.2000.vap.dat

7398:

7399:Well - looks like I did miscount, because the new files are different! And so are the Maxima:

7400:

7401:Month        Model    Max (Matlab)   Max (crutsstats)

7402:Jun 2000   TS 2.10             300                476

7403:Jun 2000   TS 3.00             358                358

7404:

7405:..so almost perfect. At least the stats for the file I'm creating match.

7406:

7407:And now the June 2000 histograms are much more interesting! And of course (for this is THIS project), much

7408:more worrying. The June 2000 plot for the new data (3.00) shows a fall at VAP ->0. This is in contrast to the

7409:other three, which show a more expotential decline from a high near 0 (though admittedly the 2.10 version does have a second

7410:peak at around 120). In fact, the June 2000 3.00 series has peaks at ~90 and ~300! Oh, help.

7411:

7412:The big question must be, why does it have so little representation in the low numbers? Especially given that I'm rounding

7413:erroneous negatives up to 1!!

7414:

7415:Oh, sod it. It'll do. I don't think I can justify spending any longer on a dataset, the previous version of which was

7416:completely wrong (misnamed) and nobody noticed for five years.

7417:

7418:

7419:So.. one week to go before handover, and I'm just STARTING the Sun/Cloud parameter, the one I thought would cause the most

7420:trouble! Oh, boy. Let's try and work out the scenario.

7421:

7422:Historically, we've issued Cloud:

7423:

7424:crua6[/cru/cruts/fromtyn1/data/cru_ts_2.10/data_all] gunzip -c cru_ts_2_10.1901-2002.cld.Z |head -10

7425:Tyndall Centre grim file created on 22.01.2004 at 13:52 by Dr. Tim Mitchell

7426:.cld = cloud cover (percentage)

7427:CRU TS 2.1

7428:[Long=-180.00, 180.00] [Lati= -90.00,  90.00] [Grid X,Y= 720, 360]

7429:[Boxes=   67420] [Years=1901-2002] [Multi=    0.1000] [Missing=-999]

7430:Grid-ref=   1, 148

7431:  725  750  750  700  638  600  613  613  663  675  713  725

7432:

7433:..so data is in % x10.

7434:

7435:Then, of course, there's the relevant read_me text (from /cru/cruts/fromdpe1a/code/idl/pro/read_me_GRIDDING.txt):

7436:

7437:"Bear in mind that there is no working synthetic method for cloud, because Mark New

7438:   lost the coefficients file and never found it again (despite searching on tape

7439:   archives at UEA) and never recreated it. This hasn't mattered too much, because

7440:   the synthetic cloud grids had not been discarded for 1901-95, and after 1995

7441:   sunshine data is used instead of cloud data anyway."

7442:

7443:So that's alright then! See also the earlier attempts to recreate TS 2.10 cloud.

7444:

7445:The main gridding prog for cloud appears to be cal_cld_gts_tdm.pro:

7446:

7447:pro cal_cld_gts_tdm,dtr_prefix,outprefix,year1,year2,info=info

7448:

7449:; calculates cld anomalies using relationship with dtr anomalies

7450:; reads coefficients from predefined files (*1000)

7451:; reads DTR data from binary output files from quick_interp_tdm2.pro (binfac=1000)

7452:; creates cld anomaly grids at dtr grid resolution

7453:; output can then be used as dummy input to splining program that also 

7454:;  includes real cloud anomaly data

7455:

7456:As for converting sun hours to cloud cover.. we only appear to have interactive, file-by-file

7457:programs. Herewith all the relevant progs I can find: 

7458:

7459:IDL

7460:./idl/pro/cal_cld_gts_tdm.pro        (synthetic cloud from DTR)

7461:./idl/pro/cloudcorr.pro              (construct cloud correlation coefficients with DTR)

7462:./idl/pro/cloudcorrspc.pro           (construct cloud correlation coefficients with sunshine %)

7463:./idl/pro/cloudcorrspcann.pro        (construct cloud correlation coefficients with sunshine %)

7464:./idl/pro/cloudcorrspcann9196.pro    (construct cloud correlation coefficients with sunshine %)

7465:

7466:(the 'ann' versions above include the assumption that the relationships remain constant through the year)

7467:

7468:F77

7469:./f77/mnew/sh2cld_tdm.for            (this one needs to be modded as for sp2cldp_m.for I think)

7470:./f77/mnew/Hsp2cldp_m.for            (one I wrote last year which seems to almost do what we need)

7471:./f77/mnew/sp2cld_m.for              (this one needs to be modded as for sp2cldp_m.for I think)

7472:./f77/mnew/sh2sp_m.for

7473:./f77/mnew/sh2sp_normal.for

7474:./f77/mnew/sh2sp_tdm.for

7475:

7476:Aaaand - another head-banging shocker! The program sh2cld_tdm.for, which describes itself thusly:

7477:

7478:      program sunh2cld    

7479:c converts sun hours monthly time series to cloud percent (n/N)

7480:

7481:Does NO SUCH THING!!! Instead it creates SUN percentages! This is clear from the variable names and

7482:user interactions.

7483:

7484:So.. if I add the sunh -> sun% process from sh2cld_tdm.for into Hsp2cldp_m.for, I should end up with a

7485:sun hours to cloud percent convertor. Possibly. Except that the sun% to cld% engine looks like it's

7486:creating oktas instead:

7487:

7488:          do im=1,12

7489:            ratio = (real(sunp(im))/100)

7490:            if (ratio.ge.0.95)           cldp(im) = 0

7491:            if (ratio.lt.0.95.and.ratio.ge.0.35)

7492:     *                                   cldp(im) = (0.95-ratio)*100

7493:            if (ratio.lt.0.35.and.ratio.ge.0.15)

7494:     *                                   cldp(im) = ((0.35-ratio)*50)+60

7495:            if (ratio.lt.0.15)           cldp(im) = ((0.15-ratio)*100)+70

7496:            if (cldp(im).gt.80.0)        cldp(im) = 80.0

7497:            if (ratio.lt.0)              cldp(im) = -9999

7498:          enddo

7499:

7500:Added the previous '*12.5' mod to approximate true percentages (*10).

7501:

7502:Looking back I see we found cloud and sunpercent databases (line counts shown):

7503:

7504:

7505:    228936 cld.0301081434.dtb

7506:    104448 cld.0312181428.dtb

7507:    111989 combo.cld.dtb

7508:     57395 spc.0301201628.dtb

7509:     51551 spc.0312221624.dtb

7510:     51551 spc.94-00.0312221624.dtb

7511:

7512:And agreed a strategy:

7513:

7514:

7515:AGREED APPROACH for cloud (5 Oct 06).

7516:

7517:For 1901 to 1995 - stay with published data. No clear way to replicate

7518:process as undocumented.

7519:

7520:For 1996 to 2002:

7521: 1. convert sun database to pseudo-cloud using the f77 programs;

7522: 2. anomalise wrt 96-00 with anomdtb.f;

7523: 3. grid using quick_interp_tdm.pro (which will use 6190 norms);

7524: 4. calculate (mean9600 - mean6190) for monthly grids, using the

7525:    published cru_ts_2.0 cloud data;

7526: 5. add to gridded data from step 3.

7527:

7528:This should approximate the correction needed.

7529:

7530:

7531:This is confusing. I can only use one (observed) cloud database in the final gridding. The above

7532:agreement seems to assume that all data after 1996 will come from sun. But dtbstat.for reports:

7533:

7534:

7535:Report for: spc.0312221624.dtb (it's similar for the other spcs, except the earlier one goes to 2002)

7536:

7537:Stations in Northern Hemisphere:     1750

7538:Stations in Southern Hemisphere:      350

7539:                          Total:     2100

7540:

7541:Maximum Timespan in Northern Hemisphere: 1889 to 2003

7542:Maximum Timespan in Southern Hemisphere: 1944 to 2003

7543:                        Global Timespan: 1889 to 2003

7544:

7545:Minimum Data Value:     0

7546:Maximum Data Value:  1000

7547:

7548:

7549:So the Sun Percent databases run for long periods. Similarly, for cloud:

7550:

7551:

7552:Report for: cld.0312181428.dtb

7553:

7554:Stations in Northern Hemisphere:     3286

7555:Stations in Southern Hemisphere:      319

7556:                          Total:     3605

7557:

7558:Maximum Timespan in Northern Hemisphere: 1905 to 1996

7559:Maximum Timespan in Southern Hemisphere: 1959 to 1996

7560:                        Global Timespan: 1905 to 1996

7561:

7562:

7563:Minimum Data Value:     0

7564:Maximum Data Value:  1000

7565:

7566:

7567:Not as long a run, and it sure ends at 1996! So 1901 to 1995 will, as agreed, remain untouched.

7568:

7569:Well.. let's try converting the MCDW and CLIMAT Sun hours to Sun percents, then adding to the

7570:SPC database (spc.0312221624.dtb). Modified Hsh2cld .for to save sun percent too. Lots of debugging..

7571:eventually dug out:

7572:

7573:Doorenbos, J., Pruitt, W.O., 1977. Guidelines for predicting crop water requirements. FAO irrigation

7574:and drainage paper no. 24. Food and Agriculture Organization of the United Nations, Rome.

7575:

7576:This was used to inform the Fortran conversion programs by indicating the latitude-potential_sun and

7577:sun-to-cloud relationships. It also assisted greatly in understanding what was wrong - Tim was in

7578:fact calculating Cloud Percent, despite calling it Sun Percent!! Just awful.

7579:

7580:And so..

7581:

7582:

7583:crua6[/cru/cruts/version_3_0/db/cld] ./Hsh2cld

7584:

7585:Hsh2cld - Convert a Sun Hours database to a Cloud Percent one

7586:

7587:Please enter the Sun Hours database: sun.0709111032.dtb

7588:Data Factor detected: *1.000

7589:

7590:Completed -     1693 stations converted.

7591:

7592:Sun Percentage Database:   spc.0711271420.dtb

7593:Cloud Percentage Database: cld.0711271420.dtb

7594:

7595:crua6[/cru/cruts/version_3_0/db/cld] ./Hsh2cld

7596:

7597:Hsh2cld - Convert a Sun Hours database to a Cloud Percent one

7598:

7599:Please enter the Sun Hours database: sun.0710151817.dtb

7600:Data Factor detected: *0.100

7601:

7602:Completed -     2020 stations converted.

7603:

7604:Sun Percentage Database:   spc.0711271421.dtb

7605:Cloud Percentage Database: cld.0711271421.dtb

7606:

7607:crua6[/cru/cruts/version_3_0/db/cld] 

7608:

7609:

7610:So, now the luxury of a little experiment.. I merged the MCDW and CLIMAT 'spc' databases into

7611:the existing one *separately*. Here were the results:

7612:

7613:MCDW:

7614:

7615:uealogin1[/cru/cruts/version_3_0/db/cld] ./newmergedb

7616:

7617:WELCOME TO THE DATABASE UPDATER

7618:

7619:Before we get started, an important question:

7620:If you are merging an update - CLIMAT, MCDW, Australian - do

7621:you want the quick and dirty approach? This will blindly match

7622:on WMO codes alone, ignoring data/metadata checks, and making any

7623:unmatched updates into new stations (metadata permitting)?

7624:

7625:Enter 'B' for blind merging, or : B

7626:Please enter the Master Database name: spc.0312221624.dtb

7627:Please enter the Update Database name: spc.0711271420.dtb

7628:

7629:Reading in both databases..

7630:Master database stations:     2100

7631:Update database stations:     1693

7632:

7633:New master database: spc.0711271504.dtb

7634:

7635:Update database stations:         1693

7636: > Matched with Master stations:   867

7637:                 (automatically:   867)

7638:                   (by operator:     0)

7639: > Added as new Master stations:   826

7640: > Rejected:                         0

7641:

7642:

7643:CLIMAT:

7644:

7645:Enter 'B' for blind merging, or : B

7646:Please enter the Master Database name: spc.0312221624.dtb

7647:Please enter the Update Database name: spc.0711271421.dtb

7648:

7649:Reading in both databases..

7650:Master database stations:     2100

7651:Update database stations:     2020

7652:

7653:   98 reject(s) from update process 0711271505

7654:

7655:New master database: spc.0711271505.dtb

7656:

7657:Update database stations:         2020

7658: > Matched with Master stations:   917

7659:                 (automatically:   917)

7660:                   (by operator:     0)

7661: > Added as new Master stations:  1005

7662: > Rejected:                        98

7663:   Rejects file:                 spc.0711271421.dtb.rejected

7664:

7665:

7666:So, as expected, a few of the CLIMAT stations couldn't be matched for metadata.. no worries.

7667:what's interestng is that roughly the same ratio of stations were matched with existing in both

7668:cases (867/1693 vs 917/2020). Slightly better for MCDW though.

7669:

7670:Now, as our updates only start in 2003, that means we've just lost between 826 and 1005 sets of

7671:data (added as new). We can't be exact as we don't know the overlap between the MCDW and the CLIMAT

7672:bulletins.. but we will have a better idea when I try the anomdtb experiment on the combined update.

7673:First, add the CLIMAT update again, this time to the MCDW-updated database:

7674:

7675:CLIMAT:

7676:

7677:Enter 'B' for blind merging, or : B

7678:Please enter the Master Database name: spc.0711271504.dtb

7679:Please enter the Update Database name: spc.0711271421.dtb

7680:

7681:Reading in both databases..

7682:Master database stations:     2926

7683:Update database stations:     2020

7684:

7685:   38 reject(s) from update process 0711271514

7686:

7687:New master database: spc.0711271514.dtb

7688:

7689:Update database stations:         2020

7690: > Matched with Master stations:  1736

7691:                 (automatically:  1736)

7692:                   (by operator:     0)

7693: > Added as new Master stations:   246

7694: > Rejected:                        38

7695:   Rejects file:                 spc.0711271421.dtb.rejected

7696:

7697:

7698:Note several bits of good news! Firstly, rejects are down to 38 (60 having matched with MCDW stations).

7699:That's not *that* good of course - those will be new and so 2003 onwards only. Similarly, (1005-246=)

7700:759 CLIMAT bulletins matched MCDW ones, they will also be 2003 onwards only. In other words, there were

7701:only (1736-759=) 977 updates to existing stations. So.. yes I'm being sidetracked again.. I found and

7702:downloaded ALL the MCDW bulletins, back to 1994!

7703:

7704:

7705:uealogin1[/cru/cruts/version_3_0/incoming/MCDW] ./mcdw2cru 

7706:

7707:MCDW2CRU: Convert MCDW Bulletins to CRU Format

7708:

7709:Enter the earliest MCDW file: ssm9409.fin

7710:Enter the latest MCDW file (or  for single files): ssm0708.fin

7711:

7712:All Files Processed

7713:tmp.0711271645.dtb: 2785 stations written   *** SEE LATER RUNS ***

7714:vap.0711271645.dtb: 2786 stations written   *** SEE LATER RUNS ***

7715:rdy.0711271645.dtb: 2781 stations written   *** SEE LATER RUNS ***

7716:pre.0711271645.dtb: 2791 stations written   *** SEE LATER RUNS ***

7717:sun.0711271645.dtb: 2184 stations written   *** SEE LATER RUNS ***

7718:

7719:Thanks for playing! Byeee!

7720:

7721:

7722:Now I'm not planning to re-run all the previous parameters! Hell, they should have had the older data

7723:in already! But for sun/cloud, this could help enormously. Here's the plan:

7724:

7725:1. Merge the CLIMAT-sourced database into the new MCDW-sourced database.

7726:2. Convert this modern sun hours database into a modern cloud percent database.

7727:3. Add normals for 95-02.

7728:4. Use the new program 'normshift.for' to calculate 95-02 normals from TS 2.10 CLD.

7729:5. Calculate difference between TS 2.10 6190 normls and the above.

7730:6. Modify the in-database normals (step 3) with the difference (step 5).

7731:7. Carry on as before?

7732:

7733:No.. this won't work. anomdtb.for calculates normals on the fly - it would have to know too much.

7734:

7735:The next opportunity comes at the output from anomdtb - the normalised values in the *.txt files that

7736:the IDL gridder reads. These are just files - one per month - with lists of coordinates and values, so

7737:ideal to add normalised values to. Decided that this will be the process:

7738:

7739:Modern SunH DB  ->  Hsh2cld.for  ->  Modern Cld% DB

7740:Modern Cld% DB  ->  newprog.for  ->  6190anomalies.txt

7741:

7742:..meanwhile, as before..

7743:

7744:Normal Cld% DB  ->  anomdtb.for  ->  6190anomalies.txt

7745:

7746:So we then just have to merge the two 6190 anomaly sets! Which could just be a concatenation.

7747:

7748:Easy, then.. the only thing we need is the miraculous 'newprog.for'! With three days before delivery.

7749:

7750:No, no, no - HANG ON. Let's not try and boil the ocean! How about:

7751:

7752:1901-2002      Static, as published, leave well alone (or recalculate with better DTR).

7753:2003-2006/7    Calc from modern SunH and use the suggested mods after gridding.

7754:

7755:This is what was originally intended. But there will be problems:

7756:

7757:1. MCDW only goes back to 2006, so what's the data density for 2003-2005? Should this also use synthetic

7758:cloud from DTR? I guess yes.

7759:

7760:2. No guarantee of continuity from 2002 to 2003. This could be the real stickler. Moving from one system

7761:to the other - this is why it might be better to re-run 1901-2002 as well.

7762:

7763:OKAY.. normshift.for now creates a gridded set of conversion data between whatever period you choose

7764:and 1961-1990. Such that it can be added to the gridded output of the process run with the 'false'

7765:normalisation period.

7766:

7767:So.. first, merge your bulletins:

7768:

7769:Well FIRSTLY, you realise that your databases don't have normals lines, so you modify mcdw2cru.for and

7770:climat2cru.for to optionally add them, then you re-run them on the bulletins, ending up with:

7771:

7772:

7773:uealogin1[/cru/cruts/version_3_0/incoming/MCDW] ./mcdw2cru 

7774:

7775:MCDW2CRU: Convert MCDW Bulletins to CRU Format

7776:

7777:Enter the earliest MCDW file: ssm9409.fin

7778:Enter the latest MCDW file (or  for single files): ssm0708.fin

7779:Add a dummy normals line? (Y/N): Y

7780:

7781:All Files Processed

7782:tmp.0711272156.dtb: 2785 stations written

7783:vap.0711272156.dtb: 2786 stations written

7784:rdy.0711272156.dtb: 2781 stations written

7785:pre.0711272156.dtb: 2791 stations written

7786:sun.0711272156.dtb: 2184 stations written

7787:

7788:Thanks for playing! Byeee!

7789:

7790:

7791:

7792:uealogin1[/cru/cruts/version_3_0/incoming/CLIMAT] ./climat2cru 

7793:

7794:CLIMAT2CRU: Convert MCDW Bulletins to CRU Format

7795:

7796:Enter the earliest CLIMAT file: climat_data_200301.txt

7797:Enter the latest CLIMAT file (or  for single file): climat_data_200707.txt

7798:Add a dummy normals line? (Y/N): Y

7799:

7800:All Files Processed

7801:tmp.0711272219.dtb: 2881 stations written

7802:vap.0711272219.dtb: 2870 stations written

7803:rdy.0711272219.dtb: 2876 stations written

7804:pre.0711272219.dtb: 2878 stations written

7805:sun.0711272219.dtb: 2020 stations written

7806:tmn.0711272219.dtb: 2800 stations written

7807:tmx.0711272219.dtb: 2800 stations written

7808:

7809:Thanks for playing! Byeee!

7810:

7811:

7812:So.. NOW can I merge CLIMAT into MCDW?!

7813:

7814:As expected, thank goodness:

7815:

7816:

7817:uealogin1[/cru/cruts/version_3_0/incoming/merge_CLIMAT_into_MCDW] ./newmergedb

7818:

7819:WELCOME TO THE DATABASE UPDATER

7820:

7821:Before we get started, an important question:

7822:If you are merging an update - CLIMAT, MCDW, Australian - do

7823:you want the quick and dirty approach? This will blindly match

7824:on WMO codes alone, ignoring data/metadata checks, and making any

7825:unmatched updates into new stations (metadata permitting)?

7826:

7827:Enter 'B' for blind merging, or : B

7828:Please enter the Master Database name: sun.0711272156.dtb

7829:Please enter the Update Database name: sun.0711272219.dtb

7830:

7831:Reading in both databases..

7832:Master database stations:     2184

7833:Update database stations:     2020

7834:

7835:Looking for WMO code matches..

7836:   28 reject(s) from update process 0711272225

7837:

7838:Writing sun.0711272225.dtb

7839:

7840:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

7841:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

7842:

7843:OUTPUT(S) WRITTEN

7844:

7845:New master database: sun.0711272225.dtb

7846:

7847:Update database stations:         2020

7848: > Matched with Master stations:  1775

7849:                 (automatically:  1775)

7850:                   (by operator:     0)

7851: > Added as new Master stations:   217

7852: > Rejected:                        28

7853:   Rejects file:                 sun.0711272219.dtb.rejected

7854:

7855:

7856:Wahey! Lots of stations to play with!

7857:

7858:So, next.. convert to cloud!

7859:

7860:

7861:crua6[/cru/cruts/version_3_0/db/cld] ./Hsh2cld 

7862:

7863:Hsh2cld - Convert a Sun Hours database to a Cloud Percent one

7864:

7865:Please enter the Sun Hours database: sun.0711272225.dtb

7866:Data Factor detected: *1.000

7867:

7868:Completed -     2401 stations converted.

7869:

7870:Sun Percentage Database:   spc.0711272230.dtb

7871:Cloud Percentage Database: cld.0711272230.dtb

7872:

7873:

7874:So.. bated breath..

7875:

7876:..and yay!

7877:

7878:

7879:crua6[/cru/cruts/version_3_0/secondaries/cld] ./anomdtb

7880: 

7881:   > ***** AnomDTB: converts .dtb to anom .txt for gridding *****

7882: 

7883:   > Enter the suffix of the variable required:

7884:.cld

7885:   > Select the .cts or .dtb file to load:

7886:cld.0711272230.dtb

7887:

7888:   > Specify the start,end of the normals period: 

7889:1995,2002

7890:   > Specify the missing percentage permitted: 

7891:12.5

7892:   > Data required for a normal:            7

7893:   > Specify the no. of stdevs at which to reject data: 

7894:3

7895:   > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 

7896:3

7897:   > Check for duplicate stns after anomalising? (0=no,>0=km range)

7898:0

7899:   > Select the generic .txt file to save (yy.mm=auto):

7900:cld.txt

7901:   > Select the first,last years AD to save: 

7902:1995,2007

7903:   > Operating...

7904:

7905: /tmp_mnt/cru-auto/cruts/version_3_0/secondaries/cld/cld.0711272230.dts         

7906:                                                                                

7907:   > NORMALS            MEAN percent      STDEV percent

7908:   >         .dtb          0     0.0

7909:   >         .cts      83961    49.3      83961    49.3

7910:   > PROCESS        DECISION percent %of-chk

7911:   > no lat/lon           95     0.1     0.1

7912:   > no normal         86174    50.6    50.7

7913:   > out-of-range         28     0.0     0.0

7914:   > accepted          83933    49.3

7915:   > Dumping years 1995-2007 to .txt files...

7916:

7917:

7918:Well.. a 'qualified' yay.. only half got normals! But I don't like to raise the 'missing percentage'

7919:limit to 25% because we're only talking about 8 values to begin with!!

7920:

7921:The output files look OK.. between 400 and 600 values in each, not a lot really but hey, better than

7922:nowt. So onto the conversion data (must stop calling 'em factors, they're not multiplicative).

7923:

7924:

7925:crua6[/cru/cruts/version_3_0/secondaries/cld] ./normshift 

7926:

7927:NORMSHIFT - Normals from any period

7928:

7929:Please enter the source file:   cru_ts_2_10.1901-2002.cld.grid

7930:Enter the start year of this file:  1901

7931:Enter the end year of this file:    2002

7932:Enter the normal period start year: 1995

7933:Enter the normal period end year:   2002

7934:Enter the 3-character parameter:    cld

7935:

7936:Normals file will be: clim.9502.to.6190.grid.cld

7937:

7938:

7939:So, erm.. now we need to create our synthetic cloud from DTR. Except that's the thing we CAN'T do because

7940:pro cal_cld_gts_tdm.pro needs those bloody coefficients (a.25.7190, etc) that went AWOL. Frustratingly we

7941:do have some of the outputs from the program (ie, a.25.01.7190.glo), but that's obviously no use.

7942:

7943:So, erm. We need synthetic cloud for 2003-2007, or we won't have enough data to run with. And yes it's

7944:taken me this long to realise that. Oh, bugger.

7945:

7946:Had a detailed search around Mark New's old disk (still online thankfully). Found this:

7947:

7948:

7949:crua6[/cru/mark1/markn/gts/cld/val] ls -l

7950:total 7584

7951:lrwxrwxrwx   1 f080     cru           25 Sep 12  2005 c1 -> /cru/u1/f080/isccp/c1_mon

7952:-rw-r--r--   1 f080     cru         1290 Mar 24  1998 cld_corr.j

7953:-rw-r--r--   1 f080     cru          938 Mar 17  1998 cld_scat.j

7954:-rw-r-----   1 f080     cru       922584 Mar 24  1998 cru_hahn_corr.ps

7955:-rw-r-----   1 f080     cru       922588 Mar 24  1998 cru_isccp_corr.ps

7956:-rw-r-----   1 f080     cru          533 Mar 27  1998 cruobs_hahn_corr.j

7957:-rw-r-----   1 f080     cru       868561 Mar 27  1998 cruobs_hahn_corr.ps

7958:-rw-r--r--   1 f080     cru          697 Mar 20  1998 dtr_corr.j

7959:-rw-r-----   1 f080     cru           50 Mar 27  1998 foo

7960:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1980

7961:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1981

7962:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1982

7963:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1983

7964:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1984

7965:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1985

7966:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1986

7967:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1987

7968:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1988

7969:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1989

7970:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1990

7971:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1991

7972:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1992

7973:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1993

7974:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1994

7975:-rw-r-----   1 f080     cru       248832 Mar 27  1998 glo25.cld.1995

7976:-rw-r-----   1 f080     cru       922592 Mar 24  1998 hahn_isccp_corr.ps

7977:-rw-r-----   1 f080     cru         2378 Mar 24  1998 test.j

7978:

7979:

7980:..which looks to me like the place where he calculated the coefficients. The *.j files are IDL 'Journal' files,

7981:so can be run from within IDL. This was my first attempt:

7982:

7983:

7984:IDL> .run cld_corr.j

7985:% Compiled module: $MAIN$.

7986:% Compiled module: RD25_GTS.

7987:YEAR:    1981

7988:% Compiled module: RDBIN.

7989:% Compiled module: STRIP.

7990:foo: Permission denied.

7991:foo: Permission denied.

7992:foo: Permission denied.

7993:% OPENR: Error opening file. Unit: 99, File: /home/cru/f098/u1/hahn/hahn25.1981

7994:  No such file or directory

7995:% Execution halted at:  RDBIN              63 /cru/u2/f080/Idl/rdbin.pro

7996:%                       RD25_GTS           11 /cru/u2/f080/Idl/rd25_gts.pro

7997:%                       $MAIN$              1 /tmp_mnt/cru-auto/mark1/f080/gts/cld/val/cld_corr.j

7998:IDL> 

7999:

8000:

8001:I then had to chase around to find three sets of missing files.. to fulfil these five conditions:

8002:

8003:if keyword_set(hgrid) eq 0 then rd25_gts,$

8004:    hgrid,'~/u1/hahn/hahn25.',1981,1991

8005:if keyword_set(rgrid) eq 0 then rd25_gts,$

8006:    rgrid,'../glo_reg_25/glo.cld.',1981,1991

8007:if keyword_set(hgrid2) eq 0 then rd25_gts,$

8008:    hgrid2,'~/u1/hahn/hahn25.',1983,1991     

8009:if keyword_set(igrid) eq 0 then rdisccp_gts,$

8010:    igrid,'c1/isccp.',1983,1991

8011:if keyword_set(rgrid2) eq 0 then rd25_gts,$

8012:    rgrid2,'../glo_reg_25/glo.cld.',1983,1991

8013:

8014:I managed to find the hahn25 files (on Mark's disk), and some likely-looking isccp files (also on Mark's disk).

8015:But although there were plenty of files with 'glo', 'cld' and '25' in them, there were none matching the filename

8016:construction above. However, as some of those were in the same directory - I'll take that chance!!

8017:

8018:I did try, honestly. Very hard. I found all the files, and put them in directories. I made a local copy of the job

8019:file, 'H_cld_corr.j', with the local directory refs in. Hell, I even precompiled the correct version of rdbin!

8020:

8021:All for nothing, as usual. It runs quite happily, zipping through things, until:

8022:

8023:% Compiled module: RDISCCP_GTS.

8024:YEAR:    1983

8025:% Compiled module: RDISCCP.

8026:c1/isccp.83.07.72

8027:c1/isccp.83.07.72.Z: No such file or directory

8028:c1/isccp.83.08.72

8029:c1/isccp.83.08.72.Z: No such file or directory

8030:c1/isccp.83.09.72

8031:c1/isccp.83.09.72.Z: No such file or directory

8032:c1/isccp.83.10.72

8033:c1/isccp.83.10.72.Z: No such file or directory

8034:c1/isccp.83.11.72

8035:c1/isccp.83.11.72.Z: No such file or directory

8036:c1/isccp.83.12.72

8037:c1/isccp.83.12.72.Z: No such file or directory

8038:YEAR:    1984

8039:c1/isccp.84.01.72

8040:c1/isccp.84.01.72.Z: No such file or directory

8041:(etc)

8042:

8043:It isn't seeing the isccp files EVEN THOUGH THEY ARE THERE. Odd. If I create Z files it says they aren't compressed.

8044:

8045:It ends with:

8046:

8047:YEAR:    1991

8048:yes

8049:filesize=      248832

8050:gridsize=      2.50000

8051:% Compiled module: MARK_CORRELATE.

8052:% Compiled module: CORRELATE.

8053:

8054:I have no idea what it's actually done though. It doesn't appear to have produced anything.. ah:

8055:

8056:

8057:IDL> help

8058:% At  $MAIN$              1 /tmp_mnt/cru-auto/cruts/version_3_0/cloud_synthetics/H_cld_corr.j

8059:CRU_HAHN_CORR   FLOAT     = Array[144, 72, 12]

8060:CRU_ISCCP_CORR  FLOAT     = Array[144, 72, 12]

8061:HGRID           LONG      = Array[144, 72, 12, 11]

8062:HGRID2          LONG      = Array[144, 72, 12, 9]

8063:IGRID           LONG      = Array[144, 72, 12, 9]

8064:ILAT            INT       =       72

8065:ILON            INT       =      144

8066:IM              INT       =       12

8067:ISCCP_HAHN_CORR FLOAT     = Array[144, 72, 12]

8068:N               LONG      = Array[5225]

8069:NN              LONG      =         5225

8070:RGRID           LONG      = Array[144, 72, 12, 11]

8071:RGRID2          LONG      = Array[144, 72, 12, 9]

8072:Compiled Procedures:

8073:    $MAIN$  DEFXYZ      RD25_GTS    RDBIN       RDISCCP     RDISCCP_GTS

8074:

8075:Compiled Functions:

8076: CORRELATE   MARK_CORRELATE          STRIP

8077:

8078:IDL> 

8079:

8080:..so this is one of a set of tools *that you have to know how to use*. All the work's done in the IDl data space.

8081:

8082:Well as we don't have any instructions, that's a complete waste of two-and-a-half days' time.

8083:

8084:Let's forget about CLD and start worrying about NetCDF.

8085:

8086:

8087:NETDCF

8088:

8089:Well now, we have to make the data available in NetCDF and ASCII grid formats. At the moment, it might be best to

8090:just post-process the final ASCII grids into NetCDF; though more elegant to have mergegrids.for produce both! As it

8091:has the data there anyway.. so I modified mergegrids.for into makegrids.for, with added NetCDF goodness. as

8092:usual, lots of problems getting the syntax right..

8093:

8094:************************************************************************

8095:BADC Work.. at RAL, 3-5 December 2007

8096:

8097:Finally got NetCDF & Fortran working on the chosen server here (damp.badc.rl.ac.uk). I am definitely not a

8098:chamaeleonic life form when it comes to unfamiliar computer systems. Shame. The elusive command line compile

8099:statement is:

8100:

8101:gfortran -I/usr/local/netcdf-3.6.2/include/ -o fileout.o filein.f /usr/local/netcdf-3.6.2/lib/libnetcdf.a

8102:

8103:Hunting for CDDs I found a potential problem with binary DTR (used in the construction of Frost Days, Vapour

8104:Pressure, and (eventually) Cloud. It looks as though there was a mistyping when the 2.5-degree binaries were

8105:constructed:

8106:

8107:IDL> quick_interp_tdm2,1901,2006,'dtrbin/dtrbin',50,gs=2.5,dumpbin='dumpbin',pts_prefix='dtrtxt/dtr.'

8108:

8109:That '50' should have been.. 750! Oh bugger. Well, might as well see if generation does work here. DTR/bin/2.5:

8110:

8111:..er.. hang on while I try and get IDL to recognise a path.. meh. As usual I find this effectively

8112:impossible, so have to issue manual .compile statements. The suite of progs required to compile for

8113:quick_interp_tdm2.pro is:

8114:

8115:glimit.pro

8116:area_grid.pro

8117:strip.pro

8118:wrbin.pro

8119:

8120:..and, of course, quick_interp_tdm2.pro. Actually, others need others.. so I wrote a

8121:generic IDL script to load all of Satan's little helpers:

8122:

8123:IDL> @../../programs/idl/loads4idl.j

8124:% Compiled module: GLIMIT.

8125:% Compiled module: AREA_GRID.

8126:% Compiled module: STRIP.

8127:% Compiled module: WRBIN.

8128:% Compiled module: RDBIN.

8129:% Compiled module: DEFXYZ.

8130:% Compiled module: FRSCAL.

8131:% Compiled module: DAYS.

8132:% Compiled module: RNGE.

8133:% Compiled module: SAVEGLO.

8134:% Compiled module: SELECTMODEL.

8135:% Compiled module: TVAP.

8136:% Compiled module: ESAT.

8137:IDL>

8138:

8139:This is just because I still don't have IDL_PATH working so

8140:having to issue each of the above as manual compile statements (in that order)

8141:was getting tedious. n00b. [this now fixed - ed] Anyway, here's the corrected

8142:binary DTR production:

8143:

8144:IDL> quick_interp_tdm2,1901,2006,'dtrbin/dtrbin',750,gs=2.5,dumpbin='dumpbin',pts_prefix='dtrtxt/dtr.'

8145:Defaults set

8146:    1901

8147:% Compiled module: MAP_SET.

8148:% Compiled module: CROSSP.

8149:% Compiled module: MEAN.

8150:% Compiled module: MOMENT.

8151:% Compiled module: STDDEV.

8152:grid 1901 non-zero    0.9415    1.8771    1.8417 cells=     5608

8153:    1902

8154:grid 1902 non-zero    0.8608    1.8713    1.8752 cells=     5569

8155:(etc)

8156:

8157:And so to regenerate FRS:

8158:

8159:

8160:IDL> frs_gts,dtr_prefix='dtrbin/dtrbin',tmp_prefix='tmpbin/tmpbin',1901,2006,outprefix='frssyn/frssyn'

8161:IDL> quick_interp_tdm2,1901,2006,'frsgrid/frsgrid',750,gs=0.5,dumpglo='dumpglo', nostn=1,synth_prefix='frssyn/frssyn'

8162:-bash-3.00$ ./glo2abs

8163:Welcome! This is the GLO2ABS program.

8164:I will create a set of absolute grids from

8165:a set of anomaly grids (in .glo format), also

8166:a gridded version of the climatology.

8167:Enter the path and name of the normals file: clim.6190.lan.frs

8168:Enter a name for the gridded climatology file: clim.6190.lan.frs.delme

8169:Enter the path and stem of the .glo files: frsgrid/frs.

8170:Enter the starting year: 1901

8171:Enter the ending year:   2006

8172:Enter the path (if any) for the output files: frsabs/

8173:Now, CONCENTRATE. Addition or Percentage (A/P)? A

8174:Do you wish to limit the output values? (Y/N): Y

8175:1. Set minimum to zero

8176:2. Set a single minimum and maximum

8177:3. Set monthly minima and maxima (for wet/rd0)

8178:4. Set all values >0, (ie, positive)

8179:Choose: 3

8180:Right, erm.. off I jolly well go!

8181:frs.01.1901.glo

8182:frs.02.1901.glo

8183:(etc)

8184:

8185:

8186:Now looking to get makegrids.for working.. managed to get the data to write by declaring REALs as

8187:DOUBLE PRECISION - later realising I could/should have changed the NetCDF interface calls to REAL

8188:instead! Ah well. Still tussling with the 'time' variable.. not clear how to handle observations.

8189:Luckily, Mike S knew what the standard was:

8190:

8191:

8192:>I need to define the time parameter in the NetCDF version of the CRUTS

8193:>dataset. I suspect I need to use 'Gregorian' (which to all intents and

8194:>porpoises is accurate although it reverts to Julian before xx/yy/1582) but

8195:>I wondered if there was a convention (in CRU, or wider) for allocating

8196:>standard timestamps to observations?

8197:

8198:For the gridded temperature we use

8199:

8200:         short time(time) ;

8201:                 time:units = "months since 1870-1-1" ;

8202:

8203:Remember to start with zero!

8204:

8205:Mike

8206:

8207:

8208:And that seems to now be working! Here's the run with the compile statement included:

8209:

8210:

8211:-bash-3.00$ gfortran -I/usr/local/netcdf-3.6.2/include/ -o makegrids ../../programs/fortran/makegrids.for /usr/local/netcdf-3.6.2/lib/libnetcdf.a

8212:-bash-3.00$ ./makegrids Welcome! This is the MAKEGRIDS program.

8213:I will create decadal and full gridded files,

8214:in both ASCII text and NetCDF formats, from

8215:the output files of (eg) glo2abs.for.

8216:

8217:Writing: cru_ts_3_00.1901.1910.frs.dat

8218:         cru_ts_3_00.1901.1910.frs.nc

8219:Writing: cru_ts_3_00.1911.1920.frs.dat

8220:         cru_ts_3_00.1911.1920.frs.nc

8221:Writing: cru_ts_3_00.1921.1930.frs.dat

8222:         cru_ts_3_00.1921.1930.frs.nc

8223:Writing: cru_ts_3_00.1931.1940.frs.dat

8224:         cru_ts_3_00.1931.1940.frs.nc

8225:Writing: cru_ts_3_00.1941.1950.frs.dat

8226:         cru_ts_3_00.1941.1950.frs.nc

8227:Writing: cru_ts_3_00.1951.1960.frs.dat

8228:         cru_ts_3_00.1951.1960.frs.nc

8229:Writing: cru_ts_3_00.1961.1970.frs.dat

8230:         cru_ts_3_00.1961.1970.frs.nc

8231:Writing: cru_ts_3_00.1971.1980.frs.dat

8232:         cru_ts_3_00.1971.1980.frs.nc

8233:Writing: cru_ts_3_00.1981.1990.frs.dat

8234:         cru_ts_3_00.1981.1990.frs.nc

8235:Writing: cru_ts_3_00.1991.2000.frs.dat

8236:         cru_ts_3_00.1991.2000.frs.nc

8237:Writing: cru_ts_3_00.2001.2006.frs.dat

8238:         cru_ts_3_00.2001.2006.frs.nc

8239:-bash-3.00$

8240:

8241:

8242:And here, for a combination of posterity and boredom, is a (curtailed) dump from ncdump:

8243:

8244:

8245:-bash-3.00$ ncdump cru_ts_3_00.1901.2006.frs.nc |head -300

8246:netcdf cru_ts_3_00.1901.2006.frs {

8247:dimensions:

8248:        lon = 720 ;

8249:        lat = 360 ;

8250:        time = UNLIMITED ; // (1272 currently)

8251:variables:

8252:        double lon(lon) ;

8253:                lon:long_name = "longitude" ;

8254:                lon:units = "degrees_east" ;

8255:        double lat(lat) ;

8256:                lat:long_name = "latitude" ;

8257:                lat:units = "degrees_north" ;

8258:        int time(time) ;

8259:                time:long_name = "time" ;

8260:                time:units = "months since 1870-1-1" ;

8261:                time:calendar = "standard" ;

8262:        double frs(time, lat, lon) ;

8263:                frs:long_name = "ground frost frequency" ;

8264:                frs:units = "days" ;

8265:                frs:scale_factor = 0.00999999977648258 ;

8266:                frs:correlation_decay_distance = 750. ;

8267:                frs:_FillValue = -9999. ;

8268:                frs:missing_value = -9999. ;

8269:

8270:// global attributes:

8271:                :title = "CRU TS 3.00 Mean Temperature" ;

8272:                :institution = "BADC" ;

8273:                :contact = "BADC " ;

8274:data:

8275:

8276: lon = -179.75, -179.25, -178.75, -178.25, -177.75, -177.25, -176.75,

8277:    -176.25, -175.75, -175.25, 

8278:                              (etc)

8279:                                    170.75, 171.25, 171.75, 172.25, 172.75,

8280:    173.25, 173.75, 174.25, 174.75, 175.25, 175.75, 176.25, 176.75, 177.25,

8281:    177.75, 178.25, 178.75, 179.25, 179.75 ;

8282:

8283: lat = -89.75, -89.25, -88.75, -88.25, -87.75, -87.25, -86.75, -86.25,

8284:    -85.75, -85.25, -84.75, 

8285:                              (etc)

8286:                                       79.75, 80.25, 80.75, 81.25, 81.75,

8287:    82.25, 82.75, 83.25, 83.75, 84.25, 84.75, 85.25, 85.75, 86.25, 86.75,

8288:    87.25, 87.75, 88.25, 88.75, 89.25, 89.75 ;

8289:

8290: time = 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385,

8291:    386, 387, 388, 389, 390,

8292:                              (etc)

8293:                                        1620, 1621, 1622, 1623, 1624, 1625,

8294:    1626, 1627, 1628, 1629, 1630, 1631, 1632, 1633, 1634, 1635, 1636, 1637,

8295:    1638, 1639, 1640, 1641, 1642, 1643 ;

8296:

8297: frs =

8298:  -999, -999, -999, -999, -999, -999, -999, -999, -999, -999, -999, -999,

8299:  (etc - probably some real data there somewhere)

8300:

8301:-bash-3.00$

8302:

8303:

8304:And VAP:

8305:

8306:IDL> vap_gts_anom,dtr_prefix='dtrbin/dtrbin',tmp_prefix='tmpbin/tmpbin',1901,2006,outprefix='vapsyn/vapsyn.',dumpbin=1

8307:% Compiled module: VAP_GTS_ANOM.

8308:% Compiled module: RDBIN.

8309:% Compiled module: STRIP.

8310:% Compiled module: DEFXYZ.

8311:Land,sea:       56016       68400

8312:Calculating tmn normal

8313:% Compiled module: TVAP.

8314:Calculating synthetic vap normal

8315:% Compiled module: ESAT.

8316:Calculating synthetic anomalies

8317:% Compiled module: MOMENT.

8318:1901 vap (x,s2,<<,>>):  1.67770e-05  6.23626e-06    -0.160509     0.222689

8319:1902 vap (x,s2,<<,>>): -0.000122533  3.46933e-05    -0.268891    0.0644855

8320:(etc)

8321:

8322:

8323:These numbers are different from the original runs - so that was a genuine mistyping. Eek, that's not

8324:very promising, is it?

8325:

8326:

8327:IDL> quick_interp_tdm2,1901,2006,'vapglo/vap.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='vapsyn/vapsyn.',pts_prefix='vaptxt/vap.'

8328:% Compiled module: QUICK_INTERP_TDM2.

8329:% Compiled module: GLIMIT.

8330:Defaults set

8331:    1901

8332:% Compiled module: RDBIN.

8333:% Compiled module: STRIP.

8334:% Compiled module: DEFXYZ.

8335:% Compiled module: MAP_SET.

8336:% Compiled module: CROSSP.

8337:% Compiled module: SAVEGLO.

8338:% Compiled module: SELECTMODEL.

8339:    1902

8340:(etc)

8341:

8342:-bash-3.00$ ./glo2abs

8343:Welcome! This is the GLO2ABS program.

8344:I will create a set of absolute grids from

8345:a set of anomaly grids (in .glo format), also

8346:a gridded version of the climatology.

8347:Enter the path and name of the normals file: clim.6190.lan.vap

8348:Enter a name for the gridded climatology file: clim.6190.lan.vap.delme

8349:Enter the path and stem of the .glo files: vapglo/vap.

8350:Enter the starting year: 1901

8351:Enter the ending year:   2006

8352:Enter the path (if any) for the output files: vapabs/

8353:Now, CONCENTRATE. Addition or Percentage (A/P)? A

8354:Do you wish to limit the output values? (Y/N): Y

8355:1. Set minimum to zero

8356:2. Set a single minimum and maximum

8357:3. Set monthly minima and maxima (for wet/rd0)

8358:4. Set all values >0, (ie, positive)

8359:Choose: 4

8360:Right, erm.. off I jolly well go!

8361:vap.01.1901.glo

8362:vap.02.1901.glo

8363:(etc)

8364:

8365:-bash-3.00$ ./makegrids

8366:Welcome! This is the MAKEGRIDS program.

8367:I will create decadal and full gridded files,

8368:in both ASCII text and NetCDF formats, from

8369:the output files of (eg) glo2abs.for.

8370:

8371:Enter a gridfile with YYYY for year and MM for month: vapabs/vap.MM.YYYY.glo.abs

8372:Enter Start Year:  1901

8373:Enter Start Month: 01

8374:Enter End Year:    2006

8375:Enter End Month:   12

8376:

8377:Please enter a sample OUTPUT filename, replacing

8378:start year with SSSS and end year with EEEE, and

8379:ending with '.dat', eg: cru_ts_3_00.SSSS.EEEE.tmp.dat : cru_ts_3_00.SSSS.EEEE.vap.dat

8380:

8381:Now please enter the 3-ch parameter code: vap

8382:Enter a generic title for this dataset, eg:

8383:CRU TS 3.00 Mean Temperature : CRU TS 3.00 Vapour Pressure

8384:Writing: cru_ts_3_00.1901.1910.vap.dat

8385:         cru_ts_3_00.1901.1910.vap.nc

8386:Writing: cru_ts_3_00.1911.1920.vap.dat

8387:         cru_ts_3_00.1911.1920.vap.nc

8388:Writing: cru_ts_3_00.1921.1930.vap.dat

8389:         cru_ts_3_00.1921.1930.vap.nc

8390:Writing: cru_ts_3_00.1931.1940.vap.dat

8391:         cru_ts_3_00.1931.1940.vap.nc

8392:Writing: cru_ts_3_00.1941.1950.vap.dat

8393:         cru_ts_3_00.1941.1950.vap.nc

8394:Writing: cru_ts_3_00.1951.1960.vap.dat

8395:         cru_ts_3_00.1951.1960.vap.nc

8396:Writing: cru_ts_3_00.1961.1970.vap.dat

8397:         cru_ts_3_00.1961.1970.vap.nc

8398:Writing: cru_ts_3_00.1971.1980.vap.dat

8399:         cru_ts_3_00.1971.1980.vap.nc

8400:Writing: cru_ts_3_00.1981.1990.vap.dat

8401:         cru_ts_3_00.1981.1990.vap.nc

8402:Writing: cru_ts_3_00.1991.2000.vap.dat

8403:         cru_ts_3_00.1991.2000.vap.nc

8404:Writing: cru_ts_3_00.2001.2006.vap.dat

8405:         cru_ts_3_00.2001.2006.vap.nc

8406:-bash-3.00$

8407:

8408:

8409:A quick look at the VAP NetCDF headers & data looked good. So - yay, that's the damage repaired,

8410:pity it took over a day of the time at RAL. But I didn't have to fix it now - it was an opportunity

8411:to get the process working in this environment.

8412:

8413:Next problem - station counts. I had this working fine in CRU - here it's insisting on stopping

8414:indefinitely at January 1957. Discovered - after 36 hours of fretting and debugging - that it's

8415:popping its clogs on the South Polar base:

8416:

8417: 890090  -900      0 2853 AMUNDSEN-SCOTT       ANTARCTICA    1957 2006 101957  -999.00

8418:

8419:And what d'you know, when I debug it, it's as simple as being too close to the pole and not having

8420:any loop restrictions in the East and West hunts for valid cells.. just looping forever! Added

8421:a few simple conditionals and all seems to run.. but outputs don't look right, the Jan 1957 station

8422:counts have missing values for the polar regions.

8423:

8424:Managed to get anomdtb compiled with gfortran, after altering a few lines (in anomdtb and its mods)

8425:where Tim had shrugged off the surly bounds of strict F90.. it must be compiled in programs/fortran/

8426:though, with the line (embedded in the anomdtb comments too):

8427:

8428:gfortran -o anomdtb filenames.f90 time.f90 grimfiles.f90 crutsfiles.f90 loadperfiles.f90 

8429:                    saveperfiles.f90 annfiles.f90 cetgeneral.f90 basicfun.f90 wmokey.f90

8430:                    gridops.f90 grid.f90 ctyfiles.f90 anomdtb.f90

8431:

8432:As part of the modifications I removed the unused options - meaning that a .dts file is no longer

8433:required (and, of course, neither is 'falsedts.for'). Ran it for the temperature database and got

8434:apparently-identical anomaly files to the ones I generated in CRU :-))) Ran quick_interp_tdm2,

8435:glo2abs and makegrids, ended up with grids very similar (though sadly not identical) to the

8436:originals (could be IDL, could be compiler, could be system).

8437:

8438:Scripting. Now this was always going to be the challenge, for a large suite of highly-interactive

8439:programs in F77, F90 and IDL which didn't follow universal file naming conventions. So to start with,

8440:I thought it might be 'fun' to compile an exhaustive/ing list of the commands needed to make a

8441:complete update. Headings begin with *, notes are in brackets

8442:

8443:* Add MCDW Updates

8444:mcdw2cru (interactive)

8445:newmergedb (per parameter, interactive)

8446:* Add CLIMAT Updates

8447:climat2cru (interactive)

8448:newmergedb (per parameter, interactive)

8449:* Add BOM Updates

8450:au2cru (unfinished, interactive, should do whole job)

8451:* Regenerate DTR Database

8452:tmnx2dtr (interactive)

8453:* Produce Primary Parameters (TMP, TMN, TMX, DTR, PRE)

8454:anomdtb (per parameter, interactive)

8455:quick_interp_tdm2 (per parameter)

8456:glo2abs (per parameter, interactive)

8457:makegrids (per parameter, interactive)

8458:* Prepare Binary Grids (TMP, DTR, PRE) for Synthetics

8459:quick_interp_tdm2 (per parameter)

8460:* Produce Secondary Parameter (FRS, uses TMP,DTR)

8461:frs_gts_tdm

8462:quick_interp_tdm2

8463:glo2abs (interactive)

8464:makegrids (interactive)

8465:* Produce Secondary Parameter (VAP, uses TMP,DTR)

8466:vap_gts_anom

8467:anomdtb (interactive)

8468:quick_interp_tdm2

8469:glo2abs (interactive)

8470:makegrids (interactive)

8471:* Produce Secondary Parameter (WET/RD0, uses PRE)

8472:rd0_gts_anom

8473:anomdtb (interactive)

8474:quick_interp_tdm2

8475:glo2abs (interactive)

8476:makegrids (interactive)

8477:

8478:

8479:*** BACK AT CRU ***

8480:

8481:Tried to compile makegrids.for on uealogin1 (as crua6 is being retired). Got an odd error:

8482:

8483:ld: warning: file /usr/local/netcdf-3.6.1/lib/libnetcdf.a(fort-attio.o): wrong ELF class: ELFCLASS64

8484:

8485:..which was cured with the addition of '-xarch=native64' to the compile statement:

8486:

8487:f77 -xarch=native64 -I/usr/local/netcdf-3.6.1/include/ -o makegrids ../../BADC_AREA/programs/fortran/makegrids.for /usr/local/netcdf-3.6.1/lib/libnetcdf.a

8488:

8489:Then had to play around to try and reduce the size of the NetCDF files - they were bigger than the uncompressedASCII ones! This

8490:was because the variable was declared as DOUBLE, which is 64 bits, or 8 bytes, per datum. A waste, since we deal with the data

8491:as integers and use factors to restore 'real' values. So redeclared as INT. Considering re-redeclaring as SHORT, which is 16

8492:bits to INT's 32.. however, that only gives me signed -32,768 to 32,767 or unsigned 0 to 65,535. That's enough for our datasets

8493:but only if precip has a positive missing value code, which I don't like the sound of.

8494:

8495:Reproduced all primaries and secondaries with INT typing for the NetCDF component.

8496:

8497:Simultaneously trying to work out why stncounts.for is apparently ignoring the South Pole station

8498:(Amundsen-Scott) even though the rest of the output looks fine.. eventually realised that the land/sea mask is blobking it!!

8499:

8500:Station counts work continues.. should the NetCDF files be written as INT to match the data files, or SHORT to save a lot of space?

8501:In fact, should the station counts be in the same NetCDF files as their data?!!

8502:

8503:Finished off the local regenration of VAP and FRS from the corrected dtrbin files:

8504:

8505:DTR fix:

8506:IDL> quick_interp_tdm2,1901,2006,'dtrbin/dtrbin',750,gs=2.5,dumpbin='dumpbin',pts_prefix='dtrtxt/dtr.'

8507:

8508:VAP binary regen:

8509:IDL> vap_gts_anom,dtr_prefix='dtrbin/dtrbin',tmp_prefix='tmpbin/tmpbin',1901,2006,outprefix='vapsyn/vapsyn.',dumpbin=1

8510:

8511:Note that, as expected, results ARE different for synthetic vap!

8512:

8513:First two lines of the CRUA6 run:

8514:

8515:1901 vap (x,s2,<<,>>):  1.53894e-05  6.16462e-06    -0.160509     0.222662

8516:1902 vap (x,s2,<<,>>): -0.000123921  3.46215e-05    -0.268891    0.0261302

8517:

8518:First two lines of the damp.badc.rl.ac.uk run:

8519:

8520:1901 vap (x,s2,<<,>>):  1.67770e-05  6.23626e-06    -0.160509     0.222689

8521:1902 vap (x,s2,<<,>>): -0.000122533  3.46933e-05    -0.268891    0.0644855

8522:

8523:Different compilers.. different architectures?.. different OS.. whichever.

8524:

8525:IDL> quick_interp_tdm2,1901,2006,'vapglo/vap.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='vapsyn/vapsyn.',pts_prefix='vaptxt/vap.'

8526:

8527:Paused here as waiting for completion of makegrids2.for (includes station counts in NetCDF files). One early ramification of this is

8528:that glo2abs.for now saves filenames with year.month, rather than replicating the loopy month.year that quick_interp_tdm2.pro gives.

8529:This will allow file lists to be compiled with 'ls >tmpfile', so that timespan and missing files can be detected at the start. Have

8530:also had to amend (and re-run at great length!) stnounts.for to do the same.

8531:

8532:Just realised I still haven't worked out how to do the station counts for the secondaries. I can't work out how Tim did it either,

8533:since he worked out counts at the same stage I do. Did he let the primary parameter's counts override? Or backfill? Well we could

8534:take the approach that the gridding routine takes, namely to use observed data and only refer to synthetic when that fails (and only

8535:refer to normals when that fails, obviously).

8536:

8537:So, stncounts will have to accept TWO sets of .txt files. At each timestep it will have to first count the secondary parameter's

8538:stations, then the primary parameter(s) will be counted and fill in any zeros in the grid. However, we will need different information

8539:for the paper - what use is the effective station count as some is of higher 'quality' than the rest? So will probably need the regular

8540:observed-only count as well.. which could be a separate run of stncounts but faaar more sensible to be a side effect of this run.

8541:

8542:Oh Gods.. I've got to modify another program. *cries*

8543:

8544:I think the mods to stncounts will be the turning point for all the programs. So far, they have all been generic, but this is not

8545:tenable if the system is to be automated - they need to be aware of the parameters and what they mean. Otherwise stncounts will have

8546:to be told how many primaries produced the synthetic grids, etc, etc - stupid. So I need to devise a directory structure and file

8547:naming schema that will support the entire update process. Eeeeeeeek.

8548:

8549:

8550:

8551:** time passes.. other projects given higher priority.. **

8552:

8553:

8554:

8555:Back to precip, it seems the variability is too low. This points to a problem with the percentage anomaly routines. See earlier

8556:escapades - will the Curse of Tim never be lifted?

8557:

8558:A reminder. I started off using a 'conventional' calculation:

8559:

8560:              absgrid(ilon(i),ilat(i)) = nint(normals(i,imo) +

8561:     *                                          anoms(ilon(i),ilat(i)) * normals(i,imo) / 100)

8562:            which is: V = N + AN/100

8563:

8564:This was shown to be delivering unrealistic values, so I went back to anomdtb to see how the anomalies were contructed in the

8565:first place, and found this:

8566:

8567:              DataA(XAYear,XMonth,XAStn) = nint(1000.0*((real(DataA(XAYear,XMonth,XAStn)) / &

8568:                				real(NormMean(XMonth,XAStn)))-1.0))

8569:            which is: A = 1000((V/N)-1)

8570:

8571:So, I reverse engineered that to get this: V = N(A+1000)/1000

8572:

8573:And that is apparently also delivering incorrect values. Bwaaaahh!!

8574:

8575:Modified anomdtb to dump the precip anomaly calculation. It seems to be working with raw values, eg:

8576:

8577:DataA:     1050, NormMean:   712.00, Anom:      475

8578:DataA:      270, NormMean:   712.00, Anom:     -621

8579:DataA:      710, NormMean:   712.00, Anom:       -3

8580:DataA:      430, NormMean:   712.00, Anom:     -396

8581:DataA:      280, NormMean:   712.00, Anom:     -607

8582:DataA:      830, NormMean:   712.00, Anom:      166

8583:DataA:        0, NormMean:   712.00, Anom:    -1000

8584:DataA:      270, NormMean:   712.00, Anom:     -621

8585:DataA:      280, NormMean:   712.00, Anom:     -607

8586:DataA:      450, NormMean:   712.00, Anom:     -368

8587:DataA:      180, NormMean:   712.00, Anom:     -747

8588:DataA:     1380, NormMean:   712.00, Anom:      938

8589:

8590:However, that -1000 is interesting for zero precip. It looks as though the anomalies are in mm*10, like

8591:the precip database raw values.. but no! These are dumped from within the program, the output .txt files

8592:have -100 instead of -1000. That's because of the CheckVariSuffix routine, which returns a factor based

8593:on the parameter code, including:

8594:

8595:else if (Suffix.EQ.".pre") then

8596:  	Variable="precipitation (mm)"

8597:  	Factor = 0.1

8598:

8599:Factor is then used when the anomaly files are written:

8600:

8601:       do XAStn = 1, NAStn

8602:         if (DataA(XAYear,XMonth,XAStn).NE.-9999) write (9,"(2f8.2,f8.1,f13.5,i7)"), &

8603:           ALat(XAStn),ALon(XAStn),AElv(XAStn),real(DataA(XAYear,XMonth,XAStn))*Factor,AStn(XAStn)

8604:       end do

8605:

8606:So, the anomalies are in real units (presumably mm/day).

8607:

8608:So.. we grid these values. The resulting anomalies (for Jan 1980) look like this:

8609:

8610:max = 1612.4

8611:min = -100

8612:

8613:These should be applied to the climatology (normals). I think they can be applied to either unscaled 'real'

8614:value normals or to normals which are mm*10. Results will be scaled accordingly. So.. let's look at glo2abs.

8615:Again.

8616:

8617:The current formula to convert to real values is:

8618:

8619:              absgrid(ilon(i),ilat(i)) = nint((rnormals(i,imo)*

8620:     *                                     (anoms(ilon(i),ilat(i))+1000)/1000)/rmult)

8621:              V = N(A+1000)/1000*rmult

8622:

8623:Not happy with that anyway. The multiplicative factor.. should that be there at all?

8624:

8625:Now, if these are 'genuine' percentage anomalies - ie, they represent the percentage change from the mean,

8626:then the formula to convert them using unscaled normals would be:

8627:

8628:     V = N + N(A/100)

8629:

8630:For instance, -100 would give V = 0, and 100 would give V = 2N. Now if the normals are *10, surely the

8631:results will be *10 too? As each term has N as a multiplicative factor anyway. This makes

8632:me wonder about the prevailing theory that the anomalies need to be *10. They are

8633:fractions of the normal, so it shouldn't matter whether the normal is real or *10. The

8634:variability should be the same (ie, the variability of the anomalies).

8635:

8636:So, a run (just for 1980) of glo2abs, then, using the following formula:

8637:

8638:              absgrid(ilon(i),ilat(i)) = nint(rnormals(i,imo) +

8639:     *                           (anoms(ilon(i),ilat(i)) * rnormals(i,imo)) / 100)

8640:              V = N + A*N/100

8641:

8642:This should give integer values of mm*10 (because the normals are mm*10 and uncorrected).

8643:

8644:So, working backwards, what *should* the original anomalising routine in anomdtb.f90 look

8645:like? It should look like this:

8646:

8647:              A = 100*(V-N)/N,  or:    A = 100(V/N) - 100, or:    A = 100((V/N)-1)

8648:

8649:The last version is REMARKABLY similar to the original (A = 1000((V/N)-1)), in fact I

8650:think we can call equivalence if V and N are in mm? Complicated. The '100' is not a

8651:scaling factor, it's the number that determines a percentage calculation. If we use

8652:1000, what are we saying? The percentage anomalies we would have got are now 10x higher.

8653:Where V=0, A will be -1000, a meaningless percentage anomaly for precip. That's where

8654:the CheckVariSuffix factor pops up, multiplying by 0.1 and getting us back where we

8655:started.

8656:

8657:So.. Tim's original looks right, once you understand the correction factor applied later.

8658:

8659:In which case, my original algorithm should have worked!!

8660:

8661:A slightly Heath-Robinson attempt to verify.. extracted the cell for Wick from the 1980

8662:output. Wick is 58.45N, 3.08W - I reckon that's (297,353). I get:

8663:

8664:   106

8665:    69

8666:    91

8667:    22

8668:    16

8669:    74

8670:    79

8671:    80

8672:   129

8673:   156

8674:   177

8675:   151

8676:

8677:The station says:

8678:

8679:1980  763  582  718  153   95  557  587  658  987 1162 1346 1113

8680:

8681:**sigh** Yes, they do follow a similar shape.. it's just that the original station data

8682:are much higher. In terms of Up/Down (following direction):

8683:

8684:Station    DUDDUUUUUUD

8685:Gridded    DUDDUUUUUUD

8686:

8687:So, once again I don't understand statistics. Quel surprise, given that I haven't had any

8688:training in stats in my entire life, unless you count A-level maths.

8689:

8690:Normals for the same cell:

8691:

8692:Grid-ref= 353, 297

8693: 1060  740  870  600  610  670  700  910  990 1070 1200 1110

8694:

8695:Now, these look like mm*10, the same units as the station data and what I expected. If I

8696:apply the anomalies to these normals, it looks like I'll get what I'm after.. the trouble

8697:is that glo2abs.for deliberately works with real values and so applies the factor in the

8698:clim header before using the values. I just don't think that works with percentages.. hmm.

8699:

8700:Actually, it does. I ran through the algorithms and because the normal is multiplicative,

8701:you can do the scaling before or after. In other words, if V' is V produced with scaled

8702:normals (N*0.1) then we do end up with V = 10V'. So I just need to include the factor in

8703:the final equation:

8704:

8705:              V = (N + A*N/100)/F

8706:

8707:Ran it, and the results were good. So - as it's the only change - I won't have to regrid

8708:precip after all! Just re-run from glo onwards.. did so, then used the old (but working)

8709:version of makegrids to produce the gridded ASCII and NetCDF files.

8710:

8711:So.. comparisons. Well I want to compare with both 2.0 and 2.1, because they do differ. So

8712:I will need to convert 2.0 to regular-gridded, as I did with 2.1. If I could only remember

8713:the program I wrote to do it!!

8714:

8715:**

8716:

8717:Another problem. Apparently I should have derived TMN and TMX from DTR and TMP, as that's

8718:what v2.10 did and that's what people expect. I disagree with publishing datasets that are

8719:simple arithmetic derivations of other datasets published at the same time, when the real

8720:data could be published instead.. but no.

8721:

8722:This introduces the problem of derivation. TMN = TMP -DTR/2 and TMX = TMP + DTR/2, but this

8723:does not tell us what to do when either or both values (TMP, DTR) are missing. One thing to

8724:check is the climatologies. Here are the first two cell normals for all four parameters:

8725:

8726:.tmp = near-surface temperature (degrees Celsius)

8727:Grid-ref=   1, 148

8728:  270  274  269  265  259  253  246  242  248  252  261  268

8729:Grid-ref=   1, 311

8730: -187 -213 -195 -137  -30   47   90   81   33  -56 -132 -186

8731:

8732:.dtr = diurnal temperature range (degrees Celsius)

8733:Grid-ref=   1, 148

8734:   56   71   54   59   55   50   54   51   57   61   61   73

8735:Grid-ref=   1, 311

8736:   74   71   72   77   59   65   64   56   49   51   63   71

8737:

8738:.tmn = near-surface temperature minimum (degrees Celsius)

8739:Grid-ref=   1, 148

8740:  242  238  242  236  232  228  219  217  220  222  230  232

8741:Grid-ref=   1, 311

8742: -224 -249 -231 -175  -59   15   58   53    8  -81 -163 -222

8743:

8744:.tmx = near-surface temperature maximum (degrees Celsius)

8745:Grid-ref=   1, 148

8746:  298  309  296  295  287  278  273  268  277  283  292  305

8747:Grid-ref=   1, 311

8748: -150 -178 -159  -98    0   80  122  109   58  -31 -100 -151

8749:Grid-ref=   1, 312

8750:

8751:Well, making allowances for rounding errors, they do seem to hold to the relationship.

8752:

8753:Wrote maketmnx.for to derive TMN and TMX from TMP and DTR grids. Works with the output files

8754:from glo2abs.for. Ran makegrids to produce .dat and .nc files ( still pre-station count

8755:inclusion).

8756:

8757:On to precip problems. Tim O ran some comparisons between 2.10 and 3.00, in general things are

8758:much improved but there are a few hair raisers (asterisked for special concern):

8759:

8760:Cape Verde Isl, MAM & DJF, P

8761:Galapagos, All, T&P

8762:Guinea, MAM 1901, P

8763:Bangladesh, All 1991-2000, P **

8764:Bhutan, DJF 1939 & 1945, P

8765:Laos/Vietnam, DJF 1991, P **

8766:

8767:Looked at Bangladesh first. Here, the 1990s show a sudden drop that really can only be some

8768:stations having data a factor of 10 too low. This ties in with the WWR station data that DL

8769:added for 1991-2000, which aprently was prone to scaling issues. Wrote stnx10.for to scale

8770:a file of WWR Bangladesh records, then manually C&P'd the decade over the erroneous ones in

8771:the database. Also fixed country name from 'BNGLADESH'!

8772:

8773:Then Laos/Vietnam. Here we have an anomalously high peak for 1991 DJF. Used getllstations.for

8774:to extract all stations in a box around Laos & Vietnam (8 to 25N, 100 to 110E), a total of 96

8775:stations from Thailand, Vietnam, Laos, Kampuchea, and China. Eeeek. Tim O's program only worked

8776:with boxes though. Also, I'm not 100% sure which year DJF belongs to in Tim's world.. hopefully

8777:it's the December year (as it was the fourth column in his plot table). However.. plotted *all*

8778:the data as overlapping years, and there is no trace of a spike in DJF. Uh-oh.

8779:

8780:I'm not actually convinced that the 'country box' approach is much cop. Better to examine each

8781:land cell and automagically mark any with excessions? Say 5 SD to begin with. Could then be

8782:extra clever and pull the relevant stations and find the source of the excession? Of course, this

8783:shouldn't happen, since there is a 4SD limit imposed by anomdtb.f90 for precip (3SD for others).

8784:

8785:Wrote vietlaos.for to run through the lists of Vietnam and Laos cells (provided by Tim O) and

8786:extract the DJF precip values for each (from the 1901-2006 gridded file). It then calculates the

8787:standard deviation of each series, normalises, and notes any values over 6.0 SDs (1991 onwards).

8788:

8789:Result.. some very high values (up to 11.3 standard deviations!) in 1991/2. Worst cells:

8790:     Row  Column   Index  StdDev

8791:     212     571     273   11.21

8792:     213     571     273   11.30

8793:     214     571     273   10.15

8794:     212     572     273   11.11

8795:     213     572     273   11.20

8796:     214     572     273   10.58

8797:     212     573     273   10.84

8798:     213     573     273   11.10

8799:     212     574     273   10.76

8800:     215     572     273   10.06

8801:     214     573     273   10.53

8802:     213     574     273   10.94

8803:     214     574     273   10.44

8804:     212     575     273   10.65

8805:     213     575     273   10.66

8806:     211     576     273   10.96

8807:     212     576     273   10.51

8808:

8809:Index 273 can be related to time as follows. The series begins in 1901 and we take three values

8810:per year (J,F,D). So 1990 would be the 90th year and the 268th-270th values. Thus 273 = Dec 1991.

8811:

8812:The cells are all contiguous, implying a single station's influence via the gridding process:

8813:

8814:      570   571   572   573   574   575   576

8815:211   n/a   n/a  6.37  7.56  8.36  9.71 10.96

8816:212   n/a 11.21 11.11 10.84 10.76 10.65 10.51

8817:213  5.52 11.30 11.20 11.10 10.94 10.66   n/a

8818:214  5.34 10.15 10.58 10.53 10.44   n/a   n/a

8819:215  4.37  9.97 10.06   n/a   n/a   n/a   n/a

8820:

8821:'n/a' means the cell isn't in the Laos or Vietnam areas.

8822:

8823:The 'epicentre' of the anomaly looks to be cell (213,571), which is in the Laos file:

8824:     Box  Column     Row     Lon     Lat

8825:  205773     571     213  105.75   16.75

8826:

8827:So we're looking for stations in the vicinity of 105.75E, 16.75N. Well the precip database has a

8828:total of EIGHT Laos stations, so that should be straightforward:

8829:

8830:4893000  1990  10210  304 LUANG PRABANG        LAOS          1951 2006   -999  -999.00

8831:4893800  1920  10170  323 SAYABOURY            LAOS          1969 2006   -999  -999.00

8832:4894000  1800  10260  170 VIENTIANE            LAOS          1941 2006   -999  -999.00

8833:4894600  1738  10465  152 THAKHEK              LAOS          1989 2006   -999  -999.00

8834:4894700  1660  10480  155 SAVANNAKHET          LAOS          1970 2006   -999  -999.00

8835:4894800  1670  10500  184 SENO                 LAOS          1951 2006   -999  -999.00

8836:4895200  1568  10643  168 SARAVANE             LAOS          1989 2006   -999  -999.00

8837:4895500  1510  10580   93 PAKSE                LAOS          1968 2006   -999  -999.00

8838:

8839:Well, SENO has to be the prime candidate. Unfortunately, this is from SENO:

8840:

8841:4894800  1670  10500  184 SENO                 LAOS          1951 2006   -999  -999.00

8842:

8843:1989    0    0-9999 1910-9999 1010 4450 2690 2880 1340    0    0

8844:1990   60 1560  150  420 1110 4830 3620 3690-9999  780   30    0

8845:1991    0    0  400    0  690 1907 1890 5308 3238  805    0  366

8846:1992  488  280   50   80 1883 2503 2644 2935 2039  131    0   89

8847:1993    0    0  139  280 2324 1163 1949 4460 2145    0   29    0

8848:

8849:A most undistinguished set. So, the net widens:

8850:

8851:4894700  1660  10480  155 SAVANNAKHET          LAOS          1970 2006   -999  -999.00

8852:

8853:1989-9999    0-9999-9999 1080-9999-9999-9999-9999 1490-9999-9999

8854:1990   30-9999  240-9999-9999-9999 1920-9999-9999-9999-9999    0

8855:1991    0    0  127   49  952 2508 1681 4034 4006 1690    0  338

8856:1992  324  338   93  691 1932 2344 2048 4464  756  607    0  197

8857:1993    0    5  335  263 2665  921 2884 2204 1834   17   23    0

8858:

8859:..nope..

8860:

8861:4894600  1738  10465  152 THAKHEK              LAOS          1989 2006   -999  -999.00

8862:

8863:1989-9999    0-9999-9999 2030-9999-9999-9999-9999 1490-9999-9999

8864:1990   10-9999  520-9999-9999-9999-9999-9999-9999-9999-9999    0

8865:1991    0    0  905  119  861 6058 3578 7092 2417  373    0  324

8866:1992  105  318  125  456 2140 2978 4623 4595 3376  425    0  854

8867:1993    0  108   52 1343 5835 2999 6285 4375 1017  467    8    0

8868:

8869:..nope.. unless these values *are* unusual? Let's look at the highest two Decembers from

8870:each station:

8871:

8872:4893000  1990  10210  304 LUANG PRABANG        LAOS          1951 2006   -999  -999.00

8873:1992  193  911    0  497  657 1246 2971 2837  929  584   95 1372

8874:1994    0   54 1107  291 1702 2436 2025 3636 1516  316  185  816

8875:

8876:4893800  1920  10170  323 SAYABOURY            LAOS          1969 2006   -999  -999.00

8877:1992  411  719    0  816  754 1252 2573 1671 1686  991  351  879

8878:1994    0  208 1695  503 2262 1607 1743 2562 3205  118  193  454

8879:

8880:4894000  1800  10260  170 VIENTIANE            LAOS          1941 2006   -999  -999.00

8881:1971    0   70  140  340 2940 2750 2890 2260 1630 1030    0  180

8882:1992  381  273   11  424 2372 4878 4381 3676 3091  630    0  212

8883:1994    0  300  921  322 2685 2725 4698 1932 4000 3031 1016  166 (inc for comparison with previous)

8884:

8885:4894600  1738  10465  152 THAKHEK              LAOS          1989 2006   -999  -999.00

8886:1991    0    0  905  119  861 6058 3578 7092 2417  373    0  324

8887:1992  105  318  125  456 2140 2978 4623 4595 3376  425    0  854

8888:1994    0  612  952  558 1697 7092 5121 4276 2428  486   20    2 (inc for comparison with previous)

8889:

8890:4894700  1660  10480  155 SAVANNAKHET          LAOS          1970 2006   -999  -999.00

8891:1991    0    0  127   49  952 2508 1681 4034 4006 1690    0  338

8892:1992  324  338   93  691 1932 2344 2048 4464  756  607    0  197

8893:1994    0  734  390  494 1381 3377 1525 5651 1881  600    0    0 (inc for comparison with previous)

8894:

8895:4894800  1670  10500  184 SENO                 LAOS          1951 2006   -999  -999.00

8896:1971    0  880  130  370 1270 4010 2200 2860 1930  410    0  140

8897:1991    0    0  400    0  690 1907 1890 5308 3238  805    0  366

8898:1992  488  280   50   80 1883 2503 2644 2935 2039  131    0   89 (inc for comparison with previous)

8899:1994    0  532  318  969 2065 1937 1197 4552 1934  197    0    0 (inc for comparison with previous)

8900:

8901:4895200  1568  10643  168 SARAVANE             LAOS          1989 2006   -999  -999.00

8902:1992  287   33   52  222 1072 5444 2998 8899 2243 1070    0    0 (inc for comparison with previous)

8903:1994    0   10  354  686 1743 3387 5829 3254 4219  408   41    4 (inc for comparison with previous)

8904:1998   26  619    0  574 2386 2871 1530 2308 2680  913  463   73

8905:2005    0    0  120 1230 1990 2860 4350 8060 3770  280  140   70

8906:

8907:4895500  1510  10580   93 PAKSE                LAOS          1968 2006   -999  -999.00

8908:1972-9999-9999-9999-9999-9999-9999-9999-9999 2610  870  280  140

8909:1992  166  101    0  210  665 1898 2574 6448 2942  648   10   31 (inc for comparison with previous)

8910:1994    0    0  134  220 2537 3596 5161 5384 7693 1513  236   94

8911:

8912:

8913:Summary: LUANG PRABANG shows a significant anomaly of 1372 for Dec 1992. Unfortunately, this

8914:finds echoes both temporal (1994 has 816) and spatial (SAYABOURY's 1992 is 879). So, if these

8915:values are causing the spike, it's genuine (if exaggerated in a way yet to be determined).

8916:

8917:Wrote vietlaos2, to gather data from the cells AND stations. It also gets the climatology. Initially

8918:it only gathered 13 stations with data in 1991/2, using 'VIETNAM' and 'LAOS' to select on country

8919:name. However, taking the cell [214,574] in December 1991 as the peak incident, we can use those

8920:coordinates (17.25N, 107.25E) to centre a bounding box for station selection. A box 10degs square

8921:yields only 17 stations, none of which have anything remotely spikey in Dec 1991. A box 20degs

8922:square (some would say unfeasibly large) yields 98 stations, one of which does have a bit of a spike

8923:in Dec 91.. not impressively so though, and it's a long way away:

8924:

8925:4855200   853   9993    3 NAKHON SI THAMMARAT  THAILAND      1912 2000   -999  -999.00

8926:

8927:Over 10.5 degrees South and over 7 degrees West of the target cell. Not very convincing, especially

8928:as closer stations are bound to have masked it.

8929:

8930:One FINAL try with vietlaos3.for. Just looking at December, now, and getting the original station

8931:normals as well as the climatological ones. The whole chain. This proves to be surprisingly

8932:complicated.

8933:

8934:On a parallel track (this would really have been better as a blog), Tim O has found that the binary

8935:grids of primary vars (used in synthetic production of secondary parameters) should be produced with

8936:'binfac' set to 10 for TMP and DTR. This may explain the poor performance and coverage of VAP in

8937:particular.

8938:

8939:Back to VietLaos.. the station output from vietlaos3.for had a couple of stations with missing

8940:anomaly values:

8941:

8942:       LAT       LON       ALT      NORM       VAL      ANOM

8943:     17.15    104.13    171.00     29.00     62.00  -9999.00

8944:     15.80    102.03    182.00     45.00     40.70  -9999.00

8945:

8946:I eventually worked out that I hadn't collapsed a universal probability, it was just the 4 standard

8947:deviation screen in anomdtb (4 for precip, 3 for temp). To confirm, I did a short anomdtb run (just

8948:for 1991) with the sd limit set to 10, and sure enough:

8949:

8950:   17.15  104.13   171.0   2037.900024835600

8951:   15.80  102.03   182.0    804.400024840300

8952:

8953:They both look high enough to trigger the 4sd cap. However, since the spike we're investigating is

8954:from a regular process run, where that limit was in place, we can't use those values. Program is thus

8955:amended to omit any stations without anomalies (for Dec 1991)

8956:

8957:Next issue is to make sense of the output. The first line from the station file is (headings added):

8958:

8959:       LAT       LON       ALT      NORM       VAL      ANOM

8960:     22.60    114.10     25.00     29.00     21.60    -25.50

8961:

8962:Remembering it's percentage anomalies! So 25.5% of 29 is 29*.255 = 7.395. Add that to 21.6? 29.0 :-)

8963:

8964:By contract, the cell file looks like this:

8965:

8966:     ROW     COL     LAT     LON     VAL    NORM

8967:     220     561   20.25  100.75   12.90   15.00

8968:

8969:There are 63 stations and 204 cells (196 when missing values (sea) eliminated). I guess one approach

8970:would be to grid the anomalies, to see if a peak is visible. I did. It is. The simple interpolation

8971:in Matlab puts the peak at 17.25N, 105.25E - matches the grid peak for lat and a little west for lon.

8972:The nearest high station anomaly is 2369.2, that's from:

8973:

8974:4838300  1653  10472  138 MUKDAHAN             THAILAND      1934 2000   -999  -999.00

8975:6190   34  127  290  907 1813 2900 2271 3353 2596  886   84   13

8976:1934    0  100    0  500 3150 2940 2460 2980 3320  350    0   20

8977:1935    0    0    0  440 1920 1560 3220  580 1770    0  170    0

8978:1936    0    0  820    0 2320 1900 3460  810 2120    0    0    0

8979:1937    0    0  350  660 3640  740 1920 2890 4470  330    0    0

8980:1938    0    0  550 1300  730 1720 2340  400 2030  810    0    0

8981:1939    0  280  700  230 1320  420 2480 4190 2130    0    0    0

8982:1940-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

8983:1941    0    0  530  590 1800 2710  420 2790  650  750    0    0

8984:1942    0    0  540  660 1650 3200 1200 1730 1990    0    0    0

8985:1943    0    0 1600 1300 1960 1880 2000 2200 2600    0    0    0

8986:1944    0-9999    0  320 2210 1040 1700 2500 2150  820   50    0

8987:1945   70  600    0  340    0 2470 3400 2780 1620   20  330  500

8988:1946    0    0 1360    0-9999 1720 1070 3330 2870 1260    0    0

8989:1947    0  180   50 1390 3200 1530 3520 1150-9999   50    0    0

8990:1948-9999-9999    0 1630 3520 1040 2900 3980 2160  380   10    0

8991:1949    0  200  170  470 3000 2720 3110 4920  360  690   90-9999

8992:1950    0   70    0  250 1610 2090 1040 1390 3500 1960   20    0

8993:1951    0  340  770 1380  530 3380 1590 1950 3580 1430   20    0

8994:1952    0    0 1170  660 1640 3160 2320 4150 3510  860   30    0

8995:1953  260  110  430  630 1010 2200 1480 2780 1180  310   10    0

8996:1954  460   10   30 1100 1950 2870 1120 2640 4220  620    0    0

8997:1955    0    0  280  580 2180 4100 2900 2570 1810  270   20    0

8998:1956    0  420  150 1000 3000 2930 3980 3840 2020  220    0    0

8999:1957    0   30 1210  630 1690 2130 2090 3030 3240  460    0    0

9000:1958   70   50   10 1090 1060 1690 2670  910 2750  700    0    0

9001:1959    0  430  730  290 2300 1540 2080 2030 3910  280    0    0

9002:1960    0  190  550  650 1230 1750 3750 5090 2190  700   90    0

9003:1961    0    0  590  660 2190 5880 2150 4310 4030 1140    0    0

9004:1962    0   20  120  880 2200 2690 2780 4770-9999  360   50    0

9005:1963    0    0  610  600 1010 3480 2130 3410 1250  220  150    0

9006:1964    0    0  740  910 3370 2600  630 2050 4700 1120   20    0

9007:1965    0  250  660  780 2120 2700 2110 2810 2210 1350    0    0

9008:1966    0  610  310  730 3340 1370 3100 4010 2020  510   40  110

9009:1967    0    0   50  870 1810 1800 1540 1960 3270  150  130    0

9010:1968   10   70   50  170 1770 2320 1140 2360 5140  700    0    0

9011:1969    0   20  290  260 1850 1990 3430 2060 3470  290   30    0

9012:1970    0   10   90 1150 2210 3620 2610 4310 1290  340   10    0

9013:1971    0  730  570  740 2130 3580 4060 2100 3240  510   30  170

9014:1972    0  550  460 1280 1040 3470 3250 3640 2980 2340   20    0

9015:1973    0    0   10 1080 2650 1990 2460 2050 2090  190    0    0

9016:1974  400    0   60 2160 1160 2520 3070 6110 1920  570  260    0

9017:1975   20  350  360  410 2200 3340 3230 3560  940  520   20   40

9018:1976    0  210  380 1700 1160 1460 2430 3720 3250  780   60    0

9019:1977   20   10   90 1000  620  620 1470 3980 4010  100    0    0

9020:1978    0  100  920  650 1710 3690 2960 4420 2190  110    0    0

9021:1979   10  140   50  900 2000 4230 1230 2540 2910    0    0    0

9022:1980    0   50  190 1040 1440 3490 3310  930 6130 1830  170   20

9023:1981    0  210  220  720 2630 4730 2490 1750  610 1260   90    0

9024:1982    0    0  290  840 1330 2160  390 4390 3400 1720  370    0

9025:1983   90   30    0  550 1360 2700  830 5200 1380 1680    0    0

9026:1984   10    0  350 1270 2030 2290 2900 3880 2130 1380  650    0

9027:1985  380   50  170  860 1100 4270 1580 3350  900 1170    0    0

9028:1986    0    0  120 1650 2120 2210 1830 2980 1760 1700  240   10

9029:1987    0  110  290  360 1090 4210 2670 3640 2140 1040   20    0

9030:1988    0   30   90 1170 1130 1790 1800 3580  740 1730    0    0

9031:1989-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

9032:1990-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999

9033:1991    0-9999  105  226 1370 2079 1452 4190 3799 1610-9999  321

9034:1992  328  314  150  637 1968 1906 2366 4973 1287  238    0  216

9035:1993-9999    5  476  768 2438 1169 3671 2463 2215   13   22-9999

9036:1994    0  781  274  409 1837 2297 1625 5755 1709  216-9999-9999

9037:1995    0  140  834  672 1556 1606 4439 2848  681  857   69    0

9038:1996   12    2  660 2394 1566 1526 1960 3350 4843  724  476-9999

9039:1997   35  321  458  642 1154 2832 1197 4071 1722 1800    0-9999

9040:1998    0  346  154  241 2174 3348  813 2153 2231  276   85   37

9041:1999   63    0  182 1025 3449 1207 3681 1570 2628  299  109-9999

9042:2000-9999   95   48 2742 2816 1852 1725 1903 2903  391    0    0

9043:

9044:Note that the Dec 1991 value is anomalous, but not as extreme as the 1945 datum,

9045:which would get the same treatment with normals and climatologies, so should

9046:produce an even bigger spike for 1945 DJF! Unless of course it's screened out by

9047:the 4SD rule.. which it is! Well - no value in pre.1945.12.txt for this location.

9048:

9049:Anyway.. this is the highest value in the Vietnam/Laos cells for Dec 1991:

9050:

9051:     ROW     COL     LAT     LON     VAL    NORM

9052:     198     571    9.25  105.75   63.50  130.00

9053:

9054:With a normal of 130, that makes the anomaly -48.85. Now I'm confused. How can

9055:an anomalously high value be well below the 61-90 mean? Aaarrgghhhh. Perhaps I

9056:should look at the highest anomaly. That turns out to be 80, from here:

9057:

9058:     216     563   18.25  101.75    1.80    1.00

9059:

9060:Not exactly a show stopper. Time to look at the .glo files, which glo2abs processes

9061:into absolutes. Here's a Far-Eastern region with a spike:

9062:

9063:>> glod3(210:216,567:573)

9064:

9065:             567          568          569          570          571          572          573

9066:216       1393.6       1791.6       1757.4       1723.2       1674.5       1553.2       1431.9

9067:215       1501.7       1899.8       1927.3       1893.1       1786.3         1665       1505.3

9068:214       1609.9       2007.9       2097.2       2019.5       1885.4       1712.8       1540.2

9069:213       1359.4       2116.1       2252.6       2092.9       1920.3       1747.7       1575.1

9070:212       80.145       1195.5       1796.1         1882       1955.2       1782.6         1610

9071:211       -6.125      -36.614       563.99       649.87       735.75       821.63        907.5

9072:210      -59.833      -90.333      -89.649      -83.283      -76.929      -70.576      -64.223

9073:

9074:The spike is at [213,569]. Yes, I know, it's the n-th set of coordinates. You should see the

9075:plots! But looking at the anomalies is the closest we'll get to what Tim's program was doing,

9076:ie, calculating DJF standard deviations. Or something. Now, the coordinates are 16.75N, 104.75E.

9077:And wouldn't you know it, our prime suspect (see above) is on top of it:

9078:

9079:4838300  1653  10472  138 MUKDAHAN             THAILAND      1934 2000   -999  -999.00

9080:

9081:So OK, here we go with the full run-down for December 1991, in the 16.75N,105.75E region:

9082:

9083:TYPE                VALUE     COMMENT

9084:Raw data:            321      Highest unscreened December for this station (67 years)

9085:Normal:               13      Looks right - of course, very low for the target data!

9086:Anomaly:            2369.2    Correctly calculated

9087:Gridded anomaly:    2252.6    Believable interpolation

9088:Gridded actual:     er...     Strangely, it seems to be 0.

9089:

9090:Ah well - had enough. It looks like it's an extreme but believable event in a Thai station, let's

9091:leave it like that. Re-running precip, with the new updated database pre.0803271802.dtb:

9092:

9093:

9094:crua6[/cru/cruts/version_3_0/primaries/precip] ./anomdtb 

9095: 

9096:   > ***** AnomDTB: converts .dtb to anom .txt for gridding *****

9097: 

9098:   > Enter the suffix of the variable required:

9099:.pre

9100:   > Will calculate percentage anomalies.

9101:   > Select the .cts or .dtb file to load:

9102:pre.0803271802.dtb

9103:   > Specify the start,end of the normals period: 

9104:1961,1990

9105:   > Specify the missing percentage permitted: 

9106:25

9107:   > Data required for a normal:           23

9108:   > Specify the no. of stdevs at which to reject data: 

9109:4

9110:   > Select outputs (1=.cts,2=.ann,3=.txt,4=.stn): 

9111:3

9112:   > Check for duplicate stns after anomalising? (0=no,>0=km range)

9113:0

9114:   > Select the generic .txt file to save (yy.mm=auto):

9115:pre.txt

9116:   > Select the first,last years AD to save: 

9117:1901,2006

9118:   > Operating...

9119: /tmp_mnt/cru-auto/cruts/version_3_0/primaries/precip/pre.0803271802.dtb        

9120:   > NORMALS            MEAN percent      STDEV percent

9121:   >         .dtb    7315040    73.8

9122:   >         .cts     299359     3.0    7613600    76.8

9123:   > PROCESS        DECISION percent %of-chk

9124:   > no lat/lon        17911     0.2     0.2

9125:   > no normal       2355275    23.8    23.8

9126:   > out-of-range      13249     0.1     0.2

9127:   > accepted        7521017    75.9

9128:   > Dumping years 1901-2006 to .txt files...

9129: 

9130:IDL> quick_interp_tdm2,1901,2006,'preglo/pregrid.',450,gs=0.5,dumpglo='dumpglo',pts_prefix='pretxt/pre.'

9131:% Compiled module: QUICK_INTERP_TDM2.

9132:% Compiled module: GLIMIT.

9133:Defaults set

9134:    1901

9135:% Compiled module: MAP_SET.

9136:% Compiled module: CROSSP.

9137:% Compiled module: STRIP.

9138:% Compiled module: SAVEGLO.

9139:% Compiled module: SELECTMODEL.

9140:    1902

9141:(etc)

9142:    2006

9143:IDL>

9144:    

9145:crua6[/cru/cruts/version_3_0/primaries/precip] ./glo2abs

9146:Welcome! This is the GLO2ABS program.

9147:I will create a set of absolute grids from

9148:a set of anomaly grids (in .glo format), also

9149:a gridded version of the climatology.

9150:Enter the path and name of the normals file: clim.6190.lan.pre

9151:Enter a name for the gridded climatology file: clim.6190.lan.pre.grid4

9152:Enter the path and stem of the .glo files: preglo/pregrid.

9153:Enter the starting year: 1901

9154:Enter the ending year:   2006

9155:Enter the path (if any) for the output files: preabs/

9156:Now, CONCENTRATE. Addition or Percentage (A/P)? P

9157:Do you wish to limit the output values? (Y/N): Y

9158:1. Set minimum to zero

9159:2. Set single minimum and maximum values

9160:3. Set minima and maxima based on days in month

9161:4. Set integer values >=1, (ie, positive)

9162:5. Changed my mind, no limits

9163:Choose: 1

9164:Right, erm.. off I jolly well go!

9165:pregrid.01.1901.glo

9166:(etc)

9167:pregrid.12.2006.glo

9168:

9169:uealogin1[/cru/cruts/version_3_0/primaries/precip] ./makegrids 

9170:Welcome! This is the MAKEGRIDS program.

9171:I will create decadal and full gridded files,

9172:in both ASCII text and NetCDF formats, from

9173:the output files of (eg) glo2abs.for.

9174:

9175:Enter a gridfile with YYYY for year and MM for month: preabs/pregrid.YYYY.MM.glo.abs

9176:Enter Start Year:  1901

9177:Enter Start Month: 01

9178:Enter End Year:    2006

9179:Enter End Month:   12

9180:

9181:Please enter a sample OUTPUT filename, replacing

9182:start year with SSSS and end year with EEEE, and

9183:ending with '.dat', eg: cru_ts_3_00.SSSS.EEEE.tmp.dat : cru_ts_3_00.SSSS.EEEE.pre.dat

9184:

9185:Now please enter the 3-ch parameter code: pre

9186:Enter a generic title for this dataset, eg:

9187:CRU TS 3.00 Mean Temperature : CRU TS 3.00 Precipitation

9188:Writing: cru_ts_3_00.1901.1910.pre.dat

9189:         cru_ts_3_00.1901.1910.pre.nc

9190:Writing: cru_ts_3_00.1911.1920.pre.dat

9191:         cru_ts_3_00.1911.1920.pre.nc

9192:Writing: cru_ts_3_00.1921.1930.pre.dat

9193:         cru_ts_3_00.1921.1930.pre.nc

9194:Writing: cru_ts_3_00.1931.1940.pre.dat

9195:         cru_ts_3_00.1931.1940.pre.nc

9196:Writing: cru_ts_3_00.1941.1950.pre.dat

9197:         cru_ts_3_00.1941.1950.pre.nc

9198:Writing: cru_ts_3_00.1951.1960.pre.dat

9199:         cru_ts_3_00.1951.1960.pre.nc

9200:Writing: cru_ts_3_00.1961.1970.pre.dat

9201:         cru_ts_3_00.1961.1970.pre.nc

9202:Writing: cru_ts_3_00.1971.1980.pre.dat

9203:         cru_ts_3_00.1971.1980.pre.nc

9204:Writing: cru_ts_3_00.1981.1990.pre.dat

9205:         cru_ts_3_00.1981.1990.pre.nc

9206:Writing: cru_ts_3_00.1991.2000.pre.dat

9207:         cru_ts_3_00.1991.2000.pre.nc

9208:Writing: cru_ts_3_00.2001.2006.pre.dat

9209:         cru_ts_3_00.2001.2006.pre.nc

9210:

9211:

9212:On to the reproduction of binaries for TMP and DTR, and subsequent regeneration of VAP and FRS.

9213:

9214:TMP Binaries:

9215:

9216:IDL> quick_interp_tdm2,1901,2006,'tmpbin/tmpbin',1200,gs=2.5,dumpbin='dumpbin',binfac=10,pts_prefix='tmp0km0705101334txt/tmp.'

9217:Defaults set

9218:    1901

9219:% Compiled module: MEAN.

9220:% Compiled module: MOMENT.

9221:% Compiled module: STDDEV.

9222:grid 1901 non-zero   -0.1127    0.9472    1.3993 cells=    46444

9223:% Compiled module: STRIP.

9224:% Compiled module: WRBIN.

9225:    1902

9226:grid 1902 non-zero   -0.4378    1.0267    1.4712 cells=    48321

9227:    1903

9228:grid 1903 non-zero   -0.3462    0.9586    1.3704 cells=    48803

9229:    1904

9230:grid 1904 non-zero   -0.4288    0.9753    1.3866 cells=    49493

9231:    1905

9232:grid 1905 non-zero   -0.2732    0.9620    1.3722 cells=    49845

9233:    1906

9234:grid 1906 non-zero   -0.1910    0.9176    1.3323 cells=    49568

9235:    1907

9236:grid 1907 non-zero   -0.5290    1.0338    1.4636 cells=    50458

9237:    1908

9238:grid 1908 non-zero   -0.3546    0.9165    1.2607 cells=    50449

9239:    1909

9240:grid 1909 non-zero   -0.4005    0.9998    1.4222 cells=    50163

9241:    1910

9242:grid 1910 non-zero   -0.3404    0.9459    1.3879 cells=    50489

9243:    1911

9244:grid 1911 non-zero   -0.3868    0.9445    1.3154 cells=    50752

9245:    1912

9246:grid 1912 non-zero   -0.4441    1.0156    1.4580 cells=    51731

9247:    1913

9248:grid 1913 non-zero   -0.3315    0.9125    1.2647 cells=    51634

9249:    1914

9250:grid 1914 non-zero   -0.1896    0.9655    1.4205 cells=    50734

9251:    1915

9252:grid 1915 non-zero   -0.1935    1.0501    1.5537 cells=    51773

9253:    1916

9254:grid 1916 non-zero   -0.3523    1.0092    1.4885 cells=    52042

9255:    1917

9256:grid 1917 non-zero   -0.5407    1.1470    1.6613 cells=    54151

9257:    1918

9258:grid 1918 non-zero   -0.4379    1.0228    1.4534 cells=    52579

9259:    1919

9260:grid 1919 non-zero   -0.3663    1.0327    1.5279 cells=    51872

9261:    1920

9262:grid 1920 non-zero   -0.2173    1.0092    1.4856 cells=    50540

9263:    1921

9264:grid 1921 non-zero   -0.1261    0.9004    1.2767 cells=    52468

9265:    1922

9266:grid 1922 non-zero   -0.2355    0.9258    1.3934 cells=    54202

9267:    1923

9268:grid 1923 non-zero   -0.1942    1.0040    1.4733 cells=    54975

9269:    1924

9270:grid 1924 non-zero   -0.1418    0.9416    1.3491 cells=    55664

9271:    1925

9272:grid 1925 non-zero   -0.1033    1.0121    1.5038 cells=    55677

9273:    1926

9274:grid 1926 non-zero    0.0471    0.9750    1.4242 cells=    55826

9275:    1927

9276:grid 1927 non-zero   -0.1290    0.9839    1.4396 cells=    57033

9277:    1928

9278:grid 1928 non-zero   -0.0254    0.9581    1.3929 cells=    56950

9279:    1929

9280:grid 1929 non-zero   -0.2651    1.1120    1.7327 cells=    58284

9281:    1930

9282:grid 1930 non-zero   -0.0233    1.0157    1.5554 cells=    57481

9283:    1931

9284:grid 1931 non-zero    0.0072    1.0705    1.6009 cells=    57932

9285:    1932

9286:grid 1932 non-zero    0.0407    1.0426    1.5664 cells=    57752

9287:    1933

9288:grid 1933 non-zero   -0.2517    1.1010    1.6794 cells=    59297

9289:    1934

9290:grid 1934 non-zero    0.0858    1.0705    1.6510 cells=    58932

9291:    1935

9292:grid 1935 non-zero   -0.0383    1.0498    1.5969 cells=    59316

9293:    1936

9294:grid 1936 non-zero   -0.0118    1.0867    1.6457 cells=    59676

9295:    1937

9296:grid 1937 non-zero    0.1841    1.0572    1.6419 cells=    59702

9297:    1938

9298:grid 1938 non-zero    0.2843    1.0094    1.4853 cells=    59478

9299:    1939

9300:grid 1939 non-zero    0.0828    1.0270    1.5633 cells=    60643

9301:    1940

9302:grid 1940 non-zero    0.1223    1.0033    1.5251 cells=    60381

9303:    1941

9304:grid 1941 non-zero    0.0049    1.0253    1.4988 cells=    63950

9305:    1942

9306:grid 1942 non-zero    0.0486    1.0061    1.5799 cells=    61984

9307:    1943

9308:grid 1943 non-zero    0.1795    1.0288    1.5243 cells=    63082

9309:    1944

9310:grid 1944 non-zero    0.1993    0.9783    1.4922 cells=    62327

9311:    1945

9312:grid 1945 non-zero   -0.0306    1.0840    1.5827 cells=    62977

9313:    1946

9314:grid 1946 non-zero   -0.0376    1.0094    1.4989 cells=    63193

9315:    1947

9316:grid 1947 non-zero    0.1326    1.0977    1.7075 cells=    64854

9317:    1948

9318:grid 1948 non-zero    0.0276    0.9783    1.4466 cells=    66490

9319:    1949

9320:grid 1949 non-zero   -0.0873    1.0422    1.5665 cells=    68159

9321:    1950

9322:grid 1950 non-zero   -0.2032    1.0344    1.5841 cells=    67736

9323:    1951

9324:grid 1951 non-zero   -0.0537    0.9777    1.4482 cells=    70202

9325:    1952

9326:grid 1952 non-zero   -0.0112    0.9952    1.5704 cells=    69668

9327:    1953

9328:grid 1953 non-zero    0.2020    1.0140    1.5735 cells=    70734

9329:    1954

9330:grid 1954 non-zero   -0.0062    1.0381    1.6387 cells=    71309

9331:    1955

9332:grid 1955 non-zero   -0.1527    1.0281    1.6167 cells=    73181

9333:    1956

9334:grid 1956 non-zero   -0.2183    1.0542    1.5788 cells=    77564

9335:    1957

9336:grid 1957 non-zero   -0.0142    0.9929    1.5067 cells=    77649

9337:    1958

9338:grid 1958 non-zero   -0.0257    1.0166    1.5491 cells=    80430

9339:    1959

9340:grid 1959 non-zero    0.0019    1.0058    1.5493 cells=    79903

9341:    1960

9342:grid 1960 non-zero   -0.0661    0.9628    1.4564 cells=    80353

9343:    1961

9344:grid 1961 non-zero    0.0016    0.9440    1.3960 cells=    81093

9345:    1962

9346:grid 1962 non-zero    0.0660    0.9249    1.4273 cells=    77733

9347:    1963

9348:grid 1963 non-zero   -0.0850    1.0164    1.5649 cells=    80869

9349:    1964

9350:grid 1964 non-zero   -0.3518    0.9639    1.4553 cells=    82284

9351:    1965

9352:grid 1965 non-zero   -0.2396    0.9097    1.3315 cells=    82512

9353:    1966

9354:grid 1966 non-zero   -0.2748    1.0171    1.5915 cells=    81405

9355:    1967

9356:grid 1967 non-zero   -0.0372    0.9385    1.4324 cells=    81573

9357:    1968

9358:grid 1968 non-zero   -0.2106    0.9665    1.5115 cells=    81706

9359:    1969

9360:grid 1969 non-zero   -0.1505    1.0751    1.7496 cells=    81490

9361:    1970

9362:grid 1970 non-zero   -0.0771    0.8111    1.1569 cells=    80462

9363:    1971

9364:grid 1971 non-zero   -0.1102    0.9128    1.3577 cells=    82451

9365:    1972

9366:grid 1972 non-zero   -0.1992    1.0147    1.5390 cells=    82070

9367:    1973

9368:grid 1973 non-zero    0.0914    0.9087    1.3303 cells=    81625

9369:    1974

9370:grid 1974 non-zero   -0.1369    0.9896    1.5273 cells=    81687

9371:    1975

9372:grid 1975 non-zero   -0.0400    0.9258    1.3720 cells=    81390

9373:    1976

9374:grid 1976 non-zero   -0.2595    0.9596    1.4088 cells=    82439

9375:    1977

9376:grid 1977 non-zero    0.0718    0.9855    1.5405 cells=    80143

9377:    1978

9378:grid 1978 non-zero   -0.0233    0.9729    1.5545 cells=    80118

9379:    1979

9380:grid 1979 non-zero   -0.0921    1.0054    1.6126 cells=    79714

9381:    1980

9382:grid 1980 non-zero    0.1600    0.9471    1.4078 cells=    80417

9383:    1981

9384:grid 1981 non-zero    0.4437    1.0207    1.5695 cells=    81226

9385:    1982

9386:grid 1982 non-zero   -0.0664    0.9502    1.4287 cells=    80230

9387:    1983

9388:grid 1983 non-zero    0.2325    0.9886    1.4907 cells=    82258

9389:    1984

9390:grid 1984 non-zero    0.0904    1.0216    1.6368 cells=    81431

9391:    1985

9392:grid 1985 non-zero    0.0625    0.9590    1.5123 cells=    81731

9393:    1986

9394:grid 1986 non-zero    0.1007    0.8952    1.3674 cells=    81016

9395:    1987

9396:grid 1987 non-zero    0.1116    0.9654    1.4412 cells=    84529

9397:    1988

9398:grid 1988 non-zero    0.3365    0.9242    1.3069 cells=    82070

9399:    1989

9400:grid 1989 non-zero    0.2451    1.0428    1.6170 cells=    79951

9401:    1990

9402:grid 1990 non-zero    0.4275    1.0581    1.5828 cells=    82418

9403:    1991

9404:grid 1991 non-zero    0.4279    0.9504    1.3005 cells=    80068

9405:    1992

9406:grid 1992 non-zero    0.0265    0.9827    1.4543 cells=    80204

9407:    1993

9408:grid 1993 non-zero    0.1614    0.9820    1.4965 cells=    78945

9409:    1994

9410:grid 1994 non-zero    0.2598    0.9699    1.3958 cells=    77509

9411:    1995

9412:grid 1995 non-zero    0.5222    1.0500    1.5943 cells=    80001

9413:    1996

9414:grid 1996 non-zero    0.3733    0.9745    1.4383 cells=    78304

9415:    1997

9416:grid 1997 non-zero    0.3947    1.0250    1.4473 cells=    80450

9417:    1998

9418:grid 1998 non-zero    0.6010    1.1707    1.5968 cells=    82794

9419:    1999

9420:grid 1999 non-zero    0.4249    1.0485    1.5105 cells=    80640

9421:    2000

9422:grid 2000 non-zero    0.4574    1.0256    1.4596 cells=    78883

9423:    2001

9424:grid 2001 non-zero    0.5695    1.0596    1.4876 cells=    78391

9425:    2002

9426:grid 2002 non-zero    0.6281    1.1662    1.6404 cells=    80200

9427:    2003

9428:grid 2003 non-zero    0.6715    1.0531    1.3475 cells=    77636

9429:    2004

9430:grid 2004 non-zero    0.5362    0.9762    1.2947 cells=    79600

9431:    2005

9432:grid 2005 non-zero    0.8050    1.1350    1.4605 cells=    80465

9433:    2006

9434:no stations found in: tmp0km0705101334txt/tmp.2006.11.txt

9435:no stations found in: tmp0km0705101334txt/tmp.2006.12.txt

9436:grid 2006 non-zero    0.6621    1.1017    1.5361 cells=    65396

9437:IDL>

9438:

9439:DTR Binaries:

9440:

9441:IDL> quick_interp_tdm2,1901,2006,'dtrbin/dtrbin',750,gs=2.5,dumpbin='dumpbin',binfac=10,pts_prefix='dtrtxt/dtr.'

9442:% Compiled module: QUICK_INTERP_TDM2.

9443:% Compiled module: GLIMIT.

9444:Defaults set

9445:    1901

9446:% Compiled module: MAP_SET.

9447:% Compiled module: CROSSP.

9448:% Compiled module: MEAN.

9449:% Compiled module: MOMENT.

9450:% Compiled module: STDDEV.

9451:grid 1901 non-zero    0.3357    0.8741    1.1669 cells=    18526

9452:% Compiled module: STRIP.

9453:% Compiled module: WRBIN.

9454:    1902

9455:grid 1902 non-zero    0.2560    0.8530    1.1640 cells=    19088

9456:    1903

9457:grid 1903 non-zero    0.2015    0.8514    1.1841 cells=    19063

9458:    1904

9459:grid 1904 non-zero    0.2647    0.8584    1.1827 cells=    19155

9460:    1905

9461:grid 1905 non-zero    0.1753    0.9056    1.2595 cells=    20808

9462:    1906

9463:grid 1906 non-zero    0.2458    0.9003    1.2127 cells=    20892

9464:    1907

9465:grid 1907 non-zero    0.2658    0.9124    1.2370 cells=    21621

9466:    1908

9467:grid 1908 non-zero    0.3003    0.8911    1.1912 cells=    22028

9468:    1909

9469:grid 1909 non-zero    0.2063    0.8791    1.1952 cells=    22253

9470:    1910

9471:grid 1910 non-zero    0.2524    0.8563    1.1561 cells=    23297

9472:    1911

9473:grid 1911 non-zero    0.2317    0.8808    1.2201 cells=    24153

9474:    1912

9475:grid 1912 non-zero    0.1748    0.9544    1.3196 cells=    24284

9476:    1913

9477:grid 1913 non-zero    0.1990    0.9047    1.2463 cells=    24366

9478:    1914

9479:grid 1914 non-zero    0.1395    0.9195    1.2747 cells=    24431

9480:    1915

9481:grid 1915 non-zero    0.0185    0.9405    1.2969 cells=    26178

9482:    1916

9483:grid 1916 non-zero    0.0178    0.8761    1.1904 cells=    27095

9484:    1917

9485:grid 1917 non-zero    0.1518    0.9108    1.2619 cells=    26614

9486:    1918

9487:grid 1918 non-zero    0.1303    0.9134    1.2533 cells=    26447

9488:    1919

9489:grid 1919 non-zero    0.1300    0.8856    1.2029 cells=    25701

9490:    1920

9491:grid 1920 non-zero    0.0500    0.8480    1.1650 cells=    26563

9492:    1921

9493:grid 1921 non-zero    0.1660    0.8213    1.1246 cells=    26549

9494:    1922

9495:grid 1922 non-zero    0.1133    0.8468    1.1707 cells=    26701

9496:    1923

9497:grid 1923 non-zero    0.1794    0.8962    1.2293 cells=    27643

9498:    1924

9499:grid 1924 non-zero    0.1309    0.8549    1.1583 cells=    28642

9500:    1925

9501:grid 1925 non-zero    0.2252    0.9167    1.2254 cells=    28841

9502:    1926

9503:grid 1926 non-zero    0.1076    0.8559    1.1687 cells=    30671

9504:    1927

9505:grid 1927 non-zero    0.1285    0.8715    1.1766 cells=    30962

9506:    1928

9507:grid 1928 non-zero    0.1279    0.8576    1.1773 cells=    31156

9508:    1929

9509:grid 1929 non-zero    0.1784    0.8826    1.1974 cells=    32021

9510:    1930

9511:grid 1930 non-zero    0.1344    0.8711    1.1830 cells=    33360

9512:    1931

9513:grid 1931 non-zero    0.0238    0.8470    1.1548 cells=    32726

9514:    1932

9515:grid 1932 non-zero    0.0872    0.8489    1.1546 cells=    33396

9516:    1933

9517:grid 1933 non-zero    0.1012    0.8560    1.1597 cells=    34574

9518:    1934

9519:grid 1934 non-zero    0.0295    0.8591    1.1676 cells=    34203

9520:    1935

9521:grid 1935 non-zero    0.1092    0.8682    1.1665 cells=    34561

9522:    1936

9523:grid 1936 non-zero    0.2106    0.8947    1.2055 cells=    35632

9524:    1937

9525:grid 1937 non-zero    0.1686    0.8463    1.1182 cells=    35921

9526:    1938

9527:grid 1938 non-zero    0.0845    0.8242    1.1125 cells=    35718

9528:    1939

9529:grid 1939 non-zero    0.1428    0.8362    1.1277 cells=    37525

9530:    1940

9531:grid 1940 non-zero    0.1891    0.8662    1.1493 cells=    38227

9532:    1941

9533:grid 1941 non-zero    0.1334    0.8502    1.1411 cells=    38486

9534:    1942

9535:grid 1942 non-zero    0.1176    0.8344    1.1001 cells=    39312

9536:    1943

9537:grid 1943 non-zero    0.1844    0.8476    1.1191 cells=    40361

9538:    1944

9539:grid 1944 non-zero    0.1564    0.8195    1.0904 cells=    40010

9540:    1945

9541:grid 1945 non-zero    0.1566    0.8164    1.0740 cells=    40172

9542:    1946

9543:grid 1946 non-zero    0.1648    0.8529    1.1288 cells=    40305

9544:    1947

9545:grid 1947 non-zero    0.0668    0.8305    1.1166 cells=    41045

9546:    1948

9547:grid 1948 non-zero    0.2204    0.8015    1.0473 cells=    41004

9548:    1949

9549:grid 1949 non-zero    0.1192    0.8185    1.1029 cells=    41329

9550:    1950

9551:grid 1950 non-zero    0.1349    0.7915    1.0675 cells=    42803

9552:    1951

9553:grid 1951 non-zero    0.1881    0.8003    1.0647 cells=    47064

9554:    1952

9555:grid 1952 non-zero    0.1751    0.7776    1.0556 cells=    46868

9556:    1953

9557:grid 1953 non-zero    0.1331    0.7763    1.0518 cells=    48129

9558:    1954

9559:grid 1954 non-zero    0.0824    0.7668    1.0349 cells=    47912

9560:    1955

9561:grid 1955 non-zero    0.0963    0.7733    1.0468 cells=    49105

9562:    1956

9563:grid 1956 non-zero    0.0901    0.7695    1.0440 cells=    50637

9564:    1957

9565:grid 1957 non-zero    0.0689    0.7536    1.0279 cells=    50456

9566:    1958

9567:grid 1958 non-zero    0.0050    0.7504    1.0291 cells=    52013

9568:    1959

9569:grid 1959 non-zero    0.0439    0.7329    1.0020 cells=    52162

9570:    1960

9571:grid 1960 non-zero    0.0674    0.7049    0.9542 cells=    52787

9572:    1961

9573:grid 1961 non-zero    0.0445    0.6810    0.9111 cells=    56188

9574:    1962

9575:grid 1962 non-zero    0.1297    0.6877    0.9156 cells=    54897

9576:    1963

9577:grid 1963 non-zero    0.1449    0.7088    0.9661 cells=    55755

9578:    1964

9579:grid 1964 non-zero    0.0955    0.6719    0.9029 cells=    54909

9580:    1965

9581:grid 1965 non-zero    0.0913    0.6638    0.8950 cells=    54906

9582:    1966

9583:grid 1966 non-zero    0.0878    0.6566    0.8813 cells=    54751

9584:    1967

9585:grid 1967 non-zero    0.0805    0.6626    0.8876 cells=    54393

9586:    1968

9587:grid 1968 non-zero    0.0826    0.6611    0.8923 cells=    54602

9588:    1969

9589:grid 1969 non-zero    0.0253    0.6787    0.9309 cells=    55176

9590:    1970

9591:grid 1970 non-zero    0.0576    0.6232    0.8301 cells=    55444

9592:    1971

9593:grid 1971 non-zero    0.0987    0.6340    0.8426 cells=    54610

9594:    1972

9595:grid 1972 non-zero    0.0472    0.6631    0.8979 cells=    55812

9596:    1973

9597:grid 1973 non-zero    0.0287    0.6424    0.8741 cells=    54755

9598:    1974

9599:grid 1974 non-zero   -0.0119    0.6782    0.9289 cells=    56105

9600:    1975

9601:grid 1975 non-zero    0.0287    0.6259    0.8458 cells=    54696

9602:    1976

9603:grid 1976 non-zero    0.0740    0.6565    0.8966 cells=    55239

9604:    1977

9605:grid 1977 non-zero   -0.0454    0.6600    0.9026 cells=    54227

9606:    1978

9607:grid 1978 non-zero   -0.1045    0.6529    0.8775 cells=    55036

9608:    1979

9609:grid 1979 non-zero   -0.0749    0.6510    0.8753 cells=    54990

9610:    1980

9611:grid 1980 non-zero   -0.0565    0.6269    0.8300 cells=    55430

9612:    1981

9613:grid 1981 non-zero   -0.0498    0.6704    0.8970 cells=    55023

9614:    1982

9615:grid 1982 non-zero   -0.0828    0.6622    0.8874 cells=    56028

9616:    1983

9617:grid 1983 non-zero   -0.1380    0.6808    0.9142 cells=    55854

9618:    1984

9619:grid 1984 non-zero   -0.1530    0.6675    0.8850 cells=    55751

9620:    1985

9621:grid 1985 non-zero   -0.1156    0.6359    0.8438 cells=    55066

9622:    1986

9623:grid 1986 non-zero   -0.0822    0.6412    0.8583 cells=    55111

9624:    1987

9625:grid 1987 non-zero   -0.0579    0.6599    0.8919 cells=    55607

9626:    1988

9627:grid 1988 non-zero   -0.0691    0.6615    0.8752 cells=    54503

9628:    1989

9629:grid 1989 non-zero   -0.0165    0.6685    0.9064 cells=    55615

9630:    1990

9631:grid 1990 non-zero   -0.0655    0.6734    0.9057 cells=    54497

9632:    1991

9633:grid 1991 non-zero   -0.0899    0.6602    0.8722 cells=    53533

9634:    1992

9635:grid 1992 non-zero   -0.1171    0.6951    0.9349 cells=    53739

9636:    1993

9637:grid 1993 non-zero   -0.1314    0.6882    0.9188 cells=    51271

9638:    1994

9639:grid 1994 non-zero   -0.0480    0.6934    0.9282 cells=    50211

9640:    1995

9641:grid 1995 non-zero   -0.0357    0.7596    1.2498 cells=    52366

9642:    1996

9643:grid 1996 non-zero   -0.0723    0.7666    1.1579 cells=    51667

9644:    1997

9645:grid 1997 non-zero   -0.1340    0.7854    1.1636 cells=    52195

9646:    1998

9647:grid 1998 non-zero   -0.1627    0.8328    1.1992 cells=    52907

9648:    1999

9649:grid 1999 non-zero   -0.1237    0.7134    0.9716 cells=    51050

9650:    2000

9651:grid 2000 non-zero   -0.1765    0.7575    1.0412 cells=    51137

9652:    2001

9653:grid 2001 non-zero   -0.1305    0.7564    1.0658 cells=    49146

9654:    2002

9655:grid 2002 non-zero   -0.0984    0.7549    1.0967 cells=    46178

9656:    2003

9657:grid 2003 non-zero   -0.1072    0.7128    1.0087 cells=    46904

9658:    2004

9659:grid 2004 non-zero   -0.1628    0.8113    1.2574 cells=    47399

9660:    2005

9661:grid 2005 non-zero   -0.1150    0.8546    1.3540 cells=    43715

9662:    2006

9663:no stations found in: dtrtxt/dtr.2006.09.txt

9664:no stations found in: dtrtxt/dtr.2006.10.txt

9665:no stations found in: dtrtxt/dtr.2006.11.txt

9666:no stations found in: dtrtxt/dtr.2006.12.txt

9667:grid 2006 non-zero   -0.0087    0.9041    1.6077 cells=    28592

9668:IDL> 

9669:

9670:VAP synthetics:

9671:

9672:IDL> vap_gts_anom,dtr_prefix='../dtrbin/dtrbin',tmp_prefix='../tmpbin/tmpbin',1901,2006,outprefix='vapsyn/vapsyn.',dumpbin=1

9673:% Compiled module: VAP_GTS_ANOM.

9674:% Compiled module: RDBIN.

9675:% Compiled module: STRIP.

9676:% Compiled module: DEFXYZ.

9677:Land,sea:       56016       68400

9678:Calculating tmn normal

9679:% Compiled module: TVAP.

9680:Calculating synthetic vap normal

9681:% Compiled module: ESAT.

9682:Calculating synthetic anomalies

9683:% Compiled module: MOMENT.

9684:1901 vap (x,s2,<<,>>):   -0.0450600     0.230521     -4.50663      4.92927

9685:% Compiled module: WRBIN.

9686:1902 vap (x,s2,<<,>>):    -0.102633     0.271200     -4.14974      4.92341

9687:1903 vap (x,s2,<<,>>):    -0.107597     0.242152     -5.74305      5.58190

9688:1904 vap (x,s2,<<,>>):    -0.123137     0.221801     -4.30042      3.71240

9689:1905 vap (x,s2,<<,>>):   -0.0799978     0.267905     -4.54584      6.04190

9690:1906 vap (x,s2,<<,>>):   -0.0343380     0.240282     -4.86007      6.45160

9691:1907 vap (x,s2,<<,>>):    -0.137421     0.284412     -5.06625      6.02255

9692:1908 vap (x,s2,<<,>>):    -0.105214     0.234139     -6.99258      5.15916

9693:1909 vap (x,s2,<<,>>):    -0.103285     0.252210     -5.48791      5.12214

9694:1910 vap (x,s2,<<,>>):    -0.104377     0.225462     -4.64360      7.19087

9695:1911 vap (x,s2,<<,>>):    -0.109270     0.255412     -4.35549      6.99751

9696:1912 vap (x,s2,<<,>>):    -0.127306     0.287857     -4.63037      5.84822

9697:1913 vap (x,s2,<<,>>):    -0.107747     0.271437     -4.73097      4.98110

9698:1914 vap (x,s2,<<,>>):   -0.0481493     0.274125     -4.43114      6.19318

9699:1915 vap (x,s2,<<,>>):   -0.0343964     0.332448     -5.31914      6.04190

9700:1916 vap (x,s2,<<,>>):   -0.0947238     0.293859     -6.31417      4.63692

9701:1917 vap (x,s2,<<,>>):    -0.170714     0.396007     -8.69476      3.86362

9702:1918 vap (x,s2,<<,>>):    -0.133214     0.311174     -5.48035      6.19711

9703:1919 vap (x,s2,<<,>>):   -0.0687798     0.307508     -6.14869      9.28416

9704:1920 vap (x,s2,<<,>>):   -0.0619862     0.262944     -5.76398      4.10598

9705:1921 vap (x,s2,<<,>>):   -0.0319013     0.301257     -5.73133      7.07691

9706:1922 vap (x,s2,<<,>>):   -0.0621843     0.231719     -5.03104      4.18061

9707:1923 vap (x,s2,<<,>>):   -0.0626035     0.285980     -5.70781      7.32851

9708:1924 vap (x,s2,<<,>>):   -0.0720660     0.281999     -4.88423      7.32851

9709:1925 vap (x,s2,<<,>>):   -0.0673457     0.292218     -5.70781      8.66163

9710:1926 vap (x,s2,<<,>>):   -0.0264075     0.297545     -4.88423