INTRODUCTORY NOTES ON DATA REDUCTION AND ERROR ANALYSIS PROGRAMS ---------------------------------------------------------------- ROBERT H. BLESSING HAUPTMAN-WOODWARD INSTITUTE 73 HIGH STREET BUFFALO, NEW YORK 14203, USA TELEPHONE: (716) 856-9600, EXTENSION 335 ELECTRONIC MAIL: blessing@hwi.buffalo.edu THE FOLLOWING PAGES GIVE: - BRIEF PROGRAM DESCRIPTIONS - A DATA REDUCTION FLOW CHART - A LISTING OF THE REQUIRED INPUT FILES FOR EACH PROGRAM - DESCRIPTIONS OF THE RECORD STRUCTURE IN THE REFLECTION DATA OUTPUT FROM EACH PROGRAM - LITERATURE REFERENCES ------------------------------------------------------------------------ BRIEF PROGRAM DESCRIPTIONS -------------------------- DETAILED PROGRAM DESCRIPTIONS ARE PROVIDED IN THE ASCII TEXT "program.use" FILES SUPPLIED WITH THE "program.f" SOURCE CODE FILES. LINE PRINTER ASCII TEXT OUTPUT FROM EACH PROGRAM IS WRITTEN TO A FILE NAMED "program.lp", WHICH CAN BE EXAMINED ON A VIDEO MONITOR AND EDITED BEFORE PRINTING. ------------------------------------------------------------------------ IN THE ORDER IN WHICH THE PROGRAMS ARE NORMALLY RUN for processing serially measured, single-reflection counter or "point" detector data: CAD4TAPE P3TAPE DECODE DIFFRACTOMETER TAPES AND WRITE "DATA.RAW" FILE REFPK REFLECTION PROFILE PEAK ANALYSIS BGLP BACKGROUND SUBTRACTION, BASED ON THE PEAK PROFILE ANALYSIS, AND LORENTZ AND POLARIZATION CORRECTIONS VIEW VIDEO TERMINAL OR LINE PRINTER PLOTS OF RAW OR PROCESSED REFLECTION PROFILES. INTERACTIVE PROFILE PROCESSING TSCALE TIME-DEPENDENT SCALING WITH (UP TO) CUBIC (3RD DEGREE) POLYNOMIALS ABSORB1 ABSORB2 ABSORPTION CORRECTIONS FROM INDEXED CRYSTAL FACES AND LENGTHS OF FACE NORMALS BY GAUSSIAN NUMERICAL INTEGRATION OVER CRYSTAL VOLUME SORTAV SORTING, INTER-SUBSET SCALING, EMPIRICAL ABSORPTION CORRECTION BASED ON MULTIPLE SYMMETRY-EQUIVALENT AND/OR AZIMUTH ROTATION-EQUIVALENT MEASUREMENTS, AVERAGING OF REPLICATE AND EQUIVALENT MEASUREMENTS, AND ANALYSIS OF VARIANCE BAYES BAYESIAN PROBABILITY EXPECTATION VALUES OF FSQ, SIGMA(FSQ), F, AND SIGMA(F) GIVEN THE SET OF UNIQUE FSQ(MEAS) AND SIGMA(FSQ(MEAS)). ALSO CALCULATES E AND SIGMA(E), WHERE E = F/SQRT(EPSILON*) IS NORMALIZED BY THE LOCAL AVERAGE MEASURED INTENSITY. (ROGERS) LEVY ABSOLUTE SCALE FACTOR AND OVERALL ANISOTROPIC MEAN-SQUARE ATOMIC DISPLACEMENT PARAMETERS BY EITHER THE METHHOD OF DONALD ROGERS OR THE METHOD OF HENRI LEVY AND WILLIAM THIESSEN. PROGRAM LEVY IS GENERALLY PREFERABLE. EVAL NORMALIZED STRUCTURE FACTOR AMPLITUDES E = K*F/SQRT(EPSILON*SUM FA**2*, WHERE E IS NORMALIZED BY THE WILSON EXPECTATION VALUE OF THE INTENSITY. TDSBG TDSCORR AN APPROXIMATE TDS CORRECTION BASED ON AN ANALYSIS OF THE SLOPES OF THE BACKGROUND PROFILE NEAR THE PEAK LIMITS. NOT RECOMMENDED FOR ROUTINE USE. ------------------------------------------------------------------------ PROGRAMS "CAD4TAPE" AND "P3TAPE" MAY NEED TO BE EXTENSIVELY MODIFIED, OR REPLACED BY LOCALLY WRITTEN PROGRAMS, IN ORDER TO DEAL WITH THE PECULIARITIES OF LOCAL DIFFRACTOMETER INSTALLATIONS AND LOCAL MAGNETIC TAPE INPUT/OUTPUT OPERATIONS. IN PROGRAM "REFPK" THERE IS A SUBROUTINE "INPUT" CONTAINING DATA STATEMENTS THAT ASSIGN DEFAULT VALUES FOR THE SCAN ANGLE UNCERTAINTY, DETECTOR DEAD TIME, AND BEAM ATTENUATOR FACTOR. THESE SHOULD BE MODIFIED TO PROVIDE THE APPROPRIATE LOCAL DEFAULT VALUES. ------------------------------------------------------------------------ For processing oscillation-method area-detector data: denzox Read and interpret a set of concatenated "Denzo.x" files. sortav bayes (rogers) levy eval As above. locscl Evaluate and apply anisotropically variable local scale factors for SIR or SAS pairs of data sets. diffe Evaluate renormalized SIR or SAS difference E-magnitudes. ------------------------------------------------------------------------ ------------------------------------------------------------------------ SERIALLY-MEASURED, SINGLE-REFLECTION COUNTER OR "POINT" DETECTOR ---------------------------------------------------------------- DATA REDUCTION FLOW CHART ------------------------- ------- PROGRAM NAMES ARE INDICATED AS PROGRAM, AND FILE NAMES AS ------- 'FILENAME.TYPE'. FILES NAMED 'DATA.TYP' ARE REFLECTION DATA FILES, AND FILES NAMED 'PROGRAM.DAT' ARE PARAMETER AND PROGRAM CONTROL DATA FILES. MOST OF THE 'PROGRAM.DAT' FILES MUST BE PREPARED BY THE USER. THE EXCEPTIONS ARE THE 'BGLP.DAT', 'ABSORB.DAT', 'EVAL.DAT', AND 'TDSCORR.DAT' FILES, WHICH ARE GENERATED BY THE PROGRAMS. DIFFRACTOMETER DATA TAPES hkl; two-theta, omega, chi, phi (or theta, phi, omega kappa); reltaive scan speed; I(theta) step-scan profile V V V V V V ------ -------- P3TAPE CAD4TAPE ------ -------- V V V V V > > > V < < < V V V ---- 'DATA.RAW' > > > VIEW > > > VT OR LP PLOTS V ---- OF RAW V REFLN. PROFILES V ----- 'REFPK.DAT' > > > REFPK + ----- 'HKLCOND.DAT' V V V 'DATA.RAW' ---- + > > > VIEW > > > VT OR LP PLOTS 'BGLP.DAT' ---- OF PROCESSED V REFLN. PROFILES V V ---- BGLP ---- V V V 'DATA.BLP' V V V V V V 'DATA.BLP' V V V ------ 'TSCALE.DAT' > > > TSCALE ------ V V V INDEXED CRYSTAL FACES V AND FACE NORMAL DISTANCES V V V V V V V ------- V ABSORB1 'DATA.SCL' ------- V V V V V V ------- 'ABSORB.DAT' > > > ABSORB2 ------- V V V 'DATA.ABS' V V V ------ 'SORTAV.DAT' > > > SORTAV ------ V V V 'DATA.SOR' V V V ----- 'BAYES.DAT' > > > BAYES ----- V V V 'DATA.BAY' V V V V V V 'DATA.BAY' V V V ---- ------ 'LEVY.DAT' > > > LEVY OR ROGERS < < < 'ROGERS.DAT' ---- ------ V V V 'DATA.BAY' + 'EVAL.DAT' V V V ---- EVAL ---- V V V 'EDATA.HKL' 'EDATA.EEE' V V V ------------------------------------------------------------------------ ------------------------------------------------------------------------ Area Detector Oscillation Data Reduction ---------------------------------------- Oscillation Frame Images v v v ----- Denzo ----- Peak-Background Integration Fsq(hkl) = (Ipk - Ibg)/(Lp), sigma[Fsq(hkl)] Full/Partial Flag, Frame Number v v v "frames.x" (concatenated set of Denzo.x files) v v v ---------------- denzox/get fulls ---------------- v v v "data.fulls" v v v ----------------------- "sortav.dat.1" > > > sortav/interframe scale ----------------------- v v v "scalek.dat" + "frames.x" v v v ---------------- denzox/sum parts ---------------- v v v "data.total" v v v v v v "data.total" v v v ------------------- "sortav.dat.2" > > > sortav/absorb/merge ------------------- v v v "data.merged" v v v ----- "bayes.dat" > > > bayes ----- v v v "data.bayes" v v v ---- ------ "levy.dat" > > > levy or rogers < < < "rogers.dat" ---- ------ v v v "eval.dat" + "data.bayes" v v v ---- eval ---- v v v "edata.hkl" "edata.eee" v v v v v v "edata.hkl" "edata.eee" v SIR or SAS cases v ---------------- v ------ "locscl.dat" > > > locscl ------ v v v "data.locscl" + "eval.dat" v v v ----- "diffe.dat" > > > diffe ----- v v v "data.diffe" v v v DM, SIR, SAS, MIR, MAD, MR,... Phasing Trials ------------------------------------------------------------------------ REQUIRED PROGRAM INPUT FILES ---------------------------- 'DATA.TYPE' FILES ARE REFLECTION DATA FILES. IN EACH PROGRAM THEY ARE READ BY A SUBROUTINE 'READ1' AND WRITTEN BY A SUBROUTINE 'WRITE1'. THE PROGRAMS 'SORTAV', 'BAYES', 'LEVY', 'ROGERS', AND 'EVAL' ALSO PERMIT FORMATTED OR UNFORMATTED EXTERNAL 'DATA.HKL' REFLECTION FILES WITH RECORDS: IH, IK, IL, FSQ OR F, SIGMA(FSQ OR F). 'PROGRAM.DAT' FILES ARE FORMATTED, ASCII PARAMETER AND CONTROL DATA FILES. SOME OF THEM, AS INDICATED, ARE WRITTEN BY THE PROGRAMS, BUT MOST MUST BE PREPARED BY THE USER ACCORDING TO INSTRUCTIONS GIVEN IN THE 'PROGRAM.USE' FILES. ------------------------------------------------------------------------ CAD4TAPE P3TAPE DIFFRACTOMETER DATA TAPES INTERACTIVE INPUT REFPK 'REFPK.DAT' 'HKLCOND.DAT' 'DATA.RAW' (FROM PROGRAM CAD4TAPE OR PROGRAM P3TAPE) BGLP 'BGLP.DAT' (FROM PROGRAM REFPK) 'HKLCOND.DAT' 'DATA.RAW' VIEW 'DATA.RAW', OR 'DATA.RAW' PLUS 'BGLP.DAT' INTERACTIVE INPUT TSCALE 'TSCALE.DAT' 'DATA.BLP' (FROM PROGRAM BGLP) ABSORB1 INTERACTIVE INPUT ABSORB2 'ABSORB.DAT' (FROM PROGRAM ABSORB1) 'DATA.SCL' (FROM PROGRAM TSCALE) OR 'DATA.BLP' SORTAV 'SORTAV.DAT' 'DATA.ABS' (FROM PROGRAM ABSORB2) OR 'DATA.SCL' OR 'DATA.BLP' OR EXTERNAL 'DATA.HKL' BAYES 'BAYES.DAT' 'DATA.SOR' (FROM PROGRAM SORTAV) OR EXTERNAL 'DATA.HKL' ROGERS 'ROGERS.DAT' 'DATA.BAY' (FROM PROGRAM BAYES) OR 'DATA.SOR' OR EXTERNAL 'DATA.HKL' LEVY 'LEVY.DAT' 'DATA.BAY' OR 'DATA.SOR' OR EXTERNAL 'DATA.HKL' EVAL 'EVAL.DAT' (FROM PROGRAM ROGERS OR PROGRAM LEVY) 'DATA.BAY' OR 'DATA.SOR' OR EXTERNAL 'DATA.HKL' TDSBG 'TDSBG.DAT' 'BGLP.DAT' (FROM PROGRAM REFPK) 'HKLCOND.DAT' 'DATA.RAW' TDSCORR 'TDSCORR.DAT' (FROM PROGRAM TDSBG) 'DATA.BAY' OR 'DATA.SOR' denzox "frames.x" (concatenated set of "Denzo.x" frames files) interactive input "scalek.dat" (frame scale factors from program sortav) The program denzox permits a default use of scale factors of unity for each frame, or it accepts a file name for a formatted, ascii file of scale factors with records [i, scalek(i), sigmak(i)] read under free format, one record per frame number i. The scale factors file can be prepared by using an editor to cut out the list of fitted scale factors from the "sortav.lp" output file from the inter-subset scaling run of the program sortav. locscl "locscl.dat" "data.eval" (from program eval) or "data.bayes" (from program bayes) or "data.merged" (from program sortav) or external "data.hkl" diffe "diffe.dat" "eval.dat" (from program levy or from program rogers) "data.locscl" (from program locscl) ------------------------------------------------------------------------ RECORD STRUCTURE IN REFLECTION DATA OUTPUT FILES ------------------------------------------------ SOME OF THE REFLECTION DATA FILES ARE UNFORMATTED, BINARY FILES. SEE INDIVIDUAL 'PROGRAM.USE' FILES FOR MORE DETAILED DESCRIPTIONS. ------------------------------------------------------------------------ CAD4TAPE P3TAPE 'DATA.RAW' II,IH,IK,IL,A1,A2,A3,A4,WIDTH,SPEED,X,X,X,X,X,XTIME, IY(96) IN THE 'DATA.RAW' FILE, IH, IK, IL, AND IY(96) ARE INTEGER*2 WORDS. IN ALL THE OTHER FILES, INTEGER WORDS ARE INTEGER*4. IN ALL FILES, REAL WORDS ARE REAL*4. II MEASUREMENT SERIAL NUMBER IH,IK,IL MILLER INDICES A1,A2,A3,A4 DIFFRACTOMETER SETTING ANGLES: INT.TAB. TWO-THETA OMEGA CHI PHI BUS.LEV. TWO-THETA OMEGA CHI PHI P3 TWO-THETA OMEGA PHI CHI CAD4 THETA PHI OMEGA KAPPA WIDTH SCAN WIDTH (DEGREES THETA) SPEED SCAN SPEED (DEGREES THETA PER MINUTE) X,X,X,X,X FIVE UNUSED WORDS XTIME RADIATION EXPOSURE TIME (HOURS) IY(96) STEP-SCAN COUNTS II IS NEGATIVE FOR STANDARD REFERENCE REFLECTION MEASUREMENTS. WIDTH IS NEGATIVE FOR MEASUREMENTS MADE USING A BEAM ATTENUATOR. BGLP TSCALE 'DATA.BLP' 'DATA.SCL' II,IH,IK,IL,A1,A2,A3,A4,FSQ,SIGFSQ,XTIME ABSORB2 'DATA.ABS' II,IH,IK,IL,A1,A2,A3,A4,FSQ,SIGFSQ,XTIME, TBAR,S0(3),S1(3) SORTAV 'DATA.SOR' IH,IK,IL,FSQ,SIGFSQ,ESD,RMSD,NMEAS, TBAR,S0(3),S1(3) (in a formatted, ASCII file) BAYES 'DATA.BAY' IH,IK,IL,FSQ,SIGFSQ,F,SIGF (in a formatted, ASCII file) EVAL 'EDATA.HKL' 'EDATA.EEE' IH,IK,IL,FSQ,SIGFSQ,F,SIGF,E,SIGE (in a formatted, ASCII file) denzox "data.fulls" "data.total" ih,ik,il,Fsq,sigmaFsq,iframe (in a formatted, ASCII file) locscl "data.locscl" ih,ik,il,F1,sigmaF1,F2,sigmaF2,E1,sigmaE1,E2,sigmaE2 (in a formatted, ASCII file) diffe "data.diffe" ih,ik,il,DiffE,sigmaDiffE (in a DiffE-ranked, formatted ASCII file) ------------------------------------------------------------------------ REFERENCES ---------- Blessing, R.H. (1987). Data Reduction and Error Analysis for Accurate Single Crystal Diffraction Intensities. Crystallography Reviews, 1, 3-58. ----------------------- = Blessing, R.H. (1986). DREAM - data reduction and error analysis routines for accurate single-crystal diffraction intensity measurements. J. Appl. Cryst. 19, 412. --------------- == Blessing, R.H., and Langs, D.A. (1987). Data Averaging with Normal Down-Weighting of Outliers. J. Appl. Cryst. 20, 427- --------------- == 428. Blessing, R.H., and Langs, D.A. (1988). A Priori Estimation of Scale and Overall Anisotropic Temperature Factors From the Patterson Origin Peak. Acta Cryst. A44, 729-735. ----------- == Blessing, R.H. (1989). DREADD - data reduction and error analysis for single crystal diffractometer data. J. Appl. -------- Cryst. 22, 396-397. ------ == Blessing, R.H. (1995). An Empirical Correction for Absorption Anisotropy. Acta Cryst. A51, 33-38. ----------- == Blessing, R.H., Guo, D.Y., and Langs, D.A. (1996). Statistical Expectation Value of the Debye-Waller Factor and |E(hkl)| Values for Macromolecular Crystals. Acta Cryst. D52, 257- ----------- == 266. Blessing, R.H. (1997). Outlier Treatment in Data Merging. J. -- Appl. Cryst. 30, 421-426. ------------ == Blessing, R.H. (1997). LOCSCL: a program to statistically optimize local scaling of single-isomorphous-replacement and single-wavelength-anomalous-scattering daia. J. Appl. Cryst. --------------- 30, 176-177. == Blessing, R.H., Guo, D.Y., and Langs, D.A. (1998). Intensity Statistics and Normalization. In Direct Methods for Solving -------------------------- Macromolecular Structures, NATO ASI Series Volume, Series C: ------------------------- Mathematical and Physical Sciences, Vol. 507, edited by S. Fortier, pp. 47-71. Dordrecht, The Netherlands: Kluwer Academic Publishers. Blessing, R.H., and Smith, G.D. (1999). Difference Structure Factor Normalization for Determining Heavy-Atom or Anomalous Scattering Substructures. J. Appl. Cryst. 32, 664-670. --------------- == ------------------------------------------------------------------------