bm2img : image preprocessing on BM2 beamline

A small program, called bm2img has ben writen for this purpose. In case of the direct illumination CCD refer to wv2ph or edf2ph programs.

Procedure

All data are stored internally as 32 bits signed integer, this choice has been done in order to reach a good dynamic without losing the counting accuracy. When the image are read they are corrected from dark current (if a dark image has been set) and from flatfield (if set). Then various operation can be done on the 32 bits image. When writing an output file, the dynamic is only modified on the resulting file not on the inernel image. If you rescale the image, remember that integer operations are order dependent.

Short example

Converting a winview CCD image (.spe) to a 32 bit EsrfDataFormat (.edf).

The following command transforms the winview file dani10.spe into the 32 bit file dani10.edf.
bm2img dani10 %X

Dark current correction

The following command transforms the winview file dani10.spe into a 32bit file corrected for dark current... dani10d.edf file.
bm2img D=dark dani10 %X
The following similar command applies to a preprocessed dark edf file that is the sum of 15 frames.
bm2img D=dark.edf,U15,X dani10 %X

A reduced view of a SAXS image can be seen below. In the central part it shows a small angle scattering image using 2*2 binning. To show the influence of the correction on 8 bit .gif images, intensities higher than 3 times the image average have been truncated; this occurs near the beamstop and/or on cosmic rays. When an images has been stored in edf format, it is alaways assumed that this image is made by only one frame, then it is mandatory to use the U parameter to correct this if the dark file contains more than one frame.

Flat field correction

The following images show the transformation of the data. In the central part they show a small angle scattering image using the same binning as for the above flatfield. To display the influence of the flatfield correction on 8 bit .gif images, intensities higher than 3 times the image mean have been truncated : this occurs near the beamstop and/or on cosmic ray spots, that have not be fully cleaned.
bm2img D=dark F=flat.edf,X dani10 %X

dark corrected dark and flatfield corected
bm2img D=dark dani10 %X bm2img D=dark F=flat.edf,X dani10 %X

Output

The output name is by default
name(d)(f).$EXTENSION
("d" is added to the name if the  dark correction has been made)
("f" is added to the name if the  flat correction has been made)
$EXTENSION indicates the format of the file.

output of an image

The key commands used to write the corrected image on disk are
command%X%Y%Z
dynamic32 bits16 bits8 bits
defaultname.edfname.edfname.edf
.edfname.edfname.edfname.edf
.gelname.gelname.gelname.gel
.tifname.tifname.tifname.tif
.txtname.txt
Most of the viewers can handle 8-bits tiff, 16-bits gel (or tiff) is also a standard format. Edf file can be read by fit2d or viewFile.

The txt output can be used to extract some binded rows or columns of the images, it does not depend on the selected dynamic. Obviously trying to output a full image using this mode may produces some disk full error!!!

Processing series of data

The first way to process a series of data is to use a script shell but it is more efficient to use bm2img internal redirection if all the images use the same dark and flat.
Use STDIN=inputfile in the bm2img.ini file and fill this file with all the order you need. Do not forget to clean the memory (%F) between each image to process or they will be added together. The following lines show an example of inputfile.
D=v2zr51.edf,U5,X
img39
%X
%R
%F
img40
%X
%R
%F
img41
...

The inline help

The command bm2img -h will display its principal key words.
....
## bm2img :  extracting info/preprocessing image
## bm2img_1.3 (20Jun02_0750)
## berar@esrf.fr (d2am),  Jul 23 2002
usage : bm2img [D=|F=]img_file[,Bb,Ee,X,Aa|Mm,Uu] [img...] [%M=val] [%[X] [img...]]
   example D=dark3,B2,A5  add frame starting at 2 in dark_files dark3 to dark8, U force used frames to u
   D=dark_file to substract, F=flatfield_file (must be defined before image
   Begin at frame b, End at frame e, X don't correct
   Add a files with increasing number (or Merge them, DMI specific)
   Inside the program the image are kept as integer with 32 bits
   %M=val (%A=val) multiply current image by val (or add to) before reading next arguments
        val can be expressed as %M=1.0e6/SPEC (call wspec2xl.pl ... to get real value)
   %C=vah[,val] cut counts higher (lower) than vah*mean before reading next arguments
   %S=val rescalling dynamics to val
   %X[[name].edf|.gel|.tif|.txt] produce outfile (edf default with 32bits)
   %Y[[name].edf|.gel|.tif|.txt] produce 16bits outfile (gel default) rescalling the output if mandatory
   %Z[[name].edf|.gel|.tif|.txt] produce 8bits outfile (tif default) rescalling the output if mandatory
   %R produce radial distribution file (.rad), one angular sector is the default
   %P[=m[,o],v]]] produce peak list file (.pks), m=5 peaks is the default, omega osc if not 0, sum>v*max, v=1.5
   %H produce an ascii map (h, k, l) of the file  using the UB matrix
   %F clean current image (but keeps dark and flat)
   %G apply grid correction as defined by GRID_DCOR, GRID_XCOR, GRID_YCOR in ini file (see bm2grid)
   %T="rule" transform according rule
   a specific statistic routine is called when the program name contains the string 'stat'
   previous keys are not valid in this case 
   see also prameters in bm2img.ini file
valid options in .ini files are : 
 Items used in the .ini file
  X_MIN = 
  Y_MIN = 
  X_MAX = 
  Y_MAX = 
  X_BIND = 
  Y_BIND = 
  BEG_FRAME = 
  END_FRAME = 
  MULT   = 
 P_EXCLUDED = x1 y1 [v] [QUIET]
 C_EXCLUDED = x1 y1 r [v] [QUIET]
 R_EXCLUDED = x1 y1 x2 y2 [v] [QUIET]
 T_EXCLUDED = x1 y1 x2 y2 x3 y3 [v] [QUIET]
 C_ROI = x1 y1 r [v] [QUIET]
 R_ROI = x1 y1 x2 y2 [v] [QUIET]
 DIR = full path to search for data file
 OUT = full path for result (.res) file
 DEF = default file extension :'.gel' or 'gel' (if not default according prg_name : spe2img, edf2img, gel2img)
 CR = 0, set to 1 if you wish cr in rad file
 DFORMAT = size of ineger field in output name, default same as entry
 STDIN = full path for redirecting stdin
 SPEC = full path to spec file for collecting info
 WSPEC2XL = script (and args) to extract normation, use * for file name 
 XL_COL = pos of the selected column in WSPEC2XL
 FLAT_NAME = full path of the flatfield file to use
 DARK_NAME = full path of the dark file to use
 SCA_DARK = scale factor to apply on dark
 ADD_DARK = offset to add to dark curent
 O_COSMIC = file_name, STDOUT or default to image_name.cosmic
 T_COSMIC = used for time removal
 S_COSMIC = used for space removal
 R_COSMIC = ratio on the standard deviation used for time and space removal
 H_COSMIC = lower cosmics on one frame for time removal
 P_COSMIC = lower cosmics on the frame sum for spacial removal
 SET_LIMIT = remove values lower than LOW_LIMIT if not zero
 LOW_LIMIT = values are replaced mean of surrounding values (SET_LIMIT=1), by LOW_LIMIT (SET_LIMIT=2)
 ADD_DARK = offset to add to dark curent
 REALTIME = 0 or waiting time in secs to ensure data completion
 PRINT   = ... +64 cosmics, +128 cor, +256 sum, +512 raw
 T_DATA  = output mode (float...) for edf file
 ORIG =  pixel_x(=0) pixel_y(=0) center of detector 
 MAT_UB = values according spec
 LAMBDA = 1 (if not given)
 GONIO =  0.00 th chi phi
 DETEC =  alpha tth Rx(=1) Ry(=0) Rz(=0)
 SCALE =  scal_xx(=1) scal_xy(=0) scal_yx(=0) scal_yy(=1) matrix form pixel to detector
 ORIG =  pixel_x(=0) pixel_y(=0) center of detector 
 PLANE = a x+ b y+ c z= d  e ecart 
 GRID_AUTO = NONE (default),AUTO apply grid correction
 GRID_MODE = NONE (default, surface corrected), FLUX is constant, no surface correction\n");
 GRID_DCOR ... see bm2grid
     following group only concerns radial distribution
 RADIUS = radius binning
 RAD_LIM  = low limit value in radial distribution (not set is no limit)
 N_SECTOR  = number of angular sectors (num 0 to N_SECTOR-1)
 A_SECTOR  = origin (degree) of the first sector
 P_SECTOR  = if not empty, binning specifications without space
 RAD_X  = if not empty, X geometrical transformation, RAD_Y
 RAD_DX  = if not empty, X geometrical density related to RAD_X, RAD_DY
    radial distribution example, N=8,A=-22.5,P=0+4,2+6,1+3+5+7 print 3 curves

Radial intensity distribution for SAXS images

In SAXS experiments, it is not always necessary to write a corrected image in a file. You can obtain the radial distribution directly using the following command. In case of anisotropy, more than one angular sector can be specified.

Then the following line can be used in many cases, this line can be included in a shell script for processing a lot of images when processing conditions are defined.

bm2img D=dark F=flat.edf,X dani10 %R

An ascii output file is created with the extension ".rad"

specific parameter in the bm2img.ini file

keyexpecteddefault
ORIG=x yCCD center float values to define the center of the image
N_SECTOR=n1number of sectors : angular resolution for anisotropic images
(one sector is obviously isotropic)
A_SECTOR=n0if anisotropic, angular origin of the first sector
(the origin is the lower side of the sector related to the X axis)
P_SECTOR=stringemptyprint statement for binning sectors, see examples below
RADIUS=n1binding radius parameter
RAD_LIM=f-Infallow to avoid Zero or Negative values in radial distribution
(values below rad_lim are set to rad_lim

Default output file

If P_SECTOR is not found, the radial distrubution file looks like:
# ./00jan147df.rad (400-900)/1*(400-800)/1  (00jan147df.gel)
# center 545.90 626.80 -> 145.90 226.80, r_max 420.51
# 16 sectors with origine at 0.00 deg
#   r      s/n       e(s/n)   n  ... 
    0        0.00    -1.00    0.0         19.75    -1.00    0.1          0.00    -1.00    0.0          0.00    -1.00    0.0          0.00    -1.00    0.0         86.57    -1.00    0.3          0.00    -1.00    0.0          0.00    -1.00    0.0          0.00    -1.00    0.0          0.00    -1.00    0.0        120.00    -1.00    0.5          0.00    -1.00    0.0          0.00    -1.00    0.0          0.00    -1.00    0.0         49.61    -1.00    0.2          0.00    -1.00    0.0   
    1       47.73    -1.00    0.2        233.25    -1.00    0.9         62.13    -1.00    0.3          0.00    -1.00    0.0         94.52    -1.00    0.4        176.43    -1.00    0.7        142.05    -1.00    0.6          0.00    -1.00    0.0        164.44    -1.00    0.6          0.00    -1.00    0.0        141.03    -1.00    0.6        141.49    -1.00    0.6          0.00    -1.00    0.0        109.12    -1.00    0.4        206.39    -1.00    0.8         64.66    -1.00    0.3   
    2      243.13    -1.00    1.0        182.34    -1.00    0.7        182.87    -1.00    0.7         85.77    -1.00    0.3        262.33    -1.00    1.0        246.62    -1.00    1.0        157.70    -1.00    0.6        169.46    -1.00    0.6        254.40    -1.00    1.0         80.24    -1.00    0.3        241.18    -1.00    1.2        255.89    -1.00    1.0        140.50    -1.00    0.5        156.90    -1.00    0.6        203.42    -1.00    0.8        261.12    -1.00    1.0   
    3      253.68    -1.00    1.0        268.60    -1.00    1.4        231.71    -1.00    0.9        277.05    -1.00    1.0        271.35    -1.00    1.2        255.20    -1.00    1.5        246.90    -1.00    1.1        254.32    -1.00    1.0        260.36    -1.00    1.0        259.13    -1.00    1.1        258.15    -1.00    1.4        273.01    -1.00    1.4        282.03    -1.00    1.0        260.58    -1.00    1.1        267.47    -1.00    1.5        277.58    -1.00    1.1   
    4      279.85    -1.00    1.0        280.01     4.29    2.3        293.07    -1.00    1.6        281.30    -1.00    1.1        265.20     4.30    2.1        249.44    -1.00    1.3        257.70    -1.00    1.5        244.29    -1.00    1.4        255.63    -1.00    1.5        254.01    -1.00    1.6        270.05    -1.00    1.5        276.85     9.92    2.1        281.49    -1.00    1.3        294.67    -1.00    1.5        286.19    -1.00    1.2        271.42     5.35    2.1   
    5      289.64     5.37    2.1        304.81    16.00    2.1        300.79    -1.00    1.3        302.34    13.46    2.1        287.04    27.20    2.0        267.82    17.94    2.4        251.79    -1.00    1.7        268.03    18.86    2.1        242.62    20.99    2.0        259.82    -1.00    1.7        279.27    11.21    2.4        288.04    20.69    2.0        301.69    22.29    2.1        318.00    -1.00    1.4        288.49     6.81    2.3        274.35     9.72    2.0   
    6      297.29    13.17    2.0        315.34     9.24    2.9        333.89    10.67    2.3        336.10    24.43    2.0        329.26    13.30    3.0        294.24    20.22    2.0        249.00     1.66    2.1        260.46    15.27    2.2        252.05    14.14    2.3        263.75    14.57    2.1        284.13    11.89    3.0        314.98     5.94    2.2        355.74    36.16    2.0        338.35    20.64    2.4        320.42    13.93    2.8        295.34    36.50    2.0   
    7      333.05    26.76    2.8        331.37    31.35    2.7        372.27    27.82    2.1        378.13    14.81    2.9        352.68     8.67    3.1        325.46    23.81    2.8        257.67     8.26    2.3        258.64    10.90    3.1        257.92     7.65    3.1        294.31    15.25    2.3        316.24    15.00    2.9        363.37    34.69    3.1        401.71    28.77    2.2        390.97    31.50    3.0        356.85    36.73    2.7        362.83    24.98    2.9   
in which
rstand for the radius of the ring
sis the sector number
s/nis the counting per pixel
e(s/n)is the standard deviation on the counting
nis the pixel number taken in account in the ring

Output file with binned sectors

The P_SECTOR string is interpreted to join the sectors in the output file. The string format consists in the number of the sectors and '+' or ',' signs, space are not allowed. The following example is consistent with sectors of 45deg, the first group is made by two sectors centered on the horizontal axis, the second on sectors from vertical axis and a third consists of the other sectors. Only the specified sectors are printed.
# ./00jan147df.rad (400-900)/1*(400-800)/1  (00jan147df.gel)
# center 545.90 626.80 -> 145.90 226.80, r_max 420.51
# 16 sectors with origine at 0.00 deg
# print sectors binned according : '0+8,4+12' 
# 4 used sectors (of 16) are binned in 2 group :  0 ( 0 8)  ,  1 ( 4 12) 
#   r       s/n       e(s/n)    n  ... 
    0        0.00    -1.00    0.0          0.00    -1.00    0.0    
    1      212.17    -1.00    0.8         94.52    -1.00    0.4    
    2      248.74     5.80    2.1        268.51    -1.00    1.5    
    3      257.01     9.49    2.0        276.33     6.17    2.2    
    4      265.64    13.20    2.5        271.52     7.71    3.3    
    5      266.17    16.05    4.1        294.42    15.03    4.1    
    6      273.35    14.83    4.3        340.04    15.23    5.0    

Normalisation using spec files

The %M command allows since Nov, 2001 to normalise the file looking into "SPEC" file. This call the script wspec2xl.pl to do it, but this argument may be changed. Using the following lines in bm2img.ini
SPEC=/mat_dsk4/visit/xx...xx/fourc.26Oct01
WSPEC2XL=wspec2xl.pl -M
XL_COL=5
the command bm2img alpdre33.edf %M=1e7/SPEC does...
# command : %M=1e7/SPEC
spec2xl 1e7/SPEC
running wspec2xl.pl -Malpdre33 /mat_dsk2/lcontact/berar/Spec/fourc.26Oct01 >bm2img.SPEC
using 1e+07/1.404e+07=0.712 as coefficient
# alpdre33.edf : 1 frame(s), mean 16742.1 sigma 60758.2 [0@0*0 - 466609@13*58] 5220!=0
It is obvious that if you wish to keep some statistical meaning, the efficient coeffficient must be of some order of magnitude (100 by example) as data are integers inside bm2img. If you wish to avoid scanning the specfile at each image, you can process like the following: