• Each XML output file size can range from a few KB to a few MB, be carefull about your quota

0.1 Using GANGA

ganga -g

Configuration file .gangarc

gangadir = /afs/cern.ch/user/LETTER/NAME/PLACE/gangadir 

Scripts have been sucessfuly tested on the current 14.2.10 version.

Notation Throughout the TWiki

Idea Structure

File Name Structure

while [ $# -gt 0 ] ; do shift ; done

bsub -R "type==SLC4&&swp>4000&&pool>2000" -q 8nh  -o ${SCRATCH}/Generation.${OUTPUT}.${EVENTS}.${ID}.Screen.txt Generation.JobTransformation.sh ${JOBOPTIONS} ${EVENTS} ${ID}

0.1 LXBATCH Simulation Submitter

6. Source CMT!

9. Et voilá! Run the job on the myTopOptions.py:

Running Athena Full Chain in Release 14

About Notation

0.0.1 Making generation script for LXBATCH step by step:

0.0.1 LXBATCH Generation Submitter

0.1 Generation on the GRID

0.1 Running Simulation

> athena.py <JOB OPTIONS> | tee Simulation.Output.txt

Screen output is saved into the Simulation.Output.txt file. Make sure you have enough disk space.

0.1.1 Running Simulation JobTransformation?

Procedures: Running simulation on LXPLUS with the JobTransformation? is very easy, however it takes incredibly long:

Notes:

• Make sure you have over 50 MB free space if you are running on LXPLUS.
• By default, the jobOptions.py script is used, which is linked to the myTopOptions.py, which you can modify to suit your needs.
• Be careful about the geometry and trigger flags in your myTopOptoins.py. Their order matters.

0.0.1 Customizing Job Options

• Reconstruction.JobTransformation.sh: reconstruction job transformation that is to be run on LXBATCH
• Batch.Reconstruction.sh: script that submits reconstruction to the LXBATCH

• DIGITIZATION POOL ROOT is a file we obtained from digitization (in our case this step is identical to simulation step). It should be in your $CASTOR_HOME/fullchain/digitization folder. All you need to do is to COPY/PASTE its name, the scripts downloads it and accesses it automatically (string).
• EVENTS is a number of events you want to reconstruct (int).
• SKIP is the number of events you want to skip (int)

• Both scripts are executable.
• All directories and files specified in the environment variables exist.

0.0.1 Making customizable LXBATCH reconstruction script

LAST=$[${EVENTS}+${SKIP}] # arithmetic evaluation

# Show the power of the processor grep MHz /var/log/dmesg

# Copy entire run directory in my working place #rfcp ${CASTOR_DIGITIZATION}/${INPUT} .

## SETUP CMT echo "###################################################################################################" echo "SETTING UP CMT" export CURRENTDIR=`pwd` # remember the current directory

# Create CMT directory mkdir cmt cd cmt

# Create REQUIREMENTS file touch requirements cat <|requirements #---- CMT HOME REQUIREMENTS FILE --------------------------------- set CMTSITE CERN set SITEROOT /afs/cern.ch macro ATLAS_DIST_AREA \${SITEROOT}/atlas/software/dist macro ATLAS_TEST_AREA /afs/cern.ch/user/m/mzeman/testarea/FullChain

apply_tag oneTest # use ATLAS working directory apply_tag setup # use working directory apply_tag 32 # use 32-bit apply_tag ${RELEASE}

use AtlasLogin? AtlasLogin? -* \$(ATLAS_DIST_AREA)

#---------------------------------------------------------------- EOF

echo "" echo "YOUR REQUIREMENTS FILE:" cat requirements

export CMT_ROOT=/afs/cern.ch/sw/contrib/CMT/${CMT_VERSION} source ${CMT_ROOT}/mgr/setup.sh

which cmt cmt config source setup.sh -tag=${RELEASE},32 export CMTPATH=/afs/cern.ch/atlas/software/releases/$RELEASE/AtlasProduction/$PCACHE source /afs/cern.ch/atlas/software/releases/${RELEASE}/AtlasOffline/${PCACHE}/AtlasOfflineRunTime/cmt/setup.sh

### RUN THE JOB # Go back to working directory echo ${CURRENTDIR} cd ${CURRENTDIR}

echo "###################################################################################################" echo "BUILDING POOL FILE CATALOG" if [ -d PoolFileCatalog? .xml ] ; then echo "###################################################################################################" echo "DELETING ALREADY EXISTENT POOL FILE CATALOG" rm -f PoolFileCatalog? .xml fi

pool_insertFileToCatalog rfio:${CASTOR_DIGITIZATION}/${INPUT} # create PoolFileCatalog.XML FCregisterLFN? -p rfio:${CASTOR_DIGITIZATION}/${INPUT} -l ${CASTOR_DIGITIZATION}/`whoami`.${INPUT}

echo "" echo "###################################################################################################" echo "RUNNING RECONSTRUCTION"

# Get the JobOption? file #get_files -jo RecExCommon? /myTopOptions.py touch myTopOptions.py

# FLAGS NEED TO COME FIRST if [ ${EVENTS} -ne 0 ] ; then echo "Setting EvtMax? to $EVENTS" echo "### NUMBER OF EVENTS" >> myTopOptions.py echo "EvtMax=${EVENTS}" >> myTopOptions.py fi

cat >> myTopOptions.py << EOF

### INPUT FILE (POOL FILE CATALOG needs to be defined) PoolRDOInput? = ["rfio:${CASTOR_DIGITIZATION}/${INPUT}"]

### GEOMETRY SELECTION DetDescrVersion? ="ATLAS-CSC-02-01-00" # new geometry for Job Transformations # DetDescrVersion? ="ATLAS-CSC-01-02-00" # default geometry

### GENERAL FLAGS # doTrigger = False # for example do not run trigger simulation # doTruth=False

### INCLUDE YOUR OWN ALGORITHMS(s) # UserAlgs? =[ "MyPackage/MyAlgorithm_jobOptions.py" ]

### ESD output CONFIGURATION # doESD=False # doWriteESD=False

### TRIGGER CNFIGURATION #(see https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerFlags) include ("TriggerRelease/TriggerFlags.py") # Trigger Flags TriggerFlags? .doLVL1=True TriggerFlags? .doLVL2=True TriggerFlags? .doEF=True

# ANALYSIS OBJECT DATA output CONFIGURATION # (see https://twiki.cern.ch/twiki/bin/view/Atlas/UserAnalysisTest#The_AOD_Production_Flags) # doAOD=False # doWriteAOD=False # doWriteTAG=False # from ParticleBuilderOptions? .AODFlags import AODFlags

### DETECTOR FLAGS # switch off Inner Detector, Calorimeters, or Muon Chambers #include ("RecExCommon/RecExCommon_flags.py") .Muon_setOff() .ID_setOff() .Calo_setOff()

### MAIN JOB OPTIONS include ("RecExCommon/RecExCommon_topOptions.py")

### USER MODIFIER ## ATLANTIS # if needed to create JiveXML? for Atlantis

echo "YOUR JOB OPTIONS:" cat myTopOptions.py echo ""

echo "RUNNING: athena.py myTopOptions.py" athena.py myTopOptions.py

### COPY OUT RESULTS IF EXIST echo "" echo "###################################################################################################" echo "COPYING SIMULATION OUTPUT" if [ -e ESD.pool.root ] ; then echo "ESD file found, copying ..." rfcp ESD.pool.root ${CASTOR_RECONSTRUCTION}/${OUTPUT}.${SKIP}-${LAST}of${TOTAL}.${ID}.esd.pool.root else echo "No ESD file found." fi

if [ -e AOD.pool.root ] ; then echo "AOD file found, copying ..." rfcp AOD.pool.root ${CASTOR_RECONSTRUCTION}/${OUTPUT}.${SKIP}-${LAST}of${TOTAL}.${ID}.aod.pool.root else echo "No AOD file found." fi

if [ -e TAG.pool.root ] ; then echo "TAG files found, copying ..." rfcp TAG.pool.root ${CASTOR_RECONSTRUCTION}/${OUTPUT}.${SKIP}-${LAST}of${TOTAL}.${ID}.tag.pool.root # ZIP ALL JiveXML? outputs to one file for FILE in `ls Jive*` do tar cf JiveXML? .tar Jive* done rfcp JiveXML? .tar ${CASTOR_RECONSTRUCTION}/${OUTPUT}.${SKIP}-${LAST}of${TOTAL}.${ID}.JiveXML.tar else echo "No TAG files found." fi

if [ -e ntuple.root ] ; then echo "NTUPLE file found, copying ..." rfcp ntuple.root ${CASTOR_RECONSTRUCTION}/${OUTPUT}.${SKIP}-${LAST}of${TOTAL}.${ID}.ntuple.root else echo "No NTUPLE file found." fi

### LIST WORKSPACE CONTENT (for debugging purposes) echo "" echo "###################################################################################################" echo "LISTING WORKSPACE CONTENT" ls -lRt

### CLEAN WORKSPACE BEFORE EXIT echo "" echo "###################################################################################################" echo "CLEANING WORKSPACE" cd .. rm -fR Reconstruction.${OUTPUT}.${SKIP}-${LAST}of${TOTAL}.${ID}

0.0.1 LXBATCH reconstruction submitter

.
.
.
still at work
.
.
.

0.1 Reconstruction on the GRID

0.1 Analysis Packages

cmt co -r AnalysisExamples-00-20-14 PhysicsAnalysis/AnalysisCommon/AnalysisExamples

0.2 ROOT

> root <MY RECONSTRUCTED FILE>

• Generation.JobTransformation.sh: generation job transformation that is to be run on LXBATCH
• Batch.Generation.sh: script that submits JobTransformation? .sh to the LXBATCH

0.1 Making generation script for LXBATCH step by step:

• Simulation.JobTransformation.sh: simulation job transformation that is to be run on LXBATCH
• Batch.Simulation.sh: script that submits JobTransformation? sh to the LXBATCH

0.1 Making simulation script for LXBATCH step by step:

Documentation: https://twiki.cern.ch/twiki/bin/view/Atlas/ReconstructionDocumentation

0.1 Running Reconstruction

Generation is run just as any job in Athena using the athena.py script on a Job Options file. It is also good to print the output on the screen and store it into a file using | tee:

> athena.py <JOB OPTIONS> | tee Generation.Output.txt

0.1.1 How to insert file to PoolFileCatalog?

> pool_insertFileToCatalog  <PATH>/<FILENAME>
> FCregisterLFN -p <PATH>/<FILENAME> -l <USERNAME>.<FILENAME> 

0.1.2 Running Reconstruction using Job Transformation

> csc_reco_trf.py <INPUT RDO.pool.root> esd.pool.root aod.pool.root ntuple.root <MAX EVENTS> <SKIP> <GEOMETRY> DEFAULT 

Again the big advantage is you don't have to source your CMT home for the Athena to understand the command.

0.2 Reconstruction on LXPLUS

> csc_evgen08_trf.py 105145 1 110 1324354657 ./MC8.105145.PythiaZmumu.py 105145.pool.root

0.3 Reconstruction on LXBATCH

• Simulation.sh: simulation job that is to be run on LXBATCH
• Batch.Simulation.sh: script that submits Simulation.sh to the LXBATCH

What you need to do ONLY ONCE is to make the scripts executable by:

> chmod +x Simulation.sh
> chmod +x Batch.Simulation.sh

Now all you need to do is run the Batch.Simulation.sh script, which submits your job to the LXBATCH. The script has four parameters you HAVE TO specify. To run the job, issue the following from the directory where you put BOTH of the scripts:

> ./Batch.Simulation.sh <GENERATION POOL ROOT> <EVENTS> <SKIP> <ID>

• GENERATION POOL ROOT is a file we obtained from generation. It should be in your $CASTOR_HOME/fullchain/generation folder. All you need to do is to COPY/PASTE its name, the scripts downloads it and accesses it automatically (string).
• EVENTS is a number of events you want to simulate (int).
• SKIP is the number of events you want to skip (for example you want to simulate between 2000 and 2200 events of your generation file)
• ID is an identifier of your choosing (string).

A few notes:

• You can run the submitter Batch.Simulation.sh from any folder (public, private - it does not matter)
• Make sure both scripts are executable before panicking.
• Make sure all directories and files specified in the environment variables exist !!! (if you followed this tutorial EXACTLY, everything should be working)
• The script creates your environment on LXBATCH machine and runs the simulation setup of your choosing
• Simulation JobTransformation? runs together with digitization, therefore it takes very long. Make sure you try to simulate 50 events at most, if you want to simulate more, you need to modify the Batch.Simulation.sh script to run in longer queues (for instance change the 8 hour queue 8nh to 2 day queue 2nd)

Documentation and download: http://cern.ch/atlantis
Online Atlantis (requires Java): http://www.hep.ucl.ac.uk/atlas/atlantis/webstart/atlantis.jnlp

0.0.1 How to create JiveXML?

include("JiveXML/JiveXML_jobOptionBase.py")
include("JiveXML/DataTypes_All.py")

OR b) change the flags directly in your package (e.g.: =RecExCommon_flags.py) as follows:

# --- write Atlantis xml file 
JiveXML = True
OnlineJiveXML = False

# --- write Atlantis geometry xml file (JiveXML needs to be set True)
AtlantisGeometry = True

Notes:

• Each XML output file has approx. 300 kB, be carefull about your quota
• Many of the JiveXML? outputs can be empty (no visible event reconstructed)
• (Of course) you need to run the reconstruction again after making these changes.

0.1 Virtual Point 1

You need to run the Virtual Point 1 on a ESD file like this:

Running General Full Chain

Its purpose is mainly creating scripts to run Athena Full Chain on LXBATCH and the GRID.

0.1 Running Generation

You will need to obtain these two files:

PDGTABLE.MeV
jobOptions.pythia.py

0.0.1 How to get Particle Data Group Table

> get_files PDGTABLE.MeV

0.0.1 Running Generation using Job Transformation

> csc_evgen08_trf.py <RUN no.> <FIRST EVENT> <MAX EVENTS> <RANDOM SEED> ./<JOB OPTIONS .py> <OUTPUT run.pool.root>

Why would we want to do this? Because the Job Transformation does not require any CMT setup and requirements files. You will only need to call these global repository CMTs and then you can run the Job:

>
  . /afs/cern.ch/atlas/software/releases/14.2.10/cmtsite/setup.sh -tag=14.2.10 
  export CMTPATH=/afs/cern.ch/atlas/software/releases/14.2.10/AtlasProduction/14.2.10 
  . /afs/cern.ch/atlas/software/releases/14.2.10/AtlasProduction/14.2.10/AtlasProductionRunTime/cmt/setup.sh 
>

1 Simulation

1 Digitization

1 Reconstruction

1 Analysis

1.1 Atlantis

http://www.hep.ucl.ac.uk/atlas/atlantis/?q=jivexml

1.2 Virtual Point 1

In order to run the program, you need to use SSH connection with enabled X11 forwarding and have your X server running. Windows users look here: http://www-hep2.fzu.cz/twiki/bin/view/ATLAS/WindowsRelated

To run the Virtual Point 1, type:

> vp1 <YOUR ESD FILE>.pool.root

# main jobOption
include ("RecExCommon/RecExCommon_topOptions.py")


include( "JiveXML/JiveXML_jobOptionBase.py" )
include( "JiveXML/DataTypes_All.py" )
# user modifier should come here
doJiveXML=True
JiveXML=True
OnlineJiveXML=False
AtlantisGeometry=True

Running General Full Chain on LXBATCH

1 Configuration

0.1 Setting up the CMT Environment

0.1 What to do every time you log on?

0.0.1 Employing Startup Script

0.1 Preparing CASTOR

0.1 Running Full Chain on LXPLUS

0.1 Using LXBATCH

1 Generation

1.1 Running Generation JobTransformation?

0.0.1 How to get Job Option files:

0.0.1 How to change minimum number of events:

0.1 Generation on LXPLUS

0.1 Generation on LXBATCH

• Generation.sh: generation job that is to be run on LXBATCH
• Batch.Generation.sh: script that submits Generation.sh to the LXBATCH

> chmod +x Generation.sh
> chmod +x Batch.Generation.sh

• You can run the submitter Batch.Generation.sh from any folder (public, private - it does not matter)
• Make sure both scripts are executable before panicking.
• Make sure all directories and files specified in the environment variables exist !!! (if you followed this tutorial EXACTLY, everything should be working)
• The script creates your environment on LXBATCH machine and runs the generation setup of your choosing

0.1 Optional - Making generation script for LXBATCH step by step:

0.1 LXBATCH Generation Submitter

## Remove all the parameters from $1, $2 and $3, so they don't get picked up by the next script while [ $# -gt 0 ] ; do shift ; done

bsub -R "type==SLC4&&swp>4000&&pool>2000" -q 8nh -o ${SCRATCH}/${OUTPUT}.${EVENTS}.${ID}.Screen.txt Generation.sh ${JOBOPTIONS} ${EVENTS} ${ID}

1 Simulation on LXBATCH

1.1 Running Simulation JobTransformation?

> csc_simul_trf.py -h

0.1 Simulation on LXPLUS

0.1 Simulation on LXBATCH

• Simulation.sh: simulation job that is to be run on LXBATCH
• Batch.Simulation.sh: script that submits Simulation.sh to the LXBATCH

What you need to do ONLY ONCE is to make the scripts executable by:

> chmod +x Simulation.sh
> chmod +x Batch.Simulation.sh

Now all you need to do is run the Batch.Simulation.sh script, which submits your job to the LXBATCH. The script has four parameters you HAVE TO specify. To run the job, issue the following from the directory where you put BOTH of the scripts:

> ./Batch.Simulation.sh <GENERATION POOL ROOT> <EVENTS> <SKIP> <ID>

• GENERATION POOL ROOT is a file we obtained from generation. It should be in your $CASTOR_HOME/fullchain/generation folder. All you need to do is to COPY/PASTE its name, the scripts downloads it and accesses it automatically (string).
• EVENTS is a number of events you want to simulate (int).
• SKIP is the number of events you want to skip (for example you want to simulate between 2000 and 2200 events of your generation file)
• ID is an identifier of your choosing (string).

A few notes:

• You can run the submitter Batch.Simulation.sh from any folder (public, private - it does not matter)
• Make sure both scripts are executable before panicking.
• Make sure all directories and files specified in the environment variables exist !!! (if you followed this tutorial EXACTLY, everything should be working)
• The script creates your environment on LXBATCH machine and runs the simulation setup of your choosing
• Simulation JobTransformation? runs together with digitization, therefore it takes very long. Make sure you try to simulate 50 events at most, if you want to simulate more, you need to modify the Batch.Simulation.sh script to run in longer queues (for instance change the 8 hour queue 8nh to 2 day queue 2nd)

0.2 Optional - Making simulation script for LXBATCH step by step:

### ENVIRONMENT SETUP
## LOCAL (your AFS environment)
# export HOME=/afs/cern.ch/user/m/mzeman # uncomment and change this if missing
# export CASTOR_HOME=/castor/cern.ch/user/m/mzeman # uncomment and change this if missing

2. Secondly, we need to process the input parameters coming from the Batch.Simulation.sh script:

### INPUT PARAMETERS
export INPUT=$1    # input POOL ROOT file
export EVENTS=$2 # number of events to process
export SKIP=$3 # number of generated events to skip
export ID=$4 # unique run identificator of your choice

### ISSUE CODE
## CREATE TEMPORARY WORKSPACE

# Run the Job

5. Finally we need to copy our generation results from the LXBATCH, the most convenient way is to put it to castor using rfcp command.

0.1 LXBATCH Simulation Submitter

bsub -R "type==SLC4&&swp>4000&&pool>2000" -q 8nh -o ${SCRATCH}/${OUTPUT}.${SKIP}-${LAST}of${TOTAL}.${ID}.Screen.txt Simulation.sh ${INPUT} ${EVENTS} ${SKIP} ${ID}

1 Digitization on LXBATCH

1 Reconstruction on LXBATCH

cmt co -r AnalysisExamples-00-20-14 PhysicsAnalysis/AnalysisCommon/AnalysisExamples

Running General Full Chain on the GRID

1 Configuration on the GRID

2 Generation on the GRID

3 Simulation on the GRID

4 Reconstruction on the GRID

1 Running General Full Chain on LXBATCH

0.1 Configuration

• MC8.105145.PythiaZmumu.py for Z->mu,mu decay,
• MC8.105144.PythiaZee.py for Z->e,e decay,

# Go back to working directory
echo ${CURRENTDIR}
cd ${CURRENTDIR}

# Run the Job

5. Finally we need to copy our generation results from the LXBATCH, the most convenient way is to put it to castor using rfcp command.

# Copy out the results if they exist

I.6. LXBATCH Generation Submitter

 
# export SCRATCH=${HOME}/scratch0/
export JOBOPTIONS=$1 #MC8.105144.PythiaZee.py                        # which file to run, look here: http://reserve02.usatlas.bnl.gov/lxr/source/atlas/Generators/EvgenJobOptions/share/
export EVENTS=$2                                       # number of events to process
export ID=$3                                          # unique run identificator of your choice

PARSE=(`echo ${JOBOPTIONS} | tr '.' ' '`)
OUTPUT=${PARSE[0]}.${PARSE[1]}.${PARSE[2]}         # name of the Screen Output

bsub -R "type==SLC4&&swp>4000&&pool>2000" -q 8nh  -o ${SCRATCH}/ScreenOutput.${OUTPUT}.${EVENTS}.${ID}.txt Generation.sh ${JOBOPTIONS} ${EVENTS} ${ID}

Running General Full Chain on LXBATCH

0. Configuration

0.1. Setting up the CMT Environment

0.2. What to do every time you log on?

0.2.1. Employing Startup Script

0.2.2. Preparing CASTOR

0.3. Running Full Chain on LXPLUS

0.4. Using LXBATCH

I. Generation

I.1. Running Generation JobTransformation?

I.2. How to get Job Option files:

> get_files -jo MC8.105145.PythiaZmumu.py

I.3. How to change minimum number of events:

evgenConfig.minevents = 100 # default is 5000

On LXBATCH we can of course leave the default 5000, however since we use get_files -jo on LXBATCH to obtain the JobOptions? file, it gets the unmodified version from the central repositories.

I.4. Generation on LXPLUS

> csc_evgen08_trf.py 105145 1 110 1324354657 ./MC8.105145.PythiaZmumu.py 105145.pool.root

I.5. Generation on LXBATCH

• Generation.sh: generation job to be submitted to the LXBATCH
• BatchGenerate.sh: Script to submit Generation.sh to the LXBATCH

A few notes:

• You can run this script from any folder (public, private - it does not matter)
• Make sure you make it executable using > chmod +x Generation.sh
• Make sure all directories and files specified in the environment variables exist !!! (if you followed this tutorial EXACTLY, everything should be working)
• The script creates your environment on LXBATCH machine and runs the generation setup of your choosing

The script has three parameters you HAVE TO specify. To run the job, issue the following from the directory where you put BOTH of the scripts:

> ./BatchGenerate.sh <JOBOPTIONS> <EVENTS> <ID>

• JOBOPTIONS is a Python script which specifies the event to be generated (string.py), all available scripts can be found here: http://reserve02.usatlas.bnl.gov/lxr/source/atlas/Generators/EvgenJobOptions/share/
• EVENTS is a number of events you want to generate (int)
• ID is an identifier of your choosing (string)

Making generation script for LXBATCH step by step: 1. First we want to specify the environment variables, so the script works generally everywhere and if something needs to be changed, it needs to be done only on one place which can be easily found at the beginning of the file:

### ENVIROMENT SETUP
## LOCAL (your AFS environment)
# export HOME=/afs/cern.ch/user/<LETTER>/<USERNAME>   # uncomment and change in case you are missing these
# export CASTOR_HOME=/castor/cern.ch/user/<LETTER>/<USERNAME>   # uncomment and change in case you are missing these
export CMT_HOME=${HOME}/cmt-fullchain
export FULL_CHAIN=${HOME}/testarea/FullChain/

# CASTOR (your CASTOR environment)
export CASTOR_GENERATION=${CASTOR_HOME}/fullchain/generation
export CASTOR_SIMULATION=${CASTOR_HOME}/fullchain/simulation
export CASTOR_DIGITIZATION=${CASTOR_HOME}/fullchain/digitization
export CASTOR_RECONSTRUCTION=${CASTOR_HOME}/fullchain/reconstruction
export CASTOR_TEMP=${CASTOR_HOME}/fullchain/temp
export CASTOR_LOG=${CASTOR_HOME}/fullchain/log

2. Secondly, we need to process the input parameters coming from the BatchGenerate.sh script:

###################################################################################################
### INPUT PARAMETERS
export JOBOPTIONS=$1 #MC8.105144.PythiaZee.py # which file to run (string)
export EVENTS=$3 # number of events to process (int)
export ID=$4 # unique run identificator of your choice (string)

# Parse environment variables amongst points
PARSE=(`echo ${JOBOPTIONS} | tr '.' ' '`)
RUN=${PARSE[1]} # generation Job Transformation requires RUN number parsed from the JobOptions filename
OUTPUT=${PARSE[0]}.${PARSE[1]}.${PARSE[2]} # name of the CASTOR output for easy orientation

## Remove all the parameters from $1, $2 and $3 (source setup.sh could pick them up)
while [ $# -gt 0 ] ; do shift ; done

3. Now

# Create temporary workspace
echo "###################################################################################################"
echo "CREATING WORKSPACE"
mkdir Generation.$JOBOPTIONS.$EVENTS.$DSN
cd Generation.$JOBOPTIONS.$EVENTS.$DSN
      
# Copy entire run directory in my working place
get_files -jo ${JOBOPTIONS}
      
# Setup Athena environment
# Have some experience with newer Athena releases,
# that first setup.sh have to be done in its directory
echo "###################################################################################################"
echo "SETTING UP CMT HOME"
export CURRENTDIR=`pwd`      # remember the current directory
cd ${CMT_HOME}

echo "Your CMT home directory:" $CMT_HOME
source ${CMT_HOME}/setup.sh -tag=14.2.10,32

. /afs/cern.ch/atlas/software/releases/14.2.10/cmtsite/setup.sh -tag=14.2.10 
export CMTPATH=/afs/cern.ch/atlas/software/releases/14.2.10/AtlasProduction/14.2.10 
. /afs/cern.ch/atlas/software/releases/14.2.10/AtlasProduction/14.2.10/AtlasProductionRunTime/cmt/setup.sh 

# Copy out the results if exist echo "###################################################################################################" echo "COPYING SIMULATION OUTPUT" if [ -e $JOBOPTIONS.$EVENTS.$DSN.pool.root ] ; then rfcp $JOBOPTIONS.$EVENTS.$DSN.pool.root ${CASTOR_GENERATION}/$JOBOPTIONS.$EVENTS.$DSN.pool.root fi

# List content of the working directory for debugging purposes ls -lRt

# Clean workspace before exit cd .. rm -fR Generation.$JOBOPTIONS.$EVENTS.$DSN

II. Simulation on LXBATCH

Digitization on LXBATCH

csc_digi_trf.py <INPUT hits.pool.root> <OUTPUT rdo.pool.root> <MAX EVENTS> <SKIP EVENTS> <GEOMETRY VERSION> <SEEDS> ...

Reconstruction on LXBATCH

Running General Full Chain on the GRID

• Generation.sh: generation job to be submitted to the LXBATCH
• BatchGenerate.sh: Script to submit Generation.sh to the LXBATCH

0. Setting up the CMT Environment

Employing Startup Script

Preparing CASTOR

> bqueues

Running General Full Chain on LX batch

> cd cmt-fullchain
> source /afs/cern.ch/sw/contrib/CMT/v1r20p20070208/mgr/setup.sh

cmt co -r AnalysisExamples-00-20-14 PhysicsAnalysis/AnalysisCommon/AnalysisExamples

Running General Full Chain on LX batch using Job Transformations

0. Preparing the LX plus environment

> cd $HOME
> mkdir cmt-fullchain

Create your $TestArea, where your packages will be installed:

> mkdir testarea
> mkdir testarea/FullChain

3. Download the CMT enviroment to your CMT_HOME. The v1r20p20070208 (an iteration for the v1r20 release) is tested well-working environment.

> cd cmt-cosmics
> source /afs/cern.ch/sw/contrib/CMT/v1r20p20070208/mgr/setup.sh

4. Create the requirements environment for the CMT.

> touch requirements
> mcedit requirements

#---- CMT HOME REQUIREMENTS FILE ---------------------------------
set   CMTSITE  CERN
set   SITEROOT /afs/cern.ch
macro ATLAS_DIST_AREA ${SITEROOT}/atlas/software/dist
macro ATLAS_TEST_AREA /afs/cern.ch/user/m/mzeman/testarea/FullChain

apply_tag oneTest  # use ATLAS working directory
apply_tag setup    # use working directory
apply_tag 32       # use 32-bit

use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)

#----------------------------------------------------------------

5. Configure the CMT to your AFS account (you do this only once per each new CMT_HOME you are using):

> cmt config

6. Logout.

7. Login. Enter your CMT_HOME directory and source the Athena setup with release specifications (14.2.0 32-bit).

> cd cmt-fullchain
> source setup.sh -tag=14.2.0,32

Congratulations, your lxplus environment is now ready

What to do every time you log on?

> cd cmt-fullchain

Load Athena environment:

> source setup.sh -tag=14.2.0,32

You can simplify this process by employing startup script with functions and environment variables, see http://www-hep2.fzu.cz/twiki/bin/view/ATLAS/AthenaRelated#LXplus_Login

Here is an example of a script employed for the FullChain? :

#---- LXPLUS STARTUP SCRIPT ---------------------------------
# Load Athena Function
function Load/Athena {
    source ${CMT_HOME}/setup.sh -tag=$*
   echo "Athena" $* "Loaded"
    shift
}

# Full Chain Setup
function FullChain {
   echo "Loading Full Chain Environment"
   
   # Specify CMT home directory
   export CMT_HOME=${HOME}/cmt-fullchain
   echo "Your CMT home directory:" $CMT_HOME

   # Use function Load/Athena
   Load/Athena 14.2.0,32
   # Load Envrioment Variables
   # LOCAL
   export FULL_CHAIN=${HOME}/testarea/FullChain/

   # CASTOR (you need to create these)
   export CASTOR_GENERATION=${CASTOR_HOME}/fullchain/generation/
   export CASTOR_SIMULATION=${CASTOR_HOME}/fullchain/simulation/
   export CASTOR_DIGITIZATION=${CASTOR_HOME}/fullchain/digitization/
   export CASTOR_RECONSTRUCTION=${CASTOR_HOME}/fullchain/reconstruction/
   export CASTOR_TEMP=${CASTOR_HOME}/fullchain/temp/
   export CASTOR_LOG=${CASTOR_HOME}/fullchain/log/
   
   echo "Environment Ready"
}

#----------------------------------------------------------------

In case you have not already done it, do create the directories on Castor:

> rfmkdir ${CASTOR_HOME}/fullchain
> rfmkdir ${CASTOR_HOME}/fullchain/generation
> rfmkdir ${CASTOR_HOME}/fullchain/simulation
.
.
.

Running Full Chain on LX plus

Disadvantages:

• LX plus will kill all your runs after 40 minutes, therefore you can simulate and reconstruct only a few events.
• Your session gets busy, therefore you loose one terminal
• Logging of lxplus, loosing connection or turning off your computer causes the run to stop

In conclusion, using lxplus on larger scale is impossible. The solution is to use LX batch or Grid.

Using LX batch

• You are starting up clean, so there is
  • NO startup script (no functions and environment variables of your own),
  • NO CMT configured,
  • NO Athena version is loaded
• Your home directory /afs/cern.ch/user// is accessible as if you were logged in (not just the public folder)
• All other ATLAS repositories are visible normally (/afs/cern.ch/atlas/software/... and other users)
• CASTOR is visible normally

Submitting jobs to LX batch is done by the following command

> bsub -q <QUEUE>[ -R "type==<OS>&&swp><# megabytes>&&pool><# megabytes>" ] <SCRIPT NAME> [ parameters ]

Listing the jobs currently running:

> bjobs [ -l ] [ -w ] [ jobID ] 

Killing a job assuming you know its ID (use bjobs to find out)

> bkill <jobID>

I. Generation on LX batch

get_files -jo <FILE NAME>

You can choose from a variety of files:

• MC8.105145.PythiaZmumu.py for Z->mu,mu
• MC8.105144.PythiaZee.py for Z->e,e
• and many others

Edit: =evgenConfig.minevents = 100 =

Run: =csc_evgen08_trf.py 105145 1 110 1324354657 ./MC8.105145.PythiaZmumu.py 105145.pool.root =

II. Simulation on LX batch

#----------------------------------------------------------------
#!/bin/zsh      
# These commands are executed on LXbatch to run the Full Chain

# ENVIROMENT SETUP
# LOCAL
# export HOME=/afs/cern.ch/user/m/mzeman
# export CASTOR_HOME=/castor/cern.ch/user/m/mzeman
export CMT_HOME=${HOME}/cmt-fullchain
export FULL_CHAIN=${HOME}/testarea/FullChain/

# CASTOR
export CASTOR_GENERATION=${CASTOR_HOME}/fullchain/generation
export CASTOR_SIMULATION=${CASTOR_HOME}/fullchain/simulation
export CASTOR_DIGITIZATION=${CASTOR_HOME}/fullchain/digitization
export CASTOR_RECONSTRUCTION=${CASTOR_HOME}/fullchain/reconstruction
export CASTOR_TEMP=${CASTOR_HOME}/fullchain/temp
export CASTOR_LOG=${CASTOR_HOME}/fullchain/log

# INPUTS
export GENERATION_INPUT=${CASTOR_GENERATION}/105144.pool.root      # input RDO file
export EVENTS=$1                                       # number of events to process
export SKIP=$2                                          # number of generated events to skip
export DSN=$3                                          # unique run identificator of your choice

# Remove all the parameters from $1, $2 and $3, otherwise
# "source setup.sh ..." would pick them up and probably fail
while [ $# -gt 0 ] ; do shift ; done

# Create temporary workspace
echo "###################################################################################################"
echo "CREATING WORKSPACE"
mkdir Temp.$DSN.$EVENTS.$SKIP
cd Temp.$DSN.$EVENTS.$SKIP
      
# Copy entire run directory in my working place
rfcp ${GENERATION_INPUT} .
      
# Setup Athena environment
# Have some experience with newer Athena releases,
# that first setup.sh have to be done in its directory
echo "###################################################################################################"
echo "SETTING UP CMT HOME"
export CURRENTDIR=`pwd`      # remember the current directory
cd ${CMT_HOME}

echo "Your CMT home directory:" $CMT_HOME
source ${CMT_HOME}/setup.sh -tag=14.2.0,32

. /afs/cern.ch/atlas/software/releases/14.2.0/cmtsite/setup.sh -tag=14.2.0 
export CMTPATH=/afs/cern.ch/atlas/software/releases/14.2.0/AtlasProduction/14.2.0.1 
. /afs/cern.ch/atlas/software/releases/14.2.0/AtlasProduction/14.2.0.1/AtlasProductionRunTime/cmt/setup.sh 

# Go back to working directory and run the job
echo "###################################################################################################"
echo "RUNNING SIMULATION"
echo $CURRENTDIR
cd $CURRENTDIR
csc_simul_trf.py 105144.pool.root hits.pool.root rdo.pool.root ${EVENTS} ${SKIP} 1324354656 ATLAS-CSC-02-01-00 100 1000 

# Copy out the results if exist
echo "###################################################################################################"
echo "COPYING SIMULATION OUTPUT"
if [ -e hits.pool.root ] ; then
    rfcp hits.pool.root ${CASTOR_SIMULATION}/hits.pool.$DSN.$SKIP.root
   rfcp rdo.pool.root ${CASTOR_SIMULATION}/rdo.pool.$DSN.$SKIP.root
fi
       
# List content of the working directory for debugging purposes
ls -lRt
      
# Clean workspace before exit
cd ..
rm -fR Temp.$DSN.$EVENTS.$SKIP

#----------------------------------------------------------------

LX batch submitter

# export SCRATCH=${HOME}/scratch0/
export EVENTS=$1   # number of events to process
export SKIP=$2      # number of events to skip
export DSN=$3      # unique run dentifier of your choice

bsub -R "type==SLC4&&swp>4000&&pool>2000" -q 8nh  -o ${SCRATCH}/ScreenOutput.$DSN.$EVENTS.$SKIP.txt Simulate.sh ${EVENTS} ${SKIP} ${DSN}

#----------------------------------------------------------------

Z->ee

-- MichalMarcisovsky - 28 Jul 2008 -- MartinZeman - 28 Jul 2008