LHC Physics or How LHC experiments will use OSG resources

Size: px

Start display at page:

Download "LHC Physics or How LHC experiments will use OSG resources"

Evan Wade
6 years ago
Views:

1 LHC Physics or How LHC experiments will use OSG resources OSG Consortium Meeting 08 / 21 / 06 Oliver Gutsche USCMS / Fermilab

2 Particle Physics Probe innermost structure and explain it from first principles 2

3 Standard Model Current description of matter: 12 elementary particles and their antiparticles 4 force particles carrying the 3 dominating forces (at scales investigated) 3

4 LHC, the discovery machine What can the LHC add to the current picture: Higgs: explain the origin of mass Super-Symmetry: beyond the standard model 4

5 LHC Large Hadron Collider Proton-Proton collisions Beam energy: 7 TeV Circumference: 27 km Bunch crossing rate at interaction regions: 40 MHz 5

6 Outline Example for Higgs discovery channel Analysis and requirements Computing CMS Metrics CMS computing model The role of the centers OSG contribution User use case Services Challenges and current status Summary & Outlook!"#$%&'$()*#+', 8 <; <9 <8 <4; <49 <48 55!! (( %% :44 ;44 -))))./ Apologies to the other 3 experiments, following talk mainly concentrates on CMS ++ 6

7 Example Signal: H ZZ (*) 4μ Cross section (fb) Signal Reconstruction strategy: Reconstruct 4 μ (2 μ +, 2 μ - ) combine each 2 μ to Z M H (GeV) combine 2 Z to H 7

8 Event display 8

9 Backgrounds and Analysis Strategy 3 main background Signal overlaid by background events which produce similar signature in detector Analysis Strategy: tt Record data with detector Zbb ZZ Simulate background and signal events Extract signal from recorded data using the background simulation 9

10 Data Taking High Level Trigger Reconstruction primary datasets MC production Data taking vs. MC production Generation and Simulation Reconstruction MC datasets 10

11 Data tiers data tier description data tier description RAW digitized detector output SimEvent generated and simulated event information RECO AOD reconstructed information extract from RECO, most used analysis quantities RECO- SimEvent AOD reconstructed information extract from RECO, most used analysis quantities FEVT (full event) SimFEVT (full simulated event) RAW RECO SimEvent RECO-SimEvent A O D A O D 11

12 Disk requirements Detector output rate after trigger: 150 Hz for all running conditions (low luminosity, high luminosity) Extrapolated beam time: year beam time number of events s s Events from data taking: data_tier event size total size in 2007 total size in 2008 RAW 1.50 MB 0.21 PB 2.10 PB Reco 0.25 MB 0.03 PB 0.35 PB AOD 0.05 MB 0.01 PB 0.07 PB total 1.80 MB 0.25 PB 2.51 PB Events from MC production: data_tier event size total size in 2007 total size in 2008 SimEvent 2.00 MB 0.28 PB 2.79 PB RecoSimEvent 0.40 MB 0.06 PB 0.56 PB AOD 0.10 MB 0.01 PB 0.14 PB total 2.50 MB 0.35 PB 3.49 PB Total: year total data volume PB PB 12

13 CPU requirements Reconstruction and MC production Assume Core in 2007: 2000 SI2K (optimistic) Single Core Pentium IV 3 GHz 1300 SI2K Time to reconstruct event 78 ksi2k s/ev On demand reconstruction at 150 Hz 11.7 MSI2K 5850 Core s Time to simulate and reconstruct event 234 ksi2k s/ev => 1 Core 270,000 ev/year Single Analysis has to access one primary dataset and the MC samples Assume: Analysis needs access to AOD Every 3 days Selection has to be finished at least after 3 days Analysis needs access to RECO Every 7 days Analysis Analysis has to be finished at least after 7 days Selection time: 0.25 ksi2k s/ev Analysis time: 0.25 ksi2k s/ev 13

14 CMS Grid Tier Structure T1: USA T1: Italy T1: France T0 T1: Germany T1: Spain T1: UK T1: Taiwan 14

15 Data flow CMS Online (HLT) ~10 online streams (RAW) ~6 Tier-1 7 T1 Centres (off-site) ~25 25 Tier-2 Centres ~10 online streams (RAW) Primary tape archive ~10 online streams (RAW) Tier 0 First pass reconstruction ~50 Datasets (RAW+RECO) CMS-CAF (CERN Analysis Facility) ~50 Datasets (RAW+RECO) shared amongst Tier 1's Average of ~8 Datasets per Tier 1 (RAW+RECO) Secondary tape archive Analysis, Tier 1 Calibration, Tier 1 skim Tier making... 1 Tier 1 Re-reconstruction, Tier 1 Tier 1 Skims: RECO, AOD s T0 distributes RAW and reconstructed data to T1 s (subset of the primary datasets, full AOD copy) s are associated to specific T1 which provides support and distributes data (simulated MC is transferred back to associated T1) substantial computing resources are provided by the T1 s and s CMS-CAF performs latency critical activities like detector problem diagnostic, trigger performance service, derivation of calibration and alignment data 15

US contribution to CMS Tier structure U.S. contribution to CMS tier structure T1 at FNAL 7 associated sites (OSG) Site Batch Slots CPU (ksi2k) Disk (TB) Caltech 256 330 55 Florida

16 US contribution to CMS Tier structure U.S. contribution to CMS tier structure T1 at FNAL 7 associated sites (OSG) Site Batch Slots CPU (ksi2k) Disk (TB) Caltech Florida MIT Nebraska Purdue UCSD Wisconsin TOTAL Nebraska Wisconsin T1 FNAL MIT Purdue CALTECH San Diego Florida 16

CMS Computing Infrastructure PhEDEx ProdAgent CRAB Core Infrastructure DBS: Data Bookkeeping System official catalog of available datasets and MC samples GRID Middleware organizes datasets and

17 CMS Computing Infrastructure PhEDEx ProdAgent CRAB Core Infrastructure DBS: Data Bookkeeping System official catalog of available datasets and MC samples GRID Middleware organizes datasets and samples in fileblock containing files DBS DLS DLS: Data Location Service catalog of which storage elements contain specific datasets and MC samples Site Resources batchsystem CPU TFC Disk TFC: Trivial File Catalog files are stored in a CMS specific namespace which is reproduced at the storage elements at every site Access to files in the CMS namespace is resolved by the CMS applications automatically prepending site specific locations and access protocol using a local site configuration 17

18 CMS Computing Infrastructure PhEDEx ProdAgent CRAB GRID Middleware Computing Systems PhEDEX Transport any dataset or samples registered in DBS between sites Site Resources DBS DLS MC production system (ProdAgent) Produce MC samples in centralized way batchsystem CPU TFC Disk User Analysis Tool (CRAB) enable user to execute his analysis code on any sample registered in DBS/ DLS 18

CMS site CPU: 1 MSI2K DISK: 200 TB Site requirements Network: 2.

19 CMS site CPU: 1 MSI2K DISK: 200 TB Site requirements Network: Gbps OSG middleware stack Computing Element (CE) Storage Element (SE) Batch system OSG sites: Condor and PBS Mass Storage system OSG sites: dcache (tapeless) CMS software installed by central instance (OSG sites: $OSG_APP) including TFC setup Opportunistic usage (OSG) Useable for MC production OSG middleware stack Computing Element (CE) Outbound connectivity of workernodes (stage-in and out) CMS software installed by central instance (in $OSG_APP) 19

20 User analysis Task Discovery of the Higgs in channel H ZZ (*) 4μ Approach: use recorded and reconstructed data from detector OLI use produced MC samples reconstruct 4μ signature and extract Higgs mass 20

21 ideal MC workflow MC samples: Produced by MC production system at level Archived at T1 which stores corresponding datasets MC production requests Events / 2 GeV H(150) H(300) ZZ Zbb tt Initiated by the physics groups 1 Samples are probably more general than needed Physics groups and users Prepare skims of the MC samples at T1 s for specific physics purposes Events / GeV ΣE CALO (GeV) for least isol. µ H(150) H(300) ZZ Zbb tt Complete skims are transported to dedicated s. User analyzes MC samples by processing skims at s Pt (GeV) 2nd lowest pt µ 21

ideal Data workflow Data Recorded by the detector Triggered by the HLT Reconstructed at T0 Split into primary datasets Distributed between T1 centers Most analysis only needs to access one of the

22 ideal Data workflow Data Recorded by the detector Triggered by the HLT Reconstructed at T0 Split into primary datasets Distributed between T1 centers Most analysis only needs to access one of the primary samples AOD is sufficient for 90% of the analysis data taking hasn t started, for now just a event display of a cosmic muon Physics groups and users Skim AOD of primary samples at T1 s Complete skims are transported to dedicated s. User analyzes data samples by processing AOD skims at s 22

23 Special tasks Alignment/Calibration In general, alignment/calibration is calculated at T0 or T1 for immediate usage during reconstruction Alignment / Calibration Parts of RAW data samples can in addition be transported to dedicated to calculate improved alignment/calibration constants Reprocessing of data Data samples are planned to be rereconstructed with improved alignment/ calibration and reconstruction algorithms 3 times a year on T0 and T1 level Reprocessing Reprocessing requires re-skimming of samples 23

24 Startup scenario During startup of LHC, no experience with the detector and its output will exist First calibration/alignment calculations will be not sufficient for analysis AOD RAW data samples will be transported to s to: Understand the detector Improve the reconstruction algorithms, calibration and alignment Extract first physics messages T1 resources Used more for analysis than skimming and rereconstruction (less resources needed by default operations). RAW Re-reprocessing will be more frequent but short running time hasn t produced large amounts of data yet 24

25 Service Challenges To prepare for the startup, the experiments exercise their systems in dedicated service challenges CMS currently runs Service Challenge 4 (SC4) which consists of the exercise of: the dataset transport system PhEDEx with emphasis on all possible transport endpoint combinations (T0, T1, ) the MC production system by running ~12,500 MC production jobs per day on LCG and OSG resources and to produce MC samples for the upcoming challenge the analysis infrastructure by running ~12,500 analysis jobs per day on samples distributed to every center CMS next challenge is CSA2006 in the fall which exercises the full workflow feeding MC samples to the HLT and reconstruction at T0 centers Reprocessing, skimming at T1 centers Analysis and MC simulation at centers Sample transportation between the individual centers 25

26 First MC production run: OSG summary LCG (2) 26% LCG (3) 15% OSG 34% First MC production run: ~2 weeks of August Total: 45 Mio. events, OSG (incl. FNAL): 17 Mio. events LCG (1) 24% More details in Ajit Mohapatra s talk on Wednesday 26

27 Analysis infrastructure: grid-wide performance 15,000 12,500 10,000 Overall Job statistics Total Pending Total Running Total Terminated 15,000 12,500 10,000 Job Success Statistics Success Failed Unknown 7,500 7,500 5,000 5,000 2,500 2, /26/06 07/27/06 07/28/06 07/29/06 07/30/06 07/31/06 08/01/06 08/02/06 08/03/06 08/04/06 08/05/06 08/06/06 08/07/06 07/26/06 07/27/06 07/28/06 07/29/06 07/30/06 07/31/06 08/01/06 08/02/06 08/03/06 08/04/06 08/05/06 08/06/06 08/07/06 15,000 12,500 10,000 7,500 5,000 GRID Success Statistics Done Cancelled Aborted Unknown Automated systems (JobRobots) achieve gridwide performance close to goal of 12,5000 jobs per day 2,500 scale at ~10,000 jobs per 0 07/26/06 07/27/06 07/28/06 07/29/06 07/30/06 07/31/06 08/01/06 08/02/06 08/03/06 08/04/06 08/05/06 08/06/06 08/07/06 day for 5 days and improving 27

Analysis infrastructure: OSG-wide performance 4,000 Overall Job statistics 4,000 Job Success Statistics 3,000 3,000 2,000 Total Pending Total Running Total Terminated 2,000 Success Failed Unknown

28 Analysis infrastructure: OSG-wide performance 4,000 Overall Job statistics 4,000 Job Success Statistics 3,000 3,000 2,000 Total Pending Total Running Total Terminated 2,000 Success Failed Unknown 1,000 1, /26/06 07/27/06 07/28/06 07/29/06 07/30/06 07/31/06 08/01/06 08/02/06 08/03/06 08/04/06 08/05/06 08/06/06 08/07/06 07/26/06 07/27/06 07/28/06 07/29/06 07/30/06 07/31/06 08/01/06 08/02/06 08/03/06 08/04/06 08/05/06 08/06/06 08/07/ GRID Success Statistics Done Cancelled Aborted Unknown Scale at ~3,000 jobs per day for 5 days OSG sites contribute large percentage of overall 0 07/26/06 07/27/06 07/28/06 07/29/06 07/30/06 07/31/06 08/01/06 08/02/06 08/03/06 08/04/06 08/05/06 08/06/06 08/07/06 analysis performance 28

29 Analysis infrastructure: OSG site performance 900 Overall Job statistics 900 Job Success Statistics Total Pending Total Running Total Terminated 450 Success Failed Unknown /26/06 07/27/06 07/28/06 07/29/06 07/30/06 07/31/06 08/01/06 08/02/06 08/03/06 08/04/06 08/05/06 08/06/06 08/07/06 07/26/06 07/27/06 07/28/06 07/29/06 07/30/06 07/31/06 08/01/06 08/02/06 08/03/06 08/04/06 08/05/06 08/06/06 08/07/ GRID Success Statistics Example: Nebraska: Done Cancelled Aborted Unknown scale between 450 and 850 jobs per day high success rates 0 07/26/06 07/27/06 07/28/06 07/29/06 07/30/06 07/31/06 08/01/06 08/02/06 08/03/06 08/04/06 08/05/06 08/06/06 08/07/06 Other OSG sites show similar performance 29

30 Summary & Outlook Computing for the LHC experiments is large and globally organized LHC computing marks the final change in high energy physics from the host laboratory centered analysis of data to a global approach GRID resources and OSG resources in particular will be used efficiently to perform standard production and analysis as well as special tasks Analysis will be a challenge, not only from the physics point of view but also from the computational requirements and specialties 30

31 In the end... Events / 2 GeV Nb bw mean bw gamma Ns m 4µ [GeV] m 4µ We hope to make many discoveries in particle physics with LHC data using OSG resources 31

32 The end 32

Run Time Tester Requirements Document

Run Time Tester Requirements Document P. Sherwood 7 March 2004 Version: 0.4 Status: Draft After review 2 March 2004 Contents 1 Introduction 2 1.1 Scope of the Requirements Document.....................