CHTC Case Studies. Impossible is (almost) nothing Challenging projects in CHTC

Transcription

1 CHTC Case Studies Impossible is (almost) nothing Challenging projects in CHTC Cathrin Weiss UW Madison 1 1

2 Most projects 2 2

3 Most projects Scientist 2 2

4 Most projects Scientist compute scripts 2 2

5 Most projects CHTC Scientist compute scripts 2 2

6 Most projects CHTC Scientist compute scripts 2 2

7 Most projects CHTC Scientist compute scripts 3 3

8 Most projects CHTC Team CHTC Scientist compute scripts 3 3

9 Most projects CHTC Team CHTC Scientist compute scripts Condor jobs 3 3

10 Most projects CHTC Scientist Condor jobs 3 3

11 Most projects CHTC Condor jobs Scientist 3 3

12 Most projects CHTC Results Scientist 4 4

13 Most projects CHTC Results Scientist 4 4

14 Most projects CHTC Results Scientist 4 4

15 But some projects CHTC Failure Scientist 5 5

16 But some projects CHTC Failure Scientist 5 5

17 But some projects CHTC Failure Scientist 5 5

18 What to do? 6

19 What to do? CHTC Team 6

20 What to do? CHTC Team Talk to us! 6

21 Example project Protein structure determination via X-ray protein crystallography *Images courtesy of Thomas Splettstoesser 7 7

22 Example project Protein structure determination via X-ray protein crystallography Protein crystal *Images courtesy of Thomas Splettstoesser 7 7

23 Example project Protein structure determination via X-ray protein crystallography X-ray Protein crystal Diffraction pattern *Images courtesy of Thomas Splettstoesser 7 7

24 Example project Protein structure determination via X-ray protein crystallography X-ray phases (FFT) Protein crystal Diffraction pattern Electron density map (3D image) *Images courtesy of Thomas Splettstoesser 7 7

25 Example project Protein structure determination via X-ray protein crystallography X-ray phases (FFT) fitting Protein crystal Diffraction pattern Electron density map (3D image) Protein structure *Images courtesy of Thomas Splettstoesser 7 7

26 Example project Protein structure determination via X-ray protein crystallography X-ray fitting phases (FFT) Protein crystal Electron Diffraction density pattern map (3D image) Protein structure *Images courtesy of Thomas Splettstoesser 7 7

27 Example project Protein structure determination via X-ray protein crystallography Handling low-resolution electron density maps: ACMI (Automatic Crystallographic Map Interpretation) X-ray fitting phases (FFT) Protein crystal Electron Diffraction density pattern map (3D image) Protein structure *Images courtesy of Thomas Splettstoesser Work by George Philipps, Siddhart Puthur, Ameet Soni, et al. Biochemistry, Medical Sciences and Computer Sciences 7 7

28 ACMI process (Protein: Sequence of amino acids) Phase 1: Local match (a priori probability of each amino acid s location) 8 8

29 ACMI process (Protein: Sequence of amino acids) Phase 1: Local match (a priori probability of each amino acid s location) Electron density map 8 8

30 ACMI process (Protein: Sequence of amino acids) Phase 1: Local match (a priori probability of each amino acid s location) Amino acid Electron density map 8 8

31 ACMI process (Protein: Sequence of amino acids) Phase 1: Local match (a priori probability of each amino acid s location) Amino acid Electron density map 8 8

32 ACMI process (Protein: Sequence of amino acids) Phase 1: Local match (a priori probability of each amino acid s location) Match found! Amino acid Electron density map 8 8

33 ACMI process (Protein: Sequence of amino acids) Phase 1: Local match (a priori probability of each amino acid s location) Amino acid loc 1: 15% loc 2: 20% loc 3: 80%

34 ACMI process (Protein: Sequence of amino acids) Phase 2: Global match (marginal probability of each amino acid s location) Amino acid 1 Amino acid

35 ACMI process (Protein: Sequence of amino acids) Phase 2: Global match (marginal probability of each amino acid s location) Location x Amino acid 1 Amino acid

36 ACMI process (Protein: Sequence of amino acids) Phase 2: Global match (marginal probability of each amino acid s location) Location x Amino acid 1 Amino acid

37 ACMI process (Protein: Sequence of amino acids) Phase 2: Global match (marginal probability of each amino acid s location) Location x Mismatch! Amino acid 1 Amino acid

38 ACMI process (Protein: Sequence of amino acids) Phase 2: Global match (marginal probability of each amino acid s location) Amino acid 1 Amino acid

39 ACMI process (Protein: Sequence of amino acids) Phase 2: Global match (marginal probability of each amino acid s location) Location y Amino acid 1 Amino acid

40 ACMI process (Protein: Sequence of amino acids) Phase 2: Global match (marginal probability of each amino acid s location) Location y Amino acid 1 Amino acid

41 ACMI process (Protein: Sequence of amino acids) Phase 2: Global match (marginal probability of each amino acid s location) Location y Match! Amino acid 1 Amino acid

42 ACMI process (Protein: Sequence of amino acids) Phase 3: Determine possible protein structures 12 12

43 ACMI process (Protein: Sequence of amino acids) Phase 3: Determine possible protein structures Protein structure 12 12

44 CHTC: Phase

45 CHTC: Phase 1 Large input data set (electron density map: 500MB) 13 13

46 CHTC: Phase 1 Large input data set (electron density map: 500MB) Task can be divided into many subprocesses (1 per amino acid or amino acid combination) 13 13

47 CHTC: Phase 1 Large input data set (electron density map: 500MB) Task can be divided into many subprocesses (1 per amino acid or amino acid combination) Every subprocess reads same input data 13 13

48 CHTC: Phase 1 Large input data set (electron density map: 500MB) Task can be divided into many subprocesses (1 per amino acid or amino acid combination) Every subprocess reads same input data Some subprocesses produce redundant output if output folder not visible upon initialization 13 13

49 CHTC: Phase

50 CHTC: Phase 1 Run in Standard Universe 14 14

51 CHTC: Phase 1 Run in Standard Universe Advantages of Standard Universe: Remote IO (input data can stay on submit node and does not need to be transferred, output data is created on submit node too), and checkpointing of job state such that it can resume from last checkpoint on evict

52 CHTC: Phase

53 CHTC: Phase 2 Only one process, large input, large output, long running (several days from beginning to end) 15 15

54 CHTC: Phase 2 Only one process, large input, large output, long running (several days from beginning to end) Large memory and disk space requirements (approximately 8GB RAM and >20GB disk space, 13GB alone for input data) 15 15

55 Phase 2: Challenges 16 16

56 Phase 2: Challenges Standard Universe is limited to 32bit (therefore: 4GB of RAM), therefore not an option 16 16

57 Phase 2: Challenges Standard Universe is limited to 32bit (therefore: 4GB of RAM), therefore not an option CHTC machines heterogenous: some 32bit, some 64bit, various disk space sizes, only very few with >20GB disk space 16 16

58 Phase 2: Challenges Standard Universe is limited to 32bit (therefore: 4GB of RAM), therefore not an option CHTC machines heterogenous: some 32bit, some 64bit, various disk space sizes, only very few with >20GB disk space Checkpointing is needed since CHTC policy only guarantees 24 hours at once, then job may be evicted 16 16

59 Phase 2: What we did 17 17

60 Phase 2: What we did Vanilla Universe (no checkpointing, no remote IO, but data transfer) 17 17

61 Phase 2: What we did Vanilla Universe (no checkpointing, no remote IO, but data transfer) Worked with ACMI developers to include software-internal checkpointing 17 17

62 Phase 2: What we did Vanilla Universe (no checkpointing, no remote IO, but data transfer) Worked with ACMI developers to include software-internal checkpointing Pre-staged input data on various CHTC machines on different disk partitions 17 17

63 Phase 2: What we did Vanilla Universe (no checkpointing, no remote IO, but data transfer) Worked with ACMI developers to include software-internal checkpointing Pre-staged input data on various CHTC machines on different disk partitions Customized job attributes to make sure jobs run on appropriate machines and that checkpoint data will be transferred back in case of job eviction 17 17

64 Phase 2: What we did # phase 2 Condor submit ######################## executable = phase2 universe = vanilla requirements = (hasacmidata==true && Arch == "x86_64") when_to_transfer_output = on_exit_or_evict... queue 18 18

65 CHTC: Phase

66 CHTC: Phase 3 Several iterations of the same job reading the same input data (each job runs several hours), low memory and disk requirements 19 19

67 CHTC: Wrap Up 20 20

68 CHTC: Wrap Up Last but not least: Combine all three phases into a DAG workflow for ease of use 20 20

69 CHTC: Wrap Up Last but not least: Combine all three phases into a DAG workflow for ease of use Phase 1 Phase 1 Phase 1 Phase 1 followed by Phase2 followed by Phase 3 Phase

70 And...? 21 21

71 And...? Scientist 21 21

72 And...? It worked! Scientist Protein structure 21 21

73 Conclusion 22 22

74 Conclusion Most CHTC projects work straightforwardly out of the box 22 22

75 Conclusion Most CHTC projects work straightforwardly out of the box But even if you get stuck, talk to us. We can work something out together! CHTC Team Scientist 22 22

76 Conclusion Most CHTC projects work straightforwardly out of the box But even if you get stuck, talk to us. We can work something out together! CHTC Team Scientist 22 22

77 Conclusion Most CHTC projects work straightforwardly out of the box But even if you get stuck, talk to us. We can work something out together! CHTC Team Scientist 22 22