Sarah Cohen-Boulakia. Université Paris Sud, LRI CNRS UMR 8623

Transcription

1 Sarah Cohen-Boulakia Université Paris Sud, LRI CNRS UMR 86

2 Part I Data Integration workflows What are scientific workflow systems Designing a workflow from scratch or by reuse workflows and reproducibility Current Challenges Part II Ranking Biological data Introducing ranking into integration solutions Consensus rankings Part III Opportunities, challenges Sarah Cohen-Boulakia, Université Paris Sud

3 A few things provided in Entrez By alphabetical order of ids, relevance (Gene) By date of publication (Medline) Biological data have specific features Data from sources reflect expertize DBs are different (reliability etc.) Cross-references are not just hypertext links Different qualities: Manually provided or automatically obtained Different meanings: More info can be found at, is very different from, is similar to Several goals to achieve when querying The most famous data, the most reliable, the freshnest Sarah Cohen-Boulakia, Université Paris Sud

4 BioZon [Birkland et al, 006], Graph-based approach (graph of entities as a support for queries) Variants of google Page-rank algorithm Difficulties To be constantly updated Google-like (page-rank, object rank): probability distribution used to represent the likelihood that a person randomly clicking on links will arrive at any particular page. Requires the knowledge of the entire graph «local pagerank» All the sources of data have the same «value» Wanted: Ranking solution exploiting links (several paths leaded to the same data) + reliability of the sources + Problem : How to combine all such criteria? Alternative: Consensus rankings? Sarah Cohen-Boulakia, Université Paris Sud

5 Generating a consensus ranking to make the most of used ranking methods applied to biological data by Putting emphasis on their common points Not putting too much importance on data classified good by only one or a few ranking methods Sarah Cohen-Boulakia, Université Paris Sud

6 Numerous applications Voting system [Borda 1781] Web information search [Dwork et al. 001] Biological data search [DeConde et al. 006] Agregation of opinions [Kittur et al. 008] Numerous communities Sociales Sciences [Ali et al. 01] Algorithmics [Ailon et al. 008] Databases [Fagin et al. 00] Biology [Sese et al. 001] Sarah Cohen-Boulakia, Université Paris Sud 6

7 Increasingly important to me 1 1 1? 1 Order elements by preference 1 1 Input rankings consensus How to define a consensus? How to compute a consensus? Sarah Cohen-Boulakia, Université Paris Sud 7

8 BioConsert [Cohen-Boulakia et al. 011] FaginDyn [Fagin et al. 00] Ailon/ [Fagin et al. 00] RepeatChoice [Ailon et al. 010] PNE (exact) [Conitzer, et al. 006] (sans égalité entre éléments) Generalized Kendall-τ distance KwikSort [Ailon et al. 008] Pick-A-Perm [Ailon et al. 008] B&B [Ali et al. 01] Chanas [Chanas et al. 1996] MEDRank [Fagin et al. 00] BordaCount [Borda 1781] ChanasBoth [Coleman et al. 009] CopelandMethod [Copeland et al. 191] MC [Dwork et al. 001] Positionnal approaches Kendall-τ distance 8/ Sarah Cohen-Boulakia, Université Paris Sud 8

9 «Find a consensus close to input rankings» The Kendall τ D(π,σ) distance counts the number of pairs of elements inversed (ie in the opposite order) between two rankings.? Sarah Cohen-Boulakia, Université Paris Sud 9

10 «Find a consensus close to input rankings» The Kendall τ D(π,σ) distance counts the number of pairs of elements inversed (ie in the opposite order) between two rankings. Sarah Cohen-Boulakia, Université Paris Sud 10

11 Kemeny Score Optimal Consensus Complexity [Dwork et al 001, Biedl et al. 009] NP-Difficult for an odd number of permutations Sarah Cohen-Boulakia, Université Paris Sud 11

12 Kemeny Score Optimal Consensus Complexity [Dwork et al 001, Biedl et al. 009] NP-Difficult for an odd number of permutations Sarah Cohen-Boulakia, Université Paris Sud 1

13 Increasingly important to me 1 1 1? 1 Order elements by preference 1 1 Sarah Cohen-Boulakia, Université Paris Sud 1

14 Increasingly important to me 1 1 Rank elements by preference 1 1 1? 1 How to define a consensus with equalities Sarah Cohen-Boulakia, Université Paris Sud 1

15 «Find a consensus close to input rankings» Generalized Kendall τ G(r,s) counts the number of pairs of elements inversed between two rankings r et s tied in only one of the two rankings? Sarah Cohen-Boulakia, Université Paris Sud 1

16 «Find a consensus close to input rankings» Generalized Kendall τ G(r,s) counts the number of pairs of elements inversed between two rankings r et s tied in only one of the two rankings Sarah Cohen-Boulakia, Université Paris Sud 16

17 Increasingly important to me 1 1 1? 1 Order elements by preference 1 1 Sarah Cohen-Boulakia, Université Paris Sud 17

18 Increasingly important to me 1 Order elements by preference 1 1? 1 Sarah Cohen-Boulakia, Université Paris Sud 18

19 Projection Unification Unrelevant data elements are removed Unrelevant data elements are placed at the end of one dedicated bucket Sarah Cohen-Boulakia, Université Paris Sud 19

20 BioConsert [Cohen-Boulakia et al. 011] FaginDyn [Fagin et al. 00] [Cohen-Boulakia et al. 011] Ailon/ [Fagin et al. 00] RepeatChoice [Ailon et al. 010] [Betzler et al. 01] PNE (exact) [Conitzer, et al. 006] (sans égalité entre éléments) [Schalekamp et al. 009] Pick-A-Perm [Ailon et al. 008] Generalized Kendall-τ KwikSort [Ailon et al. 008] Chanas [Chanas et al. 1996] B&B [Ali et al. 01] MEDRank [Fagin et al. 00] BordaCount [Borda 1781] ChanasBoth [Coleman et al. 009] CopelandMethod [Copeland et al. 191] Positionnel MC [Dwork et al. 001] [Dwork et al. 001] [Ali et al. 01] Kendall-τ [Coleman et al. 009] Sarah Cohen-Boulakia, Université Paris Sud 0

21 A few data sets reused Most of the data sets are not publicly available Different normalization methods used Similarity levels *Données en gras = mise à disposition par les auteurs Sarah Cohen-Boulakia, Université Paris Sud 1

22 Various algorithms considered in each study, different normalisations, différent data sets Incomplete results, sometimes even contradictory Equalities are not considered Same behaviour of algorithms with equalities?? Impact of similarity between data sets? Need to compare approaches in a more systematic and exhaustive way! Sarah Cohen-Boulakia, Université Paris Sud

23 BioConsert [Cohen-Boulakia et al. 011] FaginDyn [Fagin et al. 00] Adapté aux égalités Ailon/ [Fagin et al. 00] RepeatChoice [Ailon et al. 010] PNE (exact) [Conitzer, et al. 006] (sans égalité entre éléments) PNE (exact) [Brancotte, et al. 01] (avec égalité entre éléments) Pick-A-Perm [Ailon et al. 008] Generalised Kendall-τ KwikSort [Ailon et al. 008] Chanas [Chanas et al. 1996] B&B [Ali et al. 01] MEDRank [Fagin et al. 00] BordaCount [Borda 1781] ChanasBoth [Coleman et al. 009] CopelandMethod [Copeland et al. 191] MC [Dwork et al. 001] Positionnal Kendall-τ Sarah Cohen-Boulakia, Université Paris Sud

24 Sarah Cohen-Boulakia, Université Paris Sud

25 BioConsert can be used in a very large majority of the cases For very large data sets (>0.000 elements) KwikSort can be preferred If there is a need to seed up then In case of few equalities use BordaCount Otherwise use MEDRank Alternativeley: use both algorithms and pick the best Time High quality (gap) low quality Sarah Cohen-Boulakia, Université Paris Sud

26 Query NCBI so that equivalent queries provide the same results Equivalent reformulations: cervix cancer vs cervical cancer (60 vs 0 genes) Abreviations: Attention deficit hyperactivity disorders vs ADHD (109 vs 1 genes, 7 in common) Linguistics variations: tumour vs tumor (& breast cancer) : 681 vs 91 genes Finding all reformulations is timeconsuming Querying using all reformulations provide huge amounts of data sets which have to be ranked. More precise reformulations : colorectal cancer vs Lynch syndrom (+6 new genes) Sarah Cohen-Boulakia, Université Paris Sud 6

27 I)Reformulations using biomedical terminologies II) Querying NCBI to get genes ranked by relevance III) Aggregating using a series of consensus algorithms with a variant of the Generalized Kendall τ distance Sarah Cohen-Boulakia, Université Paris Sud 7

28 Gold standards Based on Physicians from Institut Curie and Childrens Hospital of Philadelphia Relevant lists of genes for 9 diseases 7 cancers (bladder, breast, cervical, colorectal, neuroblastoma, prostate, retinoblastoma) one heart disease (the Long QT Syndrome) one psychiatric disorder (ADHD) Quality measure Air under the ROC curve (AUC) measures the presence of gold standard elements on the ranked results Such a measure returns a value between 0 and 1, 1 being the highest possible score. AUC( ) = 0. AUC( ) = 0.66 AUC( ) = 0.66 AUC( ) = 0.8 relevant gene unrelevant gene Sarah Cohen-Boulakia, Université Paris Sud 8

29 Top-0 genes obtained using NCBI & ConquR-Bio AUC increases of % in average when ConQuR-Bio is used instead of classical NCBI DEMO? Sarah Cohen-Boulakia, Université Paris Sud 9

30 Faced with the number of results obtained as answer to a query ranking results is crucial Priorize experiments Very important data may be in relatively small DB Bad quality data may be highly referenced Various ranking criteria can be taken into account Freshness, Reliability, Completeness Combining criteria is difficult Consensus rankings provide good solutions Expensive (time) optimisation techniques needed Still a lot to do! Sarah Cohen-Boulakia, Université Paris Sud 0