Analyzing NIH Funding Patterns

Transcription

1 Analyzing NIH Funding Patterns with Statistical Text Analysis Margaret Blume-Kohout New Mexico Consor>um Jihyun Park Eric Nalisnick Padhraic Smyth Dept. Of Computer Science University Of California, Irvine Ralf Krestel Web Science Research Group Hasso-PlaGner-Ins>tut

2 Measuring the Impact of NIH(National Institute of Health) Funding 2 NIH invests over $30 billion each year Can we gain insight into this process using text and metadata? Our approach is to use sta>s>cal topic modeling We used grants data from NCI (Na>onal Cancer Ins>tute) 6 NCI FUNDING AMOUNT 11 NCI FUNDED GRANTS BILLIONS (US DOLLARS) NUMBER OF GRANTS (THOUSANDS) YEAR ARRA Funded YEAR

3 Overview 3 NCI Data PROJECT ID GRANT ABSTRACT GENETICS... RCDC Labels For each grant FUNDING YEAR Text ClassificaCon Techniques HUMAN GENOME 0.8 BIOENGINEERING 0.7 NANO TECHNOLOGY 0.2 Probability of each label 0 being 0.25associated 0.5 with 0.75 the 1grant Funding paaerns over Cme for each area Funding + Year informacon

4 NCI (National Cancer Institute) Data 4 Grant abstracts from 1994 through 2013 Text Processing BOW representa>on Removed 500 common stopwords Extracted noun-phrase terms using a NLP parser BOW Data Total 149,901 documents Number of documents with labels (training data) : 31,628 (2008~2011) Number of documents without labels : 118,273 Size of vocabulary (W) : 29,713

5 LDA: Topics are Represented as Distributions over Words 5 Figures from Mark Steyvers Terrorism Wall Street Firms Stock Market Bankruptcy SEPT_11 WAR SECURITY IRAQ TERRORISM NATION KILLED AFGHANISTAN ATTACKS OSAMA_BIN_LADEN WALL_STREET ANALYSTS INVESTORS FIRM GOLDMAN_SACHS FIRMS INVESTMENT MERRILL_LYNCH COMPANIES SECURITIES WEEK DOW_JONES POINTS 10_YR_TREASURY_YIELD PERCENT CLOSE NASDAQ_COMPOSITE STANDARD_POOR CHANGE FRIDAY BANKRUPTCY CREDITORS BANKRUPTCY_PROTECTION ASSETS COMPANY FILED BANKRUPTCY_FILING ENRON BANKRUPTCY_COURT KMART

6 LDA: Documents are Represented as Combinations of Topics 6 Figures from Mark Steyvers Terrorism Wall Street Firms Stock Market Bankruptcy SEPT_11 WAR SECURITY IRAQ TERRORISM NATION KILLED AFGHANISTAN ATTACKS OSAMA_BIN_LADEN WALL_STREET ANALYSTS INVESTORS FIRM GOLDMAN_SACHS FIRMS INVESTMENT MERRILL_LYNCH COMPANIES SECURITIES WEEK DOW_JONES POINTS 10_YR_TREASURY_YIELD PERCENT CLOSE NASDAQ_COMPOSITE STANDARD_POOR CHANGE FRIDAY BANKRUPTCY CREDITORS BANKRUPTCY_PROTECTION ASSETS COMPANY FILED BANKRUPTCY_FILING ENRON BANKRUPTCY_COURT KMART 70% 30% Document 1 50% 50% 90% Document 2 Document 3

7 LDA (Latent Dirichlet Allocation) 7 W T W T D D

8 LDA (Latent Dirichlet Allocation) 8 Topic Models as Factor Analysis for Count Data W T W T D D T topic weights for each document topic-word probability distribucon

9 NIH Data Representation for L-LDA 9 Words or Terms brain lung_cancer women obesity children mice experiment hbv quality glioma researcher doc doc2 1 1 doc3 2 1 Documents doc doc5 1 1 doc6 2 doc7 1 1 doc8 2 1 doc doc10 2

10 NIH Data Representation for L-LDA 10 Words or Terms Codes or Labels brain lung_cancer women obesity children mice experiment hbv quality glioma researcher brain cancer breast cancer kidney disease lung cancer mind and body doc doc doc Documents doc doc5 1 1 doc6 2 doc doc8 2 1 doc doc

11 NIH Data Representation for L-LDA 11 Words or Terms Codes or Labels brain lung_cancer women obesity children mice experiment hbv quality glioma researcher Documents brain cancer breast cancer kidney disease lung cancer mind and body Background 1 Background 2 doc doc2 1 1 doc3 2 1 doc doc5 1 1 doc6 2 doc7 1 1 doc8 2 1 doc doc

12 Examples of Topics from NCI Abstracts (5 out of 98) 12 Brain Cancer glioma brain tumor gbm malignant glioma glioblastoma brain Breast Cancer breast cancer women breast cancer cell breast breast cancer pacent brca1 Kidney Disease rcc kidney cancer renal cell carcinoma vhl renal cancer pvhl Background 1 program trainee university training candidate field Background 7 model mice work experiment human mouse model 88 Topics from RCDC labels 10 Background topics

13 Evaluation 13 Grants with RCDC labels (31,628 documents) terms TRAIN 90 % 28K docs TEST 10 % 3K docs p(code doc) AUC R-Precision p(code doc) i p(code word i,doc) Sampling probabili>es were averaged over the words in a document to calculate AUC and R-precision scores

14 Logistic Regression Classifier 14 p(code = 1 doc) p(code = 2 doc) 88 logiscc regression classifiers trained to produce calibrated probabilices using training data p(code = k doc) p(code = 87 doc) p(code = 88 doc) LogisCc Regression Classifier p LR (code = k d) L-LDA TOPIC PROBABILITY CALIBRATED TOPIC PROBABILITY

15 Evaluation Result 15 p(c d) p LR (c d) L-LDA L-LDA + LogisCc Regression AUC R-Precision

16 Analyzing Funding Patterns over Time 16 Frac>onally assign the funds in direct propor>on to the probabili>es from the logis>c regression classifiers p LR (code doc) w cd = k p LR (c d) p k (c = k d) c = 1,2,...88 F c y = d:y d =y w cd x d w cd x d y d y F c : weight for the category c for document d : amount of funding for document d (considered infla>on) : year when document d was funded : total es>mated amount of funding for category c in year y

17 Nanotechnology Networking-and-Information-Technology-RandD Human-Genome Obesity FUNDING PERCENTAGE YEAR Es>mated percentage of funding allocated to 4 general RCDC categories

18 Lung-Cancer Liver-Cancer Brain-Cancer Infectious-Diseases 1.4 FUNDING PERCENTAGE YEAR Es>mated percentage of funding allocated to 4 specific RCDC disease categories

19 19 7 Breast-Cancer Translational-Research Epidemiology-And-Longitudinal-Studies 6 FUNDING PERCENTAGE YEAR

20 Genetics Biotechnology Breast-Cancer Obesity Sleep-Research FUNDING PERCENTAGE YEAR

21 Conclusions 21 Summary Labeled topic modeling and logis>c classifiers can be combined to analyze NIH grant funding data Sta>s>cal topic modeling allows linking of text with metadata in a quan>fiable manner Future Work Jointly analyze grants and scien>fic ar>cles related to the grants (ongoing) Broader analysis of the economic and policy implica>ons Improvements on topic model How to best calibrate Selec>ng the right Hyper parameters Methods such as using seed words

22 22 THANK YOU

23 23 BACKUP SLIDES

24 NCI (National Cancer Institute) Data 24 PROJECT ID GRANT ABSTRACT NCI Data... RCDC Labels FUNDING YEAR # number of tokens Number of tokens in a document number of labels per document Number of labels in a document

25 Examples of Topics from NCI Abstracts (5 out of 88) 25 Brain Cancer glioma brain tumor gbm malignant glioma glioblastoma brain Breast Cancer breast cancer women breast cancer cell breast breast cancer pacent brca1 Kidney Disease rcc kidney cancer renal cell carcinoma vhl renal cancer pvhl HepaCCs hcv hbv liver cancer hepaccs virus hbv infeccon hbv replicacon Lung Cancer gliomalung cancer nsclc lung leading cause cancer death egfr

26 Labeled-LDA for NIC Grants Topics (RCDC Codes) 10 Background Topics Hyper parameters Dirichlet prior for word-topic distribu>on = 0.01 β w Dirichlet prior for doc-topic distribu>on Used propor>onal alphas 88 α c = 5 α b = 1 c=1 B b

27 27

28 Analyzing Funding Patterns over Time 28 Frac>onally assign the funds in direct propor>on to the probability p l (c d) w cd = k p k (c = k d) c = 1,2,...88 F c y = d:y d =y w cd x d

29 NCI (National Cancer Institute) Data ,901 grants in total for FY1994 ~ FY2013 Number of grants with labels : 31,628 (2008~2011) Number of grants without labels : 118,273 W Size of vocabulary (W) : 29, D