Vaccination and Markov Mean Field Games

Transcription

1 Vaccination and Markov Mean Field Games Gabriel Turinici, in collaboration with Laetitia Laguzet and Francesco Salvarani CEREMADE, Université Paris Dauphine Institut Universitaire de France The 11th AIMS Conference, Orlando, FL, USA Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

2 Disclaimer: What follows is a theoretical epidemiological investigation. It is not meant to be used directly for health-related decisions; if in need to take such a decision please seek professional medical advice. Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

3 Outline 1 Motivations 2 Mathematical model 3 Taking into account the individual decisions Previous works Individual dynamics Definition of individual strategy Individual-societal equilibrium Taking into account a discount 4 Imperfect vaccination: efficacy, delays The model Optimal strategy and Nash equilibria Numerical results Effects of failed vaccinations 5 Conclusion Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

4 Vaccine scares: Influenza A (H1N1) (flu) ( ) At 15/06/2010 flu (H1N1): deaths in 213 countries (WHO) France: 1334 severe forms (out of 7.7M-14.7M people infected) Vaccination in France Adjuvant suspected of some neurological undesired effects; mass vaccination uncertainty (few previous studies for this size) Very costly campaign (500M EUR), Low efficiency (8% to 10% in France with respect to e.g., 24% US or 74% Canada). Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

5 Vaccine scares: Example : Influenza A in Vaccination Coverage expected and realized in different countries on percentage of population: (See schedule 3 of the French parliamentary report number 2698) Countries Target coverage Effective rate of vaccination Germany 100 % 10% Belgium 100 % 6 % Spain 40 % < 4% France % 8.5 % Italy 40 % 1.4 % Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

6 Vaccine scares Previous vaccine scares (some have been disproved since): France: hepatitis B vaccines cause multiple sclerosis US: mercury additives are responsible for the rise in autism UK: the whooping cough (1970s), the measles-mumps-rubella (MMR) (1990s). Vaccine Scares : as cases of a disease decrease, people become complacent about their risk, and the threat of vaccines (imagined or real) seems greater than the threat of disease (C. Bauch) Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

7 Further motivation: end of compulsory vaccination context in France: discussions on the end of general compulsory vaccination How is vaccination coverage evolving? Example: nobody vaccinates then an additional individual may vaccinate; if all vaccinate an additional invididual will not vaccinate. Will the vaccination coverage become unstable or chaotic? question: what are the determinants of individual vaccination hint: individual decisions sum up to give a global response; need a model. Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

8 Outline 1 Motivations 2 Mathematical model 3 Taking into account the individual decisions Previous works Individual dynamics Definition of individual strategy Individual-societal equilibrium Taking into account a discount 4 Imperfect vaccination: efficacy, delays The model Optimal strategy and Nash equilibria Numerical results Effects of failed vaccinations 5 Conclusion Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

9 Influenza incidence historical data in France Sourse: Sentinelles network, INSERM/UPMC, Influenza A (H1N1) (flu) ( ) : see next. Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

10 Influenza A (H1N1) (flu) ( ), France Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

11 The SIR-V model Global model βsidt γidt Susceptible Infected Recovered dv Vaccinated Figure: Graphical illustration of the SIR-V model. In this model all individuals are identical. β : probability of contamination, γ : recovery rates, dv (t) : measure of vaccination (!!) I ds = βsidt dv (t) di(t) dt = βsi γi dr(t) dt = γi(t) S Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

12 Vaccination cost functional Cost for an infected person : r I. Cost of the vaccine (including side-effects) : r V. Global cost for the society (from the initial state X 0 = (S(0), I(0)) T ) : J(X 0, V ) = r I βsidt + r V dv (t) (1) 0 BUT this is not an individual perspective Abakuks, Andris, 1974 Optimal immunisation policies for epidemics,..., LL & GT, Math. Biosci., 263: , Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

13 Outline 1 Motivations 2 Mathematical model 3 Taking into account the individual decisions Previous works Individual dynamics Definition of individual strategy Individual-societal equilibrium Taking into account a discount 4 Imperfect vaccination: efficacy, delays The model Optimal strategy and Nash equilibria Numerical results Effects of failed vaccinations 5 Conclusion Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

14 the vaccination is at birth only (probability p) ; at equilibrium vaccination occurs when the probability to be infected r V /r I. no time dependence; final state stationary. equilibrium ok if everybody does the same no eradication possible through voluntary vaccination, endemic state (coherent with literature) vaccine scare behavior : let other vaccinate (when perceived risk differs from that of majority) Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43 Taking into account the individual decisions: previous literature Bauch & Earn (PNAS 2004) consider the SIR-V model with births and deaths : ds = µ(1 p) βsidt µs di = (βsi γi)dt µi dr = γidt µr dv = µp β : probability of contamination, γ : recovery rates, dv (t) : measure of vaccination µ : the birth / death rate p the probability to vaccinate at birth

15 Taking into account the individual decisions: previous literature Francis (2004) : uses a SIR-V model to find an equilibrium. Results: They identify the vaccination region although the problem is not formulated as an optimization at the individual level. Galvani, Reluga & Chapman (PNAS 2006) consider a double SIR-V periodic model of flu with two age groups (break at 65yrs). Vaccination is separated from dynamics, once at the beginning of each season. Results : show that actual vaccine coverage is consistent with individual optimum; explain impact of age-targetted campaigns (children). Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

16 Taking into account the individual decisions: previous literature Bauch (2005) : time dependent vaccination rate p(t); probability to be infected is approximated by a rule of thumb (proportional to I(t)); the corresponding dynamics is a phenomenological proposal (m a parameter) : dp(t) = kp(1 p)(mr I I(t) r V ) dt B. Buonomo : vaccination as a feedback F. Fu : taking into account the topology (networks of acquaintances) Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

17 Individual dynamics βsidt γidt Susceptible Infected Recovered dv Vaccinated Global dynamics : continuous time deterministic ODE; is the master equation of the individual dynamics. rate βi rate γ Susceptible Infected Recovered... Vaccinated Individual dynamics: continuous time Markov jumps between Susceptible, Infected, Recovered and Vaccinated classes. Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

18 Taking into account the individual decisions : individual strategy the simplest one, the pure strategy : vaccinate with certainty at some given instant τ [0, ] (τ = means no vaccination) In terms of probability to vaccinate a pure strategy is H( τ) with H( ) the Heaviside function. Here τ = 2.5. is it always optimal? NO does it give a stable equilibrium? NO Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

19 Taking into account the individual decisions : the cost of pure strategies Costs of pure strategies Π t vaccinating at t: J pure (Π 0 ) = r V, J pure (Π t ) = r I ϕ I (t) + r V (1 ϕ I (t)), J pure (Π ) = r I ϕ I ( ). more realistic mixed strategies (related game theory concepts in timing games cf. Drew & Tirole): the individual chooses a probability measure dϕ V over the space of all pure strategies The CDF function ϕ V (t) is increasing, right con tinuous with left limits (càdlàg) with ϕ V (0 ) = 0, ϕ V ( ) 1. In particular ϕ V (t) = probability to choose vaccination in the interval [0, t]. Overall cost of strategy ϕ V : J indi (ϕ V ; V ) = (1 ϕ V ( ))J pure (Π ) + 0 J pure (Π t )dϕ V (t) J indi (ϕ V ; V ) = r I ϕ I ( ) + ] 0 [r V r I ϕ I ( ) + (r I r V )ϕ I (t) dϕ V (t). Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

20 Discounted individual strategy discount factor D > 0 : use Φ I (t) = t 0 e Dτ dϕ I (τ), Φ I (0 ) = 0. J D pure(π t ) = r I Φ I (t) + e Dt r V (1 ϕ I (t)). J D pure(π ) = r I Φ I ( ). Cost of strategy ϕ V : Jindi D (ϕ V ) = r I Φ I ( )+ ] 0 [r I (Φ I (t) Φ I ( ))+r V e Dt (1 ϕ I (t)) dϕ V (t). Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

21 Taking into account the individual decisions : individual strategy Constraints : if global vaccination is at maximal rate u max : V (t + t) V (t) u max t. umax t Probability of being vaccinated in [t, t + t]: S(t). A pure strategy is transformed into a queue (exponential). Individual constraint dϕ V (t) umax S(t) (1 ϕ V (t))dt. If everybody has the same ϕ V then dv (t) = dϕ V (t) 1 ϕ V (t) S(t) (when this operation makes sense). Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

22 related works: C. Gueant (graphs), O. Cardaliaguet (HJB), D. Gomes : hyp. of superlinear costs (here linear); pure strategy set is not Gabriel Turinici compact. (CEREMADE & IUF) Vaccination and Markov MFG July / 43 Individual cost functional { ds = βsidt dv (t) di = (βsi γi)dt. vaccination at the society level with dv (t); it can have the form dv (t) = u G (t)dt. Society dynamics induces a cumulative probability of infection on [0, t] : ϕ I (t) solution of dϕ I (t) = βi(t)(1 ϕ I (t))dt, ϕ I (0) = 0. Individual decision: ϕ V (with constraints cf. above). Individual cost functional J indi (ϕ V ; V ) = r I ϕ I ( ) + ] 0 [r V r I ϕ I ( ) + (r I r V )ϕ I (t) dϕ V (t). Individual - global equilibrium condition (global strategy arise from individual decisions): dv (t) = S(t) 1 ϕ V (t) dϕ V (t) (when this operation makes sense) (Mean Field Games, cf. Lasry - Lions, Caines - Huang - Malhamé).

23 Theoretical results for discount=0 Theoretical results (GT, LL 2014): fixed global (societal) policy Optimal individual solution : vaccination until the overall infection risk βs(τ)i(τ)dτ t S(t) (depending on dv ) falls below r V ri. Theoretical results (GT, LL 2014): equilibrium a stable (mixed, Nash) equilibrium exists the domain is divided into a vaccination region and a non-vaccination region. the frontier of the vaccination region is the line: infection risk = r V ri Formula ζ(s,i) S X. = r V ri ; ζ(x) = size of a non-controlled epidemic from Comparison between Π 0 and Π is enough (cf. Francis)! Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

24 Theoretical results for discount=0 I 1 Red region: vaccination in the societal model and in the individual model, green region : vaccination for the societal model but not for the individual, blue region: no vaccination in both models. O 1 S Mean cost for an individual in the stable individual-global equilibrium is larger than the cost for the, non equilibrium, societal (non-individual) optimum state this is because in the green region, it is optimal for individuals to let other vaccinate. Conclusion: the stable strategy will be obtained even if it is more costly for everyone; the cost of anarchy in the model is non-null. Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

25 Equilibrium for D > 0, u max = Theoretical results (GT, LL 2015) I Ω n Ω i (S, I ) Ω d S White region: no vaccination, Π gray region : delayed vaccination (new behavior, Π t ) dotted region: immediate vaccination, Π 0 Comparing Π 0 and Π (Francis) is not enough, more complicated dynamics than D = 0. More complicated regions for u max <. Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

26 Application to Influenza A 2009/2010 vaccination The SIR model was fit to the observed vaccination coverage (J.P. Guthman et al. BEH 2010); the other parameters were chosen consistent with ranges from the literature (large CIs!). Time axis = weeks starting from W19Y2009; peak = W49Y2009 (30), vaccination peak = W51Y2009 (31-32), vaccination end W05Y2010 (40) Comparison of numbers of cases Real Model Comparison of cumulative numbers of cases Real Model Comparison of cumulative numbers of vaccination Real Modeled Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

27 Application to Influenza A 2009/2010 vaccination Infection risk 30 Vaccination intensity 30 Model based, optimal risk criterion Simpler criterion, model based Model free, data-driven criterion Figure: Time axis = weeks starting from W19Y2009; peak = W49Y2009 (30), vaccination peak = W51Y2009 (31-32), vaccination end W05Y2010 (40) The results indicate that the risk perception was inhomogeneous; first group to stop vaccination perceived relative risk r V /r I 5% 10%!! (week 32). The last group corresponds to r V /r I < 1% (week 40; model cannot be more precise with available data). Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

28 Outline 1 Motivations 2 Mathematical model 3 Taking into account the individual decisions Previous works Individual dynamics Definition of individual strategy Individual-societal equilibrium Taking into account a discount 4 Imperfect vaccination: efficacy, delays The model Optimal strategy and Nash equilibria Numerical results Effects of failed vaccinations 5 Conclusion Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

29 The individual model in DISCRETE TIME Failed Vaccination (F ) f λn (1 β n T In) β n T In Susceptible (S) β n T In Infected (I 0 ) 1 γ 0 T Infected (I 1 ) 1 γ 1 T... γ 0 T γ 1 T Recovered (1 f) λn (1 β n T In) α0β n T In α1β n T In αθ 1β n T In αθβ n T In (R) Vaccinated (not infected) (V 0 ) Partially Immunized (V 1 )... Partially Immunized (V Θ 1 ) Lost Immunity (V Θ ) Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

30 Individual dynamics: Dynamics: DISCRETE TIME controlled Markov chain P(M n+1 = S M n = S) = (1 λ n ) (1 β T n I n ) P(M n+1 = I 0 M n = S) = β T n I n P(M n+1 = V 0 M n = S) = (1 f ) λ n (1 β T n I n ) P(M n+1 = F M n = S) = f λ n (1 β T n I n ) P(M n+1 = R M n = I Ω ) = 1 P(M n+1 = R M n = I ω ) = γ T ω, ω = 0,..., Ω 1 P(M n+1 = I ω+1 M n = I ω ) = 1 γ T ω, ω = 0,..., Ω 1 P(M n+1 = I 0 M n = V θ ) = α θ β T n I n, θ = 0,..., Θ 1 P(M n+1 = I 0 M n = V θ ) = α θ β T n I n, θ = 0,..., Θ 1 P(M n+1 = I 0 M n = F ) = β T n I n. Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

31 Individual cost The cost for an individual has three components: the cost r I of being infected before vaccination the cost r V of vaccination plus a possible cost of being infected while immunity is still building or after the persistence period the cost r V of failed vaccination plus a possible cost of being infected For an individual starting at M 0 = S, the total cost is ( ) J indi (ξ; V ) = r V P n<n {M n+1 = V 0, M n V 0 } M 0 = S ( ) +r V P n<n {M n+1 = F, M n F } M 0 = S ( ) +r I P n<n {M n+1 = I 0, M n I 0 } M 0 = S Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

32 Individual strategy is a probability density on {t 0,..., t N 1 } { }: ξ Σ N+1 = {(x 0,..., x N ) R N+1 x k 0, x x N = 1}. ξ n : probability that the individual vaccinates at time t n (if it was not infected before t n ); before the beginning of the epidemic he knows the time t n at which he will vaccinate (unless he is already infected by that time). Conditional rates λ n are derived from ξ. Mapping between λ = (λ n ) N 1 n=0 and ξ: ξ := N 1 n 1 (1 λ n ), ξ n := λ n (1 λ k ), n N 1 n=0 ξ n k=0, if ξ n + + ξ > 0 ξ n + + ξ n N 1 : λ n = 0, otherwise Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

33 The individual minimal cost The individual cannot change the overall societal vaccination, he can only choose his own vaccination strategy ξ. J indi (ξ; V ) = ξ, g V, g V, ξ R N+1. J indi (ξ; V ) to be minimized under the constraint ξ Σ N+1. Compatibility relationship between ξ and V when all individuals follow this vaccination policy: V n (the societal vaccination) = λ n S n Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

34 Existence of equilibria Theorem The vaccination dynamics model admits a Nash equilibrium. Idea: existence: fixed point (Kakutani); uniqueness?? C ξ := g V with strategy ξ E(ξ): maximum gain obtained by an individual if he changes unilaterally its strategy (and everybody else remaining with the strategy ξ): E(ξ) = ξ, C ξ min η, C ξ 0 η Σ N+1 Σ N+1 : space of all possible strategies Equilibrium: it corresponds to a ξ such that E(ξ) = 0 Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

35 The algorithm How to compute the equilibrium? Step 1: Choose a step τ > 0 and a starting distribution ξ 1. Set iteration count k = 1. Step 2: Compute dist(η, ξ k ) 2 ξ k+1 argmin η ΣN+1 + η, C ξk. (2) 2τ Step 3: If E(ξ k+1 ) is smaller than a given tolerance then stop and exit, otherwise set k k + 1 and go back to Step 1. Remark: -flow (depends on which distance is chosen). Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

36 Short persistence, perfect efficacy Total simulation time T = 1 (one year), number of time instants: N = (three times a day) Recovery rate γ ω = γ = 365/3.2 (mean recovery time 3.2 days, Ω = 20) Reproduction number R 0 = 1.35, β = γr 0 ; Initial proportion of susceptibles S 0 = 0.94 and infected I 0 = ; Relative costs r I = 1, r V = β min = γ/s 0 and β(t) = β for t t β 2 := 1/2 (6 months) and then β(t) = β min for t > t β 2 = 1/2 Persistence of the vaccine: t 1 = 5/365, t 2 = 1/12 (one month, Θ = 93) and α θ = 1 1 [t1,t 2 ]. Vaccine efficacy: 100% (i.e., failure rate f = 0). Step τ = 0.1, 1000 iterations Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

37 9 Short persistence, perfect efficacy # Solution strategy ξ MFG supported at several time instants between 0.25 and 0.43, with 68% of nonvaccinating individuals time C Cost C ξ MFG of the optimal converged strategy ξ MFG, red line: cost of the non-vaccinating pure strategy (C ξ MFG ) N time Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

38 S Itotal S # Itotal time time Evolution of the susceptible class S n (left) and of the total infected class I n (right) Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

39 E(9) flowtime Decrease of the incentive to change strategy E(ξ k ) Cost of the solution ξ MFG, C ξ MFG : M(ξ MFG ) = Cost of anarchy > 0 for instance for Π 0 (vaccination with certainty at time t = 0.0 unless infected by that time) has cost M(Π 0 ) = Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

40 9 Long persistence, perfect efficacy # Optimal converged strategy ξ MFG. The weight of the non-vaccinating pure strategy (i.e., corresponding to time t = ) is 91% time C9 # time Cost C ξ MFG of the optimal converged strategy ξ MFG. The thin horizontal line corresponds to the cost of the non-vaccinating pure strategy (C ξ MFG ) N+1. Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

41 Effects of the failed vaccination rate on the strategy Individual vaccination policy with respect to the failed vaccination rate of the vaccine. 8 f 1 ξ failure rate (f ) r V = (the other parameters are unchanged) If f = 0.85 the probability of being infected is 14.38%. % of vaccination (1 ξ ) Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

42 Outline 1 Motivations 2 Mathematical model 3 Taking into account the individual decisions Previous works Individual dynamics Definition of individual strategy Individual-societal equilibrium Taking into account a discount 4 Imperfect vaccination: efficacy, delays The model Optimal strategy and Nash equilibria Numerical results Effects of failed vaccinations 5 Conclusion Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43

43 Final remarks Bibliography L. Laguzet & G. Turinici, Bull. Math. Biol., October 2015, Volume 77, Issue 10, pp F. Salvarani, G. Turinici, Individual vaccination equilibrium for imperfect vaccine efficacy and limited persistence, submitted (2016) Extensions a model with limited information may be more realistic structure by age groups vital dynamics (birth/death rate into the model can make it applicable to childhood diseases): work in progress with E. Hubert geographical heterogeneity additional stochastic dynamics Gabriel Turinici (CEREMADE & IUF) Vaccination and Markov MFG July / 43