Professor.

Old they're.

Really distinguished from people
who profess to have their for

the Wisconsin medicine.

Holby partners you know no harm by
a statistic and medical matters but

if you also have to do in a book or
in America for your scientists.

Believe this.

Role model for me and probably for

many of you if you listen to
the next paragraph Lysis.

Because you're pretty much or all of
these directory is getting harder and

harder to do because the directory.

He did there the senior scientists.

From your use the title to this topic.

I couldn't even I could not I got you and

unification any in
the can stream last year.

And all we are ready but
part of your response was to rework it

if you knew that it would do says that
first order logic after his Ph D.

and he started to work on.

My own mind about the Haitian maturing
starting at the Oxford University

very work with.

Several of.

These And then later on he moved back to.

Work.

In medicine for many years.

I could be called a.

Cancer Yes he has to be fanatics and
also a member of the fuel

standard if Wisconsin to them
the hard work of art for magic's.

Made me you there also took off expand
his research think you politically for

America I'm all in your article
health records to doodle pocket for.

Thank you.

Thank you thank you for that gracious
introduction Jimmy it's a real pleasure

to be here let me say right at the outset
stop me at any point if you have questions

and in case you ask a lot of questions and
we don't get to the end of the talk I'll

go ahead and do my acknowledgments now so
I'd like to thank my students and

collaborators and and just tell you what
a great pleasure it is to be here today

number of the things that I'll talk
about today people here at Georgia Tech

especially Jim ings group have been
doing better than we're doing but

I'm going to tell you about what we're
doing anyway and I look forward to getting

maybe some some collaboration's going
the first few slides I have will be.

Kind of.

Motherhood and apple pie sorts
of things to this audience but

I'm going to go ahead and and
do it anyway briefly so so

now that most of us have our data and
electronic health records we have basic

demographics data and every time we
go to the clinic we we maybe get

some die a diagnosis code or several
codes entered I'm not showing the I.C.D.

nine or ten code here but we'll get
a code entered maybe some signs and

symptoms next time we get some
other information entered in

addition to diagnoses there may be
labs vitals prescriptions procedures.

In some E.H.R.'s even genetics and
what we'd all really like to do

as patients we'd like a prediction of
when will be and again and what's going

to be going on with us and so if we had
had each E.H.R.'s twenty years ago and

I was in my first faculty job you might
have seen this for me I was in my

first semester and and working really
hard getting a research group go in and

at the time teaching two two classes
I had never taught before I think.

As teaching compilers and file structures
and woke up in the middle of the night

with my heart feeling like it was doing
a bongo beat and went into the E.R.

and I was in relation which is the most
common form of cardiac arrhythmia but

it's not real common when you're thirty
years old and and so it would have been

really nice to know that a week ahead of
time because I could have cut out caffeine

and taken an aspirin every day maybe
taken a couple days vacation and avoided

altogether so those are the predictions
we'd like to make from this kind of data

I've oversimplified a little because
of space I'm not showing procedures or

genetics but the E.H.R.
is optimized for fast transactions and

updating and reporting but
when we do research with it it's usually

downloaded into a relational database
where we call it the data warehouse a very

different view of this data
though is that one patient is

is essentially a time series
that's irregularly sampled so

we're talking about longitudinal data
instead of three variables there are on

the order of fifty thousand variables that
might that might be measured over time and

this is a slide from ten years ago but
it's really a popular topic now right now

with the precision medicine initiative
that what we'd like to do if we if we go

over here to this or this column here we'd
like to take this patient who has genetic

data clinical data and environmental data
and we like to predict this patient's

risk for things like atrial fibrillation
in the near future or if the patient

has a diagnosis we'd like to predict
the treatment response which of drugs a B.

and C.
will be most efficacious to this patient

which one of these drugs might cause
an adverse event so we don't give the drug

to that patient so we can make better
treatment decisions and of course I think

for us in this room we're all interested
in building these models by machine

learning from data on many other people
who have already had these conditions and

already had these treatments and we've had
the opportunity to see how they respond.

So I want to initially
talk about a few one off

applications that we did in our group and
then and then hopefully get to some

some broader applications and
to some some more.

Cutting edge kind of algorithm development
but the first thing I want to talk about

briefly I go back to two thousand and
nine and I was in my office minding

my own business when Michael Caldwell the
Marshfield Clinic called me up and said.

You know Michael's been interested in war
for in dosing in pharmacogenetics for

a long time and warfarin was discovered at
the University of Wisconsin years ago in

fact I'll take just a minute to tell you
the story I want tell the long version of

tell the short version but it was
originally marketed as a rat poison and

so the natural next thought
is let's give it to people

at obviously a different dose and
it turns out dosing for war for in is very

challenging because genetic factors other
factors affect the dose quite a bit so

anyway Michael calls me up in my office
and says twenty one people have twenty

one different sites have donated data and
they all have their own.

Analysis folks machine learning folks
who want to build predictive models for

the stable dose of Warford so they all
have patients whose doses have been

juggled up and down until they've reached
a target eye in our desired degree to

which the blood is being thinned so that
you can avoid stroke or heart attack but

not give an overdose where you might cause
internal bleeding a hemorrhage and so

Michael says we'd like to have somebody
essentially oversee the data analysis so

every group can try their can try their
methods and so that's how I got involved

in this were friends also called Coumadin
already the F.D.A. had on the label for

war fron that genetics can have
a big impact on war from DOS but

it left clinicians wondering well
exactly how do I use that information.

Individual sites had built dosing models
but those dosing models varied from one

another quite dramatically in part because
these sites were in different locations

around the world and had very different
populations and the just the genes and

the variants that have an impact on more
from DOS actually are very associated

with with race and that was part of
the reason that you got a big difference.

Dose is affected quite a lot by
age height weight it's affected by

genetic variations for the course Iwan
which is involved in the vitamin K.

pathway that war for an X.

on is also.

Heavily influenced by sip to see nine
which metabolize is the drug and

so we set up a a.

Division of the data five thousand
patients roughly into a validation cohort

that none of us would see I didn't even
see it until we had a single model to take

forward and then we had a set for
the for the training cohort

we had a set of ten folds for cross
validation that everyone would use so

that we could choose a winning approach
to take to the validation lots of methods

were used and I'm telling you about this
because one of the bits behind the scenes

that I can tell you about that didn't make
it into the final paper was that one point

it looked like support back to regression
the first method up here was going to be

the winning technique and the clinicians
got very bothered by this and

so I said well it might not be nonlinear
support vector regression the linear

S.V.R. is working pretty well and so
that'll look a lot like a logistic were or

a linear regression model be a vector
of coefficients on the variables but

still they didn't
understand the method and

at the end of the day it turned out
that ordinary least squares regression

was was performing roughly
equally well with a.

Report back to regression the clinicians
were much happier about that

this was the final model we won't go
through in detail but you can see some of

the variables that I was telling you about
here earlier having an effect once you

include these Gina types the effect
of races is much smaller Here's

the version that doesn't include
the genetics much simpler model and

here's a comparison of the models
five milligrams a day is what for

years people were started on in
the dose was was was changed

daily Typically all the warfront has its
peak effect four days after you take it so

this is a dangerous time period for the
patients you can potentially reduce risk

if you get a more accurate starting dose
this was the clinical data only model

if you're on five milligrams a day on
average than that's a two milligrams a day

a day mean error mean absolute error

this is down to one point two if
you use the pharmaco genetic model

This is main absolute error in the daily
dose so we're predicting we're

training on patients whose dose had been
altered until they reached the stable dose

they were in the target I N R And
we're trying to predict that dose for

the patient this is rather than mean
squared error it's mean absolute absolute

value here so I'm sorry this is.

So the patients were starting
on five milligrams a day or

thirty five milligrams a week and so
so this patient had an error of two so

some of these patients needed
seven milligrams a day and

some needed three milligrams
a day yeah and so

you can imagine that getting this cutting
that they are not quite in half but

close to in half could potentially
have a big impact on the patients and

time permitting I can say more about
the ongoing story after that but but

this was essentially
the finding that we had Yeah.

Yeah the clinicians at the outset wanted
to use that they felt it would be

more intuitive and and they felt that.

There was an argument for root mean
squared error that you want to you want to

square error because big errors are going
to have a more dramatic effect in the end

the New England Journal of Medicine pushed
us not to report either well to report

this but to focus on yet another measure
which is what fraction of patients were

within twenty percent of the final correct
dose and they felt that was a better

measure yet so I'm not showing that here
but that's and in the paper as well and

again the same ordering of the models
is reflected in that measure.

So another model we looked at models
also for predicting cardio infarction or

heart attack we looked at models for
predicting heart attack among patients

who are on Vioxx back when that was
on the market we looked at one that

was a personal interest of mine which
is predicting atrial fibrillation or

flooder A.F.F. this is the R O C curve for

that one in the in the darker blue here
a little darker than the diagonal line

area under That's point seven two turns
out you can predict among those who

stay in a true fibrillation one or three
year mortality even more accurately but

stroke among those even less significantly
less accurately which was interesting to

us in general you can predict
stroke risk better but

not in this atrial fibrillation cohort in
particular now we'd love a higher a U.C.

But even with this curve you can see about
here there's a point where you could pick

up on roughly a third of
the atrial fibrillation cases with

a false positive rate less than a tenth
now because atrial fibrillation is not all

that common thankfully you'd still your
perception would not be that great but

if your intervention is to put people
on an aspirin and cut out caffeine

that's a perfectly reasonable intervention
to do at this kind of level and

avoid some some cases I want to
avoid all of them but it would.

Void some of them so that's just
an example of some of the kinds of

predictions we could do either
continuous output or binary output but

what we'd really like
to do is to do this for

lots of other events that's the interest
in persuasion medicine to use genetics

plus clinical data to do this they result
I showed you was just using clinical

history in cancer you actually get genetic
changes with the disease if you're Gene

a type in the tumor so you may get even
more accurate predictions of treatment but

we'd love to treat to predict.

Events and treatment outcomes for
many things and

that's kind of the goal or
the the vision that we have.

So before I move on talking
more about this vision

does anybody else have a question so far.

OK I mean.

That's a that's a great question
it's certain certainly getting

you in a range because you're measuring
the degree the blood's the being to which

the the blood being thin with this eye in
are but potentially there is some room for

some wiggle room in that the dose that
gets you into that range I don't know that

there's a bias in that in that process
that's interesting Certainly year you're

within a range so that's having some
effect I don't think it's having a big

effect things so
the next result is go that

I'm going to talk about is trying to make
a start on this vision and I should say G.

Ming has already made a great start
in this with the doctor AI paper and

that approach and we're taking a different
approach but aimed at the same the same

goal so we'd like to predict every I.C.D.
coded diagnosis or at least those for

which we have sufficient patients in our
in our training data that's the goal we're

going to be using data from the Marshfield
Clinic and I had the good fortune to start

collaborating with them about fifteen
years ago if we had time I'd tell you that

interesting stories involved in dealing
with privacy and security of data and

building trust and getting data
access if time permits at the end

I can talk some about that but Marshfield
is up in the center of Wisconsin and

provides health care for much of the upper
half of the state of Wisconsin as a number

of clinics all around the state especially
the northern part of the state.

They have data going back they've had an
electronic health record at least for for

diagnoses going to the late sixty's and
and for a full E.H.R.

going back to the late eighty's and and

they have data on one point five million
patients we only of course get access to

be identified data even the dates have
been shifted just a little bit but.

And turned in for
us into patient age at events but

we can still order the dates correctly
within a for a given patient.

About half a million of these patients
just have a few entries in their record

maybe they were up from Chicago one
weekend in the north of Wisconsin either

waterskiing or snowmobiling depending on
the time of year they were up there and

they had an injury and
went into the clinic so

we limited our attention to
the million patients who had at least

four clinic encounters and we have
the sort of information that I showed you

earlier available on these patients this
is telling you what I just told you.

OK so for diagnoses in two encounters I'm
sorry we were a little more generous and

what we kept.

One challenge when I first started working
with this data fifteen years ago is not

unique to Marshfield C.H.R. data in
general I thought I want to know who had

a heart attack or cardio infarction
there's a code for that code for ten and

I'll look for a for ten I don't know who
had a heart attack but it turns out that

these codes are widely used for billing
purposes so they might put a code for

ten to bill for a trip on and test to see
if a person really had a heart attack and

then put the code in again
to verify the heart attack.

They might put a later entry of that
code in to say that today's visit is

because of the prior heart attack or
that prior heart attack is relevant and so

just phenotype in electronic phenotype you
are saying who really had the condition

is stuff we've used machine learning for
that task I won't go into that today but

for now we're using a simpler approach
which is rule of two they had at least

two occurrences of the code will say they
really had the event and we used the first

occurrence as the date of that event if
they had zero occurrences will treat them

as a control and for now we'll leave out
those who had one occurrence because for

our purposes there ambiguous We
could also do the machine learning

phenotype being in that include them we
limit ourselves to predicting codes where

they were at least five hundred cases so
that we had sufficient training data

that left us with four thousand of
the total roughly nine thousand I.C.D.

codes we started this work we did this
work with data up through two thousand and

fifteen before the switch to I.C.D.

ten so this is all I see denying codes
from one thousand nine hundred five to

twenty fifteen.

If you predict a code say
a day before the event or

a week before the event you may get
overly optimistic results because

the trick pony and
test could get entered before the M.R.I.

code gets entered or before the second
occurrence of the center and

you're really just diagnosing from the lab
test you're not predicting so we try to

predict at least a month ahead of time and
then you run the risk that if you

truncate the data on cases a month before
the event but don't truncate the data on

controls you could get an overly
optimistic accuracy because controls would

have more data or their data would go to
a later age on average and so we age and

gender matched controls to cases in fact
we sorry we actually birthday matched

roughly to within a few days controls
in cases and then truncated the data

only truncated the date on the case
thirty days before the event we so

here's the diagnosis we
truncate thirty days before we.

Go I'm sorry here are the two diagnoses
here is the first one here is thirty days

before we truncate
the control at the same day

one other complication one where we're
just predicting heart attack let's say or

stroke we can do a lot of work
to try to tailor the phenotype

now we want an automated approach for
every definition so everything we're

trying to do we want to a general purpose
way of doing it we saw that some things

like complicated complications arising
from pregnancy were a little easier to

predict than other things because all of
our cases were in fact pregnant before

the complications arose or
is many of the controls were not and so

we we also installed a general purpose
approach that if a substantial portion

of the cases have have another code
that's already occurred will control for

that as well in our control population.

OK we chose to use random Far East for
our approach for two reasons One

is while we also use Point process models
and other kinds of temporal models and

we also use relational learning methods
that can work directly on the relational

database in practice random forest
models runway faster and work just

about as well and so we use random forest
to build our models on average we're

building seven hundred fifty trees with
an average depth of one hundred per tree.

Then we're evaluating the models
with area under the R O C curve.

So if we predict a month ahead of time
this is a histogram over four thousand

I.C.D. nine codes plotting the A U C S And
you see that you get

an average day you see above point eight
our Atrial Fibrillation model just for

illustration would be here at point seven
two along with maybe one hundred other

models that are at this at this a U.C.
this one is a month ahead of time this one

is six months ahead of time so you lose
some a U.C. predicting further in advance.

Are Dresner the director of our clinical
and translational science award

which we hold jointly between Madison and
Marshfield pushed on us and

said you know Marshfield has forty years
of data why are you just predicting

six months ahead of time so
these two curves that you see here.

Are Now these two just somewhat rescale.

And this is two years ahead of time.

Five years ahead of time ten fifteen and
twenty so you can see predicting twenty

years in advance and I should quickly add
we're just predicting first occurrence so

it turns out later heart attacks
are easier to predict because

prior occurrence is a predictor
of future occurrence but

also because sometimes you just get
the code entered more so it adds

some ambiguity as well as some ease of
predictions we're always predicting first

occurrence of the event this is down near
a half where you would expect although you

can't see very well there's a heavy tell
of a dozen codes about point seven first

occurrence predicted twenty years ahead
of time they're all either I related or

neurological most of them are congenital
they have a genetic component but

still apparently it wasn't known that
they had this condition until they

got older in the symptoms started
started showing themselves but there was

enough in the record to indicate that they
were they either had the condition or

in progression to the condition so we also
said how well would we would this work

if we translated it into the clinic and
instead of waiting a year for

that we just backed our method up and
said well let's train up to twenty twelve

then let's predict on a cohort
of ten thousand randomly drawn

patients who we knew were still in
the system in two thousand and fourteen so

they were going to move away we train
the model up to two thousand and twelve.

Having those patients out we took into
account data on those patients before

twenty two but also up to two thousand and
thirteen and predicted for the next year

what events they would have that left us
in one year obviously with fewer events

for any diagnosis code than we
would have in forty years time so

that means there's going to be more
variance in our predictions but

the predictions were still quite accurate
so now this is predicting over the next

year will they have the event and so
we use the one month six month and one and

we also built a one year model to employ
for this and this is just showing

these two curves are showing the lower and
upper ninety five percent confidence

intervals on these A You sees if I'd shown
these before the confidence intervals

would have been right over the curve
because we had far more patients but here.

Here there's obviously more variance
with fewer patients to test on so

these are some promising results but

where this stands is we'd like to
translate the most accurate models into

the clinic if there's action that could be
taken we're trying to work toward that but

a big challenge is first the clinicians
want to understand these models and

remember I told you there were seven
hundred fifty trees with an average depth

of one hundred for one model so there are
techniques such as the one Brian proposed

in his random far as paper in two thousand
and one you can use to rank the top

features and we do that but a big
challenge is just comprehensibility and

giving the clinicians comfort
with the models another big

challenge though especially related
to clinician comfort with the models

is that prediction is not the same thing
as causation and so for example if you're

predicting a heart attack the fact
that the patients on a beta blocker

could indicate that the patient is already
considered to be at elevated risk and

that gives you information but it's not
causal and it causes the clinician.

To not like the models when you got the
wrong causal direction and so we've been

interested for that reason and several
other reasons I'll give in a minute and

trying to find results that are more
causally faithful models that

are more causally faithful So this I'm
going to take a turn now to another

set of models they'll be linear models
in contrast to random Far East but

we're going to do things to make the cause
and effect relationships between

predictors and the prediction a little
clearer but before I shift to that we're

just past the halfway point let me
see if anybody has questions so far.

Yeah.

Or is there something like step wise or
modeling where you might just use.

Yeah so we are interested in know
where you are interested in particular

since we're doing random forest and tree
boost but potentially other boosted mouthe

The other thing I have a student my
student Kleiman who did a lot of this work

is looking now at if you bump
up the number of trees can you

substantially decrease the depth you need
and he's getting some promising initial

results that even just with that we may
be able to get the tree depth even down

to something as narrow as small as five
because the depth is really telling you

how many feature how many features could
interact in an interaction you're capable

of picking up on but yeah boosted methods
are a good approach for that as well.

Earlier you use.

These.

Parts.

For the twenty year prediction.

I'm just wondering in general
if you look at an organ system

categories the weather
is more easily we do I

don't have it on here I wish I did but my
student also did a violin plot showing for

the one in six months what were
the A you see just within that.

Within each particular code or chapter.

And initially there were
very big differences

some things turned out to be more
predictive than I would have thought like.

Like poison ivy it turns
out to be predictive

based on surrogates for
being outdoors a lot like insect bites and

even Lyme to the then of other kinds
of things but the only one that really

stood out earlier was complicated various
types of pregnancy related things and

that's what helped us identify
there was something fishy going on

otherwise there you know you see
a range like we saw here but

there wasn't one category
that really stood out.

OK So for now we're going to go to this
problem of trying to build more cosily

accurate models and the temporal nature
of the data can give us in some insight

into causation it'll never be perfect but
you know they don't let us experiment on

the patients and decide what to give them
for the purpose of building more accurate

model says about the best that we could
do and we learned a lesson from work.

By David Madigan and Pat Ryan and
Mark suit a paper Simpson et al

they took an old idea from the early
seventy's called self-control K.

series that was actually used to show from
observational data that measles mumps and

rubella vaccine released in the U.K.
a new version of it

was having a causal effect on minute and
meningitis risk and

they took this and extended it to
something they called multiple S.C.C.S.

where you could look at many drugs
at a time for one condition and

I'll talk more in a little bit
about adverse drug events and and

something called the observational
medical outcomes partnership or

Omagh which didn't
evaluation of methods and

C.C.'s was arguably the best
performing method the key idea in M.

a C.C.'s is shown in this picture
we have a red drug here and

this is showing an exposure period for
this one patient another exposure period

here another one here and then we have
account of our injury yellow color yellow.

This looks more orange on the screen and
this looks yellow exposure for

another drug and
then we had entries of M.R.I. codes.

And that just I'll show you the details
in a minute but the gist of M.S.C. C.S.

is we're asking do I see Mark or
occurrences of during Red Orange exposure

periods that I would expect by chance and
so the patient is really serving in.

As her or his own control here their
periods of exposure and non exposure.

Spent a lot of time defining drug exposure
Windows based on prescriptions and

then increased risk windows even
after you stop your prescription and

they looked at a number of various ways
to control the sizes of those windows

based on hyper parameters We'll get
into that later but the gist is S.C.C.S.

is using each patient as their
own control under the hood

it's using passant regression and
so the X.'s are indicating

whether the patient was was on
this drug or a drug prescription

at this particular this particular
drug be at a particular time T.

and then whether an outcome
of interest was observed so

the drug might be something like Vioxx or

a general class like Cox two inhibitor and
the outcome might be M.R.I..

The trick that really makes this work so
well is we're going to learn a linear

model to predict event counts based on
just the drugs we're going to learn

coefficients on the drugs but
instead of a single baseline

here we're going to have a baseline
that changes with each outcome and

it changes with every patient so there's
a patient specific baseline because

we all have different risks baseline
risk for certain types of events and

that's kind of the magic that
makes this approach work.

So my student Charles Chua was
meeting with me one day about adverse

drug events and Jamie Thompson called
up and said you know I'd really like to

predict drug repurpose in opportunities
in particular there are lots of

numeric measurements in the E.H.R.
like cholesterol or L.D.L.

in particular blood pressure systolic or
diastolic fasting blood

glucose really be interested in
predicting if any drugs are controlling

that we don't know about that we don't
know they're controlling that and so

what my student Charles did is he said OK
look these are this is a patient these

are drug exposures like like insulin these
are fasting blood glucose measurements

and I first proposed a very naive approach
to this I said let's just take over

all patients all pairs of measurements
within a year of each other and

look at intervening drugs and average
drug effects and and we did that and

we listed the top forty drugs that had the
biggest effect on glucose we color being

green those that were known to reduce
blood glucose and read those that were

known to elevate it and there were some
in white that were nothing was known

about half of what we found were in green
and almost half of what we found were in

red like glucose glucose will elevate
obviously your blood glucose but

presumably the patient got the glucose
prescription the same day they

got the insulin prescription and so
Reserve result of confounding So

what Charles did is he said OK let's
take this idea from him a C.C.'s but

now we're doing linear regression rather
than prison regression because we're

trying to predict this continuous value.

And so he said let's add this
patient specific baseline and

sure enough we got rid
of a lot of the red but

then what he said is he must be looking at
my health record because he solver time

my glucose was going up my L.D.L.
was going up my blood.

Pressure was going up and he said OK it's
not just a patient specific baseline but

it's a time varying baseline and
so here's your basic set up for

linear regression we're adding a lasso
penalty on the betas on these drugs but

he also added something kind of like
a fuse lasso penalty if you've encountered

that it's not just that's fine this
is just a control over fitting but

the fuse last so penalty this signifies
some amount of time like six months

this is saying if the patient had two
consecutive measurements of glucose

that were within six the time.

Between them was less than six months
then the size of the baseline can't

be too big and if you have a big change
in the baseline you pay a penalty and

this is that penalty so
if a patient if a patient has his or

her glucose going up over
time we can model that but

if there's a big job within six
months then we're being pressured

to explain that instead by some
intervening drug prescription if possible

especially if that drug is shared by other
patients where we see the same phenomenon.

There's a whole world of calls on the I
want to examine this approach where you

make a call shot you're saying this is
what I'm focused on is involved relation

and then you know where I'm now that's
a big difference yet so that I should say

thank you for that like you're just trying
to mug this is like monitoring your

call kind of a meter trying to say
are there is you know like path yet

so yeah this is huge filled of causal
inference and there are great approaches

like propensity scores and graphical model
approaches like the work of Robbins and

Perl and
the folks at CMU Cooper where they're

saying under certain assumptions I can
provably recover a causal relationship for

example the graphical model you assume
I know the whole calls will model

in the form of a base and that which
also has actually now causal and

I'm just missing one edge and I want
to know should that edge be there or

not this is more in the spirit
of Granger causality and

I would classify it as something that
in Pyrrhic Lee works well but at this

point doesn't come with any
formal guarantees an honest and

the nice thing is it doesn't come with
a set of assumptions there are appropriate

assumptions were work better but
I don't have the formal guarantees but

in practice it worked really well for
the binary case and

it turns out it worked really well for
this continuous

response as well on the right is
the version that sees that you're a you

know on the right is the version with
the patient specific baseline and

then over here with all green and
white is the version over here the version

with the time varying baseline and
in fact there are some there are some

things like Dilaudid that you're not going
to give control blood glucose but there

are some here like selects on Prozac there
are some evidence in the literature that

that this class also potassium chloride
several of these there's evidence already

in the literature that maybe this class
of drugs is is controlling helping can.

Blood Glucose also a little
further down the list are some

quinine related compounds.

Let me come back though to talk
more about ma so all mop was.

A collaboration of the big
pharma industry umbrella group

with the acronym Pharma and
the foundation for the and I a church and

and I'm blanking out and the F.D.A. and
the F.D.A. aimed at evaluating methods and

really it was in response to
Congress's demand that the F.D.A.

do post-marketing surveillance of drugs
using observational data and which in

general is a possible problem but we know
some methods work better than others so

they're evaluating these methods and
they set up a competition actually

John do you care I learned was was
critical in the N.L.P. to extract some of

the ground truth from the drug labels they
set up a competition to say let's match

drug classes with events they cause
like warfarin with with bleeding.

And so they have a set of ten
health outcomes of interest and

ten drug classes and and
the goal is to match these up you use.

This money and.

Next like yeah OK thank you I should've
had that on the first flight.

So what you have is a prescription
timeline so they have no mop had methods

within their common data model to
to give you drug exposure areas and

even risk windows that go
beyond the drug exposure and

then in pictures Emma C.C.S. For example
many methods were tried but Emma C.C.S.

for example is is looking at currencies
of a particular event like bleeding and

lining it up with drug exposures or
or non exposures.

And so all mob had a ground truth and
by the way causal inference is

greatly tendered by a lack of ground
truth to go with real world data so

a lot of a lot of calls will inference
work is theoretical assumptions and

proofs that you make the right inference
is under those assumptions a lot of it

the empirical evaluation uses synthetic
data where you know the ground truth but

notice if we're computing areas under
the curve now we're not making patient

specific predictions where we can hold
out a group of patients we're making

big predictions about the world this is
this pair does this drug condition pair

go together or not and so you want to you
want to rank pairs by most likely adverse

event down to least likely
do an R O C curve and and so

you need the ground truth about the real
world which is much harder to come by and

gave us such a such a ground
truth where here the drugs

here are here here are the ups
I need to look here so

I can see OK across the top of
the drugs here on the roads

are the conditions green means
the drug actually helps reduce

the current rate of the condition
focused on the red him and blue or

red means the drug causes the condition
in fact it's on the label and

blue means we're pretty confident that
the drug does not cause the condition we

were working with data from the Marshfield
Clinic again in the past we used data in

something called the OEM OP lab at the OEM
OP labs no longer available and so we were

using data from Marshfield and I mentioned
there were many hyper parameters one could

use in the experiment such as how
you define drug exposure areas how

you define risk windows and so forth and
so the evaluation actually used

a wide range of settings and
when we use this range of settings.

Here was M.S. C.C.'s is performance and
then if we incorporate

the time varying baseline we get
a significant improvement from that.

So time is actually defined in

days or years if you like because so
we use like I knew how to use

the landing zone right if I understand
your question correctly we're saying.

I want to predict I want to
use all the drug exposures.

As predictors of this event but

also every patient has their
own baseline probability for

this event and we're actually estimating
that probability along their entire

timeline to start the first occurrence
of impeachment in the first occurrence

of the patient in the in the database and
so some of these patients we had for

forty years and so Marshfield a little
different and maybe that gives us more of

an advantage them than we would
have had in the original data sets

Marshfield an exception in that many
patients you had twenty years of data.

And so from your first encounter with this
patient you're estimating that baseline

and I should also say M.S. C.C.S.
has the big advantage that you can you

can actually solve the person regression
without estimating the patient specific

baselines now that you have a time varying
baseline there's a big computational load

in estimating that across time so
you pay for this you pay for

this gain in a sea of corn into the ground
truth in terms of run time quite heavily.

In cases where you had settings.

In cases where you had a lasting
effect rather than a shorter window

our advantage overall mop really decrease
substantially was there are over M.S.C.

C S decrease of sanctions that's
the another question OK Now we did

try another approach
as well this was work.

It's HUGE about who's who
left after this paper and

finishes them M.S. and left to do is teach
the at MIT is working with Regina bars

Elaine our natural language but
Eugenia worked with Becca Willett and

the on this on this where he took the idea
we don't want to have all these different

risk windows that we have to think about
drug exposure windows and risk window so

he said let's suppose we have these
dyadic influence functions in other words

every drug and
every period of time shown here so

progressively longer periods of
time can get its own weight and so

we can as part of the learning process
learn different weights not only for

different drugs on a condition in
different drugs on different conditions

but over different periods of time so
different drugs can have different

different with risk windows and so
that approach really is a Hox process but

it looks a lot like the Posen regression
we were doing before except now

we have we're learning
weights on these on these

influence function drug drug payers and
that approach.

Again though we do have some parameters
like what's the longest time period we're

going to consider and that approach.

Gave again substantial
improvements over him a C.C.S.

what we'd really like to do is
combine this unfortunately for

this we had to say goodbye to
the time there in baseline

we had patient specific baselines and
time varying drug effects but

we couldn't get this to work efficiently
with the time being patient baseline We'd

love to find a way to combine those but
we haven't haven't been able to yet

we'd also love to build on some of our
going to slide on this in a minute

we'd love to build on some of the great
work here at Georgia Tech such as Jim.

Things work to see if we could do some
kind of deep neural network learning

with a baseline like we've talked about
here ideally a time varying baseline

even a patient specific baseline because
that gives you the ability to pick up on

drug drug interactions right now
because this is a linear model

we're not picking up on any interactions
and we'd love to try to pick up on that

going back to the earlier
work I talked about as well.

A deep learning approach instead of our
random forest approach you might pay

in terms of run time but the nice thing is
that you would learn an integrated model

that potentially could find
common common predictors say for

example across cardiovascular
events like like stroke and

heart attack much like the Framingham
model does where as we're just treating

these all as entirely
independent prediction tasks.

And of course the big thing that we're
interested in but except for the war for

in case I have to confess we haven't
translated into this in the clinic as we'd

love to get these models into
clinical usage either to the F.D.A.

you know for system wide kind of usage or
to individual hospitals for

individual patient level predictions so
with that I actually did get to back

to my acknowledgment slide and I'll stop
there and take any questions thank you.

You've got them all in there
in the top of the great.

OK over there for the players and so for

the common room the real focus was
the real goal is really pretty probably

right but if you want to kind of somehow
you know have some causal structure

there's the miles have more explain
ability that's just one motivation for

the other motivation is that we'd like
to better find adverse drug events or

repurpose an opportunity so you really do
for that need to know is the drug causing

this event and not just predicting So
those were all motivations but

I think causation or causal modeling for
Explain ability there's a lot of

lip service to any lot of research
now to explainable models but

I don't know that anybody is taking this
causal angle and it's a very big piece of.

It is that you think there's ever
going to discover you know drugs X.

functions of drugs that are no other
words I absolutely think so and

I don't have a case yet where I don't the
case yet where we discovered something and

then they're fighting or
found the mechanism but

we are Jamie Thomson's group potentially
is going to be doing some experiments.

In in rats with a diabetic model rat with

with some of these drugs
that we've talked about so

I absolutely believe that potential is
there and hope that we'll be able to do

that since there is a lot of tension there
in this and you know the goal is just.

The temptation just use all agree or
absolute or and their competitors but

at least they let you have it right and
for some things that might be

enough you know if we if we could have
predicted my atrial fibrillation event and

given me a week's vacation
I'd have been off for

that regardless I wouldn't care how
we were predicting it I guess on this

thing you were talking about if you did
how much rest of the penalty on weight or

something you have a trade off where
you might use as much as possible and

try to drive the ones that really
don't use right and you could

do that is a process where you adjust the
land or in this case multiple land as to.

Yes everything behind no effect you
can ask your colleagues that exams

right actors Yeah absolutely.

Yeah so the part where you can get
an efficient time dang beast and

as you know the whole process yeah with
the hocks process that got us around

the drug exposure areas and
risk windows but we just couldn't get

it to the optimization to work the fish
and enough even with the casting grading

the set we couldn't get any kind of the
efficiency we needed with also estimating

the time varying baseline we'd love if any
of you are interested in say we'd love to

find a way to efficiently do both of those
to keep the dyadic influence functions and

essentially learn the risk windows and
have a time varying

patience with every baseline if you've got
some ideas I'd love to talk about that.

Yes.

Yes so the windows maybe the picture
didn't do the best job of clarifying

the windows are different periods of time
and we just had them doubling up to some

maximum window so so like one week two
weeks one month you know two months of

up to stop to a year and that all that I
was varying was the biggest amount of time

we're going to look at and so
the downside to having those windows or

those dyadic influence
functions we called them

is that you break up the drug information
between these different time windows and

so maybe you have a list a to stickle
power but we have a lot of patients and so

so the advantage is that
different drug condition

payers can have different timings like
the effect of war for an arm bleeding is

probably a very short term effect whereas
the effect of Vioxx on cardio and

far she has been speculated to be a much
potentially short term all the way to

a very long term effect after you stop the
drug and so this is given the learning.

Algorithm the ability to learn
those effects of risk windows

not only to have them different for
different drug condition payers but for

the learning algorithm to figure them out
rather than our having to pre-define.

Thanks.

Big presentation a couple
questions both related to

typing to the definitions one is.

It sounds like more fields providing
you the dataset Yes it is said.

I think the expectation would be that if
you were able to incorporate other types

of Dean it might fire ability to build
models and be Ancient to see if they were

amenable to maybe running some type of an
off the algorithms Yeah on a doc is there

right and then you translate into the
identified on mop Thank you yeah I didn't

get to talk about that piece say they
did that for some of the data types like

the older drug exposures and they would
be open to doing that in general for

privacy reasons we get the identify coded
data but it's impossible for them to

guarantee they've identified the N.L.P.
data so we don't get the techs notes but

we can drive up there and use the techs
notes and as I think the potential is to

do that and that could help dramatically
with phenotype and we'd have to try it and

see it could have a big impact on what
it can to consider the other piece

is also in the future having perspective
there's been a lot of progress from.

This kind of a now Odyssey variation of
being able to have sort of shared you

know time can run and different sites and
then look at the performance of them and

I think I want to should use that as your
yeah yeah so that each ripple ation or

really any of these items now are sort
of really beyond just the codes but

the sets of various codes and
also the validation and

different sites you say what you how it is
performing Marshfield how his performance

Columb your Stanford wherever it may
be and my students are working to make

this stuff available at Odyssey so that
we can then I've got to figure out how to

sufficiently advertise that
people run it on their site.

With that that is my hope that we get
through on this on some other place that

I had or you have a question or now.

Maybe you can watch the news then just
within these words courts of nations

here obviously all sorts
of social racial Yeah

very very Are you considering them
at all or are you just thinking.

For better and for worse Marsh fields
a pretty homogenous population Murray

brilliant who is a fabulous geneticists
is up there Marshfield tells me

most everyone's from Bavaria in the late
one thousand nine hundred and so

I think I think it's
probably safe to say that

that both in terms of underlying genetics
and to some extent socio economic factors

is less variation than you might
have elsewhere but I couldn't or and

others have pushed on me to include
more socio economic variables and

the other thing that we're doing I
didn't have time to talk about but

not only do we have genetic data on
twenty thousand patients through their

personalized medicine research program but
Scott he bring their Marshfield in my

students on why I have developed
a method from the day identified data

I can get in the details but
of constructing pedigrees family pedigrees

and we're trying to use that to also get
into some other risk factors that were not

included so this is just the coded date in
the air here in sarees In other words OK.

Have you looked to see
how transferable the.

Other population so
that's what you'll do with the role

that's our aim while on a new in our
renewal for C.T.'s A that we just

got which is to to run this on
other peak or net sites and our.

Great Plains of Korea network.

And including at Milwaukee and

potentially some other sites in
the Midwest in Madison Georgia.

Excuse me.

I'd be delighted to do that
I'd love to talk with.

Here I'd love to talk about that yeah

as we're working in the data
in the common data model so

we could if that data is in that format I
think we've made a decision to go that.

We're going to be.

OK.

Because all of the services.

So there's a long.

Sitting right directly.

Yeah that would be great to talk about
Marshfield also in Madison together and

all of us as well and
we should talk about that.

Yeah to me.

This is more philosophical.

So for the interminable model where.

We also encounter last night
that type of request review and

from yeah look laboratory.

I don't like black box now that
they like linear models that they

can view a score functions were
talking about that earlier.

You have some I have what we why that's
the case because there are not Boxing Day

leave really I think I think a great
example and there is this isn't the reason

for all and I think a great example is
what Rich car one has been talking to tell

me a few years ago but he published a
paper on this recently that years ago they

were doing an asthma prediction model for
people who came into the clinic with in

the hospital with asthma and and
it was to rank how bad their asthma was or

what their prognosis was so you could
direct resources the right way and he got

really nervous right before it actually
got filled in the clinic this is the flip

side of fielding things translating things
in the clinic really really worried that

a lot more experiments with essentially
this black box neural network and and

found that the model was essentially
predicting the best outcome for

the patients who were actually the second
because it it had learned they did

well because they got all the attention
when they came into the hospital and.

So it was going to basically be a real
harm to the treatment of those patients

as I think there's always a worry if it's
a black marks model that we don't really

know what it's doing and it could
especially bring it to a new site but

even in general if it hasn't been tested
prospectively yet it could do real harm so

I think a these models need to be
tested in clinical trials just like

devices would be we owe it to the patients
in the clinical trial to do everything we

can to understand the model help
the clinicians understand the model but

also as far as possible
to test it causally or

retrospectively to see
how it would perform.

On that in that trial.

I mean here's the conditions yanking
organizations Yeah thank you

guys expect them to think we're yeah
it was them of some tool use and

I think no one has any idea how
it's doing this is a strange and

they were right in the case
which pro-ana site is.

OK So yes thanks bicker when you thanks.