[00:00:10]
>> I'll just tell you a bit about have in fact around and out there will get started it hasn't bought a 2nd Peter science Ph d. candidate at the University of Maryland starting network security and censorship and he is an instructor for a class on how the interjections to penetration testing he is going to have a really interactive session today and that we look forward to your presentation and I welcome thanks so much Ok everybody my name is Kevin Bock before I get started actually I want to thank the entire i.s.p. Lecture Series team for putting this on they do a lot of work behind the scenes Charlie Rangle Aria Lindsay and of what else I didn't get to talk to thank all of you guys and before I also get into Geneva itself and to give a shout out to the full team of people behind me who have contributed to the work I'm about to speak to in the past it's really taken a village to get it to where it is today and everything about so you know today I'm going to be talking about censorship and specifically in that work censorship usually done by nation states that before I can get into the feeding it on a talk about slow but how it works today and how it operates though is based types of censorship the operate around the world today but some of the most pervasive is this automated censorship that operates inside the networks now the nation state operates of the ship like this with the censor physically inside the network path in practice it can often be difficult to get machines in all the paths So instead the way some sensors operate is an upper like this where they set off the side of the Never path and this makes them a man on the side attacker not the man the middle the Packer this has implications the health of the ship is done in these countries so for example if we have a user in China and they're about to make a request for some forbidden resourceless say with a pia this is censored China.

[00:02:05]
This request will move the network and you'll see the server will get this packet but the sensor will too and if the sensor once the sense of the connection it can't just drop the connection can't just drop the packets because it's not in the path so instead what it's going to do is it's going to inject its own packets into the connection and specifically it's going to inject spoofed t.c.p. resets or tear down packets these are neural packets or computer some things all the time and they basically just exist to tell the other side of the connection stop talking to me but the sensor does it's going to send 2 copies of this and it's going to send one to the client pretending to be the server it's going to send one to the server pretending to be the client but that happen is the client is going to get this packet going to think well the server just terminated my connection and the server is going to get this packet it's going to think well the client just terminated my connection and immediately both sides will stop talking to each other and just like that censorship has been achieved but in looking at this attack in order to pull this off the sensor needs to have information about the connection in order to generate these tear down packets need to know things about this and specifically needs to be tracking her flow state governor the port numbers sequence acknowledge the numbers etc And if you're going to be tracking the state of every single t.c.p. connection coming into and out of a country the size I know you're going to certainly have to take some shortcuts along the way and we as a Vader's can often take advantage of the shortcuts Let me show you an example that researchers came up in the past this is from prior work now again our client is about to make a request but before it's sensor been the quest but the Klein is going to do it's going to inject its own piece of the receptor.

[00:03:49]
And is going to send it in such a way that we set the t.t.l. or time to live in the field in the packet that it controls how long packet survive in the network the document's once per hop we're going to set the study high enough that we reach the sensor but not so high that we reach the server.

[00:04:06]
It was on this packet through just like before the censor will get a copy of the packet but the packet is not going to reach the server going to get dropped along the way for the server never saw this packet but the sensor has and the sensor says well it looks like the client just terminated the connection so I can stop tracking it now and it throws away the states and tracking a better connection and at this point we're free to communicate client to server for the rest of the flow because the sensor has no state has to censor us that it's maintaining and the server has no idea we've pulled off this trick though fantastic and this is one example of dozens of strategies that researchers have developed manually over the last decade or so the problem is mostly took a long time for people to develop and find think about and made them don't work anymore and the issue has really been how we've approached this line of research historically speaking because how historically approach this problem of censorship of Asian research has historically looked a lot like this just straight scientific model we make hypotheses we take measurements and we interview we build a good mental model get a good understanding of how the sensors operate then once we have that good understanding we apply are good human intuition we create some sort of something chip vision strategy the problem with this is it's largely a manual effort and sensors are black box stuff we're at this asymmetric information disadvantage compared to the sensor we need to spend time to understand have a sensor works but the sensor they have all their code they know how the stuff works and this largely manual effort has been giving sensors the advantage in this arms race has been playing back and forth and this is the problem that we wanted to address with this line of work the our goal at this work was to try to give evaders the advantage.

[00:05:59]
And specifically what I'm going to talk about today is all the meetings of the ship of Asian research. And what this looks like is we start by developing some way to just automate the discovery of new stuff as a base and strategies. We automate it from step one we have a good after the fact we as researchers can then use the strategies to understand how the sensor works that we don't lose any understanding about the sensor but this lets us still.

[00:06:28]
Learn how the sensor works without us our human understanding getting in the way of actual actually performing evasion we've developed such a system that we call the system Geneva and with Geneva purpose let us do is automate this and then after the fact learn how this works and that's when we're talking about the stock the Geneva stands for genetic evasion and a genetic algorithm and then briefly talk about how we built and designed.

[00:06:56]
The Genetic algorithms are biologically inspired systems you want to build a genetic algorithm you're going to need a couple things to walk through these briefly just using us humans examples Bosenova cobber them you generally need some sort of building blocks for us this is our d.n.a. bases a 50 g.b. you need some way to compose these together so for us this is how these building blocks compose into our d.n.a. need some way to mutate the static material between generations of us have just happens naturally for us and then finally some fitness metrics some way to ensure the fittest survive from generation to generation there are a couple of challenges in applying this idea of a learning algorithm to censorship of vision so let's walk through some of these.

[00:07:43]
For building blocks we're going to have Geneva be running strictly at one side of the connection and could be manipulating the packets that are coming in and out of the client but this already raises the question of how do you modify packets because this this right here was the 1st major problem we ran into and this is one of the key insights that enabled this work to be successful is if you think about it there's a lot of different ways you could give an algorithm the ability to manipulate a packet on one hand you could just let it do that manipulation any bit flip you'd like this is very versatile with enough time eventually it could learn anything you can imagine this taking forever to learn something like a checksum on the other hand we could do is we can encode known strategies or building blocks researchers have developed in the past and just let it play around with those that would be highly efficient that would let us quickly build up the things researchers found in the past but it's potentially very limiting and would be encoding our own human bias in the system that we landed on was giving Geneva the ability the building blocks to manipulate packets just like the IP layer itself can and specifically we give it for actions duplicate we take one packet you get 2 packets tamper you take one packet and change it somehow fragment or you take one packet and break into 2 packets and then drop me take a packet and just drop it now I'll call it 2 things the civically here for tamper with it alter or corrupt any t.c.p. IP header fields importantly though there's no semantic understanding of what these fields mean so for example it sees the flags field of something you can change in the p.c.b. header but it has no intuitive knowledge that changing it to sin means the start of a connection also call out the fragment of a bit of double duty here at the IP layer with your fragmentation but at the t.c.p. layer we do segmentation.

[00:09:42]
And now we have a system of building blocks that we can put together for our genetic algorithm Now we need a way to put them together and this action system actually lends itself pretty nicely to this and the reason for this is some of these actions introduce branching So for some of these actions like temper you start with one packet and you end with the same number of packets or less than you started with start with one and with one but duplicating fragment you start with one packet and you end up with 2 packets or this lets us do is start to compose these things into a tree structure so what we do here is we we put these actions together into action tree and we structure these with them actually match action pairs so every tree is assigned We call these triggers and when this trigger fires whatever outgoing packet mash this trigger gets pulled into the action tree and the action tree describes how it's modified so for this example tree up here we have the trigger that for an outbound t.c.p. packet with a flag seal the set a we pull that packet into the tree and we manipulated according to the actions the tree stump but see this running live.

[00:10:54]
Here we were a client it's about to make its written request and that's going to finish the Through a handshake with the server with the bow to send the act packet and we'll see this matches he will notice this matches. The trigger for this action tree and it pulls it into the tree and run the duplicate action we have 2 packets the left side were done so we don't do anything on the right side we temper it and change the p.c.b. flags to reset we change the t.t.l. the 2 that we do in order for versal of the leaves and we send the packet the come out and if you notice this action tree exactly implements the strategy I showed at the beginning of this talk.

[00:11:34]
And in fact we find that this this composition system is expressive enough that we were able to go back to prior work and rewrite strategies that have been described in text or other notation into the street like structure that's encouraging that for mutation This is one of the easier ones actually to do because there's just so many different ways we can you take these things so we randomly alter types who values around merge trees but trees apart there's a full rundown in our paper of the various mutations the things we exposed to the.

[00:12:08]
The last tricky thing to figure out is the fitness function and the fitness function is trying to answer the question which individuals did survive to the next generation what strategies are best that we want them to propagate their genetic material forth and this really comes down to what do we want to reward from the genetic algorithm and this is really the 2nd key insight that enabled this work to be successful to specifically what we land on is we 1st punish a 1st we 1st punish strategies if they don't trigger any packets that we punish it for bad that match triggers Next we punish it if the strategy breaks the underlying t.c.p. connection.

[00:12:52]
So if the strategy damages the p.c. connection it's punished. Next we were worried that if we can successfully obtain forbidden content through the sensor This is the encouragement for the genetic algorithm to identify strategies that excessively defeat the sensor at this point we could stop at this point this is more or less successful good enough that in this function we add one more optional rewards the genetic algorithm for conciseness and we do this really just for our own understanding how we reward the algorithm for simplifying successful strategies down this is useful for us so we can understand what this thing is doing defeating the feeding sensor with the action strategy is cool but could be difficult for us that are stand why that works so we given the decision a reward for conciseness and this in a nutshell is how we design Geneva we 1st started with this we wanted to try and do an apples to apples comparison with prior work existing work in a space.

[00:13:54]
But the problem with this is sensors are constantly changing so by the time we had developed an Eva the state of the world was different such that previous results were not necessarily apply couple if we trained against a sensor we wouldn't necessarily get the same results that the sensor is different that we did was we read their prior work and we reimplemented in the lab our researchers thought the sensor it worked and then we train Geneva in the lab against these mock sensors and within hours we were able to identify every strategy for us Geneva had a primitive for now there were several classes of very clever strategies that researchers had designed in the past that Geneva just didn't have the building blocks to express just things like changing things that the application layer because you know this restricted to t.c.p. IP or sleeping a very long time between sending packets the things that you need to cannot express we cannot read drives but everything else and it was able to identify.

[00:14:50]
With this we then moved on to doing real live sensor experiments at training Geneva against real world sensors and specifically we trained against China India Iran and Kazakhstan and this is a civically for client side results and for it should be censorship who trains leaving in each of these and we found strategies for all of them and we found enough strategies that we decided we needed to do was to find a taxonomy for how these strategies to be classified so the highest of the taxonomy is a species and this is the underlying bug we identify below that we have subspecies which is how Geneva exploits that bug and then finally baryons these are things that look functionally different on the network even if they're really doing the same thing and across all these sensors we find this number of strategies across all of these actually census work with some some even more or less talking through just some of what these some with the strategies are and by the way I know these number some of the 36 I can do math.

[00:15:52]
There's just some overlap between what's strategies work in certain countries so I was walking through with just a few of these painfully strides look like to get a feel for these the 1st pieces I'll talk about is the tear down species and this a look very familiar because this is again the strategy I open this talk with this amazingly still works today in China with his with Geneva we actually found additional variants of this which I can talk more about at the end or our in our paper.

[00:16:21]
But it's interesting that the strategy is one still alive in that Geneva can still identify these very basic types of things another strategy much more complex strategy we found is a segmentation species and this works in China still works today and what the strategy does is it takes the request and all it does that segments it twice to bite and this is.

[00:16:43]
And we saw this this was very surprising because the Great Firewall has famously had the ability to reassemble t.c.p. segments for years almost the last decade or so and it's long been thought that the segments in your request was no longer sufficient to get to the great firewall but Geneva still found the strategy and it does work very well that's question was well maybe what's happening is the strategy is just taking the forbidden keywords and splitting it up a lot so ultra surf is a keyword that censored in China is the name of the circumvention technology that the Chinese government doesn't want to people to know about and who thought maybe this key was just getting broken up a bunch but really what we find is that when we do the segmentation the entire keyword is in plain text only one packet not broken up and the request to sound like this but very strange that this works and in fact we actually find additional in strength to this bug as long as the 1st packages less than or equal to 8 bytes the sum of the 1st 2 packages that least 12 and a few other constraints we list out in the paper the strategy will work fine but we're No 100 percent sure why this works still today frankly we think this is just a bug in the Great Firewall and we're just breaking some regular expressions inside the firewall looking at the strategies we can consider what it would take for some sort of fix these the segmentation species like I said this is likely a bug Maybe it requires some 3rd party vendor to issue a patch if it's difficult to get that patch out this is realistically something that could probably still be sex this p.p.l. limited one though one of the left literally much more fundamental issue I think what it was called the eavesdropper the lemma eavesdroppers lemma which states that it's very difficult to mimic state the connection ends when you're in the middle of the connection.

[00:18:39]
What's interesting here and important is that with Geneva we can find both classes of these issues we can find implementation bugs things that we think will live and die more quickly or very serious design flaws that we think will and what difficult for sensors to fix but they're going to switch gears a little bit and talk about our 2nd our 2nd round of research with Geneva that was addressing the problem of client side evasion the censorship of Asian to date has always involved the client this intuitively makes a lot of sense if the client would like to abate censorship it makes a lot of sense the client would have to do something configuring a proxy and so on software modifying the packets exciter this requirement has always imposed a pretty significant barrier to deployment.

[00:19:26]
Installing software can often pose risks to users and you know this it can help the millions of users who don't know they're being censored in the 1st place or don't have the technical know how to set up software to Pete censorship Now ideally service could help instead of the point so what the client what we could do is instead deploy a server and the server would subvert censorship on the user's behalf without the client needing to deploy anything at all and just thinking about this the benefits of this would immediately brought them accessibility and reachability many clients connect to one server clients don't need any techno expertise or download software and immediately on sensors all those users who didn't have a test will know how were didn't know they were being censored in the 1st place the sounds amazing the problem is when we 1st approach this work it seems a little of saying this is really server side a vision it shouldn't be possible and to see why this is let's consider the waterfall diagram of packages that are exchanged leading up to the client sending a sensor keyword.

[00:20:36]
The client going to send the send normal the service going to respond with the facts. Then the client completes a through a handshake and a sensor keyword except in the service perspective though there's very little it can do between the sensor keywords before the sensor keyword ascent and in fact the sensor can't influence the connection after the sentencing that can influence the connection at all to approach this problem we took the techniques ideas from Geneva and applied them to the server side and the poor Geneva Crossman countries and protocols and amazingly it discovered multiple ways to defeat censorship so he's there with this can look like this with a waterfall diagram of one of our strategies operating in China now there's a lot going on here so I'll break this down the client search the connection just as usual sending a 7 packet which the server responds not with the syntax packet but instead with 2 Sin packets and actually this is legal in t.c.p. The 1st is a normal sim and the 2nd is a sin containing a payload to the 1st since serves to trigger a t.c.p. simultaneous open this is an archaic feature of t.c.p. that is still today supported by every major platform and that it really exists just to handle the case in which 2 computers tend to send packets each other at the exact same time now when the client receives this 1st and packet the client sends a cynic and it's actually this combination of the client sending a sent ack immediately preceded by the sim packet containing it he loaded that causes the sensor to decent cries from the connection the Great Firewall is no longer properly track your connection and for the rest of this flow the client server connection unit Kate thems are shipped free and the seems crazy but this actually works with varying degrees of success across all of the protocols all the sensor protocols in China where tests and for context the baseline success rate percentage of evasion if you do nothing at all drown 2 percent.

[00:22:37]
This is one successful strategy but it's actually one of many and in our paper we detailed 11 different attacks that we found in these 4 countries and these explain everything from not properly handling esoterically just t.c.p. correctly handling error cases just through the confusing sensors logic altogether and I want to note that none of these require any client modification at all many of these induced behavior from the client but all the behavior that's being induced this is all this is all the default behavior of that t.c.p. stack there's no software configuration changes needed on the have to climb to make this work but it walks through 2 more of these peripherally just to get a feel for these this next one is in Kazakhstan we call this knoll t.c.p. flags and with this one works it's very simple before the server response to the fin it sends a single packet with no t.c.p. flag set at all and the sensor just cannot handle unexpected to see the flags of the total lack of peace if you flags on this packet makes it to the sensor is no longer paying attention its connection one of the big takeaways for this was really motivating why we need automated techniques to discover these this is likely a bug in the sensor it may have been difficult for a human to conceptualize in the sever another crazy example I'd like to show you this is again in Kazakhstan.

[00:24:01]
Instead of sending just once an act in response with the server does it send to send x. and on each of them it attaches a payload of a well formed in c.p.p. get request what happens is when the sensor sees both of these it confuses the direction of the connection it sees these coming through it's like well maybe I missed something maybe the server is actually the client because it treats packets differently depending which side of the connection they come from confusing the sensor direction abusing the connection direction to the sensor is sufficient to defeat censorship the rest well and this works with great success and.

[00:24:41]
So we we were able to find these strategies across every major protocol we'll get them 3rd and here's the breakdown of all the countries and all the protocols we will test and I will know that even though Iran with C.N.N.'s Iran does censor d.n.a. tests but at the time of our testing the n.s.a. for t.c.p. was disabled Yes or t.c.p. was not censored in the country so we cannot test that we also tested all the strategies across all of these diverse clients even though we're doing all these crazy check the t.c.p. stack we find that they're very broadly supportive of across all these major things we found only one case where there was this class of strategy that relied on platform the private behavior and were able to very quickly rewrite it such a way to make it platform agnostic The real question is what's next now that we have this automated way to explore the space the fences the patient strategies I'd like to talk about 2 ongoing bits of research learning from Geneva strategies and our own rapid response to some ship events the 1st thing was learning from Geneva and a prime example of this was a realization we had this while testing testing out our server side server side of Asian strategies in China against many different protocols but we are finding is that all of the strategies had different success rates depending on which protocol the sensors but this is very strange because all the strategies operate strictly during the through a handshake no protocol data has been sent yet which raises the question why are different applications affected differently in China so we've always implicitly thought about the Great Firewall China works for this very very sane same network design or it singly separates the application layer from the transport layer protocols just like we always do but what's happening is we're finding different t.c.p. layer bugs for each protocol.

[00:26:41]
What that must mean is that each protocol has its own t.c.p. IP stack and the strongly suggests that the Great Firewall is running multiple independent censoring middle boxes in parallel we call this our multi box theory into the south like on the network instead of conceptualizing this as this thing will monolithic sensor instead really there's multiple sensory middle boxes all running in parallel but this itself that is the question how does it know which middle box to apply to a connection and you may think will port number but it doesn't rely important for the great firewall any of these boxes can censor effectively on any port you're making up for been able to be request on port 80 or 40000 report 32000 and still going to censor you with not relying on port number to the side which of these middle boxes step them so how does it know which metal box to apply it doesn't and we think is happening is that every middle box is independently getting copies of these packets and they're actually all independently tracking every state of each connection in all applying protocol fingerprinting to the packets they see and then all of them will just decide all this was not mine this is not for me this was not for me until one of them can finally identify this packet of this is my for call and I don't like this this is some sort that we tried to do some experiments to determine if we could find where these pro these metal boxes are located we use t.t.l. limited probes and as far as we can tell all these middle boxes are colocated at the network level so at least there are no t.t.l. documented hops between them this is one example of new information we can glean from Geneva really thinking about this or turning back to the slide even gives us the answer and that lets us work backwards to learn more about the problem this is very exciting for us to learning more about how the sensors are operating.

[00:28:43]
And the next thing going to talk to is our own rapid response to new some to ship events and 2 cases of this all the talk to the 1st memory of this year are wrong launched a new censorship system that has a protocol filter protocol weightless or. The way this operates is it censors connections that do not match protocol fingerprints so if you try to speak the protocol the Iranian regime does not approve of ahead of time it will take down your connection regardless of what information you're exchanging those do that in any any connections that do match these this protocol filter and have any Perkel that are allowed are then subject to standard censorship so if you try and if you try to talk a protocol that they allow they will they can operate censorship over their protocol so we identified this this new system while we were testing our server side strategies in Iran we actually saw when they turned it on our strategies that had previously been working suddenly stopped working and we couldn't quite figure out why and eventually we determined there's actually a whole 2nd censorship system it's just been unable.

[00:29:54]
We point to Geneva as a problem and was in just a few hours a Geneva was able to discover or different strategies to invade Iran filter which is very exciting. More recently the mess in July of this year's About a month and a half ago China begin censoring the use of encrypted s m I so China previously the way they censored each of us connections is by relying on the **** and I feel for the server name indicator this is a plain text field that's in the very 1st packet of a t.o.s. connection even though everything else is encrypted and yes connection is one bit of information that is an encrypted that's the s. and I field and s. and I is basically where the client announces to the server which website is trying to go to though right now if I connect it with a pedia in Chrome or and in China the word Wiki Pedia will appear in plain text in my initial request this is in the us and I fields and so censors have been able to use the s. and I fields for a long time the sensor issue gets connections so even though they can't see the 7 really what you're trying to do in the queue Pedia they can see that you're trying to communicate to a server they're not approving of and they can just take down the connection right there recently people have been developing T.L.'s 1.3 is the next advancement of the standard until us 1.3 gives us the ability to encrypt to the s. and I feels so finally this last bit of information can be encrypted and we can protect it and yes and I many people been looking forward to as this like finally issue the us the safe and unfortunate we learned earlier this year yes and I was not safe and China began censoring the use of yes and I across the board as a matter who you were talking to if they couldn't read us and I took down your question.

[00:31:45]
But we quickly pointed Geneva to the problem that we use them in Geneva such that we could run against the Stuart and I censorship and we very quickly discovered 6 strategies to a baby s. and I censorship and interesting really none of the strategies overlap at all with any of the other issues to be any good or if the Afghans are ship strategy be found or any of the other strategies we found that are original paper all of these are new and this really enforce reinforce for us that our our multi-block the area where the c.s. and I censorship This is not they updated their existing is to be essence of ship they didn't just beef it up they actually deployed a whole separate independent sensory system for this I like to conclude with a discussion of what the future of censorship of Asian can look like this using systems like Geneva or and the automation system machine learning except Iraq it has the potential to fast forward the r.s. race and it can do this from both sides but you should take humans months or years to develop and write peepers about we can now do with tools like this in a matter of hours the question is Where do we go from here we think it really comes down to those 2 classes of bugs that I mentioned earlier in this talk these bugs in a human patient but I think these are going to be easier for sensors to fix is or to fix these low hanging fruit things but the more systemic issues the gaps of logic the design issues the lapses of how the sensor was architected itself without a more much more expensive re architecture of the sensor system itself within this is going to be a lot harder and more expensive for sensors to fix so if we just ask what's the logical conclusion of this and we think it's that the bugs and if imitation will get patched more quickly but these longer more systemic design flaws will survive a lot longer.

[00:33:40]
To releasing the code gives users of this arms race power to previously the censors to block could fuzz their own systems right censors have it have their own code they've had their boxes there's been nothing stopping censors from funding their own implementation for years for releasing this code to the world and Geneva is fully open source you can access it on our website no point that just 2nd what we're trying to do is we're trying to democratize the ability to fund sensors India could have been funding at the tips of the last 2 decades and now anyone in this talk can do so too so this is our hope for the long term of this arms race The conclude here with Geneva how this really lets us discover new strategies quickly it gives us new insights into how the great fire works were so much server side a vision is actually possible and all of our code is open source on this website Geneva c s that you know you know if anyone has any questions I'm happy to take these now or I'm happy to talk offline that you guys in the q. and I answered how did you get vantage points in all of your countries.

[00:34:51]
Ok so getting vantage points in all these countries if it's somewhat a tricky question it's one of the things we get asked most frequently and it's one of the biggest barriers to entering research a space we've been fortunate we've been able to work with activists and people on the ground around the world to some of these machines or things we can just purchase or just for ourselves so that happen to be hosted in the censored regime we can just sign up online and get a machine there and start in sets of system but often you'll see in countries censorship of than a datacenter come up different compared to residential neighborhoods.

[00:35:30]
So oftentimes we have to rely on people within those countries giving us access those machines that we offer that be very careful with the firms we do and how these how we do these. Though it's tricky and that's really why I've been limited to just these 4 countries and the questions.

[00:35:52]
Into the can to get into the arms race how do you look at Ga strategies to block this trial strategy. Yeah it's a really good question so right now we've been I guess fortunate that this is still fairly new in the States the sensors really have not had time to respond and amounts dedicate a response to a Geneva style attack on their systems we have begun to think about what is true adversarial training in the space look like if we have a Jag for the fighting the something or what does making a fence or what is a learning some sort start to look like we don't really have answers to these questions yet it's a tricky question to answer but I think a very important one because what what a learning center will look like I think will really impact with the end goal of this of this arms race look like even just beyond fighting a learning system again learning system we also have the potential problem of and sort of having a sensor learning against us we could have sensors just trying to tamper with or messing with Geneva straining itself one of the things Geneva spin the function largely relies on this the strong signal from the sensor that when it does something that works it knows and when it does something it doesn't work it also goes but if sensors have a good way of identifying and honing in on which machines are training with Geneva if they can start modulating or censorship system to lead the genetic algorithm the wrong direction that suddenly a whole additional space of how do we train journey bug it's an adversary that doesn't want to get trained against them so it's something we haven't really dug into yet it's one of the next things we're going to be we're going to be playing with and I'm definitely open to talking about of the people of ideas or want to chat more about it.

[00:37:41]
But it's a tricky question and potentially something on the horizon. I'm just came by you m.d. dot edu. So you have any questions or want to chat more I'm happy to chat more and then I. Again the minute I wrap up every Don't be more question. Well I'm going to come back on and just thank you everyone for joining us today and will be here they will not have a great thank you everyone I.