[00:00:05]
>> My honor. Goals. So I find out yesterday that Jeff and I are going to be really Demick granduncle. Are you that for at least for through every academic. Show. That Jeff if that free for time where let him have 20 researchers an engineer over there is a machine learning your face of acceptability you guys.

[00:00:36]
Just tartan. Fly patterns. And before that you worked at i.b.m. research for a number of the research scientist. On the steering committee for the intelligent user interface of conference. That a lot of really interesting work spanning years. Excessively. Like Terry Jeffrey is going to be really interesting when you think you got it right thank you.

[00:01:05]
Thank you very much. I actually I'm originally from the Pacific Northwest Portland Oregon so I apologize it seems that the rain has followed me here from my childhood. Anyhow So yesterday I'm going to talk about. Some work that we did a few years ago eg Google. And basically assembling this data set that we call Rico I hope it's something that all of you may be interested in using at some point.

[00:01:33]
And based on this new scribe you know how we assemble this data set of few things we've used for and then what I have to talk actually I'm going to talk about things other people have used for so this talk is not all about my work so certainly.

[00:01:47]
A decent portion of it will be but but on the wall so you'll be out how people have used this data set in other ways so. And basically you know this work starts just with the motivation that I'm sure everyone you know you're aware of machine learning you know it consumes a lot of data.

[00:02:07]
And that was similar to where our motivation for building this data came from you know if we want to do machine learning work on you wise we know we're going to need a lot of data about us. And another motivation for this work just comes from the observation that you know the available the availability of called the Open Data Sets has been really important for moving you know the machine learning field forward you know things like image net you know from almost a decade ago now I think you know being available as a way to you know to to train machine learning models around images very important for moving that aspect of the field forward you know and so that what us to the question you know what if we had a data set of user interfaces of you know all sorts of user interface information that we could take we could learn over and we could do things with Seems like that would be valuable seems like not something that we could use to do any number of things and I think there are a lot of reasons we might need a dataset like this we could just use it to understand you know the world of user interfaces that are out there like.

[00:03:15]
How you know accessible or interfaces or how how many times are buttons used I mean all sorts of questions like that that are things that we really can't get out right now because it's not easy to look at you know a prod scope of user interfaces. Certainly you could also use data set of you eyes as examples to inspire a human designers designers you often use examples to inspire themselves having an availability of the data set could help with that you could also imagine then you know kind of applying more you know and all type examples like I've been talking about you know automating design assistance and in design tools also teaching machines to use an interface for perhaps explaining that interface to a human how do I do this being able to generate that kind of explanation all sorts of predictions and we can certainly imagine using many of these ideas to improve the excess ability of user interfaces.

[00:04:09]
And maybe we can even automatically design new interfaces for people based on the status so I'm sure there are lots of ideas I'd love to hear your ideas. Later on I think there's a lot of possibilities for days like this so that led us to create this day sekolah Rica which is a large open data set of mobile you eyes it's entirely Android your eye is currently 72000 screens from almost 10000.

[00:04:35]
We have screenshots we have do you hierarchies which are kind of an abstract description of the user interface we also have accessibility data and the interaction traces that you use to get between all of these different screens this data set is available publicly you can get it at the your all there I'm sure that your as later in the talk as well interaction mining dot org slash Rico and I should also mention that this is a collaboration not just folks at Google but also with a professor in chief a Kemar at u.i. you see some of her students some of whom were interned at Google.

[00:05:09]
Over the course of this work so. So that's the dataset that we created so in this talk I'm going to talk a little bit about you know how we collected this data set give you an idea of where it came from all talk about 3 different projects where we used the Rico dataset and then as I mentioned earlier I'm going to talk about you know Rico in the world how other people have used that record is set.

[00:05:35]
And some of the projects that they if they want to do. So 1st just to talk about collecting the dataset so we had some key questions where we just came to this idea Ok We want to collect a data set of user interfaces. So if there were some key questions that you know we had 1st of all you know there are a couple ways we could collect such a data set we could try to collect it statically you know we could take you know and Android app we could try to decompile it if you're aware a lot of Android apps have these x. and all files built into them so you can kind of get a static notion of what the interface looks like simply from like kind of pulling apart the app and looking at what's there ultimately you know and so that's relatively straightforward than that if we get a bunch of 80 k. files that describe the end of an application you know we can just break them apart do a bunch of analysis you know and you can kind of do that all computationally and probably automatically.

[00:06:35]
That compares to runtime analysis where we actually run the apps we actually you know have some way of clicking through them and seeing everything that that's when they're seeing what they look like in their actual stated use. But that of course has a challenge so we have to actually run the app we actually have to be able to automate the interaction with the app as you know a regular user would do.

[00:06:57]
And so you know what we decided we decided not to do static we decided to do runtime the main reason was that you know there's just a lot of dynamic aspects to a user interface that's not really captured when you're doing the static analysis. Another big question we had was whether we wanted to get real user traces or rather whether it really wanted to just kind of understand the app more completely and so for our case what we decided to do was go with complete crawls now I think this is really a decision we made based on kind of the type of data set that we happened to want which is that you know we were thinking about using this dataset as kind of a you know a training on the design of the apps themselves so we wanted to kind of see all of the screens across the map and you know if you look at I think really user traces of most apps and I'll say this a little bit of conjecture but I think that most people probably only use like maybe 20 percent of an app there's often a lot of little screens for a lot of little features and we want to capture all of that to get a understanding of like a complete app.

[00:08:04]
And eventually maybe we would come back to trying to understand you know what are the main traces how do people complete tasks but for this initial round we just focused on the complete crawls another big question that we had was you know how to do this crawl should we try to do it automatically and try to do a crawl back way or should we do it you know manually get some human users whether there crowd workers or other things Jim says no don't do that.

[00:08:32]
Unfortunately that's what we did. Actually we did it we did a hybrid of the 2 and I'll talk about that in a 2nd but I'm so we built a mining infrastructure to accomplish this crawl there's an open source platform called Open s.c.f. is actually used primarily for for testing actually stands for smartphone testing free markets with s.t.s. stands for.

[00:08:58]
And so it's usually used to like go out to meet some phones to see if you're after on sun like you know a galaxy $91011.00 you know it's like that kind of application but here we took open s.p.f. we modified it so that we could we could also use it as a platform to bring in crowd workers as well.

[00:09:19]
And so we had multiple real Android devices connected to this we've also sense been able to get emulators to run this platform as well we build build an automated crawling system so it could click through and do things automatically but we also had a a worker interface for for a crowd worker so a crowd worker could come in through a web interface and then kind of remotely controlled the phone through to that kind of interface.

[00:09:49]
And this this is work that started at Google as an interim project of the plant deck and Josh Shipman. It was later refined by the u.i.u.c. team before we before we did the chrome but that's what we started with. And what we learned so we did tried to do this automated as much as possible in the beginning but we learned is that automated crawling turns out to be pretty hard at least if you're trying to get complete Crawl's of the user interface.

[00:10:16]
So there are a number of challenges you know one kind of basic thing is logon screens also like s.m.s. 2 factor verification also comes into play these are all sorts of things that they just kind of block really naive approaches now of course you could build you know specific things to get past some of these basic items like if you see the Facebook log and always click it.

[00:10:37]
But you know that was an initial block that we offer into and I should also say for a lot of apps if you can't get past the logon screen you can see one screen so if you open the app you're at the logon screen and tell you why again you really can't see much of anything else.

[00:10:57]
But then there are also other types of issues that also block automated crime so you know complex a lot of apps feature a complex interactive sequences so like multi field forms and things like this often different elements in different fields depend on each other you know and so to get past this form you know you have to have you know a more complex understanding of you know what all these fields mean it's something that's pretty difficult for an automated crawler to do at least generally.

[00:11:27]
And then there are of course of the challenges as well many apps feature custom controls that you know are often not inspectable you know difficult for an automated system to understand some of these systems you know have just years swipes and such which also create challenges so I'm so that was why we didn't end up going with automated crawling is kind of our initial primary method so we did investigate manual crawling so we created this interface that you see here this is kind of the crowd interface that they saw So this is on a web page.

[00:12:03]
And so you have the phone over on the. On the left I guess. Jim I guess the world also doesn't like me or crawling. All right anyway so we have the phone based interface here you know some instructions in the middle one thing to point out about this and so this is it you know this interface could be embedded in Mechanical Turk We primarily use workers on up work which the different crowdsourcing platform.

[00:12:33]
You'll know one of the things that you may or may not be able to read the text in the middle there but. You know one of the things we wanted to avoid was having crowd workers record their p.i. as part of the traces so we provided them with you know a name to use in an e-mail and we had some detection sir aspects in there to try to catch wind if crowd workers enter the wrong thing accidentally.

[00:13:01]
And so basically you know so we would put prod workers in front of this there were some challenges in terms of like you know just make sure the workers had enough kind of to the. I think another big challenge that came up is while proud workers could get through the challenges that talked about before like logon screens like complicated forms they were not that great at completeness so they would kind of explore down the regular pass and maybe get a little bit outside but it didn't see you know they were often lose interest I think by the time they got to to really where the edge is for so those are some of the challenges that we dealt with.

[00:13:42]
And so that's what led us to the approach that we finally use in the end which was this kind of hybrid crawling mechanism and what we did was we started with human crawling so we had human start they would get through the majority Well maybe not the majority but certainly a lot of the user interface get through many of the hurdles and then you know we would keep track of where they ended up and then we would run our automated system and then continue the crawl from there and so you can see here in this chart.

[00:14:16]
This of the white bars are where the humans got to you and then we add the automated calling on the end and you know in a case of many of the apps that are used here as examples you know we were able to get. A pretty decent amount of additional crawl out of out of that exploration and you can also see the light green bar so there are just what the automated crawler would have done by itself and in a number of these cases you know the automated caller barely got anywhere if just used one so this is interesting one thing I would say about this is we still don't really know how complete the crawls are I mean we don't have ground truth for whether we covered all of the screens or not we just know that we were able to get a lot further with the automated crunk.

[00:15:02]
So kind of as I mentioned before but these are the there's some overall overall results of the crawling process I mentioned before that we've got you know a few more than 70000 unique screens from not quite 10000 apps we use 13 workers these are all from up work about a year's worth of crowd work in there 2450 hours cost us almost $20000.00 to do this.

[00:15:32]
And. And you can see that we've got you know apps from quite a wide range of different categories different rating levels things like that there is certainly a long tail of number of user interactions and number of clickable elements. That we saw in the course of the crawl you know and then this is just a random process of you know a bunch of the interfaces that we that we collected over the course of the of the process of course there's 70000 of these so that's only about 12 or 18 I guess.

[00:16:07]
So now I'm going to move on and talk about our uses of the record dataset so there is kind of 3 explorations that I want to talk about the 1st one we looked at trying to use you I layout patterns to retrieve different you eyes from the dataset I'm so this is actually work that was in the paper that we published about Rico that with 2017.

[00:16:30]
And so the idea here was that we trained an audio encoder. With you I layout so I'm not familiar not sure how many of you are familiar with how cultures work but basically the idea is that you start with a queer u.i. And so this case we took our we took user interfaces from our dataset and using the view hierarchies and also a little bit of human annotation we turn those into these layouts that are represented as an image so where there was an I think how this works is when there was a non-tax elements like an image you get a red box there and there was a text element you got a blue box and then you train the I don't code or by basically you know this modify this transformed image is the input and you're trying to get its output to produce the same image so we train this across.

[00:17:28]
A bunch of items and I guess you know in the case of the other coder right this is the network these are the pretty wide network you go down to a small a small layer and then back out to a large network. And the goal is to pull out this dense representation of what this looks like and then we can use that representation to try to find other things that are approximate to that in the space.

[00:17:54]
And so here are some results from the Saddam coder and so that you know we train with 90 percent of our data use their manners and how they should set. To ground her impacts to get through this which is about 5 hours and so then you can see here on the.

[00:18:10]
On the right the kind of the results so we started with. So the query images are the images and there's a transform of those which is just you know non-tax text elements and then the result images you're seeing next to that are other images from the data that that kind of closely approximate those layouts and so you can see that.

[00:18:34]
In general you get pretty similar patterns but in terms of the overall layout but the interface is themselves are also are fairly different you mean you see some icon differences and so overall this I would say worked out fairly well if you're you know you're interested in trying to find similar data across the dataset this definitely can can work and you know it kind of shows a way to begin thinking about using the Rico dataset.

[00:19:04]
A 2nd project we did with something called Swire sketch based u.i. retrieval on so this is work by Forrest Long who is a another intern with me a couple summers ago and he's currently at u.c. Berkeley on this papers published a car last year and so this idea is somewhat similar to what we just saw with the auto encoder but instead of using images as our Querrey instead we're going to use sketches as our query and some the motivation for that is that you know sketches are pretty effective for communicating design ideas obviously you can sketch something up pretty quick you don't have to you know get the full thought ality.

[00:19:43]
And it makes sense that you know we can kind of use this kind of sketch based method to drives a retrieval type of application. And so this is basically you know how that works so we took the sketch. We had somewhat similar to you know kind of a deep neural network in betting which gives us a spot in the space but we also train with images and basically be idea is to get this this imbedding to represent sketches and images in the same space as close to each other as possible so given a sketch we can get an image.

[00:20:21]
So obviously we needed a sketch dataset to go along with the existing record dataset in order to do this training. So yet again we hired workers on up work. And we created they created. Almost $4000.00 sketches from about 2000 examples so we had 4 designers but we overlap the work so we got you know generally speaking about 2 sketches per screen which about 60 to 70 hours per design or this data set is also and it was probably clear but I don't think I said this you know the sketch the things that they were sketching were chorused corresponded to you know existing screenshots in the record data set so that we could do that training.

[00:21:08]
We did as you probably see there were certain things that we did here. To make the day so little more uniform so we did not ask the designers to like sketch images when the images appeared in the u.i. Instead we had just had them represent images as boxes with a cross through it this you can kind of see in that.

[00:21:28]
Left image and also for tax either we asked them to write more maps some type text or to just use squiggles to represent text we didn't want you know specific words from the you eyes to show up in the sketches just to avoid you know that kind of crossover in the train.

[00:21:50]
So so that was this that was this some of the. I guess this goes in to to more detail on that right we gave them a screen shot they made the sketch we actually asked them to do it on paper and we use these little tags so that you know basically they could just sketch it on paper take a picture of it with their phone and then we could post process that back into you know or appropriately normalized image you know to do our comparison.

[00:22:18]
This is the network architecture and training that we use so again you know we have the sketch example now you and we have it's corresponding screen shot and then we also would randomly choose from our data set another screen shot. Which So it and we treated that screen shot as an irrelevant screen shot so there's enough diversity across the dataset that you know we randomly picking up the screenshots very likely to be very different but we didn't do any additional work to like ensure that I mean we're training over a lot of examples here so.

[00:22:52]
In the end the idea was it worked out. And so what the last option that we used then was you know we were attempting to minimize the distance in the embedding space between the sketch example in the corresponding screen shot while simultaneously trying to maximize the distance between the sketch example and the irrelevant screen shot.

[00:23:12]
And so the goal here is just trying to kind of separate you keep things that are white close together but try to separate things that are different out across the whole history. And so this these are the results that we got from the us we compared to a few other.

[00:23:31]
You know methods with our method we were able to get you know the original screenshot to appear in the search results in the top 1060 percent of the time and it was the top one screen shot about 15 percent of the time we were pretty happy with that result certainly I think you know if you're if you're into doing more sketch based retrieval you can probably improve on this result but.

[00:23:56]
I think it worked out fairly well these are some examples of. Of retrieval so you can see a like a sketch of. You know a list based interface obviously is is able to retrieve a bunch of different looks based interfaces. This is an example obviously different example again where you can see you know it's using kind of.

[00:24:20]
The image and Button aspect to you kind of see that those ball show up even though you know these the screen shots themselves certainly appear fairly differently especially with the big background image and the one. And then we so I don't have there's a lot more work that would then went into this but then we also experiment not just with having to sketch the full screen but also partial screens and some other things.

[00:24:48]
So next I want to talk about another project we did. This was more of a examining the use of material design a crossed the Ricoh dataset. So I don't know how many of you are familiar with material design this is a design language there was developed by Google I think we released it in 2014 or 2015.

[00:25:11]
And you know sense it's kind of become the main design language across Google a 2nd version of material design was released a year or 2 ago. But one of the one of the key questions you know that we had and that the material design team just in general has is like how how much is this design language being used by other users across you know the entered apps or is a question they really can't easily answer and certainly you know there's a lot of reviews of material design people talk about it and there's a lot of positive talk but there's also some criticism.

[00:25:47]
And want to answer this question is material design used and does it make apps better and so you know our goal then was to take our data set figure out ways to detect the usage of material designed crossed it. And then you know kind of examine that usage in the context of what how our apps that use material design rated how many times of the download things like that.

[00:26:15]
So we started just by trying to detect 2 specific elements which are relatively unique to material design one is the floating action button and the one hit the floating action button I thought I would get more hate. So the following action buttons this little circular button in the bottom if you can't remember how to start composing an e-mail and because it's floating action button.

[00:26:45]
I still do this time. But so there are a lot of floating action bugs those one of the things we tried to look for. In the interface the other thing is something called the navigation drawer so usually there's like a little hamburger menu in the upper left corner if you press that you get to drawer that slides out from the side it's often where set up things are a counterweight things are things like that.

[00:27:07]
So 1st we tried to see if we could just figure this out from the view hierarchy data alone. So that would have made this task much much easier if that was possible but turn out didn't work in the main reason it didn't work is that a lot of people don't use the there's an official toolkit for implementing materials I knew I wanted people don't use that and there are a bunch of unofficial toolkits the main reason I think is because the official toolkit you know only works for certain versions of Android If you want your app to work for older versions you have to comp with a different solution.

[00:27:44]
So as a result of just looking to the higher queues it's hard to figure out like is this a floating action button is this a drawer etc. So there are a lot of false positives those techniques so as a result we then fell back to attempting to use a machine learned approach.

[00:28:02]
We trained a network with a bunch of crop images of you know of known components. And were able to actually do pretty well which about 95 percent accuracy for identifying both the fab and the drawer and we did some follow on work with a few of the other elements as well.

[00:28:24]
So that these are some of the results that looking at apps that used the floating action button. And so what we found quite interesting I think is that higher rated apps seemed to use the Fab 4 more often than lower rated apps. Although I would say that there wasn't a significant.

[00:28:46]
Difference between the 2 you can kind of see obviously there those 2 box plots are fairly close the navigation driver results were similar to this. So this was like it kind of an interesting starting point. We also looked then at. Then looking kind of across apps that use material design we looked at correlations between you know usage percentage and the average rating percentile of apps so we found that usage of material design was highly correlated with reading percentile that's the top graph there.

[00:29:22]
And the downloads were somewhat correlated as well though not clearly not quite as well correlated as usage. And there's a so there's a paper about this it was never published in a in a peer review conference but it is on archive if you want to get more details about the work but.

[00:29:45]
We are just climbing a correlation I think there are a bunch of. There are a bunch of potential confounds here at think in terms of like it was for causation because you know our. Google Apps higher rated apps for example like you know Google Drive and all the related ones that certainly they are using material design.

[00:30:12]
May be maybe that's why. It yet so I think we don't have a good answers for those things so it's hard to hard to say. Right now. Are. You all or. I don't know the I don't know the answer to that question I assume that updates or don't count but I but I'm not certainly.

[00:30:41]
Right well that's true that's that's a good point I don't know how that's determined either. Yet so. So this is the the results we came up with Mathilde design team was happy. All right so now I'm going to talk a little bit about Rico in the world so all the projects from here are not my work I want to emphasize that this is work by other teams that have picked up Rico and used it for various purposes I just think it's kind of cool to see you know what what other folks are doing with the data.

[00:31:18]
So one piece of early work this from. And Ross James for very a bunch of other folks at the University of Washington they looked at. How image based buttons are labeled for accessibility in Android apps so they took the Rico data set they found a bunch of cases where there are image buttons so these are buttons that are just represented as an image like that watch e.s.p.n. button I think it's a good example of there at the top.

[00:31:45]
And they looked at you know what percentage of the time are those but labeled. You know and kind of what are other issues that that exist around these I think maybe this isn't super surprising but you know there's obviously very clearly by mobile distribution here of either you label your buttons or you don't and there's not a lot of not a lot of apps in the middle where they're only labeling half their buttons or something like that.

[00:32:13]
Perhaps not surprising but good to know. Another paper that also looked at kind of. Just like how our our interface what like what are the content content of user interface is this app looked at why good features are this paper looked at why it features across a bunch of smart apps so they had a number of questions here like just how widespread our log is log and you know what's the relationship between downloads and log and things like that.

[00:32:46]
What I thought was interesting is that. They found that only 32 percent of the apps in the record is that how to log in future that's actually surprising to me I guess maybe it's because when we were crawling the record data set logging was like our big thing that was hard to do and so it just seemed like it was something that occurred all the time but apparently not.

[00:33:09]
You know and even more surprising I guess is that it was even a smaller number of apps that use you know a social walk in like logging to Facebook or. Google or something like that. So I thought this was interesting kind of a nice you know other use of the record to set to just understand you know what are you know what's going on across the app ecosystems.

[00:33:33]
This is work. You know our collaborator on the Rico data set who was interested in looking at you know design semantics for mobile apps so you know as we talked about before you know the record is really just about the designs of the apps the content of what they're what's there we don't necessarily understand any semantic aspects about these interfaces at least from the record data set itself so you know is this a search interface is this a logging interface.

[00:34:08]
Things like that so so they took they took the record is that they use an open coating to identify this electrical database that's shared across apps a day down a 525 kind of common types of u.i. components a number of different button concepts and a router icon classes.

[00:34:28]
And then they use this label data that they have gotten to that open coding process to learn some models of they're able to then detect those semantics in the dataset so they're able to. You know basically pull out a lot more semantics about what's in the day will set and they were able to assign a manic label to about 70 percent of the visible non-redundant elements that are Enrico So that's certainly a nice additional contribution that that goes beyond where Rico is I think with the still doesn't tell you necessarily it's like what the tasks are and what the steps would be to accomplish the task.

[00:35:07]
But it gives you a sense of like this but these research and this button is used for something else. This is a paper that's about to appear at this year's Chi called good concert this is a tool for gui design that uses the Rico dataset as kind of one of its.

[00:35:28]
Inputs and so the idea is that the so this is actually for a completely inexperienced u.i. designers people have never designed a u.i. before at all. Which I think that's actually a really important use groups. Because I suspect there are a lot of people who use phones a lot but they've never actually built an app and then so they want to build an app and then they design the u.i. and then they're like wait I don't know how to do this at all.

[00:35:56]
And so what that what these folks did they you know they took a standard u.i. builder and then they added a bunch of additional metrics and tools that in August designer can use to kind of understand how their interface is being perceived including like a heat map of where people are likely to have their attention and the number of metrics that are kind of in the middle.

[00:36:21]
Top there and then they use the Rico data set as a kind of a set of examples that users could search through and use and then also extract templates from the example and bring it into their u.i. builder so they could kind of build around that template. Which I think it's it's a pretty cool tool and there's a lot of interesting little aspects here although obviously there's a lot to look at I think when you look at the 1st time but.

[00:36:51]
This is work. From Google and but also. Who's here you know a professor here at Georgia Tech as well so this is one of the 1st examples where we're seeing the record data set used to generate new design or in this case I think evaluate the design of the of the system so what this is and does is generate layouts for graphical design applications given the set of elements that you want to include in a set of basic constraints and so you kind of see an example of that right it's like I want to headline 2 piece of text into images and then I you know and then specifying some basic layout constraints for where those things should could should appear.

[00:37:33]
And basically they're able to turn this into what they call a neural design network which kind of models the constraints and then they kind of iterate over that that networks to. Basically to figure out the unspecified constraints and then to solve the design problem. And this I think this paper is either in submission or accepted to see the p.r. So it's coming up soon and you can find it on archive in a case.

[00:38:02]
And the experiments that they did demonstrated that these generated way out to visually similar to real by people so that's seems like a useful valuation. And so in this example I think I think there is one or 2 of these examples here on is a system where we're where Rico is being used as kind of a basis to understand a u.i. and then to help automate.

[00:38:30]
A user interface so this is a program by demonstration system where you can ask your device to do something if it doesn't know how to do it then asks you demonstrate it and then you know through the demonstration is able to attempt to automate that process later on.

[00:38:48]
And you know so the from as part of the system they train models to understand you know you I am and where that where they might be and to address challenge you know issues of what the interface changes how do we make it robust to that. And also I think that there is in the Re could decide a little bit to make the demonstration generalizable I believe this paper is going to peer at you why 2020 in a couple weeks so a month or so.

[00:39:18]
Mike still working Ok. And then this system also very related I think so this is a an approach to doing automated Android app testing yet again using the Rico dataset as a way to kind of understand you IOW Mintz and then in this case be able to generate new test input generators for apps that are human like so the idea is instead of you know having your testers have to figure out how to program every single u.i. test that they want to do the system's able to automatically generate some of these tests.

[00:39:56]
And you know and then of course this can happen as your u.i. changes you can generate more tests or modify your existing tests to address that. Yes I think the system's preexisting as well yep so I guess. Going to a faster those are the systems I was going to talk about today just to wrap up you know some of the time I Rico which dataset you ice cream Mehta data I think it can be used for a lot of things you've just seen how a lot of different people have used it.

[00:40:28]
And you know I think it's open some things out there that you know you can go download. And also step separated into a bunch of different parts so there's you know a bunch of metadata aspects which are relatively small that you can download I think the full data set something like 60 gigabytes but.

[00:40:47]
But there's a lot of screenshots so it's not so surprising that it's that big. I think one thing that we're working on I mean you've already seen some of these examples there are different teams that have augmented the dataset with a different piece different aspects over time there's the sketch data set.

[00:41:03]
And she says semantic iconoclast work and stuff like that these augmentations are often available as well we're trying to figure out a better way so that people can add augmentations to the data set over time and we kind of contribute to making this a useful thing that that continues to grow.

[00:41:24]
I think one thing we've learned specially from talking to a lot of people about Rico is that there is a real demand for for task related data and kind of ecologically valid interaction traces and things like that that's something we did not try to collect here was something that we had always kind of thought about as future work I think definitely something that we're going to be thinking about more going forward.

[00:41:49]
This is about 3 years old now we collected this in 2017 so we're looking now in fact into doing a 2nd round of data collection when people start probably with trying to crawl up all the apps again that were featured in this dataset We'll see how many of them still exists I see most of them well but will be interesting to find out.

[00:42:09]
And I am interested like if if there are other things that you would like to see in the Rico days that go on for certainly be interested in getting your input maybe we can add some of that stuff and. You know and obviously as a mentioned will continue think about how we can add tasks based data collection into into our process going forward.

[00:42:31]
So I just want to acknowledge a bunch of the people who work on this and mention the Kemar who. Who worked with very closely on the status that her students many of her students who have been in terms of Google on the way especially because we did most of the work on Rico for us to the work on Swire.

[00:42:52]
Josh also in turn that worked on Rico. And a bunch of other folks so I think that is all I have for now thanks very much for being here pay attention I really appreciate it. And I think we have plenty of time for questions definitely yeah. Thank. You.

[00:43:26]
Very. Yes really interesting question. I mean obviously so all the Rico was and was recorded entirely on a real phones that were you know remotely controlled by a crowd of workers or by not a minute system but that these were all on a desk in Illinois and you know it no point to these devices think that they were driving her or her up or anything like that at least I would assume they did so if these apps have context sensitive features that like change the u.i. based on activity we certainly did not see that here I guess my intuition is that there aren't too many apps that do that but maybe I might be totally wrong.

[00:44:14]
Was that yeah that's a good point here. We did not have a way to fake rotation and yeah I mean there are some apps right that kind of force you to use it horizontally which would have been super annoying in that crowd interface because you definitely could rotate the phones he would then.

[00:44:32]
It's I don't know how much that came up with that would be an interesting question so to look into the to decide I think I'd be interesting to add that as a. Force the phone to think it's in driving mode or something and see what happens. The questions are right or.

[00:44:51]
Wrong or didn't work and then roll. You know that you know and I think probably you. Have 4 years ago. Or met. Together. Because it went right which means you have. Already thought what yes to having. A man. That's right which is what you. Write about me probably.

[00:45:22]
Right well what we had from the Yes We got what we called the you hire he is out of Android and so that is a feature of the Android framework to look at whatever you want to call it. So we certainly knew like this is a button this is a this is an image I forget what exactly the abstract level is there but.

[00:45:47]
But what with his work what what she extended that she was a wide range of kind of like semantics on top of that so like a search button versus a. Trying to think of the other examples that are there. Are just a lot of what write about and yet exactly exactly how to write a book right yeah and I believe that she also was able to identify wake semantics of you ice creams as well so like this is a log in screen versus this is a search string versus this is you know a gallery or something like that but we're going to.

[00:46:25]
Get you through this already. If you get permission I guess if you want to do it. Right. I guess what we found is that. You can get a certain level at scale or you get a certain way a certain portion of the way there. Just on the basis of what the tool kit gives you but there's there's enough dynamic behavior hidden behind that stuff.

[00:46:58]
That. I mean that was one of the reasons we needed humans to help us and I think that's also why we'd like to collect more task German data like ideally I'd like to know Ok I can reserve a restaurant with this app and then ask a crowd worker reserve a restaurant with the app you know in c.v. exact steps and you know and have some maybe ask multiple people and see what the variation is in their interactions So what about other questions.

[00:47:30]
That really I just did here result here is going to be weird when you're telling a story and I was expecting to see they had a really hard time finding things they were looking for those things through my head if you ask me that this gets. Like fine nail right.

[00:47:51]
But it looks like up the hill you know there are so like fine nuance difference that right in the ill at it from the Google Docs you're all your documents from the red they're filled All right I was curious we were there certain to get artifacts that people created there were like the and it really expressed it with Zach Lee this is the birth of you know the thousands that were kind of like that right.

[00:48:22]
I'm trying to think if there was an example like that that was particularly. Insightful I mean I think our use case was more of the like generating examples for inspiration so I don't think we definitely were not thinking of you're going to do this sketch to find exactly this out because if you know exactly the op you could type in a name or something else right but.

[00:48:46]
I'm trying to remember I would have to go back and look at it again I think that there were some unique. Out you need to visualize actually the embedding space and see like what's here what's there we didn't do that so yeah I think it's a question. That.

[00:49:08]
I hear or read or very I guess I would say all of the above. I think I think I'm more worried about apps aging than I am about device changing I mean although the devices have gotten like there's a Glee bigger I don't think that the u.i. is have really changed that much.

[00:49:34]
Especially I mean you know one of the things we saw in the material design study right was that a lot of people are trying to make Android apps that work for a.p.i. levels that are quite old and I mean there's a whole challenge with Android in that people are generally not able to update to newer a.p.i. right so if you have no bones you probably have very old version of Android.

[00:49:56]
So as a result I mean maybe the interface scales up but it's not appreciably different whereas like you know there's been at least 2 versions of material design that you expect that the material design is getting more I don't know this to be certain but I expect that it's showing up more and more new eyes so that so wouldn't surprise me if.

[00:50:18]
The Absalom cells are changing in design even if they're you know not really the sizes maybe less of an aspect but I don't know would be interesting to look at. Other questions so. That. We. Could. Figure out. Where they're. Going to hear it starting. Or here. That you.

[00:51:02]
Really think that you're. Right. But that you should be aware that if you're. What I think you're. So yeah I mean we would definitely like to make it easy for external folks to augment the ricotta set with spending in Titian's or or even if you know people want to do new crawls or something and add to it I think I would be great I think.

[00:51:41]
The main challenge so far has been how to support that I mean basically the Rico Doucette is just in a cloud bucket somewhere it's not you know it's not dynamic in in that respect. And so when we've added annotations they've they've been stored somewhere else like the Swire do set is on get home randomly.

[00:52:06]
And so I think really we would need to make some kind of. Website or something that allows people to like that data and dynamically we just haven't spent the effort on that yet but I think it would be something worthwhile to do. I guess any time you're adding this kind of user generated content aspect I guess we'd also want to think about how to police whatever gets out is probably not a lot of people with add stuff trouble fully a minor concern in this case but I don't know.

[00:52:46]
That. I can tell you. Every year your track record. When your team or. Your. Club is right. For the 22nd. Job. Like. A research project. Well I mean one thing I've been working towards is trying to get the trail design to fund this on a continuing basis so that we would be able to do that I don't think.

[00:53:42]
I mean there's just been various things going on with that I don't think the money is. Committed it's not an insignificant amount of money but you know it's not it's not it's not $1000000.00 So. Yes certainly we would like to do the temple now says that you talked about things like that I think there be a lot of value there so.

[00:54:08]
I guess the other challenge is. You know finding someone to to do the work as well right because it certainly has moved on and you know it's kind of incremental amount of work it's not a Ph d. thesis anymore so. So there is a challenge there as well.

[00:54:30]
And the other questions. Thank you very much your own really appreciate.