Thank you so much on here and
Henrik for inviting me Rishi is that.

So the lesson to be learned just don't
start with philosophy do that later.

What I'd like to talk to you about today
is something that I think is going to be

really really important
area in robotics and

human robot interaction they'll be a chest
starting to get our feet in feet in what.

It's called open world robotics and
open world to human robot interaction and

I'm going to use this opportunity to
actually quote a politician I usually

don't do that but in this case there's
a really perfect fit close to our H.R.I.

is really about the known knowns
everybody knows where that's from

Rights Donald Rumsfeld made this really
interesting conceptual distinction

between the known knowns and the known
unknowns and the unknown unknowns and

this is the known knowns in the following
way read typically even be to H.R.

experiments and we both architectures for
H.R.I. we know the aspects of the task

we know all possible task based human
based variations in a setting like this.

At the beginning of the interaction
right so there you know.

There are no surprises and if there
surprises you eliminate the subject right

that's it in particular what you
know is what kinds of interactions

that could take could take place what
kinds of language interactions you can do

what kinds of gestures you can recognize
whether U.P.A. attention to I guess or

not read if touchings involved all of that
you know what the goals are the robot

knows exactly what it's supposed to
do in interaction what to search for

maybe doesn't know exactly where it is but
in those what to search for

how to get help we know the environment
ahead of time typically So

those are all examples of what I want to
call closed world H.R.I. the necessary

representations and the necessary
knowledge items to perform the tasks given

to the system at the beginning of the task
if you don't then things can of course

happen it's just him can learn what I
want to contrast this with open world.

This is the known unknowns and

I don't want to go to the unknown unknowns
I get back to that at the end but

the known unknowns other ones where
you know what the system doesn't know

which the system has to
learn during that task OK So

in this particular case some aspect of the
interaction is not known ahead of time and

some of the knowledge that the system
needs in order to complete the task

it will have to acquire
as part of the task.

Open World H.R. and
therefore requires an architecture that is

evil to deal with unknowns so
that's not easy and not a given and

I hope to show you how that's complicated
you know a little bit has to acquire

knowledge on the fly and then use
that knowledge to perform a task and

acquire it in using different
means could be trying to.

Observation discovery data mining
from explicit instructions and so

forth we should in general also not impose

any limitations of what can be acquired
right you might want to learn about new

objects that can perceive new events new
primitive actions new complex actions.

Maybe not only that maybe also concepts or
rules or

snore Mz or etiquette trite this should be
no limitations of what we want to learn

here and critically I believe
open world dealing with

open world situations is a prerequisite
for the deployment of robotic systems in

a lot of applications or means because you
will just not know all the things that

there will have to face in that context
just think of a household robot.

Now what I'd like to do in this talk is to
show you that open worlds make things all

of a sudden even harder you know robotics
is hard but to make things even harder.

Consider a simple task in which a robot
gets instructed to retrieve an object

that it does not know about in an unknown
environment several questions arise for

example how can the robot even understand
what it's supposed to get if it doesn't

know the word for that object nor
what the what kind of object it is

how does the robot then know what to look
out for if it needs to find it how can it

make a reference understand the reference
to a location that it doesn't know about.

And for the folks interested in planning
how can we even formulate the goal to

look for
something given the way our planets work

that make reference to
objects we don't know about.

And then when we finally get there and
we find it then we detect it

how do we know how to pick it
up in this particular case and

retrieve it right how to
manipulate the object that we.

Have no knowledge about.

So let me give you to start off
a motivating example from Urban Search and

Rescue Task this has been sort
of a motivating backdrop for

a lot of our work we are interested in
robots that can be instructed to perform

activities in this context and I want to
play a video of a robot that can be tasked

online in this open world setting and
then I'm going to talk in detail about.

What's involved in doing

so.

All right so this is about a dialogue ends
but notice the robot got an instruction

it had to understand instruction it had
to formulate based on instruction a goal

as its formulating the goal it realized
that it didn't know what to do

right to recognize a medical kit so
it had to ask for

more information to get more information
about it from that information it had to

configure itself in a way that it was able
to recognize things like made kits and

then on top of that it has to figure
out how to pick it up now in this

particular case we actually cheated in
that we told the robot the head of time

about things that could be picked up with
things with handles that could be picked

up by sticking your hand in that way
nevertheless it had to find the handle and

had to really pick it up OK So this is
the kind of scenario we're talking about.

I'm just going to let you watch it for
a second but in this particular case.

The control was actually bit difficult
because the Segway has ever worked

with the Segway knows that is
very wobbly you don't get to.

Access the control system so it had
all sorts of other challenges too but

the robot picks up the MIT kid
retrieves it and accomplishes the task.

All right.

So looking.

At this.

The robot was able to understand
instruction right in this particular case

with an unknown word made kit and
infer what kind of object it is

the word could be an adjective
fall you know red but

in this particular case turns out to be a
noun What are the challenges in that case

well you have to recognise it in
the speech as a first and foremost.

And then you have to infer the syntactic
category in the parser and

then you have to generate a meaning
expression of the complete utterance

that has sort of a placeholder for
whatever that thing is

at least to the extent that you can use
the semantics to do something with it.

Once you understand that you have to
learn how to take that in this particular

case to get at the scription but that the
scription now still requires you to figure

out what kinds of perception mechanisms
you have to could potentially detect it

and then you have to put them
together in a way that you generate

this new object it takes there
based on what you've got.

And you have to configure your
system to actually run it right so

you can look out some eight kids.

To resolve the reference
an unknown location.

We have to perform references solution
without assuming that we know

all the involved reference.

We might have to hypothesize new entities
as a result of that expression and

these could be objects and locations and
then if those are locations we might have

to connect them to an existing map even
though we know exactly where they are but

in a way that when we might
then explore the map and

explore unknown locations
we can identify them later.

And we have to formulate a goal to
search for me to get it watch out for

it's doing task execution and that's
tricky because we now have to introduce

new objects or object types into the plan
to model the plan to get started.

It was a particular model and now there's
a new object if to do that online and

we also have to formulate the goal in
a way that doesn't assume the existence of

something right for all we know there
might not be a Med Kit around anywhere.

And we have to generate monitoring actions
that look out for that kind of object.

And finally have to determine how to pick
it up and retrieve it and the challenges

there are to examine if you see the object
all the different possible casts that

you could use on that object and then you
have to select the appropriate ones and

generate a trajectory to them and you have
to monitor what you do in volume picking

it up to make sure you actually holding
it and then you can retrieve it.

So there's lots of interesting questions
related to just making a simple

instruction such as go get
that object open world.

So what I have worked on open
road robotics now for a while and

then different aspects I just quickly
want to give you a very short overview

without going into details a lot of our
work was focused on tasking in natural

language is giving it goals and having
dialogues about them did some hope and

planning work and that we'll talk
about that open road reference or

solution I'll mention some of that on
the learning of new perceptions and

I mentioned some of that as
well as learning of new and

primitive actions and
I'll mention some of that as well

not going to talk about learning
of new rules or plan operators and

briefly going to mention also
generalisations also learned actions so

that you can learn it for one context and
then transfer that to know the context.

So for the remainder of the talk but
I'll do is use this motivating example and

be sickly show you how we can address
these five aspects here understanding and

instruction learning how to take something
resolving references on a Locations

formulate goals and then retrieve
things so understanding with it.

And I'm not going to talk much
about the natural language but

I want you to consider very briefly
a command like what you heard in the video

command disease really
needs a medical kit.

And suppose the robot doesn't
know about command Izzy and

suppose it doesn't know what
a medical kit is it looks like and

it doesn't know where to find one or how
to retrieve one What is it that you would

want to ideally to understand
from that kind of sentence.

Well we would like the robot to understand
that if the commander has a need for

a Med Kit and
because it was communicated to the robot

the commandement have a goal
to have a made kit and

the robot should also have that goal
sort of command it to have it and

if the robot has to call for the commander
to have it it might have to have to go for

itself to get it for the commander and
the really in there indicates that

there's some urgency to it OK so we're
going to talk about the details of how

you do the natural language processing and
arrive at the semantics but

I want to point out to you is is that
senses of that sort it in general natural

language instructions have a variety of.

B.Q. it is infelicities that

a lot of people in natural process usually
find problematic but those are actually

the aspects of natural language that make
natural language to possible to people and

we have to deal with them so for
example aspects of natural destructions

are the truth of a sentence or
the generality of the rule right that may

not be a plea couple in all cases put
it maybe sort of a rule of thumb.

You might have to deal with probability
based assumptions with certain facts

obtain or not be a vacant in precise
terms if somebody says usually or

typically And
we have hedged statements or.

Present assume this existence such as
this should be but there might be or even

statement of the sort there ought to be
and so forth statements about obligation.

The question is this how do we
represent the uncertainties connected

to all of these forms and the way we do
this is reuse them to shift the theory

we are effectively instead of
representing a particular and

certain devalue with each natural
language semantics statement we

associate an interval all with each
statement where the true probability that

the statement is true lies somewhere in
the middle and the lower into of all is

a point of a Bond Three confidence
ponders on that into there probably

won't be able to talk about this here but
I can show you briefly

what's the semantic system does so if you
get the sense like command is a need for

medical kit this might get translated
into something like needs and

here's our confidence interval right these
are just America values between zero and

one.

If we assume in addition to the utterance
we hurt several facts and rules so

for example a rule that says if X.
needs why then X.

is a goal to have which we
can formalize as follows.

We can use that rule to make
inferences about what it means for

something to need something now that rules
are not generally true if you need a break

doesn't mean that you have to have a goal
to have it break right but typically you

might want to also write so we can express
that with our uncertainty interval and

we now introduce a variety of other
facts if for example command the Z.

is likely of higher rank than the robot
and usually if somebody is half

of higher rank than the robot and they
have a goal then the lower rank person or

robot in this case should have a goal for
them to have a goal can formalize that and

if the robot has a Goal for

somebody else ethical Maybe they're
all what should get that thing for

them right and again these are not always
true across the board then we can express

the fact that they're not always true
with the uncertainty intervals and it's.

You take all of these senses together you
could see how the robot might arise from

something that was not explicitly stated
as a command command Dizzee need some aid

kit at the conclusion that maybe
it should get the med kit for

the commander OK and the way this
proceeds is very straightforward

using modus ponens as
the inference rule and

be defined a special version
of modus ponens that works.

That preserve certain
logical properties so

these are our assumptions this is what
the robot got from the natural language

communication this is background
knowledge the robot has and

then in several inference steps using
modus ponens here using conjunction or

detection here using most points twice
again it arrives at the get four commands

either Mitt could so this is the kind
of inference process that can go on and

if you look at different types of rules
of how we confuse uncertain information.

You might get cases where the more
inferences you make the more uncertain

you become about what you actually believe
and then there are other fusion rules that

allow you to preserve certain properties
so in this particular case if you start

with into all assumptions that a point
into false so that each interval is just

basically like a probability exactly but
like a probability then using a particular

see if based inference that we use in
our model you preserve the point of all

that the uncertainty doesn't explode
whereas if you use some other inference

rules you can see that the in of all
gets larger and larger and similarly if

if you start with something that is not
a point of all it can potentially still

stay constrained as opposed to increasing
uncertainty over all this for example

means that you basically know nothing
right because it's almost between zero and

one and zero and one means you know
nothing so a robot that would use info for

five inference that like that would
always have to do something like asking.

Evidence that it could not really act.

So that's how we get to
the instructions and

now we know what to do how
do we detect them aid kit.

Suppose we get the information as we
did as the robot good in the example

it is a white box with the red cross
on it we can translate that into

a logical expression X.
is way than X. is a type box and Y.

is red and Y.
is of type cross and Y's on X..

And then we can search in our system for
possible detectors or

algorithms that can detect some of these
properties that is the can look for

things that can detect boxes and

process that is object to textures
right across could be both.

You might use a three D. detected it
take the box you might use in R.G.P.

detected it take the Red Cross use R.G.P.
to take the color and

then the goal of the vision system
the job of the vision system is to stitch

together the different detectors not the
vicious and should say perceptual system

stitched together these detectors as
prescribed by the logical formula and

then built this compound detected
that can detect things like kid.

Enforcing the constraint on that X.
is on Y.

and also notes that there's an intrinsic
ambiguity here between a texture and

an object because the Make It could
have been a three D. cross on it and

it's really a box right where it could
have been as read across on a flat

box texture so in that particular case
the natural language expression alone

is not going to resolve the ambiguity you
would have to see examples of it right for

the robot knows either one can work and
here's an example of how we do that

in a vision system we have an architecture
where we have a set of a veil is available

detectors and also trackers
the advertized to keep abilities and

then the vision system doesn't match
up in a logical formula comes in.

Looks for these matching detectors and
then performs conjoint searches that will

stitch together the compound object and
then we can track it.

OK moving on to resolving references
to possibly unknown locations

let's assume that we have in a distributed
robotic architecture different databases

where we store and entities which is
a very natural assumption in a robotic

architecture Typically you will have
a map the map stores maybe locations and

objects but you might also have an image
database where you store your images right

in the visions of them so those different
object to pieces here are just intended to

reflect that in a distributed architecture
of different locations that store and

to this.

Now will be proposed here in order to
do references illusion is to have a set

of consultants these are interfaces
to the general reference

resolution algorithm that abstract
over the details of what it takes

to recognize a location or represent
a location and recognize an object or

represent an object or recognize a face or
person right and provide a unified

interface to the reference or solution to
them so you can just deal with objects and

least recognition to
the specific modalities and

then what you want these consultants to
have a certain capabilities they should be

able to advertise what kind of object
they can recognize a location an object

the face should provide a list
of Tommy candidate objects so

we can look at complex expressions
that should give us the probability if

we make a clear either the constraint
holds of a set of entities and

they should be able to modify their
world model according to new information

we want to assert so if you consider
the expression for example my office which

is across the hall from the kitchen
in this particular case advertising.

Constraint might be a mapping component
yes I know how to deal with kitchens.

You can give me certain location
types that I understand

this is how I would access it and then you
can give me new facts of that form and

I will store those in my map.

So I do we do now open world references
solution we look at the set of consultants

we are given and we look at an utterance
like the one I just showed you and

the first parse it to pull out
the variables that we're looking at

the kinds of entities we
have in their exes and

office was a Halls is a kitchen
the ones across from the other.

You find an ordering of the variable and

the ordering of the variable basically is
based on the magical parsing the sentence

to see what depends on what they'll see
show in a second why that's important and

then we try to find a mapping from.

These variables on to the consultants
that can handle them and

it could be the consultant might
be able to handle multiple ones or

it could be that one could handle
be handled by multiple consultants.

And then here's the nutshell
of our probabilistic

open world entity resolution algorithm.

We look at this mapping and we find
the most probable variable assignment

from variables into entities that are
managed by the knowledge basis to which

the consultants provide the interface So
in other words everything that's mentioned

in the natural language expression the
kitchen the hallway right and the office

should find an entity if you do find
an entity assignment for all of them you

know you can look at which one is the size
the most likely want to go with that but

if you don't find it so in other words
if it is not above some threshold.

That is pre-defined then
maybe that entity is

not in our knowledge base right and that's
where the open world aspect comes in

maybe we have to hypothesize
that there's an.

One and
that's where you have to generate it.

So let me show you how to do that suppose
the robot has a partial map just like

that it knows about
three rooms one two and

three this is the hallway there
across from each other and it gets

a description of the sort Met Office which
is across the hall from the kitchen.

So the question is this how does that
map onto what the robot already knows.

And we consider.

Four different ways in which it
could map in the first case.

All known entities in the second case Z.

by are known but X. is not.

In the third case X. and Z. are not
known why is known in the last case

all of these are new places.

So let me go through an example in this
particular case we're interested in

proposing a variable assignment that
produces a probability about some

threshold and we do that by just
multiplying the conditional probabilities

of assignments given certain hypotheses So
for example you might assume X.

is a room one this.

And why is.

Room to.

Room three right and then we just look
at the relationship here between.

X. and Z..

If you multiply this out.

We have non-zero probabilities for
a cross and Hall but one is a room and

not an office so this is zero probability
and therefore the product is zero and

that assignment cannot succeed so we move
on to the next assignment in this case we

leave out X. and we look at the rest
of the expressions so for example here.

Again we make an assignment for Hall but
Room three is not the kitchen so

we get a zero probability and
that drops out.

And finally.

Assuming we leave out X. and Z.
and just look at why be confined by.

The two which is a hallway there's a
ninety percent ninety percent chance that

that's a hallway so we go with that
because that's great it's our.

Cutoff threshold so now we've basically
resolved one entity to an existing one and

we have to have pot the size two new
entities which we will do by attaching

to a new room somewhere at B.

don't know where we just know
that we have to posit them.

And then what we can assert is if we
believe the information we got that

four is certainly an office and
five is certainly a kitchen and

they're certainly across from each other
based on the instruction we got right.

If you don't trust your source then you
can also change that probability right and

at the end the system will still work.

OK So this is how we can deal with it and
we've in the past shown that we

can actually have a dialogue about
these locations we can say things like

they're connected to the hall and
we can potentially discover them when B.

drives down the hall and we have
a way to identify rooms and so forth.

All right how do we
then formulate goals so

we've dealt with how the initial
destruction we can detect the object

we can resolve references how do
we formulate a call to search.

I suppose we get the following three
instructions and they don't have to

come in natural language but suppose those
are the facts you know medical kits can

be found in rooms they are writing
called with the Red Cross on them and

the goal is to try to find
all of these made kids.

Who post this is trying to work with our
come about it is you but we've proposed is

to represent them as open world
quantified goals and I'm quickly going

going to show you how the school
representation corresponds to the fact.

You were given.

Can also be actually used the extension
cord if it goes so sense like medical kits

can be found inside rooms
will on the one hand tell

you the find the found will tell you that
there's something you have to look for and

to look for in predicate something of and
kid that can be looked for

in something that's R.
and R. is of the type room.

And moreover there's an interesting
aspect here that says

you need to perform a sensing action so
you can make the look for predicate

true right if you don't look for it you
will not be able to have to look for it so

this is what the sensing indicates and
it tells you that the variable in case of

type made kit so
this is part of the formulate the goal.

This is what I showed you before formulate
it slightly differently here we have

a clause here that tells you what has to
be true of that object has to be why it

has of a symbol on it and then finally
we get to CLOS try to find all of them

right we don't know if they exist we
don't know which room has them if any but

we can formulate it as soft coal that
the robot should attempt to accomplish but

doesn't have to accomplish right because
there might not be any and so we make

the distinction between hard and soft
goals where the hard goals are the goals

that you have to accomplish when you put
generate a plan you have to generate

a plan plan that meets all of the articles
but because we don't know if there is any

way to accomplish them because the amount
of intimate kids formulate a soft goal

where you get a bonus some
utility of one hundred say.

If you find that make it right so

if the plan opportunistically finds a past
that will let you go to someplace where

you could find a med kit it will accrue
additional benefit and by doing so

we can plan at the very least
up to the sensing action.

And then let's.

You know and then the planet can wait and
see what happens when the robot senses and

updated state.

So what we do is when the robot
finds a room off the right type in

this case anyone will do then the planner
automatically after the State Update

will insert a pseudo object
a runtime object made kids

that's them at the kid you want to find
in the room we don't know if it exists or

not right but we just put it there the
moment we put it there now we can plan for

it now we can say if there's a mint kid in
there of course you go into that room you

look around right for the right kind of
object and then you will have found it so

the plan in addition to generating this
hypothetical made kid puts in things like

properties of the made kids what it
expects to have happened afterwards right

and they call it has to look for
the made kid.

If the robot can afford to go in and
search it will go in it will look for and

at that point we get another state update
hopefully we will see a Med Kit and

if we do see a Med Kit right then we can
eliminate all of these objects that we

generated the fake objects and replaced
them by the actual perceptions we had and

we do that also if we don't find any made
kit in there right we removed that as well

and so by hypothesizing that there
could be one allows us to plan in

a closed world fashion
using a standard platter.

All right so.

Coming closer to the end here
suppose we found a Med Kit and

now we need to pick it up how do we do
that there's multiple ways in which

we could learn actions online you could
do physics simulations we can look at

human demonstrations we could use Cross
are saying this is multiple different ways

will be if you don't know what to do
we could learn it right there and then.

In my lap if Explorer two
different ways of doing this

one is an explicit verbal instruction
tell you how to do it and this.

Of course you can understand natural
language expression and so forth and

then we generate a control off from it or

I give you explicit instruction and
explicit demonstration.

Which in this particular case wouldn't
really work because there's no human

around say so the first one would work but
the second one might not work because

you can actually watch a human perform it
you might be able to look at them a video

on youtube and see how to do it
rather than and infer from that.

Or we can actually use some
sort of mental simulation and

beef looked into using existing
knowledge of actions and

seeing if we can generalize the action and
adapt maybe it will apply to

a new situation as well and that's
another way of how we could do it online.

So let me demonstrate explicit instruction
really quickly Suppose I give you

a command such as instruction such
as to follow somebody means that you

should stay within one metre of me they're
all but doesn't know what to follow means

it gets that instruction Well we would
like then is assuming it knows all these

terms we would like it's to immediately
be able to derive a control and do it.

So what we've demonstrated in the past and
again I'm not going be able to go through

the details of the natural language
understanding here the robot will generate

a very simple script that looks like this
there's an actor there's another actor and

follow means that you make this truth or
there's a maintenance goal

of this expression that this sense of X.
and Y. should be less than one and

this is something that the actual
interpreter can run and

the robot can do that immediately.

Lips.

And immediately that would do it OK So

that's one example of
an explicit instruction but

of course as lots of simplifications here
it's not our general notion of following

right if the person we had to run off
real quick there really were no I would

have no idea of how to maintain that right
it would have to do problem solving but

it gets across the basic idea that you
could learn it from instruction alone

assuming you know all the primitives.

A more complex way of doing a more
complicated we have doing it where you

might be able to learn things better
is when a human actually demonstrates

an action and at the same time narrates
it because you get in a way the benefits

of the natural language if combined
with vision in the appropriate way and

here in this particular case you want to
teach a robot to pick up the Med Kit It

doesn't know how to do it and then the
person gives it sort of simple step wise

instructions put your hand above the Med
Kit with your palms down like this close

your hand lift your hand right it's very
simple straightforward algorithm but

notice that the robot has to understand
that your hand in this particular case

refers to actually DID demonstrate his
hand right as to that it's the robots and

the robot does it but
while the demonstrated demonstrate it

demonstrates it it's
the demonstrators hand right so

there's some interesting complications
of perspective taking here.

And put you with your
palm down down like this

you have to understand what the like this
refers to that refers to the Palm and

not some other aspects.

And then at the end of that instruction
you would hope that there are about

is able to pick this up.

Here's an example of that.

OK So at this point the robot has a simple
model of the Med Kit It needs to put it in

its perspective he needs to find the
relative position of where the handle is

that it observed in its coronet system
right and then planets ejector and

this is a little slow because of motion
planning and the period two because we're

very close to the edge of the table so
a lot of the objectors don't work.

But it eventually will get there.

And then.

Simplifying assumptions for
example that hand really maps onto gripper

right that beat that this grasp
is really the pinch grasp and so

forth.

Because it understands that words pick up
right and it tries to retrieve a script

a skill and it doesn't that fails
right and then it knows so it has.

You don't need a lot of reflection for
that kind of stuff but

in general it actually is just
capabilities that are largely noticed and

things of that sort.

All right so

now suppose the robot has something like
this would be nice is to take the robot.

Learn how to pick it up and then learn
that it's not restricted to a Med Kit

maybe a suitcase will work as well right
because it's got enough features similar

to the Med Kit it's going to handle the
color doesn't matter and then potentially.

We want to say well if you.

Do this pick up maybe you can
pick up all the objects using

different kinds of pick up action suit
because in the end what matters is is

that the thing is in the air right.

So if the robot could go from
here through that sequence and

generalize that would be nice because
if it had to pick up a tub as part of

the task it will be able to do that and
so we just started doing

work in that direction using
analogical generalisations where we

take part of the robotic architecture so
this is our goal monitor and

there's a peer to component with
different period to libraries and

there's a vision system and then be
duplicated in the virtual world and

connected to the simulated peer
to root robot in gazebo and

up top here is a generalization
algorithm that takes.

A known script and tries to generalize it
and then at the apt it's to a new case.

So in this particular case
imagine new objects for

example we might imagine
a suitcase that looks like this or

object with a head like this or cup or a.

Cylinder with no handles right and
we try to find

different ways of performing the same kind
of script that it it before with these

different objects some of which require
you to perform different actions and

find different grass points and so forth.

So let me show you the final one
here which is taking a pick and

place script that the robot has learned
through an interaction that for

language ever that it now takes and

can apply to a new object in a new
context using this mental simulation.

So here on the right
you can see the robots.

Mental simulation it's starting by
executing what it already knows which is

to pick up a Med Kit and
place it somewhere.

And then what you'll see is the robot and.

Imagining trying to generalize both
to objects that he can pick up but

also the way he can pick up the object
still preserving the overall course

steps which are taking place.

And this is a little slow because of
the way to set up base here but you can

see in this particular case it imagine
another object but with the other arm so

it doesn't matter which arm you take
you can generalize over the arms.

Imagines a new object this object is
too far you cannot use the same script

to do that it would actually
have to move over right and

pick it up so this is a different script
it doesn't generalize it right it's not

reachable it requires additional
actions Here's another object and

the handle is not on top but on the side
it became grasp it on the site and

still lift it and pick it up.

So that's that's falls
on at the same script.

And the handle doesn't matter if it's
like this or that's a now be imagining.

Another object that has a handle and

here that actually tries out a different
grasp turns out the small object you don't

you can also pick up
like this with two arms.

And it learns that it doesn't necessarily
have to be picked up by the handle even

though there is a handle and
it can generalize over that.

They put that down and
then the final one is.

An object that looks similar to
this one but it's smaller and

it learns that you can pick
those up with two arms.

This is that of course this
is I want to say also very

preliminary work because there's lots of
interesting questions of how you generally

generate these examples read.

How much fantasy you have to sing things
new objects at the Imagine and different

task and exactly how you represent those
generalisations so now here's the.

The object that it picked up and
then what you will see is.

This is a new object that the robot
supposed to pick up and it's learnt.

By applying this generalized
case how to pick this one up and

he can do it immediately of course
that the laser not something

that we happy with it takes a while right
now to do this because this runs more or

less in real terms it takes a while but
you could imagine this

running on a cluster in parallel but all
of these cases independent of each other

you could do it offline if you knew what
kind of object it to deal with it doesn't

have to be online as it
is done in this case.

So let me just go to the place where it.

So here before it performs exactly that
pick up action that it be from this last

object but the pinch and
you can do a pick a place.

All right.

So to wrap up I showed you hope that.

Open World Open world requires

various changes to existing algorithms
from the natural language side so

that we represent into this
division process saying.

Basically all components of the
architecture really are affected by it and

I showed you how we can in a very
rudimentary sense handle some open world

tasking with new goals on the fly and
references to unknown objects and

locations where you acquire
the necessary information on the fly

through instructions or simulations as
in this case you're currently working on

better integrating the architecture to
handle the different uncertainties you get

to make this much more robust and
then we also want to really demonstrate

that all of these bits and pieces can
come together in one integrated system so

that we can go from the unknown words to
the unknown action all in one big system.

Lots of open problems.

And clearly we have only started
scratching the surface of what

I think is going to be a really
really fruitful research as agenda.

Roughly in the order of difficulty I have
a couple of problems that I find very

interesting for example how to learn
object to for insists and manipulate

objects from observations instructions and
simulations How do you learn those.

How do you learn complex activities
quickly from scenario to

demonstrations ideally from one example
we have an interest in one shot learning

how to use problem solving in Anya's
this to improve your knowledge and

performance in open worlds or
to for example handle failures and

ideally if one robot learned something
how can we take that knowledge now and

dump it on the other robot and
other work and

immediately do it that doesn't have to
be trained again and from of a practical

perspective if you want to get any of this
published right the first thing reviewable

say to us is well where's the evaluation
What part of the problem is this we

don't even know how to evaluate this
right it's open world what kinds of cases

would you have to generate to convince
reviewers that the algorithm is general.

And then ultimately want to move
into mixed human robot teams so

how do you do this in open worlds.

So thank you very much for

listening I have obviously several people
help with all of this various students and

postdocs who helped current
students collaborate as I mentioned

come about it is you who are planning
a person who's been working in the for

almost ten years come up
this group at Miami who is.

There with Titian and them so
Shaffer before and for about ten years and

Mark is Vince's group at Vienna who is
a machine vision person in robotics and

before also been working for six or
seven years together finding you know U.S.

National Science Foundation Office
of Naval Research papers and videos.

At these places Thank you.

And they stayed within

ten.

You know I didn't show you all of
the ones that it actually Madge and

it imagine many more.

So that robot started with a script that
it had learned from a demonstration for

example that one pick up in a place which
it in show you which is just to pick up

in the place it takes that
script right and then it tries

different kinds of objects and different
kinds of actions on the object so

this particular case we already know what
crass points are on the object so we can

go here or we can go here and we can do
it with one arm or with two arms right so

we know all of that so now the question
is is try it out see what succeeds and

what doesn't succeed right and then we
want to find the properties that had

the same across different kinds of mental
simulations such that we can abstract

Those So for example objects that
are cylinders right and the height doesn't

matter we can pick up with two arms
that's what the robot learned and

then if it finds a cell in there
became immediately a play that.

But it needs so that the key point is that
it needs to know that it has two arms and

that it can experiment with the other
arm if it doesn't know that it will

not discover it.

So in this case the demo that just showed
you we actually annotated the grass

positions on different types of objects
it imagines that that was knowledge

recently used code from
Platt who has trained

convolution that works on large
sets of objects so that we can

online to take now a completely new
objects that cross points and then.

We can make them subject to constraints
and filter out the ones we're interested

in so for example the points that
are on top of the object and then cross

that.

Down.

So there is this currently no notion of
optimality right success criteria and

for the simulation is basically what the
script ought to accomplish now if if you

impose additional constraints such as it
has to be done in five seconds you know

then then you will several things will
fall out right and drop out as possible

generalization because they won't work
moving the arm like this which is optimal

might allow you to pick in place but
it's not going to be the time get here so

the general abstraction ideas here is
if I know what it is that matters then

anything that accomplishes what I want to
accomplish under those constraints is good

enough is it the best possible way well
then you have to give me that performance.

But if the code here in this case
is this you need to pick it up and

deliver it you know in five minutes.

Then I want maybe the fastest way of
picking up the optimal route through

the environment and all of that.

So we're not we're not we're
not optimizing necessarily.

Thank you thank you.