Building a 3D model for affordances

One of my interests has always been to build a more accurate 3D model using computer vision. In the not too distant past, I have thought that it would be cool to do, but it won't help me graduate, and I'm not sure what I would use it for anyway. Now, I still think that it probably won't help me graduate, but at least I came up with a use for it besides the arm/mobile manipulation project.

I tend to see two sides in the papers on affordances I'm reading. Some go with a rich model approach and learn to recognize a rather small set of objects. Some go with no model and use observation. It seems like people use a little of both. Or perhaps people learn a rich model using observation. At any rate, people don't exclusively do one thing... we use models and we learn. The big question is how to build up the model.

It seems like one component of that model would be 3D structure. I have looked at a few "environment reconstruction from cameras" papers, and they don't look trivial to implement, as well as having results that are a little lacking. While reading today, I remembered a relatively old algorithm that Philip and I implemented in part for a class project... voxel coloring. Now it's ridiculously slow and consumes a lot of memory (slight improvement with paper called real-time voxel coloring), but the end results are better than any stereo vision algorithm I have come across (although I haven't looked in depth at multi-view reconstruction). The big trick is that you have to segment out the object of interest, and you have to know where the cameras are with a rather high degree of accuracy. It's the kind of algorithm that's simple in theory (and simulation), but becomes extremely difficult to make robust in practice.

Limitations/assumptions notwithstanding, I think this might be a good approach for a robot to build up a 3d model of an object it's interested in. The robot requires mobility (or big-brother multiple cameras), but I think that's a reasonable requirement for any project that truly wants to learn affordances of objects. I envision an approach where the robot identifies an object of interest, takes a picture, then moves around it and takes pictures at say, 10 degree intervals while tracking the object. From that point voxel coloring (or another reconstruction algorithm) can be applied to build a 3D model.

I actually had this idea before we got the SwissRanger in our lab as a way to build up a good 3D model with only a single monocular camera. But, as I said before, it probably won't help me graduate to implement such code, so whether I actually implement such a thing is still undecided.

Inverse affordances

While thinking of hand manipulation affordances, an idea came to mind. I'll call it inverse affordances for now. The idea is something like this: the user wants to perform such and such an action on an object, so what tool or procedure would be useful for doing such an action? An example case would be unscrewing a bolt, so what size wrench (8mm? 12mm?) would be best for that task? Or a screwdriver... what size screwdriver do I need? I can see this being especially useful for telemanipulation, even something like space maintenance/construction. Hey, I'd even like to pull out a cameraphone, point it at whatever bolt I'm trying to unscrew, and have it tell me what size wrench I need. From that standpoint it could be a "mechanic's assistant."

Now, I'm not going to try to solve this problem from start to finish right now, it might be interesting to setup a sort of Woz study, where all of the objects are annotated manually ahead of time, and then the user interacts with the system.

Affordance Maps

We have come up with an overarching theme for my past, present, and future research: affordance maps. In terms of interacting with the physical world, objects afford certain actions. A common example is that a chair affords sitting. In computer vision, classifying a chair is a difficult problem, because chairs have highly varied form. For any object that affords sitting, saying that the object is a chair is quite likely not completely wrong, even if it isn't the best answer. For example, a table affords sitting, so it could be classified as a chair. Such an answer might get a chuckle out of another person, but only because it's a somewhat unusual answer, and not entirely wrong.

In this sense we can see that most objects afford many actions, and some actions are afforded by several things. A chair affords sitting, standing, pushing, pounding, kicking, etc. Throwing is afforded by almost all objects, at least for some person or machine (most people cannot throw a car, but a construction crane could). Any object will have an affordance ranking or preference pattern. For a cup, this might be drinking, pouring, drumming, trapping insects, in order from highest to lowest preference.

Extending this idea, we can consider the notion of social affordances. That is, given the current state of the world, what social actions are permitted or acceptable? Discovering this kind of thing automatically is certainly daunting, and the only idea I have so far is to classify human social actions then study and learn from humans interacting with each other and with the robot.