One of my interests has always been to build a more accurate 3D model using computer vision. In the not too distant past, I have thought that it would be cool to do, but it won't help me graduate, and I'm not sure what I would use it for anyway. Now, I still think that it probably won't help me graduate, but at least I came up with a use for it besides the arm/mobile manipulation project.
I tend to see two sides in the papers on affordances I'm reading. Some go with a rich model approach and learn to recognize a rather small set of objects. Some go with no model and use observation. It seems like people use a little of both. Or perhaps people learn a rich model using observation. At any rate, people don't exclusively do one thing... we use models and we learn. The big question is how to build up the model.
It seems like one component of that model would be 3D structure. I have looked at a few "environment reconstruction from cameras" papers, and they don't look trivial to implement, as well as having results that are a little lacking. While reading today, I remembered a relatively old algorithm that Philip and I implemented in part for a class project... voxel coloring. Now it's ridiculously slow and consumes a lot of memory (slight improvement with paper called real-time voxel coloring), but the end results are better than any stereo vision algorithm I have come across (although I haven't looked in depth at multi-view reconstruction). The big trick is that you have to segment out the object of interest, and you have to know where the cameras are with a rather high degree of accuracy. It's the kind of algorithm that's simple in theory (and simulation), but becomes extremely difficult to make robust in practice.
Limitations/assumptions notwithstanding, I think this might be a good approach for a robot to build up a 3d model of an object it's interested in. The robot requires mobility (or big-brother multiple cameras), but I think that's a reasonable requirement for any project that truly wants to learn affordances of objects. I envision an approach where the robot identifies an object of interest, takes a picture, then moves around it and takes pictures at say, 10 degree intervals while tracking the object. From that point voxel coloring (or another reconstruction algorithm) can be applied to build a 3D model.
I actually had this idea before we got the SwissRanger in our lab as a way to build up a good 3D model with only a single monocular camera. But, as I said before, it probably won't help me graduate to implement such code, so whether I actually implement such a thing is still undecided.
Showing posts with label vision. Show all posts
Showing posts with label vision. Show all posts
Tuesday, November 17, 2009
Tuesday, January 13, 2009
Filtering the 3D scan
For the 3D scan display, we don't necessarily want to show every point that the camera sees. For example, the robot arm itself. We already display a graphical version of the arm that we draw based on outputs from the arm. When the arm and 3d points overlap, it's impossible to see exactly what's going on.
We can also remove things like the table the whole setup sits on and replace it with a plane. This will be easier to see, and give more contrast to items of interest, such as the blocks or pipe cleaners we will be using. The ground plane is simple enough to filter... since the coordinate system for the robot arm is what's used in the actual program. Anything below zero altitude is simply eliminated.
In the case of the robot arm, though, it's a little bit trickier. There are two ways I can think of to filter out the points. The first would be to put bounding boxes around the larger parts of the arm, then move the bounding boxes along with the arm and test to see if points are inside. The second method is to use OpenGL to render my relatively detailed model of the robot arm from the viewpoint of the stereo camera and in that way determine which pixels correspond to the robot arm. We have a mapping between 3D points and image pixels, so that's not the hard part. The second way is a little bit more nebulous to me... I'm not sure how difficult it will be. I do think the second way is better, though. Certainly faster, and probably more accurate as well. We can get a "closer trim" with the second method.
We can also remove things like the table the whole setup sits on and replace it with a plane. This will be easier to see, and give more contrast to items of interest, such as the blocks or pipe cleaners we will be using. The ground plane is simple enough to filter... since the coordinate system for the robot arm is what's used in the actual program. Anything below zero altitude is simply eliminated.
In the case of the robot arm, though, it's a little bit trickier. There are two ways I can think of to filter out the points. The first would be to put bounding boxes around the larger parts of the arm, then move the bounding boxes along with the arm and test to see if points are inside. The second method is to use OpenGL to render my relatively detailed model of the robot arm from the viewpoint of the stereo camera and in that way determine which pixels correspond to the robot arm. We have a mapping between 3D points and image pixels, so that's not the hard part. The second way is a little bit more nebulous to me... I'm not sure how difficult it will be. I do think the second way is better, though. Certainly faster, and probably more accurate as well. We can get a "closer trim" with the second method.
Dirty 3D rectangles!
It would be cool (and possibly useful) for the 3D scan in my interface to be dynamically updated in real-time. This is no small and trivial task, because it's a 640x480 image that the model is based on, so about 300k points. In the camera API, these points are stored in a big image where each point takes up 16 bytes (int, float, float, float). That's nearly 5 megabytes of data to sift through every camera update. And this camera can update at 30fps.
That's alotta data.
Well, it's a lot for stuff that needs to be processed in software, anyway. The bottleneck is not in transferring the data, mind you, it's in loading the data to the video card. Copying 5 megabytes to the video card 30 times per second, or even 5 times per second simply won't happen today.
So how do we solve this? I've been processing this at about 0.5% brainpower for the past several months, and today it dawned on me. I had been thinking all along of using vertex arrays, but I couldn't see a way to only update vertices that have changed. It didn't seem possible really. The answer is to use vertex arrays, but lots of vertex arrays instead of one. You break up the image into smaller pieces, so it's really something like 40x30 images that are each 16x16 pixels in size. Then you create a vertex array (or better, a vertex buffer object or VBO) for each sub-image. Then, if nothing is happening in a part of the image, you don't bother uploading that portion of data to the card.
This solution means that if the whole image changes, you have to upload the whole thing all over again, but for a camera that's sitting still, that should only happen if someone trips over the tripod, in which case you don't really care about the image anymore. If your camera is mounted on a mobile robot, the 3d display will only be dynamic if the robot's sitting still, and if it's looking at a static scene. For today's tasks and technology, these are reasonable constraints, I think.
Now that I've said all of this, I may or may not actually implement it. That all depends on how many grandkids I want to have by the time I graduate.
That's alotta data.
Well, it's a lot for stuff that needs to be processed in software, anyway. The bottleneck is not in transferring the data, mind you, it's in loading the data to the video card. Copying 5 megabytes to the video card 30 times per second, or even 5 times per second simply won't happen today.
So how do we solve this? I've been processing this at about 0.5% brainpower for the past several months, and today it dawned on me. I had been thinking all along of using vertex arrays, but I couldn't see a way to only update vertices that have changed. It didn't seem possible really. The answer is to use vertex arrays, but lots of vertex arrays instead of one. You break up the image into smaller pieces, so it's really something like 40x30 images that are each 16x16 pixels in size. Then you create a vertex array (or better, a vertex buffer object or VBO) for each sub-image. Then, if nothing is happening in a part of the image, you don't bother uploading that portion of data to the card.This solution means that if the whole image changes, you have to upload the whole thing all over again, but for a camera that's sitting still, that should only happen if someone trips over the tripod, in which case you don't really care about the image anymore. If your camera is mounted on a mobile robot, the 3d display will only be dynamic if the robot's sitting still, and if it's looking at a static scene. For today's tasks and technology, these are reasonable constraints, I think.
Now that I've said all of this, I may or may not actually implement it. That all depends on how many grandkids I want to have by the time I graduate.
Monday, January 12, 2009
Second experiment
Because the results from my first study weren't jump out of your seat exciting and significant, we're working toward a second study to get at least a shift-weight-in-chair result.
The most important thing to fix is the alignment between the 3D scan and robot arm. Just last week I managed to fully incorporate the new Videre/SRI STOC camera including calibration, and the alignment is lots better than the SwissRanger. Still, something's slightly off, so I might just cheat a little. I'm thinking of artificially inserting points in the scan for where I know the object to be. This way, the critical points will be aligned perfectly, and I won't have to spend weeks developing methods to get a better alignment. It's really not a perfect solution though, because the robot arm model could still be off, but it does get rid of one point of error.
Another thing that needs doing is smooth arm control. Currently the arm sort of jerks around while moving in a "vroom screech" fashion. Today I worked in a solution that works well for the most part, but it feels hackish and I've noticed a problem that I think is actually due to noisy readings from my servo position feedback, but I never saw it with the jerky control. So of course my adviser comments that I should be using a PD controller for this. But today I equipped my t-shirt of minimum coding effort, so I followed the path I trod earlier to get this thing whipped up quick. And now that I've got a solution mostly working, I know that my adviser's right, and I'll probably have weird problems crop up or a lot of work to remove the quirks unless I make a PD controller (sigh, grumble, sigh). It's sad to throw away code, but sometimes it just has to be done.
That's really all of the hard stuff... there are lots of little things, like improving the subjective user surveys (Likert scale and all of that), but the big stuff is getting to be under control. I can definitely see the study starting up again within a couple weeks (knock knock knock).
Wednesday, December 17, 2008
Seeing double
We recently got a Videre STOC stereo camera, and I've been working on integrating it with our robot arm setup. It has been somewhat of a challenge, mostly because the API provided by Videre/SRI is in C++, and our interface is in C#. A couple of my labmates mentioned that they had been able to work with C++ dlls before, so I looked into it.
SRI's API unfortunately is not just function calls, they also have custom classes that need to be created as well. So I needed to make a C++ wrapper class that would use the objects as needed to get me the information I wanted, and make everything function-based.
The next hurdle was that I wanted to get some 50,000+ 3D points from the camera, and doing a single GetPoint call for each point would be a massive bottleneck... it would be better to get the entire list of points with one call. Several hours later, I discovered that I didn't create a normal C++ wrapper class, but a C++/CLI project. That means that I have some managed code concepts to use inside C++.
After some research, I discovered that there are basically two main approaches to writing interop code with C# and C++ in windows. The more widely documented way is to do stuff with P/Invoke and marshaling data structures and write a COM dll in vanilla C++. The way I stumbled onto with CLI is to use an extended version of C++ that adds managed structures that can be used directly in C#. Basically it's a .NET-specific dll, but it really reduces the headache associated with getting funny data structures across to C++, at least if you're content with duplicating the contents of memory rather than using the existing C++ memory in-place. There are probably ways to marshal the data better than what I'm doing so far, but I will leave that to the optimization zealots. Here ( http://www.codeproject.com/KB/mcpp/quickcppcli.aspx ) is a link to the resource I used to learn what I needed for the CLI stuff. Very concise quick reference with examples, though I doubt it's complete.
The end result of all of this is that I have a working integration of the new stereo camera into the interface, although there are still a few adjustments left to make.
It looks to me like the objects will be unrecognizable from a viewpoint other than the camera's, but I'm hoping that the position will be accurate. So standard video can be used to identify what objects are, and the 3D model can be used to determine where they are. Of course by the time I'm done with this research, the 3D model will be a perfect (down to the subatomic particles) representation of reality ;)
SRI's API unfortunately is not just function calls, they also have custom classes that need to be created as well. So I needed to make a C++ wrapper class that would use the objects as needed to get me the information I wanted, and make everything function-based.
The next hurdle was that I wanted to get some 50,000+ 3D points from the camera, and doing a single GetPoint call for each point would be a massive bottleneck... it would be better to get the entire list of points with one call. Several hours later, I discovered that I didn't create a normal C++ wrapper class, but a C++/CLI project. That means that I have some managed code concepts to use inside C++.
After some research, I discovered that there are basically two main approaches to writing interop code with C# and C++ in windows. The more widely documented way is to do stuff with P/Invoke and marshaling data structures and write a COM dll in vanilla C++. The way I stumbled onto with CLI is to use an extended version of C++ that adds managed structures that can be used directly in C#. Basically it's a .NET-specific dll, but it really reduces the headache associated with getting funny data structures across to C++, at least if you're content with duplicating the contents of memory rather than using the existing C++ memory in-place. There are probably ways to marshal the data better than what I'm doing so far, but I will leave that to the optimization zealots. Here ( http://www.codeproject.com/KB/mcpp/quickcppcli.aspx ) is a link to the resource I used to learn what I needed for the CLI stuff. Very concise quick reference with examples, though I doubt it's complete.
The end result of all of this is that I have a working integration of the new stereo camera into the interface, although there are still a few adjustments left to make.
It looks to me like the objects will be unrecognizable from a viewpoint other than the camera's, but I'm hoping that the position will be accurate. So standard video can be used to identify what objects are, and the 3D model can be used to determine where they are. Of course by the time I'm done with this research, the 3D model will be a perfect (down to the subatomic particles) representation of reality ;)
Thursday, October 9, 2008
Slight improvement
Instead of discarding what I've already done I decided to try getting a better calibration with linear fitting. The way it works right now is I've mounted a target on the robot arm, and using forward kinematics I know where the target is located in the robot arm's coordinate system. From the SR-3000 I can find the target (currently done by human), so I have correlated points from two coordinate systems. All that's left is to linear least-squares solve to find the transformation between the two.
Today I added some calibration targets on the ground plane (a wooden board) that correspond to a height of zero. The reason for this is that yesterday, targets low to the ground were very poorly aligned (2-3 cm off). I thought one reason might be because the target was mounted somewhat high on the robot arm, so any skew happening down low was missed.
The end result is that it works somewhat better. It's still off by an average of 0.5 cm I'd say, but it appears to be consistent. With the "guess and check" method I could align it for one setup where two blocks were about 0.2 cm off, but the third would be 2 cm off. When I changed the setup, everything ended up being off by 1 cm or more. I think if I sample just the right space, and then keep all items of interest in that space, I may not have to worry about nonlinear distortion so much. After all, everything's linear in the limit, right? :)
Today I added some calibration targets on the ground plane (a wooden board) that correspond to a height of zero. The reason for this is that yesterday, targets low to the ground were very poorly aligned (2-3 cm off). I thought one reason might be because the target was mounted somewhat high on the robot arm, so any skew happening down low was missed.
The end result is that it works somewhat better. It's still off by an average of 0.5 cm I'd say, but it appears to be consistent. With the "guess and check" method I could align it for one setup where two blocks were about 0.2 cm off, but the third would be 2 cm off. When I changed the setup, everything ended up being off by 1 cm or more. I think if I sample just the right space, and then keep all items of interest in that space, I may not have to worry about nonlinear distortion so much. After all, everything's linear in the limit, right? :)
Wednesday, October 8, 2008
M(is)aligned decomposition
So I finished coding up the 3D camera calibration routines using Singular Value Decomposition and the like, and the results were better than I expected for a first attempt, but they more or less still stink. I think the problems come from three things: nonlinear distortions, inadequate sample space, and backlash in the robot arm.
I really do think the SR-3000 has some nonlinear distortion to it. It's not extreme, but it is there. So I should learn how to account for nonlinearity and calibrate accordingly. Now don't get me wrong, the SR-3000 is a pretty nice sensor, especially considering its size, power requirements, and illumination independence. It's just not quite good enough for the precision I'm looking for. If the scan is misaligned by 0.5cm in any direction, the task is extremely difficult to do.
The inadequate sample space comes from the calibration target being mounted in a place on the arm that cannot go below about 10cm (where the highest reach of the arm is 40cm). The result appears to be that objects sitting at around 15cm are aligned perfectly, while objects below 10cm are typically off by about 1 cm. Of course, this is after playing with it for only half an hour. One solution to this may be to mount the target in/on the gripper instead of the "wrist". Another possibility would be to put a few targets on the ground plane, although that could introduce some discrepancies between my robot arm forward kinematics model and the real world. In other words, where the arm really is and where it thinks it is.
Arm backlash is probably more important than I would like it to be. The base joint basically doesn't rotate for about 2-5 degrees when you change directions. I tried to hack a software solution together, but it's not good enough. To fix that I think I'll have to modify the hobbyist servos to give noisy analog position feedback, then run it through a filter to get the actual position. Not so difficult, but it takes time. Probably a better solution would be to put digital encoders on there, but while my servo controller has analog inputs, it has no digital inputs. That means I would have to get into AVR programming. While I would love to... it all comes back to time.
I really do think the SR-3000 has some nonlinear distortion to it. It's not extreme, but it is there. So I should learn how to account for nonlinearity and calibrate accordingly. Now don't get me wrong, the SR-3000 is a pretty nice sensor, especially considering its size, power requirements, and illumination independence. It's just not quite good enough for the precision I'm looking for. If the scan is misaligned by 0.5cm in any direction, the task is extremely difficult to do.
The inadequate sample space comes from the calibration target being mounted in a place on the arm that cannot go below about 10cm (where the highest reach of the arm is 40cm). The result appears to be that objects sitting at around 15cm are aligned perfectly, while objects below 10cm are typically off by about 1 cm. Of course, this is after playing with it for only half an hour. One solution to this may be to mount the target in/on the gripper instead of the "wrist". Another possibility would be to put a few targets on the ground plane, although that could introduce some discrepancies between my robot arm forward kinematics model and the real world. In other words, where the arm really is and where it thinks it is.
Arm backlash is probably more important than I would like it to be. The base joint basically doesn't rotate for about 2-5 degrees when you change directions. I tried to hack a software solution together, but it's not good enough. To fix that I think I'll have to modify the hobbyist servos to give noisy analog position feedback, then run it through a filter to get the actual position. Not so difficult, but it takes time. Probably a better solution would be to put digital encoders on there, but while my servo controller has analog inputs, it has no digital inputs. That means I would have to get into AVR programming. While I would love to... it all comes back to time.
Subscribe to:
Posts (Atom)
Posts
