Trying to find the relative translation between 2 markers.

[jkh13]

Hi, I am testing a marker detection process I have written in order to do AR similar to the glyph detection tutorial on this website. I have managed to extract out the location of the four corners of my 2 markers correctly from my test image and I used the coplanar posit object to generate the translation and rotation relative to the camera for each.

Test image:



The two markers are on the same plane with marker 2 100mm away from marker 1.

I am now trying to find the relative transformation between my two markers by multiplying the inverse of the transformation of marker 1 by the transformation of marker 2 (moving into the coordinate system of marker 1 and finding the transform to get to 2) . I am using XNA 4 to do the matrix maths similar to the 3d AR tutorial. However the result I am getting is not what I expect. The resulting matrix is telling me that the translation is {X:-43.27767 Y:-26.33978 Z:-34.37939}, I expected Z to be 0 and either X or Y to be 0 as well.

The code is as follows for my calculations.
I am creating XNA matrixes from the results of the CoPlanarPOSIT.
There are only the 2 markers in the markers list.
The image size is 640x480
The marker size is 100mm

The posit object:

Code: Select all

posit = new CoplanarPosit(new AForge.Math.Vector3[]
            {new AForge.Math.Vector3(-50,0,50),
                new AForge.Math.Vector3(50,0,50),
                new AForge.Math.Vector3(50,0,-50),
                new AForge.Math.Vector3(-50,0,-50)}, 640);

Code: Select all

        private void computePOSIT()
        {
            List<Matrix> matrixes = new List<Matrix>();

            AForge.Math.Matrix3x3 matrix;
            AForge.Math.Vector3 trans;

            foreach (Marker m in markers)
            {

                AForge.Point[] points = new AForge.Point[4];


                for (int i = 0; i < 4; i++)
                {
                    points[i] =
                        new AForge.Point(m.corners[i].X - 320,240 - m.corners[i].Y);
                }

                posit.EstimatePose(m.corners, out matrix, out trans);

                float yaw, pitch, roll;

                matrix.ExtractYawPitchRoll(out yaw, out pitch, out roll);

                Matrix rotation =
                    Matrix.CreateFromYawPitchRoll(-yaw, -pitch, roll);

                Matrix translation =
                    Matrix.CreateTranslation(new Vector3(trans.X, trans.Y, -trans.Z));

                Matrix complete = rotation * translation;
                matrixes.Add(complete);

            }

            Matrix res = Matrix.Multiply(Matrix.Invert(matrixes[0]), matrixes[1]);
            Vector3 scaleR;
            Vector3 translationR;
            Quaternion rotationR;
            res.Decompose(out scaleR, out rotationR, out translationR);
        }

[andrew.kirillov]

Hello,

There are few thigns to note:
1) Although the matrix may look suspicious, it is allways easy to make a fast test - apply the transformation to an object and see where it ends up in the XNA scene.
2) Since you know position/rotation of both objects, you may try to compute the matrix yourselves, just to see if it works at all and then get a reference matrix, so you know what to expect from the routine which multiplies inverse with another matrix. If you know that one object has coordinates (x1, y1, z1) and another has (x2, y2, z2), then you may create translation, which shifts from position to another. Similar should be doable with rotation.
3)

I am now trying to find the relative transformation between my two markers by multiplying the inverse of the transformation of marker 1 by the transformation of marker 2 (moving into the coordinate system of marker 1 and finding the transform to get to 2) . I am using XNA 4 to do the matrix maths similar to the 3d AR tutorial. However the result I am getting is not what I expect. The resulting matrix is telling me that the translation is {X:-43.27767 Y:-26.33978 Z:-34.37939}, I expected Z to be 0 and either X or Y to be 0 as well.

It seems like the resulting matrix gives you translation in the world's coordinate system, but not in coordinate system of the plane, which contains both objects. If you apply the result matrix to object 1, then it should get to position of object 2, but it works in such way that it is first moves to the origin of world’s coordinates system (result of inverse of the first matrix) and then it is moved to position of 2nd object. Obviously everything happens in one step. But the point is that the result translation is in world’s coordinate system. I would think so ... (but maybe missed something )

[jkh13]

Thanks for the reply,

I created an axes model and used the transformation matrixes to put them onto the scene. I got some very odd results however:



The transformation is mostly correct however one of the transforms is usually off center from the marker, sometimes by quite a lot:



I have double checked the co-ordinates of the corners and I am using the following to draw the scene:

Code: Select all

            foreach (List<Matrix> m in matrixes)
            {
                Vector3 cameraPosition = new Vector3(0.0f,0f, 3);
                foreach (ModelMesh mesh in myModel.Meshes)
                {
                    foreach (BasicEffect effect in mesh.Effects)
                    {
                        effect.EnableDefaultLighting();
                        effect.World = transforms[mesh.ParentBone.Index] * Matrix.CreateScale(1f) * m[0] * m[1];
                        effect.View = Matrix.CreateLookAt(cameraPosition,
                            Vector3.Zero, Vector3.Up);
                        effect.Projection =
                            Matrix.CreatePerspectiveFieldOfView(
                            MathHelper.ToRadians(45), aspectRatio,
                            1.0f, 10000.0f);
                         
                    }
                    mesh.Draw();
                }
            }

where m[0] is the translation and m[1] is the rotation

The rotation is slightly off as well most of the time. I tried running the posit on a flat square (corner to screen coords : (0,0),(0,100),(100,100),(0,100)) and the overlaid axes were slightly rotated.

The values after the call "matrix.ExtractYawPitchRoll(out yaw, out pitch, out roll);" for the above corners show:

yaw: -3.126024
pitch: -1.562007
roll: -3.14159

which is very slightly off.

Am I missing something to do with how the POSIT algorithm works? is it something to do with perspective?

Thanks,
Jkh13

[andrew.kirillov]

yaw: -3.126024
pitch: -1.562007
roll: -3.14159

which is very slightly off.

Am I missing something to do with how the POSIT algorithm works? is it something to do with perspective?

You may need to be prepared that Coplanar POSIT will not be able to provide 100% correct values. It is estimation only, which is close enough, but may not be exact.

The transformation is mostly correct however one of the transforms is usually off center from the marker, sometimes by quite a lot:

Try playing a bit with camera focal length, which is specified for POSIT. Also check that your model definition is correct.

[jkh13]

Thanks for the reply Andrew, you have been an invaluable help so far,

I realized that the focal length affects the results greatly. The estimate I am getting seems completely off however. I have double checked the model definition for the input to Posit as well.

I ran two images with 640 focal length and obtained the two following results.


I then reran the same 2 images with 565 focal length and got these results:


I can tweak the focal length to work for a particular frame but no single focal length works for everything. I was also wondering what measure the focal length is? The camera I am using is the PS eye camera which has a fixed focal length less than 20mm and has very low image distortion, sticking in 20 results in a result that is totally off. I am guessing the focal length is not measured in mm seeing as you used 640 in your guide?

Thanks,
Jonathan.

[andrew.kirillov]

I can tweak the focal length to work for a particular frame but no single focal length works for everything. I was also wondering what measure the focal length is? The camera I am using is the PS eye camera which has a fixed focal length less than 20mm and has very low image distortion, sticking in 20 results in a result that is totally off. I am guessing the focal length is not measured in mm seeing as you used 640 in your guide?

This part of the POSIT algorithm seems to be the most tricky. Unfortunately there is not much said about it in the original papers describing POSIT and Coplanar POSIT (you can find titles of those paper in documentation or in my tutorials). From those papers and also from some POSIT info on OpenCV page I found that effective focal length is always set to image's width. But I did not find much information about how this is related to lens’s focal length provided in specification of camera.

One thing you can try ... Just to make sure XNA does not add extra issues, you can check a sample application from AForge.NET framework (3D Pose Estimation (2)), which draws X/Y/Z coordinates on top of an object. That sample does not use XNA/OpenGL/whatever, but just computes everything on its own.

[jkh13]

Very useful, thanks.

Using the tool I think I found the problem.

The labeling of the corners seems to matter alot when it comes to pose estimation.



By marking the left-most corner as the first corner (moving clockwise) the pose comes out looking correct, whereas placing it on the bottom-most corner (moving clockwise) generates a bad estimation for both the given and the alternative estimation. I didn't think the POSIT algorithm cared about the labeling of the corners as long as they are in order but the above seems to show otherwise.

I have managed to replicate the issue for a number of images but not for others. For example the below doesn't exhibit the same problem for any starting corner:



It seems like quite an odd problem but the solution seems to be to enforce that the leftmost corner is the first corner.

[andrew.kirillov]

I didn't think the POSIT algorithm cared about the labeling of the corners as long as they are in order but the above seems to show otherwise.

I would not think this either that it cares about which is the first point. This seems to be an interesting finding, which may need further investigation to make the entire algorithm more robust.

[Cayman]

Hi guys.

Forgive me, this is the first post I have ever written on any forum ever. Its long, but I'm almost certain it is relevant!

I'm having the same problem. Here's where I got with it so far....

SETUP
1 - I have adapted the Glyph Recognition Studio sample by removing the 3d stuff and the image stuff, leaving only the GDI drawn borders. After all, I'm just trying to confirm accurate pose estimation via GDI drawings before I delve in to images and renders.

2 - I have adapted the Glyph Recognition Studio sample by adding the code from the Pose Estimation 2 sample that (1) extracts YPR (TransformationMatrix.ExtractYawPitchRoll), (2) draws the four colored corner points (of the recognized quadrilateral) and (3) projects a model using a transformation matrix (in this case I will use the GlyphData.TransformationMatrix of a glyph recognized by Glyph Recognition Studio).

3 - I have printed a glyph and phyically painted a white dot in the center of the glyph, so that locating the actual center of the glyph in the image requires no guess work.

THE TEST IMAGE
Using this test app, I have observed (via webcam in real time) the same distortions described in this thread by jkh13. However, the ability to project GDI-drawn models in the live video stream allows me to show in real time the colored corner points of the recognized quadrilateral and the axes of the pose estimation in real time. It also allows me to project the outline and cross hairs of the pose esimation.

So, the app presents a webcam image that contains:
1 - the recognized glyph (physically painted white dot in the physical center)
2 - the outline of the recognized quadrilateral per the Glyph Recognition Studio sample, which is *not* the outline of the pose estimation, because the quadrilateral is recognized *before* any pose estimation is done (right Andrew?)
3 - the four GDI-drawn quadrilateral points (like the Pose Estimation 2 sample)
4 - the three GDI-drawn axes (like the Pose Estimation 2 sample)
5 - a GDI-drawn outline of the pose estimation using 4 model points of the same size as the glyph modelPoints
6 - a GDI-drawn crosshairs of the pose estimation (cross at pose-estimated glyph center) using the same model points in item 5.

OBSERVATIONS
The image shows patterns in the inaccuracy of pose estimation.
1 - When point#1 (yellow) is far away, the outline renders smaller than the recognized quadrilateral. If point#1 is left of image center, then the rendered outline is skewed left of the observed quadrilateral - exaggerated. If point#1 is right of image center, then the rendered outline is skewed right of the observed quadrilateral - exaggerated.
2 - When point#1 is near, the outline renders larger than the recognized quadrilateral. If point#1 is left of image center, then the rendered outline is skewed right of the observed quadrilateral - not far enough. If point#1 is right of image center, then the rendered outline is skewed left of the observed quadrilateral - not far enough.
3 - When point#1 is mid-way between the furthest and nearest other corners (either on the left or the right), the outline renders the same size as the recognized quadrilateral. If point#1 is left of image center, then the rendered outline is pretty much right on the observed quadrilateral. If point#1 is right of image center, then the rendered outline is pretty much right on the observed quadrilateral.

So, as jkh13 pointed out, point#1 does appear to relate to the skew between estimated pose and the recognized quadrilateral. My wife is about to kill me if I don't call it a night, but I can upload screen shots of the obsered "skews" including all the CDI drawings if you like....definitely has something to do with point1 in pose estimation though...

One more thing - the pose esimtation outline is jumpy - like some kind of rounding or smoothing is causing it to hop / jerk - not smooth transition as the glpyh moves smoothly in the image. Might have something to do with it too.

All the best.
Pat

[Cayman]

Here are some screen shots showing the recognized quadrilateral (with colored points) and the bounds of model points projected using the estimated pose (with cross and axes). See how rotation affects the accuracy of the estimated pose?

Screenshot01.jpg (81.41 KiB) Viewed 1195 times

Screenshot02.jpg (80.04 KiB) Viewed 1195 times

Screenshot03.jpg (78.81 KiB) Viewed 1195 times

Once more screen shot to upload, because the site is limiting me to three.

Pat