Visual servo control, or visual servoing, makes use of computer vision to control robot motion. There exist two types of visual servo control: image-based visual servoing and position-based visual servoing. Image-based visual servo control relies on the ability of a camera to locate image points in its field of view. Position-based visual servoing uses camera vision data to perform a robot manipulator pose.
Visual Servo Control
The goal of any visual servo control system is to minimize error in image feature values. This error can be defined as:(1)
The image features s(m(t),a) are determined by the vector m(t), a set of image measurements, and a, a parameter set that contains other information about the system, such as the intrinsic parameters of the camera. The image measurements could include the coodinates of important points in the image or the coordinates of the centroids of objects in the camera view. The intrinsic parameters are constant for a given camera, and, therefore, would not change when the camera moves. s* contains the desired parameter values of the image or the desired coordinates of image points, and represents what the operator wants to see in the final image (camera's field of view).
Vision-based control systems use one of two camera positioning techniques. These are a fixed camera or a eye-in-hand configuration. For the fixed camera configuration, the camera position is fixed relative to the global reference frame or workspace. The camera is able to observe the robot arm and any objects that will be moved. One problem with this configuration is that the robot arm could block the camera's view of its end effector. Without the essential information about the end effector's position and orientation, the execution of a desired task is very difficult. For the eye-in-hand configuration, the camera is attached to the robot arm. It is usually positioned above the wrist of the manipulator, so that the end effector can be observed without any obstructions in the field of view. One possible problem with this technique is the change in the camera's field of view that occurs when the robot arm moves through the workspace.
Image-Based Visual Servoing (IBVS)
Image-based servo control seeks to determine the error in what the camera "sees" in its field of view and what the operator wants to see. For image-based visual servoing, the image feature values can be readily determined from the camera's image data. To carry out this type of control, the most common strategy is to find the desired spatial velocity of the camera, given by $\xi$, and use it as an input to a velocity controller. The spatial camera velocity can be modeled as(2)
where vc is the linear velocity of the origin of the camera frame and $\omega$c is the angular velocity of the camera frame. Both of the velocity components are instantaneous. s(t), introduced above, is a vector of measurable feature values in an image. The derivative of s(t) is known as the image feature velocity. If the only feature is a single image point, its feature values would be given by(3)
The feature values denote the coordinates of an image point in the image plane of the camera. A linear relationship exists between the image feature velocity and the camera velocity. The camera velocity is given by(4)
where $\xi$ is the camera velocity, v is the linear velocity, and $\omega$ is the angular velocity.
The image plane velocity of the image point would be $\dot s(t)$ and its relationship to the camera velocity is shown below:
where L(s,q) (or Ls) is the interaction matrix, and is a function of the robot configuration and the image feature values, s. The interaction matrix is also known as feature Jacobian or the image Jacobian matrix.
The above equation can be separated into spatial and angular velocity components by the following relation:(7)
Note that Lv(u,v,z) includes the first three columns of the interaction matrix. L$\omega$(u,v) includes the last three columns of the interaction matrix, but is not depth-dependent. (Consult the following section for an overview of the interaction matrix.)
The Image Jacobian
The interaction matrix can be created for one image point, shown in Eqs. (8) and (9). It is dependent on the image coodinates of the point and focal length of the camera, $\lambda$.(8)
The principle for finding the interaction matrix for one image point can be applied to multiple points. The parameter set, s, will contain the feature values of each point and a vector containing the depth values of the points, z.(10)
A composite interaction matrix will relate the image feature velocity $\dot s(t)$ to the camera velocity, $\xi$. It is shown below that the composite interaction matrix is a function of the image feature and depth values for the points.(11)
For multiple image points, it is necessary to stack the rows from the interaction matrices of the image points into one matrix. A generalized form for n interaction matrices for multiple points is given below.(12)
For a further discussion of the formation of the interaction matrix, see .
The image-based servo control system encounters problems for the case in which the camera undergoes large rotations about its optical axis (the axis normal to the image plane of the camera). This situation occurs because image-based servoing does not account for camera motion. A solution to this problem employs a partitioned method, which controls a portion of the degrees of freedom using the interaction matrix. The remaining degrees of freedom are controlled using other methods. From Eq. (9), the following result can be obtained:(14)
$\dot s$xy = Lxy$\xi$xy gives the image plane velocity component for the translation along and rotation about the x and y axes. $\dot s$z = Lz$\xi$z gives the image plane velocity component due to camera translation along and rotation about the z (optical) axis. (See  for a further discussion of the partitioned method.)
Position-Based Visual Servoing (PBVS)
Position-based servo control uses a set of parameters s. This set has to be found from image measurements, in which 3-D parameters are found. With this type of control, it is desired to create a 3-D representation of the workspace and target points (relative to the camera frame) in real time. Errors in camera calibration can hamper the effectiveness of this method. Because the camera motion can result in an object leaving the camera's field of view, there is no direct control over the image.
Example: Robot Arm Manipulator with Image-Based Servo Control
A robot arm manipulator was created that employed image-based visual servoing using a eye-in-hand camera at the end effector. This example uses a gradient search to move in the direction of minumum error until the target point is reached. The goal of the task was to move the arm of a robot manipulator so that its end effector would reach and press a button on an elevator panel. The image points consisted of the centroids of two buttons on the panel.
Figure1. This is a 6 degree of freedom robot arm with 3 prismatic and 3 rotation joints. The camera is shown from the orange link 6 toward the end effector.
Figure2. This is the camera view. Many tools in Matlab are used to set its position, target, and up vector for the camera.
This example is based in visual servoing with a gradient search algorithm instead of using the image Jacobian. The first step is to create the 6 images of the different joint movements in the positive and negative step directions. Then with each case, calculate and save the error. The goal was to line up the center of the camera with the center of the button, so the camera xy error is just the distance equation. The depth error is harder to get. By placing two buttons and using the perspective camera view. The distance between the two buttons could be compared to its true distance. This is how the depth error was found.
Figure3. After the image processing, the distance from the center of the screen can be compared to the target button, and the distance between the buttons can be saved for the depth calculation.
%% Here are some settings for the Matlab view
axis vis3d off
campos(v) % this is the position of the camera
camtarget([T06(1,4),T06(2,4),T06(3,4)]) % this is the direction the camera is facing
camup([T06(1,1),T06(2,1),T06(3,1)]) % this sets the orientation of the camera frame
drawnow % this refreshes the figure
pos1=[20,35,500,500]; % [left bottom width height]
set(12,'Position',pos1) % where and what size the figure will be
M=getframe(12,rect); % where and what to capture
Next is finding the lowest error among the steps and taking the corresponding step to which movement it came from. The gradient search method leads the robot on the path of least error to the target. In figure 4, the robot reached the tolerance level in the error of the button around 15 iterations and stopped taking steps in the x and y directions. As the robot continues to move in the z direction the little error was compounded with the zooming perspective view. In figure 5 the robot continues to move in the depth direction until it reaches the set tolerance.
%% sample code for image processing and error calculation
I = M.cdata; % load image
% Color of object (RGB value)
obj_color=[0,0,0]; % filter black
%Subtract object color from image
I(:,:,1) = I(:,:,1) - obj_color(1);
I(:,:,2) = I(:,:,2) - obj_color(2);
I(:,:,3) = I(:,:,3) - obj_color(3);
%Convert to black and white image
% Filter out any objects that are smaller than certain size
I = bwareaopen(~I, 50);
I = bwlabel(I);
% find centroids in image data
s = regionprops(I, 'Centroid');
% find center of camera view
b=size(M.cdata); % order is y then x
centroids = cat(1, s.Centroid);
% calculate error
camerr=((l(2)-centroids(1,1))^2 + (l(1)-centroids(1,2))^2)^.5;
%% sample code for gradient search
for b=1:6 % load different theta steps to find error
if flag1 == 1
d3 = d3r+dstep;
d3 = d3r-dstep;
d2 = d2r+dstep;
d2 = d2r-dstep;
if flag2 == 1
d1 = d1r+dstep;
d1 = d1r-dstep;
% now plot theta step to image M
[M] = rob6rot(ptb,d1,d2,d3,l1,l1b,l2,l3,l4,l5,l6,t1,t2,t3, …
% find camera error based on image M
% save camera error
if b < 5
Et(j,b) = camerr; % xy error
Ex(j,1) = cam2err; % depth error
Ex(j,2) = cam2err; % depth error
Figure4. Using the gradient search method to drive the robot arm, the code compares the 6 different error values from the movements to pick the path of least error.
% sample code for choosing least error path
% now move in the dir of least Et
Emin(j)= min(Et(j,:)); % check for xy error in 1 to 4 moves
Extar(j)= min(Ex(j,:)); % check for depth error in 5 and 6 moves
if j < jlimit % if too many iterations
% move in the x,y
if flag1 == 1
if Emin(j) >= tol
if Et(j,1) == Emin(j)
d3 = d3r+dstep;
elseif Et(j,2) == Emin(j)
d3 = d3r-dstep;
elseif Et(j,3) == Emin(j)
d2 = d2r+dstep;
elseif Et(j,4) == Emin(j)
d2 = d2r-dstep;
flag1 = 0; % within tol, end the while loop
fprintf('after %d iterations, INside of xy tol %d \n',j,tol)
Figure5. The depth error is driven to the tolerance level.