The main goal of this thesis is to investigate techniques for solving whole manipulation tasks using uncalibrated visually guided manipulation. On the high task specification level, this translates into subdividing a task into different modes of visual control. On the low level, suitable visual servoing behaviors are needed, as well as on-line visual-motor model estimation. The latter is particularly useful since the different sub-behaviors involved in solving a complete task have different visual and motor calibrations. Using the estimated models in the other direction we have shown how to perform image based visual simulation or view synthesis. The main contributions on both low and high level aspects of visually guided manipulation are summarized below.