We propose a novel direct visual odometry algorithm formicro-lens-array-based light field cameras. The algorithm calculates adetailed, semi-dense 3D point cloud of its environment. Thisis achievedby establishing probabilistic depth hypotheses based on stereo obser-vations between the micro images of different recordings. Tracking isperformed in a coarse-to-fine process, working directly on the recordedraw images. The tracking accounts for changing lighting conditions andutilizes a linear motion model to be more robust. A novel scale optimiza-tion framework is proposed. It estimates the scene scale, on the basisof keyframes, and optimizes the scale of the entire trajectory by filter-ing over multiple estimates. The method is tested based on a versatiledataset consisting of challenging indoor and outdoor sequences and iscompared to state-of-the-art monocular and stereo approaches. The al-gorithm shows the ability to recover the absolute scale of the scene andsignificantly outperforms state-of-the-art monocular algorithms with re-spect to scale drifts.