Tracing just-in-time compilation is a technique used by virtual machines to optimize the execution of a program at runtime. This is done by recording a linear sequence of frequently executed operations, compiling them to native machine code and executing them. This is opposed to traditional just-in-time (JIT) compilers that work on a per-method basis.
It's a JIT that traces runtime execution, rather than looking at whole method or function bodies. In particular it eliminates control flow, which simplifies optimisation.
Basically you observe loops and loops that satisfy certain criteria are traced, that is all operations performed within one iteration are stored. Such a trace is then optimized, compiled to machine code and then executed for every iteration when the loop is encountered again.